2024-09-21T18:37:15.707 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T18:37:15.723 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T18:37:15.827 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119 branch: wip-adk-testing-2024-09-21-1014 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: adking@redhat.com first_in_suite: false job_id: '7914119' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-09-21-1014 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: 0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 0ecbb435437226bc091a0c2d4549ba25bf724efe selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-adk-testing-2024-09-21-1014 sha1: 0ecbb435437226bc091a0c2d4549ba25bf724efe owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9557 sha1: 0ecbb435437226bc091a0c2d4549ba25bf724efe sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-adk-testing-2024-09-21-1014 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_0ecbb435437226bc091a0c2d4549ba25bf724efe/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 0ecbb435437226bc091a0c2d4549ba25bf724efe targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLjIMx+v0kw5RIe4fOoo1FOZ3eqclEYnP3uHtC3k/Rs8A8UHXciwugpLC6hk/z7+5NfPY0Yx9/hbajCOV025LeE= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMa0rwH9WXQKnRKW87j0/DF4vKqB7oNFG75aHmiJRfdCicin4uGxu8DwZ7QC7TSoQ03g6wBmbItkLkHwrm6KOLk= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_16:57:11 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T18:37:15.827 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_0ecbb435437226bc091a0c2d4549ba25bf724efe/qa; will attempt to use it 2024-09-21T18:37:15.827 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_0ecbb435437226bc091a0c2d4549ba25bf724efe/qa/tasks 2024-09-21T18:37:15.827 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T18:37:15.829 INFO:teuthology.task.internal:Checking packages... 2024-09-21T18:37:15.847 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0ecbb435437226bc091a0c2d4549ba25bf724efe' 2024-09-21T18:37:15.847 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T18:37:15.847 INFO:teuthology.packaging:ref: None 2024-09-21T18:37:15.847 INFO:teuthology.packaging:tag: None 2024-09-21T18:37:15.848 INFO:teuthology.packaging:branch: wip-adk-testing-2024-09-21-1014 2024-09-21T18:37:15.848 INFO:teuthology.packaging:sha1: 0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:37:15.848 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-09-21-1014 2024-09-21T18:37:15.991 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5129.g0ecbb435 2024-09-21T18:37:15.992 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T18:37:16.005 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T18:37:16.005 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T18:37:16.026 INFO:teuthology.task.internal:Saving configuration 2024-09-21T18:37:16.038 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T18:37:16.059 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T18:37:16.085 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 18:32:19.319176', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLjIMx+v0kw5RIe4fOoo1FOZ3eqclEYnP3uHtC3k/Rs8A8UHXciwugpLC6hk/z7+5NfPY0Yx9/hbajCOV025LeE='} 2024-09-21T18:37:16.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 18:32:19.317245', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMa0rwH9WXQKnRKW87j0/DF4vKqB7oNFG75aHmiJRfdCicin4uGxu8DwZ7QC7TSoQ03g6wBmbItkLkHwrm6KOLk='} 2024-09-21T18:37:16.105 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T18:37:16.190 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T18:37:16.190 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-21T18:37:16.190 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T18:37:16.334 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fde875f4280>, signals=[15]) 2024-09-21T18:37:16.334 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T18:37:16.340 INFO:teuthology.task.internal:Opening connections... 2024-09-21T18:37:16.340 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-09-21T18:37:16.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:37:16.415 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T18:37:16.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:37:16.489 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T18:37:16.495 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-09-21T18:37:16.513 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-09-21T18:37:16.514 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-09-21T18:37:16.570 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-09-21T18:37:16.570 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-09-21T18:37:16.570 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T18:37:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T18:37:16.572 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-09-21T18:37:16.604 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-09-21T18:37:16.622 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-09-21T18:37:16.622 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-09-21T18:37:16.680 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-09-21T18:37:16.680 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T18:37:16.681 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T18:37:16.682 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-09-21T18:37:16.703 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T18:37:16.712 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T18:37:16.880 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T18:37:16.880 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-09-21T18:37:16.882 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-09-21T18:37:16.899 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T18:37:16.906 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T18:37:16.906 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-09-21T18:37:16.940 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-09-21T18:37:17.048 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T18:37:17.104 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T18:37:17.118 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T18:37:17.118 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T18:37:17.118 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T18:37:17.119 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T18:37:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:37:17.136 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-09-21T18:37:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:37:17.137 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-21T18:37:17.194 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-503.el9.x86_64 2024-09-21T18:37:17.194 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-503.el9.x86_64 2024-09-21T18:37:17.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-09-21T18:37:17.195 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-503.el9.x86_64 2024-09-21T18:37:17.196 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-503.el9.x86_64 2024-09-21T18:37:17.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-09-21T18:37:19.717 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T18:37:19.740 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T18:37:21.533 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-21T18:37:21.534 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-21T18:37:25.917 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-21T18:37:25.934 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-21T18:37:27.302 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 116 kB/s | 19 kB 00:00 2024-09-21T18:37:27.451 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-21T18:37:28.539 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-21T18:37:28.539 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-21T18:37:33.936 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T18:37:33.978 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T18:37:35.851 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T18:37:35.851 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T18:37:35.901 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T18:37:35.901 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T18:37:35.966 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:37:35.977 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:37:35.977 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-09-21T18:37:35.978 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:37:35.979 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 79 M 2024-09-21T18:37:35.979 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 126 M 2024-09-21T18:37:35.979 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:37:36.011 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-09-21T18:37:36.016 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:37:36.017 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 79 M 2024-09-21T18:37:36.017 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 126 M 2024-09-21T18:37:36.017 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:37:36.146 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 260 kB/s | 36 kB 00:00 2024-09-21T18:37:36.271 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 310 kB/s | 36 kB 00:00 2024-09-21T18:37:36.389 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-21T18:37:36.914 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 38 MB/s | 35 MB 00:00 2024-09-21T18:37:37.322 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-21T18:37:37.322 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:37:37.323 INFO:teuthology.orchestra.run.smithi081.stdout:Total 59 MB/s | 79 MB 00:01 2024-09-21T18:37:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:37:38.337 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:37:38.337 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:37:38.573 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 6.5 MB/s | 16 MB 00:02 2024-09-21T18:37:39.286 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:37:39.286 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:37:39.463 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 9.0 MB/s | 29 MB 00:03 2024-09-21T18:37:40.205 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 8.5 MB/s | 35 MB 00:04 2024-09-21T18:37:40.205 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:37:40.205 INFO:teuthology.orchestra.run.smithi104.stdout:Total 19 MB/s | 79 MB 00:04 2024-09-21T18:37:40.889 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:37:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:37:42.180 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:37:42.180 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:37:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T18:37:42.475 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T18:37:44.622 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:37:45.249 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T18:37:45.281 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T18:37:45.362 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T18:37:45.378 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T18:37:48.117 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T18:37:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T18:37:50.215 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T18:37:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:37:53.097 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T18:37:53.139 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:37:55.150 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:37:58.027 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:33.806 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:33.849 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:34.311 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:34.311 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T18:38:34.311 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T18:38:34.311 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T18:38:35.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:35.180 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:38:35.180 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:38:35.180 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T18:38:35.181 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T18:38:35.181 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T18:38:35.181 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T18:38:35.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:38:35.181 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:38:35.515 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T18:38:35.601 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T18:38:35.602 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T18:38:35.602 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T18:38:35.602 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T18:38:35.602 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T18:38:35.602 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T18:38:35.602 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T18:38:35.602 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T18:38:35.660 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T18:38:35.660 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T18:38:35.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 82.el9 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8274670 2024-09-21T18:38:35.692 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T18:38:35.693 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-09-21T18:38:35.694 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:38:35.694 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T18:38:35.694 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T18:38:35.694 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-508.el9.x86_64 2024-09-21T18:38:35.695 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T18:38:36.061 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:36.349 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-09-21T18:38:36.475 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:36.475 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T18:38:36.476 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T18:38:36.476 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T18:38:37.316 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:38:37.317 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:38:37.670 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T18:38:37.741 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T18:38:37.746 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T18:38:37.746 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T18:38:37.746 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T18:38:37.746 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T18:38:37.746 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T18:38:37.747 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T18:38:37.747 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T18:38:37.747 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T18:38:37.769 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-09-21T18:38:37.771 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-09-21T18:38:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.9qEVCD5H6s 2024-09-21T18:38:37.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9qEVCD5H6s 2024-09-21T18:38:37.805 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T18:38:37.806 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T18:38:37.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-09-21T18:38:37.837 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-09-21T18:38:37.837 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-09-21T18:38:37.837 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 82.el9 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8274670 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-09-21T18:38:37.838 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:38:37.839 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T18:38:37.840 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T18:38:37.840 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-508.el9.x86_64 2024-09-21T18:38:37.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T18:38:37.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.9qEVCD5H6s 2024-09-21T18:38:37.994 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.9qEVCD5H6s is 6KB 2024-09-21T18:38:38.042 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.9qEVCD5H6s 2024-09-21T18:38:38.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T18:38:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T18:38:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T18:38:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T18:38:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T18:38:38.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T18:38:38.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-09-21T18:38:38.527 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-09-21T18:38:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T18:38:39.880 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-09-21T18:38:39.895 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-21T18:38:39.911 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.aIN1mUv07g 2024-09-21T18:38:39.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aIN1mUv07g 2024-09-21T18:38:39.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.aIN1mUv07g 2024-09-21T18:38:40.113 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.aIN1mUv07g is 6KB 2024-09-21T18:38:40.134 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.aIN1mUv07g 2024-09-21T18:38:40.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T18:38:40.211 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T18:38:40.211 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T18:38:40.211 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T18:38:40.211 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T18:38:40.213 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T18:38:40.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-09-21T18:39:08.310 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T18:39:08.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:39:08.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:39:10.390 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T18:39:10.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:39:10.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:39:26.703 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-21T18:39:28.755 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-21T18:39:35.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:39:35.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:39:37.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:39:37.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:39:38.771 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-21T18:39:40.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-21T18:39:50.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:39:50.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:39:52.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:39:52.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:39:53.839 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-21T18:39:55.887 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-21T18:40:08.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:40:08.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:40:10.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:40:10.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:40:11.919 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-21T18:40:13.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-21T18:40:29.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:40:29.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:40:30.345 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-09-21T18:40:30.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:40:30.711 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T18:40:30.712 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-09-21T18:40:30.728 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-508.el9.x86_64 2024-09-21T18:40:30.728 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T18:40:30.728 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T18:40:30.728 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T18:40:31.729 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T18:40:31.729 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T18:40:31.795 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-09-21T18:40:31.817 DEBUG:teuthology.parallel:result is None 2024-09-21T18:40:31.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:40:31.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:40:32.615 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-21T18:40:33.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:40:33.013 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T18:40:33.013 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-21T18:40:33.031 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-508.el9.x86_64 2024-09-21T18:40:33.031 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T18:40:33.031 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T18:40:33.031 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T18:40:34.032 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T18:40:34.033 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T18:40:34.387 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-09-21T18:40:34.412 DEBUG:teuthology.parallel:result is None 2024-09-21T18:40:34.412 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T18:40:34.419 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T18:40:34.420 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T18:40:34.422 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T18:40:34.440 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T18:40:34.462 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T18:40:34.532 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T18:40:34.532 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T18:40:34.535 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T18:40:34.595 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T18:40:34.602 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T18:40:34.603 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T18:40:34.605 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T18:40:34.637 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T18:40:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T18:40:34.649 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T18:40:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T18:40:34.654 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T18:40:34.661 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T18:40:34.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T18:40:34.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T18:40:34.724 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T18:40:34.732 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T18:40:34.732 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T18:40:34.765 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T18:40:34.781 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T18:40:34.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T18:40:34.912 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T18:40:34.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T18:40:35.038 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:40:35.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T18:40:35.104 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T18:40:35.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T18:40:35.208 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T18:40:35.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T18:40:35.336 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:40:35.336 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T18:40:35.404 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-09-21T18:40:35.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-09-21T18:40:35.445 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T18:40:35.492 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T18:40:35.963 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T18:40:35.982 INFO:teuthology.task.internal:Starting timer... 2024-09-21T18:40:35.983 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T18:40:36.024 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T18:40:36.031 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-21T18:40:36.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-09-21T18:40:36.085 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-09-21T18:40:36.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-09-21T18:40:36.133 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-09-21T18:40:36.135 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T18:40:36.135 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-09-21T18:40:36.172 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-09-21T18:40:36.172 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-09-21T18:40:36.206 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-09-21T18:40:36.206 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-09-21T18:40:36.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T18:40:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:40:36.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T18:40:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:40:36.275 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T18:40:36.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-09-21T18:40:36.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-09-21T18:40:36.353 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T18:40:36.362 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T18:40:36.373 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T18:40:36.374 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T18:44:19.593 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-09-21T18:44:19.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:44:19.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:44:19.670 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-09-21T18:44:19.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-21T18:44:19.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:44:19.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:44:19.779 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-21T18:44:19.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T18:44:19.862 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T18:44:19.884 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T18:44:19.884 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T18:44:19.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:44:19.887 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T18:44:19.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:44:19.913 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T18:44:19.927 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T18:44:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T18:44:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T18:44:19.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-09-21T18:44:19.962 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-09-21T18:44:19.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-09-21T18:44:19.976 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T18:44:19.978 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-09-21T18:44:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T18:44:19.992 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T18:44:20.006 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T18:44:20.048 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-09-21T18:44:20.052 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T18:44:20.053 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-09-21T18:44:20.053 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.053 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.053 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.053 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-09-21T18:44:20.065 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T18:44:20.065 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-09-21T18:44:20.065 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.065 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.066 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.066 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T18:44:20.067 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T18:44:20.080 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T18:44:20.081 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi081.front.sepia.ceph.com 2024-09-21T18:44:20.081 DEBUG:teuthology.orchestra.run.smithi081:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T18:44:20.081 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T18:44:20.082 DEBUG:teuthology.orchestra.run.smithi104:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T18:44:20.440 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:44:20.457 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:44:20.458 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:20.458 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-21T18:44:20.458 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:20.458 INFO:teuthology.orchestra.run.smithi081.stdout:Removing: 2024-09-21T18:44:20.458 INFO:teuthology.orchestra.run.smithi081.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout:Removing unused dependencies: 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout:Remove 2 Packages 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout:Freed space: 5.5 M 2024-09-21T18:44:20.459 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:44:20.463 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:44:20.463 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout:Removing: 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout:Removing unused dependencies: 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T18:44:20.478 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:20.479 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:44:20.479 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:20.479 INFO:teuthology.orchestra.run.smithi104.stdout:Remove 2 Packages 2024-09-21T18:44:20.479 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:20.479 INFO:teuthology.orchestra.run.smithi104.stdout:Freed space: 5.5 M 2024-09-21T18:44:20.479 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:44:20.482 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:44:20.482 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:44:20.527 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:44:20.527 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:44:20.541 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:44:20.541 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:44:20.682 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:44:20.687 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:44:20.760 INFO:teuthology.orchestra.run.smithi104.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T18:44:20.763 INFO:teuthology.orchestra.run.smithi081.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T18:44:20.822 INFO:teuthology.orchestra.run.smithi081.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T18:44:20.827 INFO:teuthology.orchestra.run.smithi104.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T18:44:21.324 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T18:44:21.324 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T18:44:21.355 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T18:44:21.355 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T18:44:21.583 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T18:44:21.583 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:21.583 INFO:teuthology.orchestra.run.smithi081.stdout:Removed: 2024-09-21T18:44:21.583 INFO:teuthology.orchestra.run.smithi081.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T18:44:21.583 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:21.583 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:44:21.629 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T18:44:21.629 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:21.629 INFO:teuthology.orchestra.run.smithi104.stdout:Removed: 2024-09-21T18:44:21.629 INFO:teuthology.orchestra.run.smithi104.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T18:44:21.629 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:21.629 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:44:22.437 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 06:43:33 PM UTC. 2024-09-21T18:44:22.465 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 06:43:33 PM UTC. 2024-09-21T18:44:22.571 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:22.572 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-09-21T18:44:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:22.580 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.0 M 2024-09-21T18:44:22.580 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 9.2 M 2024-09-21T18:44:22.580 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:44:22.600 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T18:44:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T18:44:22.602 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T18:44:22.602 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:22.602 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:44:22.602 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:22.602 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-09-21T18:44:22.602 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:22.603 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.0 M 2024-09-21T18:44:22.603 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 9.2 M 2024-09-21T18:44:22.604 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:44:22.695 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 477 kB/s | 44 kB 00:00 2024-09-21T18:44:22.721 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 892 kB/s | 106 kB 00:00 2024-09-21T18:44:22.763 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.0 MB/s | 72 kB 00:00 2024-09-21T18:44:22.788 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.2 MB/s | 84 kB 00:00 2024-09-21T18:44:22.821 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.0 MB/s | 906 kB 00:00 2024-09-21T18:44:22.857 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 371 kB/s | 44 kB 00:00 2024-09-21T18:44:22.881 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.0 MB/s | 837 kB 00:00 2024-09-21T18:44:22.881 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:44:22.881 INFO:teuthology.orchestra.run.smithi081.stdout:Total 6.7 MB/s | 2.0 MB 00:00 2024-09-21T18:44:22.882 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 730 kB/s | 106 kB 00:00 2024-09-21T18:44:22.935 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 924 kB/s | 72 kB 00:00 2024-09-21T18:44:22.953 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:44:22.960 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.1 MB/s | 84 kB 00:00 2024-09-21T18:44:22.967 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:44:22.967 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:44:22.985 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.6 MB/s | 906 kB 00:00 2024-09-21T18:44:23.069 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.1 MB/s | 837 kB 00:00 2024-09-21T18:44:23.069 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:44:23.069 INFO:teuthology.orchestra.run.smithi104.stdout:Total 4.3 MB/s | 2.0 MB 00:00 2024-09-21T18:44:23.143 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:44:23.156 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:44:23.157 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:44:23.240 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:44:23.240 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:44:23.447 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:44:23.447 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:44:23.666 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:44:23.759 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-21T18:44:23.802 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-21T18:44:23.830 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-21T18:44:23.899 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:44:23.989 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-21T18:44:24.032 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-21T18:44:24.035 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-21T18:44:24.068 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-21T18:44:24.069 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T18:44:24.294 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-21T18:44:24.341 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T18:44:24.502 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T18:44:24.530 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T18:44:24.781 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T18:44:24.810 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T18:44:25.523 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T18:44:25.523 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-21T18:44:25.523 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-21T18:44:25.523 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-21T18:44:25.524 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-21T18:44:25.524 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-21T18:44:25.811 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T18:44:25.811 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-21T18:44:25.811 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-21T18:44:25.811 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-21T18:44:25.811 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-21T18:44:25.811 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-21T18:44:25.886 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-21T18:44:25.886 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:25.886 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:44:25.887 INFO:teuthology.orchestra.run.smithi081.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T18:44:25.887 INFO:teuthology.orchestra.run.smithi081.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T18:44:25.887 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T18:44:25.887 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:25.887 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:44:26.120 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:26.121 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:44:26.253 DEBUG:teuthology.parallel:result is None 2024-09-21T18:44:26.475 DEBUG:teuthology.parallel:result is None 2024-09-21T18:44:26.475 INFO:teuthology.run_tasks:Running task install... 2024-09-21T18:44:26.483 DEBUG:teuthology.task.install:project ceph 2024-09-21T18:44:26.483 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe'}} 2024-09-21T18:44:26.483 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe'} 2024-09-21T18:44:26.483 INFO:teuthology.task.install:Using flavor: default 2024-09-21T18:44:26.488 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T18:44:26.489 INFO:teuthology.task.install:extra packages: [] 2024-09-21T18:44:26.489 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe', 'tag': None, 'wait_for_package': False} 2024-09-21T18:44:26.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:44:26.491 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe', 'tag': None, 'wait_for_package': False} 2024-09-21T18:44:26.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:44:26.628 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/ 2024-09-21T18:44:26.629 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5129.g0ecbb435 2024-09-21T18:44:26.632 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/ 2024-09-21T18:44:26.632 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5129.g0ecbb435 2024-09-21T18:44:26.737 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T18:44:26.737 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:44:26.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T18:44:26.742 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T18:44:26.742 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:44:26.742 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T18:44:26.767 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T18:44:26.767 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0ecbb435437226bc091a0c2d4549ba25bf724efe/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T18:44:26.772 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T18:44:26.773 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0ecbb435437226bc091a0c2d4549ba25bf724efe/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T18:44:26.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T18:44:26.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T18:44:26.910 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T18:44:26.918 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T18:44:26.975 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-09-21T18:44:26.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-09-21T18:44:26.983 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-09-21T18:44:26.985 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-09-21T18:44:27.204 INFO:teuthology.orchestra.run.smithi081.stdout:45 files removed 2024-09-21T18:44:27.219 INFO:teuthology.orchestra.run.smithi104.stdout:45 files removed 2024-09-21T18:44:27.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2024-09-21T18:44:27.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-09-21T18:44:27.795 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 303 kB/s | 84 kB 00:00 2024-09-21T18:44:27.807 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 307 kB/s | 84 kB 00:00 2024-09-21T18:44:28.070 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 104 kB/s | 24 kB 00:00 2024-09-21T18:44:28.108 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-21T18:44:28.338 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-21T18:44:28.383 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-21T18:44:30.138 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-21T18:44:30.150 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-21T18:44:31.901 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-21T18:44:32.439 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-21T18:44:36.385 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.5 MB 00:00 2024-09-21T18:44:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-21T18:44:38.002 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-21T18:44:39.090 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-21T18:44:39.499 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-21T18:44:40.620 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-21T18:44:44.528 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-09-21T18:44:46.127 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T18:44:46.852 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:44:46.854 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:46.854 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:44:46.854 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:46.854 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:44:46.854 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 11 M 2024-09-21T18:44:46.855 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-09-21T18:44:46.855 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 3.5 M 2024-09-21T18:44:46.855 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 3.2 M 2024-09-21T18:44:46.855 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:44:46.855 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T18:44:46.855 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 5.5 M 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 22 M 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 25 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 754 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T18:44:46.856 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 508 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 5.5 M 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 45 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 150 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 162 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 321 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 307 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 100 k 2024-09-21T18:44:46.857 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout:Install 31 Packages 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-09-21T18:44:46.858 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:46.859 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 62 M 2024-09-21T18:44:46.859 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:44:47.463 INFO:teuthology.orchestra.run.smithi104.stdout:(1/33): ceph-base-19.3.0-5129.g0ecbb435.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-21T18:44:47.521 INFO:teuthology.orchestra.run.smithi104.stdout:(2/33): ceph-selinux-19.3.0-5129.g0ecbb435.el9. 431 kB/s | 25 kB 00:00 2024-09-21T18:44:47.671 INFO:teuthology.orchestra.run.smithi104.stdout:(3/33): ceph-radosgw-19.3.0-5129.g0ecbb435.el9. 16 MB/s | 11 MB 00:00 2024-09-21T18:44:47.705 INFO:teuthology.orchestra.run.smithi104.stdout:(4/33): libcephfs2-19.3.0-5129.g0ecbb435.el9.x8 4.0 MB/s | 754 kB 00:00 2024-09-21T18:44:47.739 INFO:teuthology.orchestra.run.smithi104.stdout:(5/33): libradosstriper1-19.3.0-5129.g0ecbb435. 7.5 MB/s | 508 kB 00:00 2024-09-21T18:44:47.822 INFO:teuthology.orchestra.run.smithi104.stdout:(6/33): python3-ceph-argparse-19.3.0-5129.g0ecb 541 kB/s | 45 kB 00:00 2024-09-21T18:44:47.931 INFO:teuthology.orchestra.run.smithi104.stdout:(7/33): librgw2-19.3.0-5129.g0ecbb435.el9.x86_6 24 MB/s | 5.5 MB 00:00 2024-09-21T18:44:47.956 INFO:teuthology.orchestra.run.smithi104.stdout:(8/33): python3-ceph-common-19.3.0-5129.g0ecbb4 1.1 MB/s | 150 kB 00:00 2024-09-21T18:44:47.981 INFO:teuthology.orchestra.run.smithi104.stdout:(9/33): python3-cephfs-19.3.0-5129.g0ecbb435.el 3.2 MB/s | 162 kB 00:00 2024-09-21T18:44:48.007 INFO:teuthology.orchestra.run.smithi104.stdout:(10/33): python3-rados-19.3.0-5129.g0ecbb435.el 6.2 MB/s | 321 kB 00:00 2024-09-21T18:44:48.032 INFO:teuthology.orchestra.run.smithi104.stdout:(11/33): python3-rbd-19.3.0-5129.g0ecbb435.el9. 6.0 MB/s | 307 kB 00:00 2024-09-21T18:44:48.057 INFO:teuthology.orchestra.run.smithi104.stdout:(12/33): python3-rgw-19.3.0-5129.g0ecbb435.el9. 1.9 MB/s | 100 kB 00:00 2024-09-21T18:44:48.192 INFO:teuthology.orchestra.run.smithi104.stdout:(13/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.2 MB/s | 171 kB 00:00 2024-09-21T18:44:48.391 INFO:teuthology.orchestra.run.smithi104.stdout:(14/33): ceph-common-19.3.0-5129.g0ecbb435.el9. 16 MB/s | 22 MB 00:01 2024-09-21T18:44:48.402 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:44:48.404 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:48.404 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:44:48.404 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:48.404 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:44:48.404 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 11 M 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 3.5 M 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 3.2 M 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 5.5 M 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 22 M 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 25 k 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 754 k 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T18:44:48.405 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 508 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 5.5 M 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 45 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 150 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 162 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 321 k 2024-09-21T18:44:48.406 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 307 k 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 100 k 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Install 31 Packages 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 2 Packages 2024-09-21T18:44:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:44:48.408 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 62 M 2024-09-21T18:44:48.408 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:44:48.416 INFO:teuthology.orchestra.run.smithi104.stdout:(15/33): boost-program-options-1.75.0-8.el9.x86 278 kB/s | 107 kB 00:00 2024-09-21T18:44:48.442 INFO:teuthology.orchestra.run.smithi104.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 642 kB/s | 160 kB 00:00 2024-09-21T18:44:48.508 INFO:teuthology.orchestra.run.smithi104.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-21T18:44:48.542 INFO:teuthology.orchestra.run.smithi104.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-21T18:44:48.576 INFO:teuthology.orchestra.run.smithi104.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-21T18:44:48.601 INFO:teuthology.orchestra.run.smithi104.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 458 kB/s | 42 kB 00:00 2024-09-21T18:44:48.626 INFO:teuthology.orchestra.run.smithi104.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-21T18:44:48.652 INFO:teuthology.orchestra.run.smithi104.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 475 kB/s | 36 kB 00:00 2024-09-21T18:44:48.677 INFO:teuthology.orchestra.run.smithi104.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-21T18:44:48.735 INFO:teuthology.orchestra.run.smithi104.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-21T18:44:48.761 INFO:teuthology.orchestra.run.smithi104.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-21T18:44:48.786 INFO:teuthology.orchestra.run.smithi104.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 277 kB/s | 26 kB 00:00 2024-09-21T18:44:48.811 INFO:teuthology.orchestra.run.smithi104.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T18:44:48.845 INFO:teuthology.orchestra.run.smithi104.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-21T18:44:48.912 INFO:teuthology.orchestra.run.smithi104.stdout:(29/33): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-09-21T18:44:48.945 INFO:teuthology.orchestra.run.smithi104.stdout:(30/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 839 kB 00:00 2024-09-21T18:44:48.961 INFO:teuthology.orchestra.run.smithi081.stdout:(1/33): ceph-base-19.3.0-5129.g0ecbb435.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-21T18:44:49.019 INFO:teuthology.orchestra.run.smithi081.stdout:(2/33): ceph-selinux-19.3.0-5129.g0ecbb435.el9. 434 kB/s | 25 kB 00:00 2024-09-21T18:44:49.037 INFO:teuthology.orchestra.run.smithi104.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T18:44:49.119 INFO:teuthology.orchestra.run.smithi081.stdout:(3/33): libcephfs2-19.3.0-5129.g0ecbb435.el9.x8 7.4 MB/s | 754 kB 00:00 2024-09-21T18:44:49.354 INFO:teuthology.orchestra.run.smithi104.stdout:(32/33): librbd1-19.3.0-5129.g0ecbb435.el9.x86_ 8.0 MB/s | 3.2 MB 00:00 2024-09-21T18:44:49.404 INFO:teuthology.orchestra.run.smithi104.stdout:(33/33): librados2-19.3.0-5129.g0ecbb435.el9.x8 7.0 MB/s | 3.5 MB 00:00 2024-09-21T18:44:49.409 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:44:49.409 INFO:teuthology.orchestra.run.smithi104.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-21T18:44:49.453 INFO:teuthology.orchestra.run.smithi081.stdout:(4/33): ceph-radosgw-19.3.0-5129.g0ecbb435.el9. 12 MB/s | 11 MB 00:00 2024-09-21T18:44:49.525 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:44:49.561 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:44:49.561 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:44:49.637 INFO:teuthology.orchestra.run.smithi081.stdout:(5/33): libradosstriper1-19.3.0-5129.g0ecbb435. 983 kB/s | 508 kB 00:00 2024-09-21T18:44:49.846 INFO:teuthology.orchestra.run.smithi081.stdout:(6/33): ceph-common-19.3.0-5129.g0ecbb435.el9.x 16 MB/s | 22 MB 00:01 2024-09-21T18:44:49.879 INFO:teuthology.orchestra.run.smithi081.stdout:(7/33): python3-ceph-argparse-19.3.0-5129.g0ecb 186 kB/s | 45 kB 00:00 2024-09-21T18:44:49.913 INFO:teuthology.orchestra.run.smithi081.stdout:(8/33): python3-ceph-common-19.3.0-5129.g0ecbb4 2.2 MB/s | 150 kB 00:00 2024-09-21T18:44:49.946 INFO:teuthology.orchestra.run.smithi081.stdout:(9/33): python3-cephfs-19.3.0-5129.g0ecbb435.el 2.4 MB/s | 162 kB 00:00 2024-09-21T18:44:49.972 INFO:teuthology.orchestra.run.smithi081.stdout:(10/33): python3-rados-19.3.0-5129.g0ecbb435.el 5.4 MB/s | 321 kB 00:00 2024-09-21T18:44:49.997 INFO:teuthology.orchestra.run.smithi081.stdout:(11/33): python3-rbd-19.3.0-5129.g0ecbb435.el9. 6.0 MB/s | 307 kB 00:00 2024-09-21T18:44:50.031 INFO:teuthology.orchestra.run.smithi081.stdout:(12/33): python3-rgw-19.3.0-5129.g0ecbb435.el9. 1.7 MB/s | 100 kB 00:00 2024-09-21T18:44:50.061 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:44:50.061 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:44:50.106 INFO:teuthology.orchestra.run.smithi081.stdout:(13/33): librgw2-19.3.0-5129.g0ecbb435.el9.x86_ 8.4 MB/s | 5.5 MB 00:00 2024-09-21T18:44:50.465 INFO:teuthology.orchestra.run.smithi081.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 229 kB/s | 107 kB 00:00 2024-09-21T18:44:50.532 INFO:teuthology.orchestra.run.smithi081.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 377 kB/s | 160 kB 00:00 2024-09-21T18:44:50.557 INFO:teuthology.orchestra.run.smithi081.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 324 kB/s | 171 kB 00:00 2024-09-21T18:44:50.815 INFO:teuthology.orchestra.run.smithi081.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 129 kB/s | 45 kB 00:00 2024-09-21T18:44:51.116 INFO:teuthology.orchestra.run.smithi081.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 523 kB/s | 292 kB 00:00 2024-09-21T18:44:51.149 INFO:teuthology.orchestra.run.smithi081.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 127 kB/s | 42 kB 00:00 2024-09-21T18:44:51.216 INFO:teuthology.orchestra.run.smithi081.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 968 kB/s | 662 kB 00:00 2024-09-21T18:44:51.434 INFO:teuthology.orchestra.run.smithi081.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 126 kB/s | 36 kB 00:00 2024-09-21T18:44:51.484 INFO:teuthology.orchestra.run.smithi081.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-21T18:44:51.546 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:44:51.592 INFO:teuthology.orchestra.run.smithi081.stdout:(23/33): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-21T18:44:51.609 INFO:teuthology.orchestra.run.smithi081.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 121 kB/s | 47 kB 00:00 2024-09-21T18:44:51.626 INFO:teuthology.orchestra.run.smithi081.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 775 kB/s | 26 kB 00:00 2024-09-21T18:44:51.637 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T18:44:51.660 INFO:teuthology.orchestra.run.smithi081.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T18:44:51.675 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T18:44:51.685 INFO:teuthology.orchestra.run.smithi081.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 643 kB/s | 49 kB 00:00 2024-09-21T18:44:51.719 INFO:teuthology.orchestra.run.smithi081.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-21T18:44:51.721 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T18:44:51.744 INFO:teuthology.orchestra.run.smithi081.stdout:(29/33): qatlib-24.02.0-1.el9.x86_64.rpm 351 kB/s | 221 kB 00:00 2024-09-21T18:44:51.748 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T18:44:51.769 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T18:44:51.778 INFO:teuthology.orchestra.run.smithi081.stdout:(30/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.1 MB/s | 839 kB 00:00 2024-09-21T18:44:51.828 INFO:teuthology.orchestra.run.smithi081.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T18:44:51.908 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T18:44:51.961 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T18:44:51.988 INFO:teuthology.orchestra.run.smithi081.stdout:(32/33): librados2-19.3.0-5129.g0ecbb435.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-21T18:44:52.097 INFO:teuthology.orchestra.run.smithi081.stdout:(33/33): librbd1-19.3.0-5129.g0ecbb435.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-09-21T18:44:52.101 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:44:52.101 INFO:teuthology.orchestra.run.smithi081.stdout:Total 17 MB/s | 62 MB 00:03 2024-09-21T18:44:52.217 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:44:52.255 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:44:52.255 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:44:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T18:44:52.410 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 7/35 2024-09-21T18:44:52.486 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 7/35 2024-09-21T18:44:52.574 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 8/35 2024-09-21T18:44:52.595 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 9/35 2024-09-21T18:44:52.667 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 9/35 2024-09-21T18:44:52.712 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T18:44:52.749 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:44:52.749 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:44:52.756 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:19.3.0-5129.g0ecbb435.el9. 11/35 2024-09-21T18:44:52.826 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 12/35 2024-09-21T18:44:52.856 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_6 13/35 2024-09-21T18:44:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_6 13/35 2024-09-21T18:44:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T18:44:53.063 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T18:44:53.097 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T18:44:53.152 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T18:44:53.187 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T18:44:53.216 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T18:44:53.249 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T18:44:53.286 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T18:44:53.608 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T18:44:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 23/35 2024-09-21T18:44:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 23/35 2024-09-21T18:44:53.767 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 24/35 2024-09-21T18:44:53.820 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T18:44:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:44:54.311 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T18:44:54.340 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 27/35 2024-09-21T18:44:54.356 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T18:44:54.394 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T18:44:54.412 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 27/35 2024-09-21T18:44:54.419 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T18:44:54.441 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T18:44:54.463 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T18:44:54.493 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 28/35 2024-09-21T18:44:54.534 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:19.3.0-5129.g0ecbb435.el9.x8 29/35 2024-09-21T18:44:54.661 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T18:44:54.714 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T18:44:55.076 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T18:44:55.187 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 7/35 2024-09-21T18:44:55.272 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 7/35 2024-09-21T18:44:55.378 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 8/35 2024-09-21T18:44:55.406 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 9/35 2024-09-21T18:44:55.469 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 9/35 2024-09-21T18:44:55.497 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T18:44:55.550 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.3.0-5129.g0ecbb435.el9. 11/35 2024-09-21T18:44:55.628 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 12/35 2024-09-21T18:44:55.649 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_6 13/35 2024-09-21T18:44:55.697 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_6 13/35 2024-09-21T18:44:55.734 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T18:44:55.848 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T18:44:55.891 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T18:44:55.938 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T18:44:55.972 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T18:44:56.002 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T18:44:56.034 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T18:44:56.081 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T18:44:56.408 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T18:44:56.457 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 23/35 2024-09-21T18:44:56.557 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 23/35 2024-09-21T18:44:56.586 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 24/35 2024-09-21T18:44:56.622 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T18:44:56.838 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 30/35 2024-09-21T18:44:56.888 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 30/35 2024-09-21T18:44:57.111 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T18:44:57.167 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 27/35 2024-09-21T18:44:57.239 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 27/35 2024-09-21T18:44:57.321 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 28/35 2024-09-21T18:44:57.353 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.3.0-5129.g0ecbb435.el9.x8 29/35 2024-09-21T18:44:57.460 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 30/35 2024-09-21T18:44:57.499 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 31/35 2024-09-21T18:44:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 31/35 2024-09-21T18:44:57.578 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T18:44:57.578 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T18:44:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:44:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 32/35 2024-09-21T18:44:59.649 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 30/35 2024-09-21T18:44:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 30/35 2024-09-21T18:45:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 30/35 2024-09-21T18:45:00.295 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 31/35 2024-09-21T18:45:00.370 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 31/35 2024-09-21T18:45:00.370 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T18:45:00.370 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T18:45:00.370 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:00.413 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 32/35 2024-09-21T18:45:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 32/35 2024-09-21T18:45:07.716 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-09-21T18:45:07.716 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-09-21T18:45:07.716 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-09-21T18:45:07.716 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-09-21T18:45:07.717 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-09-21T18:45:07.717 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-09-21T18:45:07.717 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-09-21T18:45:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:08.733 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/35 2024-09-21T18:45:08.759 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/35 2024-09-21T18:45:08.759 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:08.759 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T18:45:08.759 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:45:08.759 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:45:08.759 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:08.760 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T18:45:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T18:45:08.835 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T18:45:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T18:45:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/35 2024-09-21T18:45:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/35 2024-09-21T18:45:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 5/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_6 6/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 7/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5129.g0ecbb435.el9. 8/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:19.3.0-5129.g0ecbb435.el9.x8 9/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 10/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 11/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 12/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 13/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T18:45:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T18:45:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 32/35 2024-09-21T18:45:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T18:45:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 34/35 2024-09-21T18:45:10.494 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 32/35 2024-09-21T18:45:10.494 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-09-21T18:45:10.495 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:10.899 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T18:45:10.899 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:10.899 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-09-21T18:45:10.899 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.900 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.900 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:45:10.900 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T18:45:10.900 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.900 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.900 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T18:45:10.901 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T18:45:10.902 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T18:45:10.902 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T18:45:10.902 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.902 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T18:45:10.902 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.902 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T18:45:10.903 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:10.904 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:11.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-09-21T18:45:11.505 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/35 2024-09-21T18:45:11.532 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/35 2024-09-21T18:45:11.532 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:11.532 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T18:45:11.532 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:45:11.532 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:45:11.532 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:11.534 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T18:45:11.624 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T18:45:11.625 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T18:45:11.832 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:11.968 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 46 M 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 161 k 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:11.969 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-09-21T18:45:11.970 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:11.970 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 46 M 2024-09-21T18:45:11.970 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 201 M 2024-09-21T18:45:11.970 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:45:12.316 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): libcephsqlite-19.3.0-5129.g0ecbb435.el9. 747 kB/s | 161 kB 00:00 2024-09-21T18:45:12.649 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 553 kB/s | 304 kB 00:00 2024-09-21T18:45:12.691 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-21T18:45:12.927 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T18:45:12.927 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 5/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_6 6/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 7/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5129.g0ecbb435.el9. 8/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.3.0-5129.g0ecbb435.el9.x8 9/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 10/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 11/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 12/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 13/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T18:45:12.928 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T18:45:12.929 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T18:45:12.930 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T18:45:12.930 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T18:45:12.930 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T18:45:12.931 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 32/35 2024-09-21T18:45:12.931 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T18:45:12.931 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 34/35 2024-09-21T18:45:13.508 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): ceph-test-19.3.0-5129.g0ecbb435.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-21T18:45:13.510 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:13.511 INFO:teuthology.orchestra.run.smithi104.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-21T18:45:13.538 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:45:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:45:13.559 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.826 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.827 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T18:45:13.828 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T18:45:13.829 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:13.829 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:13.931 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:45:13.931 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:45:14.190 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2024-09-21T18:45:14.434 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:45:14.529 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T18:45:14.573 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T18:45:14.601 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/4 2024-09-21T18:45:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:28 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:14.912 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 46 M 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 161 k 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:14.913 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:45:14.914 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:14.914 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-09-21T18:45:14.914 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:14.914 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 46 M 2024-09-21T18:45:14.914 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 201 M 2024-09-21T18:45:14.914 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:45:15.174 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-21T18:45:15.199 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): libcephsqlite-19.3.0-5129.g0ecbb435.el9. 679 kB/s | 161 kB 00:00 2024-09-21T18:45:15.224 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T18:45:16.424 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): ceph-test-19.3.0-5129.g0ecbb435.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-21T18:45:16.427 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:16.427 INFO:teuthology.orchestra.run.smithi081.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-21T18:45:16.455 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:45:16.475 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:45:16.475 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:45:16.842 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:45:16.843 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:45:17.346 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:45:17.443 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T18:45:17.481 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T18:45:17.509 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/4 2024-09-21T18:45:18.951 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/4 2024-09-21T18:45:19.048 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/4 2024-09-21T18:45:21.406 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/4 2024-09-21T18:45:21.406 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/4 2024-09-21T18:45:21.406 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/4 2024-09-21T18:45:21.406 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T18:45:21.808 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T18:45:21.808 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:21.808 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:45:21.809 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:21.809 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:21.809 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T18:45:21.809 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T18:45:21.809 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:21.809 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:21.895 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/4 2024-09-21T18:45:21.986 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/4 2024-09-21T18:45:22.098 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-09-21T18:45:22.692 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:22.830 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:22.832 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:22.832 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:45:22.832 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 6.6 k 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 2.4 M 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 1.5 M 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 293 k 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 4.8 M 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 17 M 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T18:45:22.833 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T18:45:22.834 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T18:45:22.835 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout:Install 37 Packages 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 31 M 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 109 M 2024-09-21T18:45:22.836 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:45:23.674 INFO:teuthology.orchestra.run.smithi104.stdout:(1/37): ceph-19.3.0-5129.g0ecbb435.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-21T18:45:23.824 INFO:teuthology.orchestra.run.smithi104.stdout:(2/37): ceph-mgr-19.3.0-5129.g0ecbb435.el9.x86_ 5.3 MB/s | 1.5 MB 00:00 2024-09-21T18:45:23.883 INFO:teuthology.orchestra.run.smithi104.stdout:(3/37): ceph-mds-19.3.0-5129.g0ecbb435.el9.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-09-21T18:45:23.983 INFO:teuthology.orchestra.run.smithi104.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5129.g0ecb 2.9 MB/s | 293 kB 00:00 2024-09-21T18:45:24.042 INFO:teuthology.orchestra.run.smithi104.stdout:(5/37): ceph-mon-19.3.0-5129.g0ecbb435.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-21T18:45:24.209 INFO:teuthology.orchestra.run.smithi104.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-21T18:45:24.384 INFO:teuthology.orchestra.run.smithi104.stdout:(7/37): ceph-osd-19.3.0-5129.g0ecbb435.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-21T18:45:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/4 2024-09-21T18:45:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/4 2024-09-21T18:45:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/4 2024-09-21T18:45:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T18:45:24.409 INFO:teuthology.orchestra.run.smithi104.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-21T18:45:24.459 INFO:teuthology.orchestra.run.smithi104.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-21T18:45:24.510 INFO:teuthology.orchestra.run.smithi104.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-09-21T18:45:24.535 INFO:teuthology.orchestra.run.smithi104.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-21T18:45:24.577 INFO:teuthology.orchestra.run.smithi104.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 701 kB/s | 135 kB 00:00 2024-09-21T18:45:24.652 INFO:teuthology.orchestra.run.smithi104.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-21T18:45:24.678 INFO:teuthology.orchestra.run.smithi104.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-09-21T18:45:24.703 INFO:teuthology.orchestra.run.smithi104.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-21T18:45:24.728 INFO:teuthology.orchestra.run.smithi104.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-21T18:45:24.753 INFO:teuthology.orchestra.run.smithi104.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-21T18:45:24.779 INFO:teuthology.orchestra.run.smithi104.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 550 kB/s | 42 kB 00:00 2024-09-21T18:45:24.785 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T18:45:24.785 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:24.785 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:45:24.785 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:24.785 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:24.786 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T18:45:24.786 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T18:45:24.786 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:24.786 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:24.821 INFO:teuthology.orchestra.run.smithi104.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-21T18:45:24.854 INFO:teuthology.orchestra.run.smithi104.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.3 MB/s | 172 kB 00:00 2024-09-21T18:45:24.879 INFO:teuthology.orchestra.run.smithi104.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-21T18:45:24.905 INFO:teuthology.orchestra.run.smithi104.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 126 kB/s | 22 kB 00:00 2024-09-21T18:45:24.930 INFO:teuthology.orchestra.run.smithi104.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-21T18:45:24.964 INFO:teuthology.orchestra.run.smithi104.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 888 kB/s | 23 kB 00:00 2024-09-21T18:45:24.989 INFO:teuthology.orchestra.run.smithi104.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 831 kB/s | 19 kB 00:00 2024-09-21T18:45:25.014 INFO:teuthology.orchestra.run.smithi104.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-21T18:45:25.040 INFO:teuthology.orchestra.run.smithi104.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-21T18:45:25.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2024-09-21T18:45:25.065 INFO:teuthology.orchestra.run.smithi104.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-21T18:45:25.090 INFO:teuthology.orchestra.run.smithi104.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-09-21T18:45:25.115 INFO:teuthology.orchestra.run.smithi104.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T18:45:25.141 INFO:teuthology.orchestra.run.smithi104.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-21T18:45:25.166 INFO:teuthology.orchestra.run.smithi104.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-21T18:45:25.191 INFO:teuthology.orchestra.run.smithi104.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-21T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T18:45:25.242 INFO:teuthology.orchestra.run.smithi104.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-21T18:45:25.267 INFO:teuthology.orchestra.run.smithi104.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T18:45:25.292 INFO:teuthology.orchestra.run.smithi104.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T18:45:25.296 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:25.296 INFO:teuthology.orchestra.run.smithi104.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-21T18:45:25.450 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:45:25.489 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:45:25.489 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:45:25.655 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:39 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:25.797 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 6.6 k 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 2.4 M 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 1.5 M 2024-09-21T18:45:25.800 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 293 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 4.8 M 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 17 M 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T18:45:25.801 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T18:45:25.802 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout:Install 37 Packages 2024-09-21T18:45:25.803 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:25.804 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 31 M 2024-09-21T18:45:25.804 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 109 M 2024-09-21T18:45:25.804 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:45:25.941 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:45:25.942 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:45:26.805 INFO:teuthology.orchestra.run.smithi081.stdout:(1/37): ceph-19.3.0-5129.g0ecbb435.el9.x86_64.r 44 kB/s | 6.6 kB 00:00 2024-09-21T18:45:26.981 INFO:teuthology.orchestra.run.smithi081.stdout:(2/37): ceph-mgr-19.3.0-5129.g0ecbb435.el9.x86_ 4.7 MB/s | 1.5 MB 00:00 2024-09-21T18:45:27.031 INFO:teuthology.orchestra.run.smithi081.stdout:(3/37): ceph-mds-19.3.0-5129.g0ecbb435.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-21T18:45:27.115 INFO:teuthology.orchestra.run.smithi081.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5129.g0ecb 3.4 MB/s | 293 kB 00:00 2024-09-21T18:45:27.224 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:45:27.294 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T18:45:27.334 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T18:45:27.372 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T18:45:27.431 INFO:teuthology.orchestra.run.smithi081.stdout:(5/37): ceph-mon-19.3.0-5129.g0ecbb435.el9.x86_ 7.7 MB/s | 4.8 MB 00:00 2024-09-21T18:45:27.444 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T18:45:27.505 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T18:45:27.537 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T18:45:27.607 INFO:teuthology.orchestra.run.smithi081.stdout:(6/37): ceph-osd-19.3.0-5129.g0ecbb435.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-21T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T18:45:27.674 INFO:teuthology.orchestra.run.smithi081.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 453 kB/s | 253 kB 00:00 2024-09-21T18:45:27.715 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T18:45:27.757 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T18:45:27.791 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T18:45:27.821 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T18:45:27.856 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T18:45:27.889 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T18:45:27.938 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T18:45:27.972 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T18:45:28.032 INFO:teuthology.orchestra.run.smithi081.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 250 kB/s | 106 kB 00:00 2024-09-21T18:45:28.056 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T18:45:28.088 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T18:45:28.124 INFO:teuthology.orchestra.run.smithi081.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 300 kB/s | 135 kB 00:00 2024-09-21T18:45:28.161 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T18:45:28.299 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T18:45:28.316 INFO:teuthology.orchestra.run.smithi081.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2024-09-21T18:45:28.379 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T18:45:28.431 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T18:45:28.479 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T18:45:28.500 INFO:teuthology.orchestra.run.smithi081.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 267 kB/s | 125 kB 00:00 2024-09-21T18:45:28.525 INFO:teuthology.orchestra.run.smithi081.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 903 kB/s | 188 kB 00:00 2024-09-21T18:45:28.536 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T18:45:28.584 INFO:teuthology.orchestra.run.smithi081.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-21T18:45:28.592 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T18:45:28.634 INFO:teuthology.orchestra.run.smithi081.stdout:(14/37): python3-urllib3-1.26.5-6.el9.noarch.rp 421 kB/s | 215 kB 00:00 2024-09-21T18:45:28.651 INFO:teuthology.orchestra.run.smithi081.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-09-21T18:45:28.676 INFO:teuthology.orchestra.run.smithi081.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 838 kB/s | 77 kB 00:00 2024-09-21T18:45:28.702 INFO:teuthology.orchestra.run.smithi081.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 759 kB/s | 51 kB 00:00 2024-09-21T18:45:28.752 INFO:teuthology.orchestra.run.smithi081.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-21T18:45:28.777 INFO:teuthology.orchestra.run.smithi081.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-09-21T18:45:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T18:45:28.803 INFO:teuthology.orchestra.run.smithi081.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-21T18:45:28.833 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T18:45:28.836 INFO:teuthology.orchestra.run.smithi081.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 518 kB/s | 43 kB 00:00 2024-09-21T18:45:28.870 INFO:teuthology.orchestra.run.smithi081.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 449 kB/s | 11 kB 00:00 2024-09-21T18:45:28.919 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T18:45:28.920 INFO:teuthology.orchestra.run.smithi081.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 367 kB/s | 18 kB 00:00 2024-09-21T18:45:28.954 INFO:teuthology.orchestra.run.smithi081.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 977 kB/s | 172 kB 00:00 2024-09-21T18:45:28.979 INFO:teuthology.orchestra.run.smithi081.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-09-21T18:45:29.004 INFO:teuthology.orchestra.run.smithi081.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-21T18:45:29.015 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T18:45:29.030 INFO:teuthology.orchestra.run.smithi081.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T18:45:29.054 INFO:teuthology.orchestra.run.smithi081.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-21T18:45:29.065 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T18:45:29.080 INFO:teuthology.orchestra.run.smithi081.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T18:45:29.114 INFO:teuthology.orchestra.run.smithi081.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-21T18:45:29.139 INFO:teuthology.orchestra.run.smithi081.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-09-21T18:45:29.178 INFO:teuthology.orchestra.run.smithi081.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 930 kB/s | 36 kB 00:00 2024-09-21T18:45:29.194 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T18:45:29.206 INFO:teuthology.orchestra.run.smithi081.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T18:45:29.231 INFO:teuthology.orchestra.run.smithi081.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2024-09-21T18:45:29.256 INFO:teuthology.orchestra.run.smithi081.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 809 kB/s | 20 kB 00:00 2024-09-21T18:45:29.290 INFO:teuthology.orchestra.run.smithi081.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.0 MB/s | 230 kB 00:00 2024-09-21T18:45:29.316 INFO:teuthology.orchestra.run.smithi081.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T18:45:29.321 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:29.321 INFO:teuthology.orchestra.run.smithi081.stdout:Total 8.7 MB/s | 31 MB 00:03 2024-09-21T18:45:29.363 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5129.g0ecbb435.el9. 32/37 2024-09-21T18:45:29.392 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/37 2024-09-21T18:45:29.418 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/37 2024-09-21T18:45:29.419 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:29.419 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T18:45:29.419 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T18:45:29.419 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T18:45:29.419 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:29.475 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:45:29.515 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:45:29.515 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:45:29.973 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:45:29.973 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:45:30.884 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 34/37 2024-09-21T18:45:30.912 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 34/37 2024-09-21T18:45:30.912 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:30.912 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T18:45:30.912 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T18:45:30.912 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T18:45:30.912 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:31.275 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:45:31.344 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T18:45:31.377 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 35/37 2024-09-21T18:45:31.389 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T18:45:31.401 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 35/37 2024-09-21T18:45:31.401 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:31.401 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T18:45:31.401 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T18:45:31.401 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T18:45:31.401 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:31.421 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T18:45:31.491 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T18:45:31.528 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T18:45:31.569 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T18:45:31.627 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 36/37 2024-09-21T18:45:31.651 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 36/37 2024-09-21T18:45:31.651 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:31.651 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T18:45:31.651 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T18:45:31.651 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T18:45:31.652 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:31.669 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T18:45:31.717 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 37/37 2024-09-21T18:45:31.748 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T18:45:31.788 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T18:45:31.823 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T18:45:31.862 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T18:45:31.904 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T18:45:31.938 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T18:45:31.979 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T18:45:32.013 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T18:45:32.048 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T18:45:32.096 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T18:45:32.129 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T18:45:32.201 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T18:45:32.341 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T18:45:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T18:45:32.489 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T18:45:32.528 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T18:45:32.585 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T18:45:32.640 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T18:45:32.826 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T18:45:32.873 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T18:45:32.943 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T18:45:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T18:45:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T18:45:33.209 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T18:45:33.379 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5129.g0ecbb435.el9. 32/37 2024-09-21T18:45:33.409 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/37 2024-09-21T18:45:33.435 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 33/37 2024-09-21T18:45:33.436 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:33.436 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T18:45:33.436 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T18:45:33.436 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T18:45:33.436 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:33.539 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 37/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 5/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5129.g0ecbb435.el9. 6/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T18:45:33.540 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T18:45:33.541 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T18:45:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T18:45:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T18:45:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T18:45:33.542 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T18:45:33.543 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T18:45:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T18:45:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T18:45:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:34.526 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 34/37 2024-09-21T18:45:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 34/37 2024-09-21T18:45:34.929 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:34.929 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T18:45:34.929 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T18:45:34.929 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T18:45:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:34.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-09-21T18:45:35.385 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 35/37 2024-09-21T18:45:35.409 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 35/37 2024-09-21T18:45:35.409 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:35.410 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T18:45:35.410 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T18:45:35.410 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T18:45:35.410 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:35.555 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:35.636 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 36/37 2024-09-21T18:45:35.643 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:45:35.659 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 36/37 2024-09-21T18:45:35.659 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:35.660 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T18:45:35.660 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T18:45:35.660 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T18:45:35.660 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:35.686 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:35.687 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:45:35.687 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:35.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 37/37 2024-09-21T18:45:35.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-09-21T18:45:36.332 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:36.463 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:36.463 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:36.463 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:45:36.464 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:36.464 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:45:36.464 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 784 k 2024-09-21T18:45:36.464 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:36.465 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:45:36.465 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:36.465 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:45:36.465 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:36.466 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 784 k 2024-09-21T18:45:36.466 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 791 k 2024-09-21T18:45:36.466 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:45:36.745 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-19.3.0-5129.g0ecbb435.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-21T18:45:36.745 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:36.746 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-21T18:45:36.746 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:45:36.751 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:45:36.751 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:45:36.757 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:45:36.757 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:45:36.832 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:45:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:37.373 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:37.609 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 37/37 2024-09-21T18:45:37.609 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/37 2024-09-21T18:45:37.609 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/37 2024-09-21T18:45:37.609 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 3/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 4/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 5/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5129.g0ecbb435.el9. 6/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T18:45:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T18:45:37.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T18:45:37.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T18:45:37.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T18:45:37.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T18:45:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T18:45:37.614 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T18:45:37.614 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T18:45:37.614 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T18:45:37.615 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T18:45:37.742 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:37.990 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:37.990 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:37.990 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:45:37.991 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:37.991 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:37.991 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:38.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:38.671 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T18:45:38.672 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T18:45:38.673 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T18:45:38.674 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T18:45:38.674 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T18:45:38.674 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:38.799 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:38.932 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:38.932 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:38.932 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-21T18:45:38.932 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 145 k 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:45:38.933 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:38.934 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 145 k 2024-09-21T18:45:38.934 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 432 k 2024-09-21T18:45:38.934 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:45:39.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base 2024-09-21T18:45:39.134 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-19.3.0-5129.g0ecbb4 724 kB/s | 145 kB 00:00 2024-09-21T18:45:39.135 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:39.135 INFO:teuthology.orchestra.run.smithi104.stdout:Total 718 kB/s | 145 kB 00:00 2024-09-21T18:45:39.135 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:45:39.142 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:45:39.142 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:45:39.200 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:45:39.201 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:45:39.343 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:45:39.410 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435. 1/1 2024-09-21T18:45:39.435 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435. 1/1 2024-09-21T18:45:39.435 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:39.435 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T18:45:39.435 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:39.649 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:39.738 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:45:39.779 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:39.781 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:45:39.781 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:39.860 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install cephadm 2024-09-21T18:45:40.367 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435. 1/1 2024-09-21T18:45:40.367 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:40.367 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:45:40.367 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:40.367 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:40.367 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:40.435 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:40.565 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:40.565 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 784 k 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:45:40.566 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:40.567 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 784 k 2024-09-21T18:45:40.567 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 791 k 2024-09-21T18:45:40.567 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:45:40.597 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-09-21T18:45:40.860 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm-19.3.0-5129.g0ecbb435.el9.noarch.rpm 2.6 MB/s | 784 kB 00:00 2024-09-21T18:45:40.860 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:40.861 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-21T18:45:40.861 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:45:40.866 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:45:40.866 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:45:40.872 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:45:40.872 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:45:40.943 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:45:41.187 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:41.278 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:45:41.320 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:41.321 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:45:41.321 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:41.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T18:45:41.438 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:41.813 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:41.972 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:42.057 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:45:42.057 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:42.057 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:45:42.057 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:42.057 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:42.057 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:42.104 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 14 M 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:45:42.105 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 34 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 16 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:45:42.106 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:45:42.107 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-09-21T18:45:42.107 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:42.107 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 19 M 2024-09-21T18:45:42.107 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 159 M 2024-09-21T18:45:42.107 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:45:42.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T18:45:42.846 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): ceph-prometheus-alerts-19.3.0-5129.g0ec 100 kB/s | 16 kB 00:00 2024-09-21T18:45:42.871 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): ceph-grafana-dashboards-19.3.0-5129.g0e 180 kB/s | 34 kB 00:00 2024-09-21T18:45:42.871 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:43.007 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 145 k 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:45:43.008 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:43.009 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 145 k 2024-09-21T18:45:43.009 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 432 k 2024-09-21T18:45:43.009 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:45:43.096 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-21T18:45:43.138 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-09-21T18:45:43.213 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.1 MB/s | 548 kB 00:00 2024-09-21T18:45:43.239 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-21T18:45:43.258 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-19.3.0-5129.g0ecbb4 582 kB/s | 145 kB 00:00 2024-09-21T18:45:43.259 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:43.259 INFO:teuthology.orchestra.run.smithi081.stdout:Total 578 kB/s | 145 kB 00:00 2024-09-21T18:45:43.259 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:45:43.267 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:45:43.267 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:45:43.324 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:45:43.324 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:45:43.381 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): ceph-mgr-dashboard-19.3.0-5129.g0ecbb43 20 MB/s | 14 MB 00:00 2024-09-21T18:45:43.439 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 10 MB/s | 2.0 MB 00:00 2024-09-21T18:45:43.463 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:45:43.473 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-09-21T18:45:43.498 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 525 kB/s | 31 kB 00:00 2024-09-21T18:45:43.531 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 961 kB/s | 144 kB 00:00 2024-09-21T18:45:43.551 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435. 1/1 2024-09-21T18:45:43.565 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-21T18:45:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:43.569 INFO:teuthology.orchestra.run.smithi104.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-21T18:45:43.575 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435. 1/1 2024-09-21T18:45:43.575 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:45:43.576 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T18:45:43.576 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:45:43.636 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:45:43.636 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:45:43.911 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:45:43.911 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:45:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:45:44.550 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435. 1/1 2024-09-21T18:45:44.550 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:44.550 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:45:44.550 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:45:44.550 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:44.550 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:44.599 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T18:45:44.634 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T18:45:44.680 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T18:45:44.719 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T18:45:44.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2024-09-21T18:45:44.836 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T18:45:44.970 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T18:45:45.064 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T18:45:45.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T18:45:45.136 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T18:45:45.168 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5129.g0ecbb435.el9 10/12 2024-09-21T18:45:45.378 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:45.467 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-mgr-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:45:45.509 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:45.510 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:45:45.510 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:45.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T18:45:46.154 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:46.285 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 14 M 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 34 k 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 16 k 2024-09-21T18:45:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-09-21T18:45:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 19 M 2024-09-21T18:45:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 159 M 2024-09-21T18:45:46.288 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:45:46.779 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): ceph-grafana-dashboards-19.3.0-5129.g0e 225 kB/s | 34 kB 00:00 2024-09-21T18:45:46.804 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): ceph-prometheus-alerts-19.3.0-5129.g0ec 93 kB/s | 16 kB 00:00 2024-09-21T18:45:46.946 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-21T18:45:47.005 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-09-21T18:45:47.072 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-09-21T18:45:47.097 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 788 kB/s | 19 kB 00:00 2024-09-21T18:45:47.164 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-21T18:45:47.190 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-21T18:45:47.215 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T18:45:47.241 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-21T18:45:47.383 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): ceph-mgr-dashboard-19.3.0-5129.g0ecbb4 19 MB/s | 14 MB 00:00 2024-09-21T18:45:47.446 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.7 MB/s | 863 kB 00:00 2024-09-21T18:45:47.449 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:47.449 INFO:teuthology.orchestra.run.smithi081.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-21T18:45:47.497 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:45:47.516 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:45:47.516 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:45:47.811 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:45:47.812 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:45:48.358 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5129.g0ecbb435.el 11/12 2024-09-21T18:45:48.405 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noa 12/12 2024-09-21T18:45:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:45:48.491 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T18:45:48.535 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T18:45:48.582 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T18:45:48.612 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T18:45:48.702 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T18:45:48.829 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T18:45:48.915 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T18:45:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T18:45:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T18:45:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5129.g0ecbb435.el9 10/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noa 12/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5129.g0ecbb435.el 1/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noa 2/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5129.g0ecbb435.el9 3/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T18:45:50.410 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T18:45:50.411 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T18:45:50.718 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:50.719 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:45:50.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T18:45:51.574 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:07 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:45:51.707 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 7.4 M 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T18:45:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-09-21T18:45:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:45:51.710 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 38 M 2024-09-21T18:45:51.710 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 212 M 2024-09-21T18:45:51.710 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:45:52.037 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T18:45:52.079 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-21T18:45:52.154 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-21T18:45:52.196 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-21T18:45:52.212 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5129.g0ecbb435.el 11/12 2024-09-21T18:45:52.238 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 922 kB/s | 38 kB 00:00 2024-09-21T18:45:52.274 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noa 12/12 2024-09-21T18:45:52.338 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-51 15 MB/s | 7.4 MB 00:00 2024-09-21T18:45:52.522 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.8 MB/s | 3.0 MB 00:00 2024-09-21T18:45:52.555 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-21T18:45:52.647 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-09-21T18:45:52.673 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-09-21T18:45:52.756 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-09-21T18:45:53.040 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-21T18:45:53.040 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:45:53.041 INFO:teuthology.orchestra.run.smithi104.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-21T18:45:53.399 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:45:53.425 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:45:53.425 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:45:53.856 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:45:53.856 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noa 12/12 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5129.g0ecbb435.el 1/12 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noa 2/12 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5129.g0ecbb435.el9 3/12 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T18:45:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T18:45:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T18:45:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T18:45:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T18:45:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T18:45:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T18:45:54.607 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:54.608 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:45:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:45:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T18:45:54.757 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T18:45:54.852 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T18:45:54.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T18:45:54.897 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T18:45:55.082 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T18:45:55.112 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T18:45:55.192 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T18:45:55.455 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:09 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:45:55.598 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================= 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================= 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 7.4 M 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T18:45:55.599 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================= 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-09-21T18:45:55.600 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:45:55.601 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 38 M 2024-09-21T18:45:55.601 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 212 M 2024-09-21T18:45:55.601 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:45:55.839 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T18:45:56.258 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-51 17 MB/s | 7.4 MB 00:00 2024-09-21T18:45:56.333 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 383 kB/s | 191 kB 00:00 2024-09-21T18:45:56.525 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-21T18:45:56.808 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 59 kB/s | 33 kB 00:00 2024-09-21T18:45:57.008 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-21T18:45:57.190 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T18:45:57.367 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 69 kB/s | 38 kB 00:00 2024-09-21T18:45:57.693 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-09-21T18:45:58.220 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T18:45:58.253 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb 12/12 2024-09-21T18:45:58.259 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 274 kB/s | 245 kB 00:00 2024-09-21T18:45:58.485 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 3.3 MB/s | 4.9 MB 00:01 2024-09-21T18:45:59.068 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.7 MB/s | 5.1 MB 00:01 2024-09-21T18:45:59.093 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 301 kB/s | 250 kB 00:00 2024-09-21T18:46:00.377 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.6 MB/s | 16 MB 00:01 2024-09-21T18:46:00.378 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:00.379 INFO:teuthology.orchestra.run.smithi081.stdout:Total 8.0 MB/s | 38 MB 00:04 2024-09-21T18:46:00.741 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:00.766 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:00.766 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:00.862 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb 12/12 2024-09-21T18:46:00.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb 1/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T18:46:00.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T18:46:01.203 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:01.203 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T18:46:01.288 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:01.289 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:01.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-09-21T18:46:02.004 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:02.083 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T18:46:02.131 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T18:46:02.234 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T18:46:02.276 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T18:46:02.354 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:18 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:02.461 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T18:46:02.486 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 49 k 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T18:46:02.487 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:02.488 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-09-21T18:46:02.489 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:02.489 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.2 M 2024-09-21T18:46:02.489 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-09-21T18:46:02.489 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:02.492 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T18:46:02.569 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T18:46:02.661 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 213 kB/s | 26 kB 00:00 2024-09-21T18:46:02.686 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 19 kB 00:00 2024-09-21T18:46:02.711 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): ceph-mgr-rook-19.3.0-5129.g0ecbb435.el9 289 kB/s | 49 kB 00:00 2024-09-21T18:46:02.795 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-21T18:46:02.820 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-21T18:46:02.862 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 773 kB/s | 32 kB 00:00 2024-09-21T18:46:02.887 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 801 kB/s | 54 kB 00:00 2024-09-21T18:46:02.913 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 282 kB/s | 14 kB 00:00 2024-09-21T18:46:02.946 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-21T18:46:02.988 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-21T18:46:03.013 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.7 MB/s | 90 kB 00:00 2024-09-21T18:46:03.039 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 653 kB/s | 59 kB 00:00 2024-09-21T18:46:03.072 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-21T18:46:03.074 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:03.075 INFO:teuthology.orchestra.run.smithi104.stdout:Total 3.8 MB/s | 2.2 MB 00:00 2024-09-21T18:46:03.139 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:03.154 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:03.154 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T18:46:03.244 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:03.244 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:03.339 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T18:46:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:03.916 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T18:46:03.972 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T18:46:04.027 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T18:46:04.057 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T18:46:04.098 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T18:46:04.200 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T18:46:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T18:46:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T18:46:04.608 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T18:46:05.074 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T18:46:05.122 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T18:46:05.156 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T18:46:05.203 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T18:46:05.226 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 13/13 2024-09-21T18:46:05.645 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T18:46:05.675 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb 12/12 2024-09-21T18:46:05.991 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 13/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 1/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T18:46:05.992 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T18:46:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:06.316 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:06.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T18:46:07.190 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:23 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:07.322 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 156 k 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T18:46:07.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 7.0 M 2024-09-21T18:46:07.324 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 33 M 2024-09-21T18:46:07.325 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:07.582 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-21T18:46:07.607 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): ceph-mgr-cephadm-19.3.0-5129.g0ecbb435.e 680 kB/s | 156 kB 00:00 2024-09-21T18:46:07.665 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-21T18:46:07.691 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 689 kB/s | 58 kB 00:00 2024-09-21T18:46:07.716 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-21T18:46:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb 12/12 2024-09-21T18:46:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb 1/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T18:46:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T18:46:08.659 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T18:46:08.660 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T18:46:08.660 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T18:46:08.660 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T18:46:08.660 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T18:46:08.660 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:08.660 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:08.741 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-09-21T18:46:08.744 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:08.744 INFO:teuthology.orchestra.run.smithi104.stdout:Total 4.9 MB/s | 7.0 MB 00:01 2024-09-21T18:46:08.814 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:08.826 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:08.826 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:08.926 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:08.929 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:09.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2024-09-21T18:46:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:09.424 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T18:46:09.514 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T18:46:09.639 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:23 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:09.771 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:09.772 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:09.772 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:46:09.773 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:09.773 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:09.773 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 49 k 2024-09-21T18:46:09.773 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:46:09.773 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2024-09-21T18:46:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:09.775 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.2 M 2024-09-21T18:46:09.775 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-09-21T18:46:09.775 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T18:46:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T18:46:10.231 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T18:46:10.251 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 227 kB/s | 26 kB 00:00 2024-09-21T18:46:10.255 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 6/6 2024-09-21T18:46:10.275 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 19 kB 00:00 2024-09-21T18:46:10.301 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): ceph-mgr-rook-19.3.0-5129.g0ecbb435.el9 302 kB/s | 49 kB 00:00 2024-09-21T18:46:10.393 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-21T18:46:10.418 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-21T18:46:10.463 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-21T18:46:10.485 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T18:46:10.510 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-21T18:46:10.544 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-21T18:46:10.578 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-21T18:46:10.611 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.8 MB/s | 90 kB 00:00 2024-09-21T18:46:10.636 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-09-21T18:46:10.670 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-21T18:46:10.673 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:10.673 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-21T18:46:10.738 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:10.753 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:10.753 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:10.840 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:10.842 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:11.044 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 6/6 2024-09-21T18:46:11.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/6 2024-09-21T18:46:11.045 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T18:46:11.045 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T18:46:11.045 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T18:46:11.045 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T18:46:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T18:46:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:11.313 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:11.314 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:11.346 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:11.511 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T18:46:11.563 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T18:46:11.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-09-21T18:46:11.628 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T18:46:11.665 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T18:46:11.706 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T18:46:11.809 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T18:46:11.927 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T18:46:11.983 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T18:46:12.190 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:12.323 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:12.323 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:12.323 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:46:12.323 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:12.323 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:12.323 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 900 k 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:12.324 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 979 k 2024-09-21T18:46:12.325 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.9 M 2024-09-21T18:46:12.325 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:12.720 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T18:46:12.772 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T18:46:12.805 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T18:46:12.854 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T18:46:12.892 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 13/13 2024-09-21T18:46:12.893 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): ceph-fuse-19.3.0-5129.g0ecbb435.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-21T18:46:13.076 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 168 kB/s | 80 kB 00:00 2024-09-21T18:46:13.077 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:13.077 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.3 MB/s | 979 kB 00:00 2024-09-21T18:46:13.091 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:13.115 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:13.115 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:13.198 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:13.199 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:13.389 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T18:46:13.601 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/2 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 13/13 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 1/13 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T18:46:13.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T18:46:13.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T18:46:13.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T18:46:13.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T18:46:13.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T18:46:13.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T18:46:13.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T18:46:14.079 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T18:46:14.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T18:46:14.080 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T18:46:14.081 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:14.081 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:14.302 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/2 2024-09-21T18:46:14.302 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/2 2024-09-21T18:46:14.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T18:46:14.568 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T18:46:14.569 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:14.569 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:14.569 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T18:46:14.569 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:14.569 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:14.904 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:14.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-volume 2024-09-21T18:46:15.037 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 156 k 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T18:46:15.038 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-09-21T18:46:15.039 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:15.040 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 7.0 M 2024-09-21T18:46:15.040 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 33 M 2024-09-21T18:46:15.040 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:15.497 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:15.548 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-09-21T18:46:15.615 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-21T18:46:15.633 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:15.634 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:15.634 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-09-21T18:46:15.634 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:15.634 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:15.634 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 285 k 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 285 k 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 1.3 M 2024-09-21T18:46:15.635 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:15.640 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-21T18:46:15.666 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): ceph-mgr-cephadm-19.3.0-5129.g0ecbb435.e 656 kB/s | 156 kB 00:00 2024-09-21T18:46:15.691 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-21T18:46:15.783 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-21T18:46:15.785 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:15.785 INFO:teuthology.orchestra.run.smithi081.stdout:Total 9.4 MB/s | 7.0 MB 00:00 2024-09-21T18:46:15.855 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:15.863 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-volume-19.3.0-5129.g0ecbb435.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T18:46:15.864 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:15.864 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T18:46:15.864 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:15.866 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:15.866 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:15.873 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:15.873 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:15.903 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:15.904 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:15.967 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:15.969 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:16.118 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:16.182 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:46:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:46:16.199 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:46:16.199 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T18:46:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:16.275 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:16.457 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T18:46:16.518 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T18:46:17.042 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T18:46:17.128 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T18:46:17.206 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T18:46:17.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:46:17.230 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:17.230 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:17.230 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:17.230 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:17.230 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:17.239 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 6/6 2024-09-21T18:46:17.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-09-21T18:46:18.094 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 6/6 2024-09-21T18:46:18.094 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 1/6 2024-09-21T18:46:18.094 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T18:46:18.094 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T18:46:18.094 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T18:46:18.094 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T18:46:18.133 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:18.270 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 127 k 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:18.271 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 127 k 2024-09-21T18:46:18.272 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 456 k 2024-09-21T18:46:18.272 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:18.370 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:18.460 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-19.3.0-5129.g0ecbb435.el9.x86_64 673 kB/s | 127 kB 00:00 2024-09-21T18:46:18.460 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:18.461 INFO:teuthology.orchestra.run.smithi104.stdout:Total 668 kB/s | 127 kB 00:00 2024-09-21T18:46:18.461 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:18.466 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:18.466 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:18.531 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:18.531 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:18.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2024-09-21T18:46:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:18.800 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:19.228 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:19.266 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:19.361 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:19.361 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:19.361 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 900 k 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-09-21T18:46:19.362 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:19.363 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 979 k 2024-09-21T18:46:19.363 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.9 M 2024-09-21T18:46:19.363 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:19.532 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:19.532 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:19.543 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 502 kB/s | 80 kB 00:00 2024-09-21T18:46:19.702 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-fuse-19.3.0-5129.g0ecbb435.el9.x86_ 2.8 MB/s | 900 kB 00:00 2024-09-21T18:46:19.702 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:19.703 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.8 MB/s | 979 kB 00:00 2024-09-21T18:46:19.715 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:19.737 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:19.737 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:19.820 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:19.820 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:19.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-09-21T18:46:20.003 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:20.138 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T18:46:20.210 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/2 2024-09-21T18:46:20.436 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:20.530 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:20.575 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:20.576 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:20.576 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:20.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-09-21T18:46:20.889 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2/2 2024-09-21T18:46:20.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/2 2024-09-21T18:46:21.190 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T18:46:21.190 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:21.190 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:21.190 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T18:46:21.190 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:21.190 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:21.233 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 32 k 2024-09-21T18:46:21.366 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 32 k 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 142 k 2024-09-21T18:46:21.367 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:21.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-volume 2024-09-21T18:46:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-19.3.0-5129.g0ecbb435.el9.x86_6 207 kB/s | 32 kB 00:00 2024-09-21T18:46:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:21.522 INFO:teuthology.orchestra.run.smithi104.stdout:Total 205 kB/s | 32 kB 00:00 2024-09-21T18:46:21.522 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:21.547 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:21.548 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:21.641 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:21.727 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:22.075 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:22.124 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:22.213 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.3.0-5129.g0ecbb435.el9 ceph-noarch 285 k 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:46:22.214 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:22.215 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 285 k 2024-09-21T18:46:22.215 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.3 M 2024-09-21T18:46:22.215 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:22.352 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:22.352 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:22.352 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:22.352 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:22.352 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:22.352 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:22.457 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-volume-19.3.0-5129.g0ecbb435.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-09-21T18:46:22.458 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:22.458 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-21T18:46:22.458 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:22.466 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:22.467 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:22.499 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:22.500 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:22.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-09-21T18:46:22.719 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:22.784 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:46:22.800 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:46:22.800 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:46:22.800 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T18:46:22.800 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:23.123 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:23.212 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:23.255 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:23.256 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:23.256 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:23.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-09-21T18:46:23.820 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 1/1 2024-09-21T18:46:23.820 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:23.820 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:23.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.3.0-5129.g0ecbb435.el9.noarch 2024-09-21T18:46:23.820 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:23.820 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:23.923 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:24.012 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:24.055 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:24.056 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:24.056 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:24.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-09-21T18:46:24.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel 2024-09-21T18:46:24.726 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:24.769 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:24.818 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:24.862 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:24.863 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:24.863 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:24.905 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 127 k 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:46:24.906 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:24.907 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 127 k 2024-09-21T18:46:24.907 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 456 k 2024-09-21T18:46:24.907 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:24.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-09-21T18:46:25.097 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-19.3.0-5129.g0ecbb435.el9.x86_64 669 kB/s | 127 kB 00:00 2024-09-21T18:46:25.097 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:25.097 INFO:teuthology.orchestra.run.smithi081.stdout:Total 663 kB/s | 127 kB 00:00 2024-09-21T18:46:25.098 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:25.103 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:25.103 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:25.168 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:25.168 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:25.318 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:25.436 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:25.517 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:25.607 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:25.651 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:25.651 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:25.652 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:25.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-09-21T18:46:25.939 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:26.245 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:26.246 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:26.246 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:26.246 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:26.246 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:26.246 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:26.320 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:26.409 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:26.452 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:26.453 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:26.453 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:26.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-09-21T18:46:26.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2024-09-21T18:46:27.123 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:27.175 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:27.211 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:27.255 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:27.257 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-09-21T18:46:27.257 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:27.265 INFO:teuthology.orchestra.run.smithi081.stdout:Package libcephfs2-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:27.309 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:27.310 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:27.310 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:27.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-09-21T18:46:27.404 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2024-09-21T18:46:27.926 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:27.987 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:28.059 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:28.059 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:28.059 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-21T18:46:28.059 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 85 k 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:28.060 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 85 k 2024-09-21T18:46:28.061 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 231 k 2024-09-21T18:46:28.061 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:28.121 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 32 k 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:28.122 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:28.123 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:28.123 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:46:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:28.123 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 32 k 2024-09-21T18:46:28.123 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 142 k 2024-09-21T18:46:28.123 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:28.249 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-19.3.0-5129.g0ecbb435.el9.x86_64.rpm 452 kB/s | 85 kB 00:00 2024-09-21T18:46:28.249 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:28.250 INFO:teuthology.orchestra.run.smithi104.stdout:Total 449 kB/s | 85 kB 00:00 2024-09-21T18:46:28.250 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:28.257 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:28.257 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:28.268 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-devel-19.3.0-5129.g0ecbb435.el9.x86_6 218 kB/s | 32 kB 00:00 2024-09-21T18:46:28.269 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:28.269 INFO:teuthology.orchestra.run.smithi081.stdout:Total 216 kB/s | 32 kB 00:00 2024-09-21T18:46:28.269 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:28.273 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:28.273 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:28.296 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:28.296 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:28.321 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:28.321 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:28.492 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:28.589 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:29.109 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:29.110 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:29.110 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:29.110 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:29.110 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:29.110 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:29.187 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:29.187 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:29.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2024-09-21T18:46:29.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-09-21T18:46:29.873 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:29.963 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:30.006 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:30.007 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:30.007 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:30.076 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2024-09-21T18:46:30.144 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:30.287 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:30.287 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 3.2 M 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.2 M 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 14 M 2024-09-21T18:46:30.288 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:30.636 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-19.3.0-5129.g0ecbb435.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-21T18:46:30.636 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:30.637 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-09-21T18:46:30.637 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:30.646 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:30.647 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:30.662 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:30.729 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:30.730 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:30.752 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:30.795 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:30.796 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:30.796 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:30.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados 2024-09-21T18:46:31.249 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:31.320 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:31.345 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:31.345 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:46:31.345 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T18:46:31.346 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T18:46:31.346 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T18:46:31.346 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:31.451 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:31.539 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rados-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:31.583 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:31.584 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:31.584 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:31.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw 2024-09-21T18:46:32.241 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:32.329 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rgw-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:32.372 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:32.373 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:32.373 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:32.437 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:32.437 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:32.437 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:32.437 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:32.437 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:32.437 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:32.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs 2024-09-21T18:46:32.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-09-21T18:46:33.029 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:33.117 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-cephfs-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:33.160 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:33.161 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:33.161 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:33.251 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd 2024-09-21T18:46:33.286 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 06:44:44 PM UTC. 2024-09-21T18:46:33.417 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 170 k 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T18:46:33.418 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T18:46:33.419 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-09-21T18:46:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:33.419 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 170 k 2024-09-21T18:46:33.419 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 490 k 2024-09-21T18:46:33.419 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T18:46:33.638 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-19.3.0-5129.g0ecbb435.el9.x86_64.rpm 780 kB/s | 170 kB 00:00 2024-09-21T18:46:33.639 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:33.639 INFO:teuthology.orchestra.run.smithi104.stdout:Total 773 kB/s | 170 kB 00:00 2024-09-21T18:46:33.639 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T18:46:33.647 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T18:46:33.647 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T18:46:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T18:46:33.711 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T18:46:33.835 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:33.866 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T18:46:33.926 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 is already installed. 2024-09-21T18:46:33.969 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:33.970 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-21T18:46:33.971 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:33.978 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:34.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2024-09-21T18:46:34.399 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:48 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:34.670 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T18:46:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:46:34.671 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T18:46:34.756 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:34.756 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:34.756 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 85 k 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:34.757 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 85 k 2024-09-21T18:46:34.758 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 231 k 2024-09-21T18:46:34.758 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:34.903 DEBUG:teuthology.parallel:result is None 2024-09-21T18:46:35.097 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-19.3.0-5129.g0ecbb435.el9.x86_64.rpm 251 kB/s | 85 kB 00:00 2024-09-21T18:46:35.097 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:35.097 INFO:teuthology.orchestra.run.smithi081.stdout:Total 250 kB/s | 85 kB 00:00 2024-09-21T18:46:35.098 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:35.104 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:35.105 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:35.171 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:35.171 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:35.319 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:35.428 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:35.811 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:36.079 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:36.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:36.079 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:36.079 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:36.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:36.079 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:36.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror 2024-09-21T18:46:36.941 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:50 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:37.077 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 3.2 M 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:37.078 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:46:37.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:37.079 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.2 M 2024-09-21T18:46:37.079 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 14 M 2024-09-21T18:46:37.079 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:37.436 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-19.3.0-5129.g0ecbb435.el9.x86_64.rpm 8.9 MB/s | 3.2 MB 00:00 2024-09-21T18:46:37.437 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:37.437 INFO:teuthology.orchestra.run.smithi081.stdout:Total 8.9 MB/s | 3.2 MB 00:00 2024-09-21T18:46:37.438 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:37.447 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:37.447 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:37.529 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:37.529 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:38.020 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:38.095 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:38.121 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T18:46:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T18:46:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T18:46:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T18:46:38.122 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:39.255 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:39.255 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:39.255 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:39.255 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:39.255 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:39.255 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:39.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd 2024-09-21T18:46:40.110 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:54 ago on Sat 21 Sep 2024 06:44:46 PM UTC. 2024-09-21T18:46:40.247 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.3.0-5129.g0ecbb435.el9 ceph 170 k 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-21T18:46:40.248 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-21T18:46:40.249 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-21T18:46:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:40.249 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 170 k 2024-09-21T18:46:40.249 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 490 k 2024-09-21T18:46:40.249 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-21T18:46:40.466 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-19.3.0-5129.g0ecbb435.el9.x86_64.rpm 789 kB/s | 170 kB 00:00 2024-09-21T18:46:40.466 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:46:40.466 INFO:teuthology.orchestra.run.smithi081.stdout:Total 783 kB/s | 170 kB 00:00 2024-09-21T18:46:40.467 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-21T18:46:40.474 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-21T18:46:40.474 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-21T18:46:40.540 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-21T18:46:40.540 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-21T18:46:40.706 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-21T18:46:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:41.232 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:41.454 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 1/1 2024-09-21T18:46:41.454 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:41.455 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-21T18:46:41.455 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.3.0-5129.g0ecbb435.el9.x86_64 2024-09-21T18:46:41.455 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:46:41.455 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-21T18:46:41.688 DEBUG:teuthology.parallel:result is None 2024-09-21T18:46:41.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:46:41.839 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T18:46:41.860 INFO:teuthology.orchestra.run.smithi081.stdout:19.3.0-5129.g0ecbb435.el9 2024-09-21T18:46:41.860 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5129.g0ecbb435.el9 2024-09-21T18:46:41.860 INFO:teuthology.task.install:The correct ceph version 19.3.0-5129.g0ecbb435 is installed. 2024-09-21T18:46:41.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:46:42.013 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T18:46:42.038 INFO:teuthology.orchestra.run.smithi104.stdout:19.3.0-5129.g0ecbb435.el9 2024-09-21T18:46:42.038 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5129.g0ecbb435.el9 2024-09-21T18:46:42.038 INFO:teuthology.task.install:The correct ceph version 19.3.0-5129.g0ecbb435 is installed. 2024-09-21T18:46:42.040 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T18:46:42.040 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:46:42.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T18:46:42.065 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:46:42.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T18:46:42.107 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T18:46:42.107 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:46:42.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T18:46:42.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T18:46:42.197 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:46:42.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T18:46:42.225 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T18:46:42.291 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T18:46:42.291 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:46:42.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T18:46:42.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T18:46:42.380 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:46:42.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T18:46:42.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T18:46:42.471 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T18:46:42.472 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:46:42.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T18:46:42.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T18:46:42.560 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:46:42.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T18:46:42.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T18:46:42.650 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T18:46:42.738 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe', 'cephadm_mode': 'root'} 2024-09-21T18:46:42.738 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:46:42.739 INFO:tasks.cephadm:Cluster fsid is d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:46:42.739 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T18:46:42.739 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.81:6789]', 'mon.c': '[v1:172.21.15.81:6790]', 'mon.b': '[v1:172.21.15.104:6789]'} 2024-09-21T18:46:42.739 INFO:tasks.cephadm:First mon is mon.a on smithi081 2024-09-21T18:46:42.739 INFO:tasks.cephadm:First mgr is y 2024-09-21T18:46:42.739 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T18:46:42.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo hostname $(hostname -s) 2024-09-21T18:46:42.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-09-21T18:46:42.789 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T18:46:42.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:46:42.943 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83206/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5129-g0ecbb435', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5129.g0ecbb435'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 15:40:29.818446', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-09-21-1014', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-21T18:46:43.085 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-adk-testing-2024-09-21-1014, sha1 0ecbb435437226bc091a0c2d4549ba25bf724efe from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:46:43.086 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/x86_64/flavors/default/cephadm 2024-09-21T18:46:43.087 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/x86_64/flavors/default/cephadm 2024-09-21T18:46:43.087 DEBUG:teuthology.orchestra.run.smithi081:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T18:46:43.603 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r--. 1 ubuntu ubuntu 804323 Sep 21 18:46 /home/ubuntu/cephtest/cephadm 2024-09-21T18:46:43.604 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-09-21-1014/0ecbb435437226bc091a0c2d4549ba25bf724efe/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T18:46:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 804323 Sep 21 18:46 /home/ubuntu/cephtest/cephadm 2024-09-21T18:46:44.107 DEBUG:teuthology.orchestra.run.smithi081:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T18:46:44.127 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T18:46:44.157 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe on all hosts... 2024-09-21T18:46:44.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe pull 2024-09-21T18:46:44.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe pull 2024-09-21T18:46:45.808 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe... 2024-09-21T18:46:45.835 INFO:teuthology.orchestra.run.smithi081.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe... 2024-09-21T18:47:31.222 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:47:31.222 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)", 2024-09-21T18:47:31.222 INFO:teuthology.orchestra.run.smithi081.stdout: "image_id": "3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25", 2024-09-21T18:47:31.223 INFO:teuthology.orchestra.run.smithi081.stdout: "repo_digests": [ 2024-09-21T18:47:31.223 INFO:teuthology.orchestra.run.smithi081.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5" 2024-09-21T18:47:31.223 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:47:31.223 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:47:38.369 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:47:38.370 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)", 2024-09-21T18:47:38.370 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25", 2024-09-21T18:47:38.370 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-09-21T18:47:38.370 INFO:teuthology.orchestra.run.smithi104.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5" 2024-09-21T18:47:38.370 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:47:38.370 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:47:38.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2024-09-21T18:47:38.431 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-09-21T18:47:38.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /etc/ceph 2024-09-21T18:47:38.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-09-21T18:47:38.523 INFO:tasks.cephadm:Writing seed config... 2024-09-21T18:47:38.524 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T18:47:38.524 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T18:47:38.524 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-21T18:47:38.524 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-21T18:47:38.524 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T18:47:38.525 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-21T18:47:38.526 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-21T18:47:38.526 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-21T18:47:38.526 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-21T18:47:38.526 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-21T18:47:38.526 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T18:47:38.526 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-21T18:47:38.526 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:47:38.526 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T18:47:38.557 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d839dc86-7849-11ef-baf6-efdc52797490 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-21T18:47:38.557 DEBUG:teuthology.orchestra.run.smithi081:mon.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T18:47:38.599 DEBUG:teuthology.orchestra.run.smithi081:mgr.y> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T18:47:38.642 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T18:47:38.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe -v bootstrap --fsid d839dc86-7849-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.81:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T18:47:38.847 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-21T18:47:38.847 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe', '-v', 'bootstrap', '--fsid', 'd839dc86-7849-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.81:6789]', '--skip-admin-label'] 2024-09-21T18:47:38.847 INFO:teuthology.orchestra.run.smithi081.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T18:47:38.848 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying podman|docker is present... 2024-09-21T18:47:38.871 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 5.2.2 2024-09-21T18:47:38.872 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying lvm2 is present... 2024-09-21T18:47:38.872 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying time synchronization is in place... 2024-09-21T18:47:38.879 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T18:47:38.879 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T18:47:38.886 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T18:47:38.886 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-09-21T18:47:38.893 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout enabled 2024-09-21T18:47:38.899 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout active 2024-09-21T18:47:38.899 INFO:teuthology.orchestra.run.smithi081.stdout:Unit chronyd.service is enabled and running 2024-09-21T18:47:38.899 INFO:teuthology.orchestra.run.smithi081.stdout:Repeating the final host check... 2024-09-21T18:47:38.922 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 5.2.2 2024-09-21T18:47:38.923 INFO:teuthology.orchestra.run.smithi081.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T18:47:38.923 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl is present 2024-09-21T18:47:38.923 INFO:teuthology.orchestra.run.smithi081.stdout:lvcreate is present 2024-09-21T18:47:38.929 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T18:47:38.929 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T18:47:38.935 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T18:47:38.935 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-09-21T18:47:38.942 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout enabled 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout active 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Unit chronyd.service is enabled and running 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Host looks OK 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Cluster fsid: d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Acquiring lock 140387735452016 on /run/cephadm/d839dc86-7849-11ef-baf6-efdc52797490.lock 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Lock 140387735452016 acquired on /run/cephadm/d839dc86-7849-11ef-baf6-efdc52797490.lock 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying IP 172.21.15.81 port 6789 ... 2024-09-21T18:47:38.949 INFO:teuthology.orchestra.run.smithi081.stdout:Base mon IP(s) is [172.21.15.81:6789], mon addrv is [v1:172.21.15.81:6789] 2024-09-21T18:47:38.953 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.81 metric 100 2024-09-21T18:47:38.953 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.81 metric 100 2024-09-21T18:47:38.955 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T18:47:38.955 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T18:47:38.955 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d753/64 scope link noprefixroute 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T18:47:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:Mon IP `172.21.15.81` is in CIDR network `172.21.0.0/20` 2024-09-21T18:47:38.959 INFO:teuthology.orchestra.run.smithi081.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-21T18:47:38.959 INFO:teuthology.orchestra.run.smithi081.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T18:47:38.960 INFO:teuthology.orchestra.run.smithi081.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe... 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe... 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:21b5a2bb4fd2b634e899a996bd2c361c29a2bbdb6dc6de9dcb7e17a30e3d130c 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying config sha256:3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 2024-09-21T18:47:39.399 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T18:47:41.460 INFO:teuthology.orchestra.run.smithi081.stdout:ceph: stdout ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev) 2024-09-21T18:47:41.460 INFO:teuthology.orchestra.run.smithi081.stdout:Ceph version: ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev) 2024-09-21T18:47:41.460 INFO:teuthology.orchestra.run.smithi081.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T18:47:51.106 INFO:teuthology.orchestra.run.smithi081.stdout:stat: stdout 167 167 2024-09-21T18:47:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial keys... 2024-09-21T18:47:53.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQDZFO9mnrxOABAAC8gIW1ObFi89z6PB483VBg== 2024-09-21T18:47:54.486 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQDZFO9mO3/hMhAAfRvl0iSvyla81CQB9nUK9A== 2024-09-21T18:47:55.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQDaFO9mEC5QLhAARNp7rFw/MUOdA+sKIvCtvw== 2024-09-21T18:47:55.359 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial monmap... 2024-09-21T18:47:56.265 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T18:47:56.265 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T18:47:56.265 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:47:56.265 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T18:47:56.265 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool for a [v1:172.21.15.81:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T18:47:56.266 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = quincy 2024-09-21T18:47:56.266 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: set fsid to d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:47:56.266 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T18:47:56.266 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:47:56.266 INFO:teuthology.orchestra.run.smithi081.stdout:Creating mon... 2024-09-21T18:47:57.455 INFO:teuthology.orchestra.run.smithi081.stdout:create mon.a on 2024-09-21T18:47:57.659 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T18:47:57.797 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T18:47:57.961 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d839dc86-7849-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-d839dc86-7849-11ef-baf6-efdc52797490.target. 2024-09-21T18:47:57.961 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d839dc86-7849-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-d839dc86-7849-11ef-baf6-efdc52797490.target. 2024-09-21T18:47:58.196 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a 2024-09-21T18:47:58.196 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T18:47:58.449 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d839dc86-7849-11ef-baf6-efdc52797490.target.wants/ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-d839dc86-7849-11ef-baf6-efdc52797490@.service. 2024-09-21T18:47:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:58 smithi081 podman[28989]: 2024-09-21 18:47:58.591616034 +0000 UTC m=+0.016239080 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:47:59.366 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 podman[28989]: 2024-09-21 18:47:59.099358279 +0000 UTC m=+0.523981318 container create 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T18:47:59.366 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 podman[28989]: 2024-09-21 18:47:59.31837473 +0000 UTC m=+0.742997777 container init 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T18:47:59.366 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 podman[28989]: 2024-09-21 18:47:59.32153545 +0000 UTC m=+0.746158491 container start 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T18:47:59.394 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T18:47:59.395 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-09-21T18:47:59.402 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T18:47:59.403 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-09-21T18:47:59.403 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-09-21T18:47:59.403 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T18:47:59.403 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mon to start... 2024-09-21T18:47:59.403 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mon... 2024-09-21T18:47:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 ceph-mon[29016]: mkfs d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:47:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 ceph-mon[29016]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T18:47:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 bash[28989]: 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 2024-09-21T18:47:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:47:59 smithi081 systemd[1]: Started Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:48:00.480 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T18:48:00.480 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout id: d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout services: 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.555689s) 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout data: 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:mon is available 2024-09-21T18:48:00.481 INFO:teuthology.orchestra.run.smithi081.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T18:48:00.594 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T18:48:00.594 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: monmap epoch 1 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: last_changed 2024-09-21T18:47:55.713547+0000 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: min_mon_release 19 (squid) 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: election_strategy: 1 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: fsmap 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: mgrmap e1: no daemons active 2024-09-21T18:48:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:00 smithi081 ceph-mon[29016]: from='client.? v1:172.21.15.81:0/1676358690' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T18:48:01.563 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:01.563 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T18:48:01.563 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout fsid = d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:01.563 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.81:6789] 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T18:48:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:Generating new minimal ceph.conf... 2024-09-21T18:48:01.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:01 smithi081 ceph-mon[29016]: from='client.? v1:172.21.15.81:0/688038001' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T18:48:01.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:01 smithi081 ceph-mon[29016]: from='client.? v1:172.21.15.81:0/688038001' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T18:48:02.532 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:02 smithi081 ceph-mon[29016]: from='client.? v1:172.21.15.81:0/3045462202' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:48:02.801 INFO:teuthology.orchestra.run.smithi081.stdout:Restarting the monitor... 2024-09-21T18:48:03.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:02 smithi081 systemd[1]: Stopping Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:48:03.469 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29012]: 2024-09-21T18:48:03.066+0000 7f5eb9309640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T18:48:03.469 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29012]: 2024-09-21T18:48:03.066+0000 7f5eb9309640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T18:48:03.469 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 podman[29260]: 2024-09-21 18:48:03.220103143 +0000 UTC m=+0.254312363 container died 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T18:48:03.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 podman[29260]: 2024-09-21 18:48:03.566866302 +0000 UTC m=+0.601075561 container cleanup 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T18:48:03.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 bash[29260]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a 2024-09-21T18:48:04.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 podman[29274]: 2024-09-21 18:48:03.72605774 +0000 UTC m=+0.499618137 container remove 04dcc59aa35b5531e7359f27610c9818c53285bcfc99d6974df9bde260267434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T18:48:04.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T18:48:04.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 systemd[1]: Stopped Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:48:04.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:03 smithi081 systemd[1]: Starting Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 podman[29365]: 2024-09-21 18:48:04.039546701 +0000 UTC m=+0.015764796 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 podman[29365]: 2024-09-21 18:48:04.143747502 +0000 UTC m=+0.119965584 container create 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 podman[29365]: 2024-09-21 18:48:04.329609907 +0000 UTC m=+0.305828007 container init 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 podman[29365]: 2024-09-21 18:48:04.332628654 +0000 UTC m=+0.308846728 container start 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev), process ceph-mon, pid 2 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: pidfile_write: ignore empty --pid-file 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: load: jerasure load: lrc 2024-09-21T18:48:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: RocksDB version: 7.9.2 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Git sha 0 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Compile date 2024-09-21 14:38:11 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: DB SUMMARY 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: DB Session ID: WNKNZMZICCPORZH70SE3 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: CURRENT file: CURRENT 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92888 ; 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.error_if_exists: 0 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.create_if_missing: 0 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.paranoid_checks: 1 2024-09-21T18:48:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.env: 0x56139cb5d120 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.info_log: 0x56139e4db6a0 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.statistics: (nil) 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.use_fsync: 0 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_log_file_size: 0 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T18:48:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.allow_fallocate: 1 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.use_direct_reads: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.db_log_dir: 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.wal_dir: 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T18:48:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T18:48:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.write_buffer_manager: 0x56139e4dfb80 2024-09-21T18:48:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T18:48:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T18:48:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T18:48:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T18:48:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T18:48:04.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T18:48:04.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T18:48:04.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T18:48:04.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.unordered_write: 0 2024-09-21T18:48:04.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T18:48:04.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.row_cache: None 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.wal_filter: None 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.two_write_queues: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.wal_compression: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.atomic_flush: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.log_readahead_size: 0 2024-09-21T18:48:04.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_background_jobs: 2 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_background_compactions: -1 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_subcompactions: 1 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T18:48:04.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_open_files: -1 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_background_flushes: -1 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Compression algorithms supported: 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kZSTD supported: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kXpressCompression supported: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kBZip2Compression supported: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T18:48:04.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kLZ4Compression supported: 1 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kZlibCompression supported: 1 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: kSnappyCompression supported: 1 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.merge_operator: 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_filter: None 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T18:48:04.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56139e4db2c0) 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: cache_index_and_filter_blocks: 1 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: pin_top_level_index_and_filter: 1 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_type: 0 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: data_block_index_type: 0 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_shortening: 1 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: checksum: 4 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: no_block_cache: 0 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache: 0x56139e4cf610 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_name: BinnedLRUCache 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_options: 2024-09-21T18:48:04.417 INFO:journalctl@ceph.mon.a.smithi081.stdout: capacity : 536870912 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: num_shard_bits : 4 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: strict_capacity_limit : 0 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: high_pri_pool_ratio: 0.000 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_compressed: (nil) 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: persistent_cache: (nil) 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_size: 4096 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_size_deviation: 10 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_restart_interval: 16 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_block_restart_interval: 1 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: metadata_block_size: 4096 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: partition_filters: 0 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: use_delta_encoding: 1 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: filter_policy: bloomfilter 2024-09-21T18:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout: whole_key_filtering: 1 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: verify_compression: 0 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: read_amp_bytes_per_bit: 0 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: format_version: 5 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: enable_index_compression: 1 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_align: 0 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: max_auto_readahead_size: 262144 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: prepopulate_block_cache: 0 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: initial_auto_readahead_size: 8192 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression: NoCompression 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T18:48:04.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.num_levels: 7 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T18:48:04.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T18:48:04.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T18:48:04.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.inplace_update_support: 0 2024-09-21T18:48:04.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.bloom_locality: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.max_successive_merges: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.ttl: 2592000 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.enable_blob_files: false 2024-09-21T18:48:04.424 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.min_blob_size: 0 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4caae806-8971-4e64-aa57-34e66dd59d65 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944484358643, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944484359569, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88874, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 287, "table_properties": {"data_size": 86930, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11809, "raw_average_key_size": 48, "raw_value_size": 80402, "raw_average_value_size": 326, "num_data_blocks": 11, "num_entries": 246, "num_filter_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726944484, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4caae806-8971-4e64-aa57-34e66dd59d65", "db_session_id": "WNKNZMZICCPORZH70SE3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T18:48:04.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944484359661, "job": 1, "event": "recovery_finished"} 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56139e4fee00 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: DB pointer 0x56139e600000 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** DB Stats ** 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T18:48:04.426 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: L0 2/0 88.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: Sum 2/0 88.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T18:48:04.427 INFO:journalctl@ceph.mon.a.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative compaction: 0.00 GB write, 27.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval compaction: 0.00 GB write, 27.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Block cache BinnedLRUCache@0x56139e4cf610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T18:48:04.428 INFO:journalctl@ceph.mon.a.smithi081.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: starting mon.a rank 0 at public addrs v1:172.21.15.81:6789/0 at bind addrs v1:172.21.15.81:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???) e1 preinit fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).mds e1 new map 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).mds e1 print_map 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: e1 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: btime 2024-09-21T18:47:59:361071+0000 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: legacy client fscid: -1 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout: No filesystems configured 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T18:48:04.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:48:04.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:48:04.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:48:04.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T18:48:04.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 bash[29365]: 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 2024-09-21T18:48:04.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 systemd[1]: Started Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:48:04.430 INFO:teuthology.orchestra.run.smithi081.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T18:48:04.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T18:48:04.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: monmap epoch 1 2024-09-21T18:48:04.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:04.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: last_changed 2024-09-21T18:47:55.713547+0000 2024-09-21T18:48:04.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:48:04.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: min_mon_release 19 (squid) 2024-09-21T18:48:04.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: election_strategy: 1 2024-09-21T18:48:04.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:48:04.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: fsmap 2024-09-21T18:48:04.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T18:48:04.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:04 smithi081 ceph-mon[29384]: mgrmap e1: no daemons active 2024-09-21T18:48:05.366 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T18:48:05.387 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T18:48:05.387 INFO:teuthology.orchestra.run.smithi081.stdout:Creating mgr... 2024-09-21T18:48:05.388 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T18:48:05.388 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T18:48:05.633 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y 2024-09-21T18:48:05.633 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T18:48:05.783 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d839dc86-7849-11ef-baf6-efdc52797490.target.wants/ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-d839dc86-7849-11ef-baf6-efdc52797490@.service. 2024-09-21T18:48:06.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1135154525' entity='client.admin' 2024-09-21T18:48:06.309 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:06 smithi081 podman[29597]: 2024-09-21 18:48:06.209709204 +0000 UTC m=+0.310396866 container start 03c9b587d393ad02898829aee9d2be18bc7212a435a7bd59c3d71d37b66d1105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T18:48:06.324 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T18:48:06.324 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-09-21T18:48:06.332 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T18:48:06.332 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-09-21T18:48:06.333 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-09-21T18:48:06.333 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T18:48:06.342 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T18:48:06.342 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-09-21T18:48:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T18:48:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-09-21T18:48:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-09-21T18:48:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T18:48:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr to start... 2024-09-21T18:48:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr... 2024-09-21T18:48:06.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:06 smithi081 bash[29597]: 03c9b587d393ad02898829aee9d2be18bc7212a435a7bd59c3d71d37b66d1105 2024-09-21T18:48:06.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:06 smithi081 systemd[1]: Started Ceph mgr.y for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:48:07.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1602112635' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T18:48:07.485 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:07.487 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:07.487 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T18:48:07.487 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:07.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T18:48:07.489 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T18:48:07.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T18:47:59:361071+0000", 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T18:48:07.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T18:47:59.361856+0000", 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T18:48:07.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:07.495 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (1/15)... 2024-09-21T18:48:10.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1363886213' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T18:48:10.619 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T18:48:10.620 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T18:48:10.621 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T18:48:10.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T18:47:59:361071+0000", 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T18:48:10.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T18:47:59.361856+0000", 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:10.626 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (2/15)... 2024-09-21T18:48:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: Activating manager daemon y 2024-09-21T18:48:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: mgrmap e2: y(active, starting, since 0.00316813s) 2024-09-21T18:48:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: Manager daemon y is now available 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' 2024-09-21T18:48:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:11 smithi081 ceph-mon[29384]: from='mgr.14100 v1:172.21.15.81:0/2030711243' entity='mgr.y' 2024-09-21T18:48:13.284 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:13 smithi081 ceph-mon[29384]: mgrmap e3: y(active, since 1.00666s) 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T18:48:13.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T18:48:13.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T18:48:13.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T18:48:13.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T18:48:13.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T18:48:13.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T18:48:13.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T18:47:59:361071+0000", 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T18:48:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T18:47:59.361856+0000", 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T18:48:13.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:13.876 INFO:teuthology.orchestra.run.smithi081.stdout:mgr is available 2024-09-21T18:48:14.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:14 smithi081 ceph-mon[29384]: mgrmap e4: y(active, since 2s) 2024-09-21T18:48:14.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3263794628' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T18:48:15.024 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout fsid = d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.81:6789] 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T18:48:15.025 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T18:48:15.026 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T18:48:15.026 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling cephadm module... 2024-09-21T18:48:15.447 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1209369795' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T18:48:16.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/903540692' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T18:48:16.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:16 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ignoring --setuser ceph since I am not root 2024-09-21T18:48:16.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:16 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ignoring --setgroup ceph since I am not root 2024-09-21T18:48:17.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/903540692' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T18:48:17.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:17 smithi081 ceph-mon[29384]: mgrmap e5: y(active, since 5s) 2024-09-21T18:48:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2272074263' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T18:48:18.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:18.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T18:48:18.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T18:48:18.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T18:48:18.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T18:48:18.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:18.320 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for the mgr to restart... 2024-09-21T18:48:18.320 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr epoch 5... 2024-09-21T18:48:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: Active manager daemon y restarted 2024-09-21T18:48:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: Activating manager daemon y 2024-09-21T18:48:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T18:48:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: mgrmap e6: y(active, starting, since 0.00324164s) 2024-09-21T18:48:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:48:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T18:48:21.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:48:21.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:48:21.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:48:21.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:21 smithi081 ceph-mon[29384]: Manager daemon y is now available 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: Found migration_current of "None". Setting to last migration. 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:48:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='client.14122 v1:172.21.15.81:0/981940707' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: mgrmap e7: y(active, since 1.00759s) 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='client.14122 v1:172.21.15.81:0/981940707' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:48:22.706+0000 7f2775971640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:18:48:22] ENGINE Error in HTTPServer.serve 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: Traceback (most recent call last): 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:48:22.817 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._connections.run(self.expiration_interval) 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._run(expiration_interval) 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: s = self.context.wrap_socket( 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: return self.sslsocket_class._create( 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self.do_handshake() 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:48:22.818 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._sslobj.do_handshake() 2024-09-21T18:48:22.819 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:48:22.819 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:48:22.849 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:22.849 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T18:48:22.849 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T18:48:22.849 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:22.849 INFO:teuthology.orchestra.run.smithi081.stdout:mgr epoch 5 is available 2024-09-21T18:48:22.849 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying orchestrator module is enabled... 2024-09-21T18:48:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:23 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:22] ENGINE Bus STARTING 2024-09-21T18:48:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:23 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:22] ENGINE Serving on http://172.21.15.81:8765 2024-09-21T18:48:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:23 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:22] ENGINE Serving on https://172.21.15.81:7150 2024-09-21T18:48:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:23 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:22] ENGINE Bus STARTED 2024-09-21T18:48:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:23 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:22] ENGINE Error in HTTPServer.serve 2024-09-21T18:48:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout: Traceback (most recent call last): 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._connections.run(self.expiration_interval) 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._run(expiration_interval) 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: s = self.context.wrap_socket( 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: return self.sslsocket_class._create( 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:48:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: self.do_handshake() 2024-09-21T18:48:24.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:48:24.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._sslobj.do_handshake() 2024-09-21T18:48:24.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:48:24.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:23 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1546302150' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T18:48:24.340 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T18:48:24.341 INFO:teuthology.orchestra.run.smithi081.stdout:Setting orchestrator backend to cephadm... 2024-09-21T18:48:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1546302150' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T18:48:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:24 smithi081 ceph-mon[29384]: mgrmap e8: y(active, since 2s) 2024-09-21T18:48:26.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:25 smithi081 ceph-mon[29384]: from='client.14132 v1:172.21.15.81:0/474449127' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:26.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:25 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:26.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:25 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:48:26.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T18:48:26.645 INFO:teuthology.orchestra.run.smithi081.stdout:Generating ssh key... 2024-09-21T18:48:27.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:26 smithi081 ceph-mon[29384]: from='client.14134 v1:172.21.15.81:0/4207845648' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: Generating public/private rsa key pair. 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: Your identification has been saved in /tmp/tmp27ikndk_/key 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: Your public key has been saved in /tmp/tmp27ikndk_/key.pub 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: The key fingerprint is: 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: SHA256:pYZ5fxBDF0djghXzTo1S/Wi57WoFiiIYpVb60WDrtVM ceph-d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: The key's randomart image is: 2024-09-21T18:48:27.707 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: +---[RSA 3072]----+ 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | .oB+*. | 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | = ... B +.| 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | * + + . +oo| 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | = ooooEo += .| 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | . =ooSo.. o.+ | 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | . +o+.... . o| 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | . o. . o | 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | . . .| 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: | ... | 2024-09-21T18:48:27.708 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: +----[SHA256]-----+ 2024-09-21T18:48:27.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-mon[29384]: from='client.14136 v1:172.21.15.81:0/1373681099' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:27.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-mon[29384]: Generating ssh key... 2024-09-21T18:48:27.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:27.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:27 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:29.142 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7WRqdYBG68dxZLbGPERLDTO0SdH3AJ708uMbmaT+1yn9lXAqaO0iT6iftTlIWpWfzc6ab3vDY/p2Qv8Qy2JmpSUG7oRl+RRPiUuq8lHJZ24TR+xZepEU8pI5d/3HvfLAHCeTKSzGzpPcgrDoa9hfvf5o67UyHID2JMfEESaBpDS5XOvNkp44Hlx3jDbmVjq5tcNHn+VA20pIzPXkwsBVrNrtAGonuxDSK1dHRceLoCw8IuQySrOme+gAsHyRxZGvZ9w8JeLWpHh88yHuitPe0F+kuP+0B8GhTy13SF75n7DKT4wr85CoWcFnHRP4fzWzC6MJulR0v2KIQUTAiNxCayZ0p5LCvR85mMrbO2aCn8kDNw60jWSQDUIvQwcr/pr8jrOpHGoyTdPQ2+NGCzXaQV3bTk/9zWnzP+EvErATIAvygyQGVUd6RVK+s3CLSdq9LeTaRKE5I8Nxl3edVvyntpZFmN1RDN036jKT4EGv5vB68b02eUB9Pk1wQQdPwc= ceph-d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:29.142 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T18:48:29.142 INFO:teuthology.orchestra.run.smithi081.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T18:48:29.147 INFO:teuthology.orchestra.run.smithi081.stdout:Adding host smithi081... 2024-09-21T18:48:29.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:29 smithi081 ceph-mon[29384]: from='client.14138 v1:172.21.15.81:0/3656747987' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:30.505 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:30 smithi081 ceph-mon[29384]: from='client.14140 v1:172.21.15.81:0/88142797' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi081", "addr": "172.21.15.81", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:31.517 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:31 smithi081 ceph-mon[29384]: Deploying cephadm binary to smithi081 2024-09-21T18:48:33.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Added host 'smithi081' with addr '172.21.15.81' 2024-09-21T18:48:33.493 INFO:teuthology.orchestra.run.smithi081.stdout:Deploying unmanaged mon service... 2024-09-21T18:48:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:33 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:33 smithi081 ceph-mon[29384]: Added host smithi081 2024-09-21T18:48:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:33 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:48:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:35 smithi081 ceph-mon[29384]: from='client.14142 v1:172.21.15.81:0/203585750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:35 smithi081 ceph-mon[29384]: Saving service mon spec with placement count:5 2024-09-21T18:48:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:35 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:35 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:35.549 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T18:48:35.549 INFO:teuthology.orchestra.run.smithi081.stdout:Deploying unmanaged mgr service... 2024-09-21T18:48:36.783 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T18:48:37.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:37 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:37.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:37 smithi081 ceph-mon[29384]: from='client.14144 v1:172.21.15.81:0/3919984654' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:37.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:37 smithi081 ceph-mon[29384]: Saving service mgr spec with placement count:2 2024-09-21T18:48:37.365 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:37 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:38.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3799497034' entity='client.admin' 2024-09-21T18:48:39.155 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling the dashboard module... 2024-09-21T18:48:39.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/431854909' entity='client.admin' 2024-09-21T18:48:40.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1123084183' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T18:48:40.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:40 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:40.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:40 smithi081 ceph-mon[29384]: from='mgr.14118 v1:172.21.15.81:0/3026184966' entity='mgr.y' 2024-09-21T18:48:40.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:40 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ignoring --setuser ceph since I am not root 2024-09-21T18:48:40.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:40 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ignoring --setgroup ceph since I am not root 2024-09-21T18:48:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1123084183' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T18:48:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:41 smithi081 ceph-mon[29384]: mgrmap e9: y(active, since 19s) 2024-09-21T18:48:42.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:42.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T18:48:42.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T18:48:42.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T18:48:42.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T18:48:42.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:42.524 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for the mgr to restart... 2024-09-21T18:48:42.524 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr epoch 9... 2024-09-21T18:48:42.704 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1623094348' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T18:48:46.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: Active manager daemon y restarted 2024-09-21T18:48:46.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: Activating manager daemon y 2024-09-21T18:48:46.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T18:48:46.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: mgrmap e10: y(active, starting, since 0.00409613s) 2024-09-21T18:48:46.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:48:46.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T18:48:46.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:48:46.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:48:46.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:48:46.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:45 smithi081 ceph-mon[29384]: Manager daemon y is now available 2024-09-21T18:48:47.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:48:47.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:48:47.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:48:47.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:46 smithi081 ceph-mon[29384]: mgrmap e11: y(active, since 1.00618s) 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:48:47.212+0000 7fc0ada16640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:18:48:47] ENGINE Error in HTTPServer.serve 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: Traceback (most recent call last): 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._connections.run(self.expiration_interval) 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._run(expiration_interval) 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:48:47.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: s = self.context.wrap_socket( 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: return self.sslsocket_class._create( 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self.do_handshake() 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._sslobj.do_handshake() 2024-09-21T18:48:47.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:48:47.319 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:48:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-09-21T18:48:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T18:48:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T18:48:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-09-21T18:48:47.329 INFO:teuthology.orchestra.run.smithi081.stdout:mgr epoch 9 is available 2024-09-21T18:48:47.329 INFO:teuthology.orchestra.run.smithi081.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T18:48:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:47] ENGINE Bus STARTING 2024-09-21T18:48:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:47] ENGINE Serving on https://172.21.15.81:7150 2024-09-21T18:48:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:47] ENGINE Error in HTTPServer.serve 2024-09-21T18:48:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: Traceback (most recent call last): 2024-09-21T18:48:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:48:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._connections.run(self.expiration_interval) 2024-09-21T18:48:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:48:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._run(expiration_interval) 2024-09-21T18:48:48.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:48:48.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:48:48.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:48:48.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:48:48.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:48:48.068 INFO:journalctl@ceph.mon.a.smithi081.stdout: s = self.context.wrap_socket( 2024-09-21T18:48:48.069 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:48:48.069 INFO:journalctl@ceph.mon.a.smithi081.stdout: return self.sslsocket_class._create( 2024-09-21T18:48:48.069 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:48:48.069 INFO:journalctl@ceph.mon.a.smithi081.stdout: self.do_handshake() 2024-09-21T18:48:48.069 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:48:48.069 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._sslobj.do_handshake() 2024-09-21T18:48:48.070 INFO:journalctl@ceph.mon.a.smithi081.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:48:48.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:47] ENGINE Serving on http://172.21.15.81:8765 2024-09-21T18:48:48.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:47 smithi081 ceph-mon[29384]: [21/Sep/2024:18:48:47] ENGINE Bus STARTED 2024-09-21T18:48:48.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:48 smithi081 ceph-mon[29384]: mgrmap e12: y(active, since 2s) 2024-09-21T18:48:48.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:48 smithi081 ceph-mon[29384]: from='client.14164 v1:172.21.15.81:0/4236461349' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:48.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:48 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:48:49.111 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIULfRvl2jtCxgQdJZrpiJNFGIS6YgwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTE4NDg0OFoXDTM0\nMDkyMjE4NDg0OFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjgxMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEA5bP93I+DeaehePeeITuRwKFoqyu7eH4GETG/\nusQz2C59CNIG2vnOx5meMwM4DaqyRwuotTJUzHYvlrOWk1dUAZfAW6cwX92+TCcQ\n0VLhkJYJApThLwHNWozPntCDNxC5DWWDZoEqZUAr1LZrB6U/BNs5FlgFbpxUBY77\nameL1HoAGDpa7LJFaB8RxQXGVUswzzcuKYkHZIzvlRb20dkw2FticVAbud/glE95\nlpx1u1p0H7WrAQOul4H3F6yXp6rYkN9ZizkTPqFhsbX9U3VUv6A+g/UaYcCQgt08\nrnN+YnyqGvbJJVktjBz4hSQ3kSY+g1l+ZgLtSuvkbnaLVkWXwKqmiBhVZxAI+t9G\nDPiPrD3PXWq7pAnWJQg3k8kDfPR/uu83ghPhIQnGoAN4mizz9QKzM/Hx9YjXYOxM\nXkCAYAhVF9293uh8L18SbpEse4P3a6k52XMcRaTCrvDlTXJ/AMl4ut7QGSI2UDoR\nXPgDNEJSSXscneDsBBrk1gl3UmZBeyKjr6B1aDW90qC5YW7ddCWoOMtoblXx0vTA\nlLZFfSIC5KlBLlnnj8ND/QsYJ4lblUqtpJfOAF4EO3tNXbJwtAqfLhhdNd9i5BvC\nASlCXP8GwVRz3tBrLnmiOIkUDOBzJ/nCJ1o9BTQFiW7/vBwdtc4or+LJX0S4DZ1Z\nZ8eqnD0CAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDgxghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPUTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAa\ntqmMWrIfXBLJ1jYMFR5ch+o1UuQKdVZ48x/xHFsC+9N5byPdV2wlbCTgTrdB+Qgk\nSJPPd1lclphiDaR6ZxP0LM2FiLGJ676aKJXlZMkgYREIgQeVV7/tebSZ0Z5W0CfO\n8NbX/CKy/e7hjqoIuCDe4xoH5SFzsXMrbeFjejrwO200yEWLIwn/m9+ntwp1HxwM\nRbsUr4ib4snvddaqhouOpPXYhsXeR+rP8FWNEN5EgFOb3JTTWqwmAlUuxChigD3S\nJYUz28ytIAlcS/FQ0NLkXhQeu6lcF3EJ274wwUWBbcZzJ5E6PDhp931juiiMLV9k\nQ/PXTN/bWom0iuKF0VRMorXDa9uqW2aUrma1UiUYqqyFAV0Hs0vgErulfgBF4I6q\nu0Z+ybae83oRfpLeMwVdbhvWis0x3alm+H2tMzxa2QPj0oOc32PEhEJb0IXJuR2P\nmsh1kWPQc61+FGxuPOJNDgzy+PMMRGDpG6M976wrZsyVGfSI4w+pUP1PgWmLLMsJ\neR+hStYth05621KrZfBeEdQQF9d1GX4nOj+j2heDMOgU6yXx1gFZXs/m7GKM4LdN\nGaKwbCyhUw/olhmI1Mo/jcgbUJau+j9dE+Mov8YI964V+U6ykEKKspaWXRx2dbLl\nQHPvqjJamlipTa2Yu9IB4C5XTYwYnNNBV1xt7LjGlA==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEA5bP93I+DeaehePeeITuRwKFoqyu7eH4GETG/usQz2C59CNIG\n2vnOx5meMwM4DaqyRwuotTJUzHYvlrOWk1dUAZfAW6cwX92+TCcQ0VLhkJYJApTh\nLwHNWozPntCDNxC5DWWDZoEqZUAr1LZrB6U/BNs5FlgFbpxUBY77ameL1HoAGDpa\n7LJFaB8RxQXGVUswzzcuKYkHZIzvlRb20dkw2FticVAbud/glE95lpx1u1p0H7Wr\nAQOul4H3F6yXp6rYkN9ZizkTPqFhsbX9U3VUv6A+g/UaYcCQgt08rnN+YnyqGvbJ\nJVktjBz4hSQ3kSY+g1l+ZgLtSuvkbnaLVkWXwKqmiBhVZxAI+t9GDPiPrD3PXWq7\npAnWJQg3k8kDfPR/uu83ghPhIQnGoAN4mizz9QKzM/Hx9YjXYOxMXkCAYAhVF929\n3uh8L18SbpEse4P3a6k52XMcRaTCrvDlTXJ/AMl4ut7QGSI2UDoRXPgDNEJSSXsc\nneDsBBrk1gl3UmZBeyKjr6B1aDW90qC5YW7ddCWoOMtoblXx0vTAlLZFfSIC5KlB\nLlnnj8ND/QsYJ4lblUqtpJfOAF4EO3tNXbJwtAqfLhhdNd9i5BvCASlCXP8GwVRz\n3tBrLnmiOIkUDOBzJ/nCJ1o9BTQFiW7/vBwdtc4or+LJX0S4DZ1ZZ8eqnD0CAwEA\nAQKCAgAhD6rIieR1yF2anGNHa7xDAkhfKLR7N4JWi7dJoa9HnItVD2u6rU1ol7D4\nLSyudOTjimdOKzAe3JzlXzRbQSy/bPDcEECLb85RxiaqxjfemGW0GCsgGa2NrKo+\ndMsKDC9iYPvMb2djetzEKTxW9Z8QKyB4mZMwlDIYubdwnXY4DxY/5nhLTK5mOeCU\nH8xn3uJn1MyhYE5u18UtC4jbc9wOq/qLjijrdDBuEFFTDvJ4FKEVpk92XioNNwDS\nSL5sPNAea/NGsWCWeWJnEmdjkYXw7dbJpefV7YIvP7oZ7tggQ0q1hlM1u7PtUIcp\nnKidzOge/izN8Hpod/LhPlE4Sdx5MhcxCBSPNUFIcUf0RF+Vn2jhC+q5yJkzQ+Fw\nuywNGfYdIleakMpoiH6g2A17DAXUAAPXF5NuPjuRxbCifYDeh7j7jf8ZjDQNHlpa\nHM+P75MR731fP6wwo/bK6yq4JnzdSatzDw83/J0e291S1SpltBdgeG+7Q4pp6xQh\nUsQCB1eV48kJI78R4TvR/Zy0OESVtyQGVET4xvttJV6yPmrbqed9ZiLrUEqGaKUb\nlX8y4iDErz2ypPd6xUGlXTvJwXlJFhwT0hBdPJkMsy7T3Zps6YHUX6PApCSkAy4X\nhrw4GKFFOeRnu8yVBlxO0Z2caUUjLlZMLz+BOh2q6g03zhowAQKCAQEA9VjKv8dr\nMzIbVAF318GQ0Kecae0Z1mL+TtLxKqzGaZDnBipssN4ndTceNW0OefrWmPF0+vUu\n4Uq4swCs60wK9xYztWL42i7kwM9pMzwkw5zgRCmjZ29DC4o43Chpui0+5L8sVos4\n6FHzxXPIeb6x3bWrLf5KrdIj0RiQK/48jnQbKd7lh8xxjVetNMM/XBAw6BHZD34/\nUbnXkGjwyOdTAxzyjNm74cMfzqsnTL1PMRNgQo2aB5uynoyb0ho3roi9YSCEfo0B\nMaYsvuerSgA8s7hfKVZJQ9iEKMoWE5jibeRe/2tLgz/nNAPh+o/bIYurHCDAxoiP\nvc3ITYUQFpZsPQKCAQEA761O2osj0uSZtl5DECHFgL/8viGaAfmXJbb8qg4pasKP\niJq/VI+ATak7ngNR2eh5knaumF1ucJ/MD64HGnp3c1rFAIfMu7h2nuZ/IXCufVic\nTRIJW/Z+0YzlN+LzaiFOUat8S1jQu+oWnMplQ5ezxE9rHUzLwmgEw5JOPZCe73U/\nJmmdyk15Dgfs5FqNw4FqP1oLL8If37eiyX0cL5zQV8rAIw7mLfl4V/60HWq0YEwz\npbo0FQw2uiup7EVaQ+ZfzaWcDDAKbGsQ2U6ZhOcualHvaPFTW9YYjfItFrXMaA7y\ny6DweU0rfWpxDIyJkpBYzba+45gnKfcZobSpPLfwAQKCAQEAv6EemzQzMsWMOsz6\nwfZaGd8QeFvMtnqTjxWLcH2Ka6HWECbKxkp+HztxyODpUbTuMH9Hk0w11Zn7MTTx\n2saGOf+JW792B1RhL4fQ70emn3LEZ/jtvDhW8YiAaNTX36a/XQNEqkQLqsl30V1h\n1FVxr23Zyg64XZOojgTVdscYnVtqbAXA2ZSBjJhSnMdzkYDD8j8efs9ovXieOhj9\nGK41VFUjzYezC97P1SgxherHE9nCcbkP+alOli/U9QGtRBFheZ9gXUfdCt50TJe+\nXgpZXzY+fZb7mpZeg010eNIM6TBaUr2HOXiMKUBvHLmHumfQuiOK7oZzXxh/vqyT\nqFDNMQKCAQBH2Q4wWVGN4NVrY+p6pYUNSYA8htoiCW4T8jzfxYsI4/YfHzSH5tSx\ncy/Z+GjLB/OZwlYoeqmmvWBwutyKRXgVbnpr5EmtZGi6zaICpkLIdI22VNfVZ5kw\nvCnijPgnMoDF3sBN3fwEuPG6Yrnkf092Y9BaYjEawA/diLhHoFjks6MQosyYXo0R\ndbK/6Z1hpCPzhcygrnLbn7L+77BJXZrfFnpuiALceVJoHoRj5R7ZVz6MSbtfK+Ak\nAv7XQxsCC8dgYGrdBHG7eoT8f6Fpw9uYJ6O7M8ynVRDbTGPT0P/Lmd+Sx1YPJuEI\noaXt+xsqgnb8g/HJTE1BOT/Hx96LwyABAoIBAA4l6G4qt5z5zPq3qNq9g3KVpNCF\nnOxYa9s62EoUscyJWKmZ1cqs+kdVf3WoqeqKWqpD/CBjQEzOzi/RZD9XKKWNJhKm\nRyFrGhlAEsGi6PVq3Wk/Lp78EDVusrX2jTqhN62xLFLtvial4xrqjiyycbWfpjZt\ngBQ3S3h8W+ajCl0/tHKNWx93z5p7z9zIYsr+GxexCuj221XsqxqtWyntAjgqqBdK\nbva+p3mLqzc6dgEf7HllBc8EityPrX5nu3hra542+oBOYuWlIwX3GPXhH3bOBF2I\nYuzxTXsPso/houA5GkKcXG/WfN9bupBHXJjbYlr8dDCtMnh4VvgupeMGH9s=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T18:48:50.334 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T18:48:50.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:50 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:48:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:50 smithi081 ceph-mon[29384]: from='client.14166 v1:172.21.15.81:0/3418842990' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:50 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:48:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:52 smithi081 ceph-mon[29384]: from='client.14168 v1:172.21.15.81:0/3589998788' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:52 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:48:52.557 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T18:48:52.557 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial admin user... 2024-09-21T18:48:54.024 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6ITRmZk/mqWyrcw6y5BO5eI3RwXcXJXAkvVvedrZBFyTaBAFDJc9.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726944533, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T18:48:54.024 INFO:teuthology.orchestra.run.smithi081.stdout:Fetching dashboard port number... 2024-09-21T18:48:54.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:54 smithi081 ceph-mon[29384]: from='client.14170 v1:172.21.15.81:0/437544999' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:48:54.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:54 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:48:55.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T18:48:55.257 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T18:48:55.258 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-09-21T18:48:55.265 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T18:48:55.265 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-09-21T18:48:55.265 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-09-21T18:48:55.265 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T18:48:55.267 INFO:teuthology.orchestra.run.smithi081.stdout:Ceph Dashboard is now available at: 2024-09-21T18:48:55.267 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:55.267 INFO:teuthology.orchestra.run.smithi081.stdout: URL: https://smithi081.front.sepia.ceph.com:8443/ 2024-09-21T18:48:55.267 INFO:teuthology.orchestra.run.smithi081.stdout: User: admin 2024-09-21T18:48:55.268 INFO:teuthology.orchestra.run.smithi081.stdout: Password: c9zdnhbcth 2024-09-21T18:48:55.268 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:55.268 INFO:teuthology.orchestra.run.smithi081.stdout:Saving cluster configuration to /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config directory 2024-09-21T18:48:55.505 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3524112068' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T18:48:56.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout: ceph telemetry on 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout:For more information see: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:48:56.647 INFO:teuthology.orchestra.run.smithi081.stdout:Bootstrap complete. 2024-09-21T18:48:56.686 INFO:tasks.cephadm:Fetching config... 2024-09-21T18:48:56.686 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:48:56.686 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T18:48:56.702 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T18:48:56.703 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:48:56.703 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T18:48:56.759 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T18:48:56.759 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:48:56.759 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T18:48:56.824 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T18:48:56.824 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:48:56.824 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T18:48:56.879 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T18:48:56.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7WRqdYBG68dxZLbGPERLDTO0SdH3AJ708uMbmaT+1yn9lXAqaO0iT6iftTlIWpWfzc6ab3vDY/p2Qv8Qy2JmpSUG7oRl+RRPiUuq8lHJZ24TR+xZepEU8pI5d/3HvfLAHCeTKSzGzpPcgrDoa9hfvf5o67UyHID2JMfEESaBpDS5XOvNkp44Hlx3jDbmVjq5tcNHn+VA20pIzPXkwsBVrNrtAGonuxDSK1dHRceLoCw8IuQySrOme+gAsHyRxZGvZ9w8JeLWpHh88yHuitPe0F+kuP+0B8GhTy13SF75n7DKT4wr85CoWcFnHRP4fzWzC6MJulR0v2KIQUTAiNxCayZ0p5LCvR85mMrbO2aCn8kDNw60jWSQDUIvQwcr/pr8jrOpHGoyTdPQ2+NGCzXaQV3bTk/9zWnzP+EvErATIAvygyQGVUd6RVK+s3CLSdq9LeTaRKE5I8Nxl3edVvyntpZFmN1RDN036jKT4EGv5vB68b02eUB9Pk1wQQdPwc= ceph-d839dc86-7849-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T18:48:56.954 INFO:teuthology.orchestra.run.smithi081.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7WRqdYBG68dxZLbGPERLDTO0SdH3AJ708uMbmaT+1yn9lXAqaO0iT6iftTlIWpWfzc6ab3vDY/p2Qv8Qy2JmpSUG7oRl+RRPiUuq8lHJZ24TR+xZepEU8pI5d/3HvfLAHCeTKSzGzpPcgrDoa9hfvf5o67UyHID2JMfEESaBpDS5XOvNkp44Hlx3jDbmVjq5tcNHn+VA20pIzPXkwsBVrNrtAGonuxDSK1dHRceLoCw8IuQySrOme+gAsHyRxZGvZ9w8JeLWpHh88yHuitPe0F+kuP+0B8GhTy13SF75n7DKT4wr85CoWcFnHRP4fzWzC6MJulR0v2KIQUTAiNxCayZ0p5LCvR85mMrbO2aCn8kDNw60jWSQDUIvQwcr/pr8jrOpHGoyTdPQ2+NGCzXaQV3bTk/9zWnzP+EvErATIAvygyQGVUd6RVK+s3CLSdq9LeTaRKE5I8Nxl3edVvyntpZFmN1RDN036jKT4EGv5vB68b02eUB9Pk1wQQdPwc= ceph-d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:56.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7WRqdYBG68dxZLbGPERLDTO0SdH3AJ708uMbmaT+1yn9lXAqaO0iT6iftTlIWpWfzc6ab3vDY/p2Qv8Qy2JmpSUG7oRl+RRPiUuq8lHJZ24TR+xZepEU8pI5d/3HvfLAHCeTKSzGzpPcgrDoa9hfvf5o67UyHID2JMfEESaBpDS5XOvNkp44Hlx3jDbmVjq5tcNHn+VA20pIzPXkwsBVrNrtAGonuxDSK1dHRceLoCw8IuQySrOme+gAsHyRxZGvZ9w8JeLWpHh88yHuitPe0F+kuP+0B8GhTy13SF75n7DKT4wr85CoWcFnHRP4fzWzC6MJulR0v2KIQUTAiNxCayZ0p5LCvR85mMrbO2aCn8kDNw60jWSQDUIvQwcr/pr8jrOpHGoyTdPQ2+NGCzXaQV3bTk/9zWnzP+EvErATIAvygyQGVUd6RVK+s3CLSdq9LeTaRKE5I8Nxl3edVvyntpZFmN1RDN036jKT4EGv5vB68b02eUB9Pk1wQQdPwc= ceph-d839dc86-7849-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T18:48:57.145 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7WRqdYBG68dxZLbGPERLDTO0SdH3AJ708uMbmaT+1yn9lXAqaO0iT6iftTlIWpWfzc6ab3vDY/p2Qv8Qy2JmpSUG7oRl+RRPiUuq8lHJZ24TR+xZepEU8pI5d/3HvfLAHCeTKSzGzpPcgrDoa9hfvf5o67UyHID2JMfEESaBpDS5XOvNkp44Hlx3jDbmVjq5tcNHn+VA20pIzPXkwsBVrNrtAGonuxDSK1dHRceLoCw8IuQySrOme+gAsHyRxZGvZ9w8JeLWpHh88yHuitPe0F+kuP+0B8GhTy13SF75n7DKT4wr85CoWcFnHRP4fzWzC6MJulR0v2KIQUTAiNxCayZ0p5LCvR85mMrbO2aCn8kDNw60jWSQDUIvQwcr/pr8jrOpHGoyTdPQ2+NGCzXaQV3bTk/9zWnzP+EvErATIAvygyQGVUd6RVK+s3CLSdq9LeTaRKE5I8Nxl3edVvyntpZFmN1RDN036jKT4EGv5vB68b02eUB9Pk1wQQdPwc= ceph-d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:48:57.159 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T18:48:57.181 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3509083744' entity='client.admin' 2024-09-21T18:48:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:48:58.692 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T18:48:58.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T18:48:59.216 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:48:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:58 smithi081 ceph-mon[29384]: mgrmap e13: y(active, since 12s) 2024-09-21T18:48:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:48:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2607924353' entity='client.admin' 2024-09-21T18:49:00.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:00.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:00.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"} : dispatch 2024-09-21T18:49:00.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:00.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:00.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:01.092 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-09-21T18:49:01.092 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:49:01.092 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-09-21T18:49:01.111 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:49:01.111 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:01.169 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-09-21T18:49:01.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch host add smithi104 2024-09-21T18:49:01.643 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:49:01.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='client.14178 v1:172.21.15.81:0/2452382215' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:01.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:02.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:02 smithi081 ceph-mon[29384]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:02.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:02 smithi081 ceph-mon[29384]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:49:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:03 smithi081 ceph-mon[29384]: from='client.14180 v1:172.21.15.81:0/947554141' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:04.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:04 smithi081 ceph-mon[29384]: Deploying cephadm binary to smithi104 2024-09-21T18:49:05.309 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-09-21T18:49:05.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T18:49:06.208 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:49:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:06 smithi081 ceph-mon[29384]: Added host smithi104 2024-09-21T18:49:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:06.822 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:49:06.823 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-09-21T18:49:07.403 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T18:49:07.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T18:49:07.643 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:49:07.758 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:07 smithi081 ceph-mon[29384]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:07.758 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-09-21T18:49:08.971 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:08 smithi081 ceph-mon[29384]: from='client.14182 v1:172.21.15.81:0/2061049390' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:49:08.972 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/954667891' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T18:49:09.270 INFO:tasks.cephadm:Adding mon.a on smithi081 2024-09-21T18:49:09.270 INFO:tasks.cephadm:Adding mon.c on smithi081 2024-09-21T18:49:09.270 INFO:tasks.cephadm:Adding mon.b on smithi104 2024-09-21T18:49:09.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b' 2024-09-21T18:49:09.514 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T18:49:09.560 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/954667891' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"} : dispatch 2024-09-21T18:49:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:49:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:09 smithi081 ceph-mon[29384]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:10.302 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2024-09-21T18:49:10.892 DEBUG:teuthology.orchestra.run.smithi081:mon.c> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T18:49:10.895 DEBUG:teuthology.orchestra.run.smithi104:mon.b> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T18:49:10.898 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T18:49:10.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T18:49:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:49:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='client.14186 v1:172.21.15.104:0/2398812047' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: Saving service mon spec with placement smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b;count:3 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:10 smithi081 ceph-mon[29384]: Deploying daemon mon.b on smithi104 2024-09-21T18:49:11.610 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:12.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:49:12.660 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","modified":"2024-09-21T18:47:55.713547Z","created":"2024-09-21T18:47:55.713547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T18:49:12.660 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-09-21T18:49:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:12 smithi081 ceph-mon[29384]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/2871493673' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:15.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.unordered_write: 0 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.row_cache: None 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.wal_filter: None 2024-09-21T18:49:15.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T18:49:15.258 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T18:49:15.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T18:49:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:14 smithi081 ceph-mon[29384]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T18:49:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.two_write_queues: 0 2024-09-21T18:49:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T18:49:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.wal_compression: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.atomic_flush: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.log_readahead_size: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_background_jobs: 2 2024-09-21T18:49:15.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_background_compactions: -1 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_subcompactions: 1 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_open_files: -1 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T18:49:15.509 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_background_flushes: -1 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Compression algorithms supported: 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kZSTD supported: 0 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kXpressCompression supported: 0 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kBZip2Compression supported: 0 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kLZ4Compression supported: 1 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kZlibCompression supported: 1 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: kSnappyCompression supported: 1 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-21T18:49:15.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.merge_operator: 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_filter: None 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d8d644b2a0) 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_type: 0 2024-09-21T18:49:15.511 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_index_type: 0 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_shortening: 1 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: checksum: 4 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: no_block_cache: 0 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache: 0x55d8d643f610 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_options: 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: capacity : 536870912 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_shard_bits : 4 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: strict_capacity_limit : 0 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_compressed: (nil) 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: persistent_cache: (nil) 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size: 4096 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size_deviation: 10 2024-09-21T18:49:15.512 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_restart_interval: 16 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_block_restart_interval: 1 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: metadata_block_size: 4096 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: partition_filters: 0 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: use_delta_encoding: 1 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: filter_policy: bloomfilter 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: whole_key_filtering: 1 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: verify_compression: 0 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: format_version: 5 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_index_compression: 1 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_align: 0 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: max_auto_readahead_size: 262144 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: prepopulate_block_cache: 0 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: initial_auto_readahead_size: 8192 2024-09-21T18:49:15.513 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T18:49:15.514 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression: NoCompression 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.num_levels: 7 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T18:49:15.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T18:49:15.516 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T18:49:15.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T18:49:15.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.inplace_update_support: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.bloom_locality: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.max_successive_merges: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.ttl: 2592000 2024-09-21T18:49:15.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.enable_blob_files: false 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.min_blob_size: 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T18:49:15.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a6dce989-97dd-48d2-bfad-e39a1615ad8a 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944555182304, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944555182860, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726944555, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a6dce989-97dd-48d2-bfad-e39a1615ad8a", "db_session_id": "NVFBQPWMMH55YXR4FXY9", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944555182996, "job": 1, "event": "recovery_finished"} 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d8d646ee00 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: DB pointer 0x55d8d6570000 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** DB Stats ** 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T18:49:15.521 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T18:49:15.522 INFO:journalctl@ceph.mon.b.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Block cache BinnedLRUCache@0x55d8d643f610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T18:49:15.523 INFO:journalctl@ceph.mon.b.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: using public_addrv v1:172.21.15.104:6789/0 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: starting mon.b rank -1 at public addrs v1:172.21.15.104:6789/0 at bind addrs v1:172.21.15.104:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(???) e0 preinit fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: e1 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: btime 2024-09-21T18:47:59:361071+0000 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: legacy client fscid: -1 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T18:49:15.524 INFO:journalctl@ceph.mon.b.smithi104.stdout: No filesystems configured 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='client.14182 v1:172.21.15.81:0/2061049390' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/954667891' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:15.525 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/954667891' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"} : dispatch 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:49:15.526 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='client.14186 v1:172.21.15.104:0/2398812047' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: Saving service mon spec with placement smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b;count:3 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: Deploying daemon mon.b on smithi104 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/2871493673' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:15.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 ceph-mon[32438]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T18:49:15.528 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 bash[32419]: 70529628d81389e8318cab5aae4faf354bd82d1117839e67ed1e53df6d16a5f2 2024-09-21T18:49:15.528 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:15 smithi104 systemd[1]: Started Ceph mon.b for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:49:15.528 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:49:20.485 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:49:20.486 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","modified":"2024-09-21T18:49:15.205812Z","created":"2024-09-21T18:47:55.713547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T18:49:20.486 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-09-21T18:49:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:49:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: mon.a calling monitor election 2024-09-21T18:49:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: mon.b calling monitor election 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: monmap epoch 2 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: last_changed 2024-09-21T18:49:15.205812+0000 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:49:20.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: min_mon_release 19 (squid) 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: election_strategy: 1 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: 1: v1:172.21.15.104:6789/0 mon.b 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: fsmap 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: mgrmap e13: y(active, since 34s) 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: overall HEALTH_OK 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:20.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: mon.a calling monitor election 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: mon.b calling monitor election 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: monmap epoch 2 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: last_changed 2024-09-21T18:49:15.205812+0000 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: min_mon_release 19 (squid) 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: election_strategy: 1 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: 1: v1:172.21.15.104:6789/0 mon.b 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: fsmap 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: mgrmap e13: y(active, since 34s) 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: overall HEALTH_OK 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:20.760 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:20.760 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:20.760 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:21 smithi104 ceph-mon[32438]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:21 smithi104 ceph-mon[32438]: Deploying daemon mon.c on smithi081 2024-09-21T18:49:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/1720366576' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:21.562 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:21 smithi081 ceph-mon[29384]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:21.562 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:21 smithi081 ceph-mon[29384]: Deploying daemon mon.c on smithi081 2024-09-21T18:49:21.562 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/1720366576' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:21.562 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:22.145 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T18:49:22.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T18:49:22.397 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:49:23.050 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:49:23.050 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","modified":"2024-09-21T18:49:15.205812Z","created":"2024-09-21T18:47:55.713547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T18:49:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-09-21T18:49:23.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 systemd[1]: Starting Ceph mon.c for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:49:23.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[29384]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/396409115' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:23.482 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 podman[39173]: 2024-09-21 18:49:23.119041641 +0000 UTC m=+0.015458002 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 podman[39173]: 2024-09-21 18:49:23.219536131 +0000 UTC m=+0.115952485 container create 003e5b6bfcba94bdb53bde6a24381279822a609ecd7fea3a0a3b9aa4e7688747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 podman[39173]: 2024-09-21 18:49:23.43037805 +0000 UTC m=+0.326794403 container init 003e5b6bfcba94bdb53bde6a24381279822a609ecd7fea3a0a3b9aa4e7688747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 podman[39173]: 2024-09-21 18:49:23.434445224 +0000 UTC m=+0.330861576 container start 003e5b6bfcba94bdb53bde6a24381279822a609ecd7fea3a0a3b9aa4e7688747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev), process ceph-mon, pid 2 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pidfile_write: ignore empty --pid-file 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: load: jerasure load: lrc 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: RocksDB version: 7.9.2 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Git sha 0 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Compile date 2024-09-21 14:38:11 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: DB SUMMARY 2024-09-21T18:49:23.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: DB Session ID: HS9IM8UEU1LBT8EXVHDH 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: CURRENT file: CURRENT 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.error_if_exists: 0 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.create_if_missing: 0 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.paranoid_checks: 1 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.env: 0x55dfeff1e120 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.info_log: 0x55dff248d6e0 2024-09-21T18:49:23.484 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.statistics: (nil) 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.use_fsync: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_log_file_size: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.allow_fallocate: 1 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.use_direct_reads: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.db_log_dir: 2024-09-21T18:49:23.485 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.wal_dir: 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.write_buffer_manager: 0x55dff2491b80 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T18:49:23.486 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.unordered_write: 0 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.row_cache: None 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.wal_filter: None 2024-09-21T18:49:23.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.two_write_queues: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.wal_compression: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.atomic_flush: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.log_readahead_size: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T18:49:23.489 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_background_jobs: 2 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_background_compactions: -1 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_subcompactions: 1 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T18:49:23.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_open_files: -1 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_background_flushes: -1 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Compression algorithms supported: 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kZSTD supported: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kXpressCompression supported: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kBZip2Compression supported: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kLZ4Compression supported: 1 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kZlibCompression supported: 1 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: kSnappyCompression supported: 1 2024-09-21T18:49:23.491 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.merge_operator: 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_filter: None 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dff248d300) 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout: cache_index_and_filter_blocks: 1 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T18:49:23.492 INFO:journalctl@ceph.mon.c.smithi081.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: pin_top_level_index_and_filter: 1 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: index_type: 0 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: data_block_index_type: 0 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: index_shortening: 1 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: checksum: 4 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: no_block_cache: 0 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_cache: 0x55dff2481610 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_cache_name: BinnedLRUCache 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_cache_options: 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: capacity : 536870912 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: num_shard_bits : 4 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: strict_capacity_limit : 0 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: high_pri_pool_ratio: 0.000 2024-09-21T18:49:23.493 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_cache_compressed: (nil) 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: persistent_cache: (nil) 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_size: 4096 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_size_deviation: 10 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_restart_interval: 16 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: index_block_restart_interval: 1 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: metadata_block_size: 4096 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: partition_filters: 0 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: use_delta_encoding: 1 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: filter_policy: bloomfilter 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: whole_key_filtering: 1 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: verify_compression: 0 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: read_amp_bytes_per_bit: 0 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: format_version: 5 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: enable_index_compression: 1 2024-09-21T18:49:23.494 INFO:journalctl@ceph.mon.c.smithi081.stdout: block_align: 0 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout: max_auto_readahead_size: 262144 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout: prepopulate_block_cache: 0 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout: initial_auto_readahead_size: 8192 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression: NoCompression 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.num_levels: 7 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T18:49:23.495 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T18:49:23.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T18:49:23.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T18:49:23.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.inplace_update_support: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.bloom_locality: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.max_successive_merges: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T18:49:23.499 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.ttl: 2592000 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.enable_blob_files: false 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.min_blob_size: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T18:49:23.500 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a4b26db6-a01d-48b7-b3ab-3937ce8ebace 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944563461110, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944563461498, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726944563, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a4b26db6-a01d-48b7-b3ab-3937ce8ebace", "db_session_id": "HS9IM8UEU1LBT8EXVHDH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726944563461591, "job": 1, "event": "recovery_finished"} 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dff24b0e00 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: DB pointer 0x55dff25b2000 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout: ** DB Stats ** 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T18:49:23.501 INFO:journalctl@ceph.mon.c.smithi081.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: ** Compaction Stats [default] ** 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: ** Compaction Stats [default] ** 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T18:49:23.502 INFO:journalctl@ceph.mon.c.smithi081.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Block cache BinnedLRUCache@0x55dff2481610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T18:49:23.503 INFO:journalctl@ceph.mon.c.smithi081.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: using public_addrv v1:172.21.15.81:6790/0 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: starting mon.c rank -1 at public addrs v1:172.21.15.81:6790/0 at bind addrs v1:172.21.15.81:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(???) e0 preinit fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: e1 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: btime 2024-09-21T18:47:59:361071+0000 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: legacy client fscid: -1 2024-09-21T18:49:23.504 INFO:journalctl@ceph.mon.c.smithi081.stdout: 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout: No filesystems configured 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.14182 v1:172.21.15.81:0/2061049390' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/954667891' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.505 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/954667891' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"} : dispatch 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:49:23.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.14186 v1:172.21.15.104:0/2398812047' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Saving service mon spec with placement smithi081:[v1:172.21.15.81:6789]=a;smithi081:[v1:172.21.15.81:6790]=c;smithi104:[v1:172.21.15.104:6789]=b;count:3 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Deploying daemon mon.b on smithi104 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/2871493673' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:23.507 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.a calling monitor election 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.b calling monitor election 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: monmap epoch 2 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:23.508 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: last_changed 2024-09-21T18:49:15.205812+0000 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: min_mon_release 19 (squid) 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: election_strategy: 1 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: 1: v1:172.21.15.104:6789/0 mon.b 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: fsmap 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mgrmap e13: y(active, since 34s) 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: overall HEALTH_OK 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:23.509 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: Deploying daemon mon.c on smithi081 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/1720366576' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/396409115' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:23 smithi104 ceph-mon[32438]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:23.510 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:23 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/396409115' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 ceph-mon[39192]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-09-21T18:49:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 bash[39173]: 003e5b6bfcba94bdb53bde6a24381279822a609ecd7fea3a0a3b9aa4e7688747 2024-09-21T18:49:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:23 smithi081 systemd[1]: Started Ceph mon.c for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:49:24.693 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T18:49:24.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T18:49:24.936 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:49:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:49:28.783 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","modified":"2024-09-21T18:49:23.489748Z","created":"2024-09-21T18:47:55.713547Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6790","nonce":0}]},"addr":"172.21.15.81:6790/0","public_addr":"172.21.15.81:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T18:49:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 3 2024-09-21T18:49:28.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:49:28.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: mon.a calling monitor election 2024-09-21T18:49:28.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:28.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: mon.b calling monitor election 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: mon.c calling monitor election 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: monmap epoch 3 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: last_changed 2024-09-21T18:49:23.489748+0000 2024-09-21T18:49:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: min_mon_release 19 (squid) 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: election_strategy: 1 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: 1: v1:172.21.15.104:6789/0 mon.b 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: 2: v1:172.21.15.81:6790/0 mon.c 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: fsmap 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: mgrmap e13: y(active, since 42s) 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: overall HEALTH_OK 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: mon.a calling monitor election 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: mon.b calling monitor election 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: mon.c calling monitor election 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T18:49:28.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: monmap epoch 3 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: last_changed 2024-09-21T18:49:23.489748+0000 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: min_mon_release 19 (squid) 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: election_strategy: 1 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: 1: v1:172.21.15.104:6789/0 mon.b 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: 2: v1:172.21.15.81:6790/0 mon.c 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: fsmap 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: mgrmap e13: y(active, since 42s) 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: overall HEALTH_OK 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:28.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:49:29.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: mon.a calling monitor election 2024-09-21T18:49:29.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:49:29.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: mon.b calling monitor election 2024-09-21T18:49:29.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:29.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: mon.c calling monitor election 2024-09-21T18:49:29.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:29.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: monmap epoch 3 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: last_changed 2024-09-21T18:49:23.489748+0000 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: created 2024-09-21T18:47:55.713547+0000 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: min_mon_release 19 (squid) 2024-09-21T18:49:29.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: election_strategy: 1 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: 0: v1:172.21.15.81:6789/0 mon.a 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: 1: v1:172.21.15.104:6789/0 mon.b 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: 2: v1:172.21.15.81:6790/0 mon.c 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: fsmap 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: mgrmap e13: y(active, since 42s) 2024-09-21T18:49:29.011 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: overall HEALTH_OK 2024-09-21T18:49:29.012 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:29.012 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:29.012 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:29.012 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:29.346 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T18:49:29.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T18:49:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:49:29.607 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:29 smithi104 ceph-mon[32438]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:29.607 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:29.607 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/2137147044' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:29.607 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[29384]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:29.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:29.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/2137147044' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:29.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:29.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[39192]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:29.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:29.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/2137147044' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:49:29.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:49:30.425 INFO:teuthology.orchestra.run.smithi081.stdout:# minimal ceph.conf for d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:30.425 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-09-21T18:49:30.425 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:49:30.425 INFO:teuthology.orchestra.run.smithi081.stdout: mon_host = 172.21.15.81:6789/0 172.21.15.104:6789/0 v1:172.21.15.81:6790/0 2024-09-21T18:49:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/311832807' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:30.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[39192]: Reconfiguring daemon mon.a on smithi081 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/311832807' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:30.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:30 smithi081 ceph-mon[29384]: Reconfiguring daemon mon.a on smithi081 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/311832807' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:31.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T18:49:31.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:31.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:31.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:31.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:30 smithi104 ceph-mon[32438]: Reconfiguring daemon mon.a on smithi081 2024-09-21T18:49:31.097 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T18:49:31.097 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:49:31.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T18:49:31.124 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:49:31.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:31.189 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:49:31.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T18:49:31.217 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:49:31.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:49:31.282 INFO:tasks.cephadm:Adding mgr.y on smithi081 2024-09-21T18:49:31.282 INFO:tasks.cephadm:Adding mgr.x on smithi104 2024-09-21T18:49:31.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi081=y;smithi104=x' 2024-09-21T18:49:31.559 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:49:31.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:31 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:32.145 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mgr update... 2024-09-21T18:49:32.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:31 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:32.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:31 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:32.734 DEBUG:teuthology.orchestra.run.smithi104:mgr.x> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T18:49:32.736 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T18:49:32.736 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T18:49:32.737 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:49:32.737 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T18:49:32.752 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T18:49:32.753 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-21 18:49:22.948720008 +0000 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:32.808 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:32.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T18:49:32.875 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-21T18:49:32.875 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-21T18:49:32.876 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000276209 s, 1.9 MB/s 2024-09-21T18:49:32.876 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T18:49:32.932 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 976 Links: 1 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-21 18:49:22.949719989 +0000 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:32.987 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:32.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T18:49:33.050 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-21T18:49:33.050 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-21T18:49:33.050 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000192203 s, 2.7 MB/s 2024-09-21T18:49:33.051 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T18:49:33.107 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-21 18:49:22.949719989 +0000 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:33.162 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-21 18:44:24.614178332 +0000 2024-09-21T18:49:33.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T18:49:33.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:33.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: Reconfiguring mon.c (monmap changed)... 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: Reconfiguring daemon mon.c on smithi081 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='client.24109 v1:172.21.15.104:0/1922031000' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi081=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: Saving service mgr spec with placement smithi081=y;smithi104=x;count:2 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.219 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: Reconfiguring mon.c (monmap changed)... 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: Reconfiguring daemon mon.c on smithi081 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='client.24109 v1:172.21.15.104:0/1922031000' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi081=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: Saving service mgr spec with placement smithi081=y;smithi104=x;count:2 2024-09-21T18:49:33.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.226 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-21T18:49:33.226 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-21T18:49:33.226 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000253226 s, 2.0 MB/s 2024-09-21T18:49:33.228 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T18:49:33.284 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-09-21T18:49:33.339 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T18:49:33.339 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:33.339 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-21T18:49:33.339 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:33.339 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:33.339 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-21 18:49:22.949719989 +0000 2024-09-21T18:49:33.340 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-21 18:44:24.613178350 +0000 2024-09-21T18:49:33.340 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-21 18:44:24.613178350 +0000 2024-09-21T18:49:33.340 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-21 18:44:24.613178350 +0000 2024-09-21T18:49:33.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T18:49:33.401 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-21T18:49:33.402 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-21T18:49:33.402 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000183663 s, 2.8 MB/s 2024-09-21T18:49:33.403 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T18:49:33.459 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:49:33.459 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T18:49:33.476 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T18:49:33.476 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-09-21T18:49:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: Reconfiguring mon.c (monmap changed)... 2024-09-21T18:49:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:33.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:33.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:33.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: Reconfiguring daemon mon.c on smithi081 2024-09-21T18:49:33.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='client.24109 v1:172.21.15.104:0/1922031000' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi081=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:33.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: Saving service mgr spec with placement smithi081=y;smithi104=x;count:2 2024-09-21T18:49:33.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T18:49:33.523 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:33.523 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-21T18:49:33.523 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:33.523 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:33.524 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 18:49:14.619849289 +0000 2024-09-21T18:49:33.524 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:33.524 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:33.524 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:33.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T18:49:33.587 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T18:49:33.587 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T18:49:33.587 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000219996 s, 2.3 MB/s 2024-09-21T18:49:33.589 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T18:49:33.646 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-09-21T18:49:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T18:49:33.701 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:33.701 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-21T18:49:33.701 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:33.701 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:33.701 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 18:49:14.620849278 +0000 2024-09-21T18:49:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 18:44:24.888007875 +0000 2024-09-21T18:49:33.702 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 18:44:24.888007875 +0000 2024-09-21T18:49:33.702 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 18:44:24.888007875 +0000 2024-09-21T18:49:33.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T18:49:33.764 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T18:49:33.764 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T18:49:33.764 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.00020178 s, 2.5 MB/s 2024-09-21T18:49:33.766 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T18:49:33.822 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 18:49:14.620849278 +0000 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:33.878 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:33.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T18:49:33.942 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T18:49:33.942 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T18:49:33.942 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000196727 s, 2.6 MB/s 2024-09-21T18:49:33.943 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T18:49:33.999 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-09-21T18:49:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 18:49:14.621849266 +0000 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:34.055 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 18:44:24.889007864 +0000 2024-09-21T18:49:34.055 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T18:49:34.119 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T18:49:34.119 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T18:49:34.119 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000197278 s, 2.6 MB/s 2024-09-21T18:49:34.121 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T18:49:34.177 INFO:tasks.cephadm:Deploying osd.0 on smithi081 with /dev/vg_nvme/lv_4... 2024-09-21T18:49:34.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T18:49:34.418 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: Reconfiguring mon.b (monmap changed)... 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[39192]: Reconfiguring daemon mon.b on smithi104 2024-09-21T18:49:34.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: Reconfiguring mon.b (monmap changed)... 2024-09-21T18:49:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:34 smithi081 ceph-mon[29384]: Reconfiguring daemon mon.b on smithi104 2024-09-21T18:49:34.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:34.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:34.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: Reconfiguring mon.b (monmap changed)... 2024-09-21T18:49:34.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T18:49:34.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T18:49:34.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:34.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:34 smithi104 ceph-mon[32438]: Reconfiguring daemon mon.b on smithi104 2024-09-21T18:49:35.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:35.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T18:49:35.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:35 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:35.968 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T18:49:35.969 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:35 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:36.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:36 smithi081 ceph-mon[39192]: Deploying daemon mgr.x on smithi104 2024-09-21T18:49:36.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:36 smithi081 ceph-mon[29384]: Deploying daemon mgr.x on smithi104 2024-09-21T18:49:36.826 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:49:36.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi081:vg_nvme/lv_4 2024-09-21T18:49:36.856 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:36 smithi104 ceph-mon[32438]: Deploying daemon mgr.x on smithi104 2024-09-21T18:49:36.857 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:36 smithi104 systemd[1]: Starting Ceph mgr.x for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:49:37.095 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:49:37.150 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:36 smithi104 podman[33925]: 2024-09-21 18:49:36.854311802 +0000 UTC m=+0.101288196 container create 571615ecb52535a5d695d8641f69f75e8f1929243e255e2b0e879cd5bc787f14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T18:49:37.150 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:36 smithi104 podman[33925]: 2024-09-21 18:49:36.768655699 +0000 UTC m=+0.015632099 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T18:49:37.150 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:37 smithi104 podman[33925]: 2024-09-21 18:49:37.039728597 +0000 UTC m=+0.286705003 container init 571615ecb52535a5d695d8641f69f75e8f1929243e255e2b0e879cd5bc787f14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T18:49:37.150 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:37 smithi104 podman[33925]: 2024-09-21 18:49:37.042467787 +0000 UTC m=+0.289444175 container start 571615ecb52535a5d695d8641f69f75e8f1929243e255e2b0e879cd5bc787f14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T18:49:37.150 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mgr[33944]: -- 172.21.15.104:0/2235770286 <== mon.1 v1:172.21.15.104:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 1355105829 0 0) 0x560b34ae8d20 con 0x560b34ac7000 2024-09-21T18:49:37.150 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:37 smithi104 bash[33925]: 571615ecb52535a5d695d8641f69f75e8f1929243e255e2b0e879cd5bc787f14 2024-09-21T18:49:37.507 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:49:37 smithi104 systemd[1]: Started Ceph mgr.x for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[29384]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[39192]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:37.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:37.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:37 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mon[32438]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:37 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:38.782 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:49:38.783 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T18:49:38.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T18:49:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[29384]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[29384]: from='client.14226 v1:172.21.15.81:0/4048644020' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[29384]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[29384]: Reconfiguring daemon mgr.y on smithi081 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[39192]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[39192]: from='client.14226 v1:172.21.15.81:0/4048644020' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[39192]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T18:49:39.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:39 smithi081 ceph-mon[39192]: Reconfiguring daemon mgr.y on smithi081 2024-09-21T18:49:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:39 smithi104 ceph-mon[32438]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:39 smithi104 ceph-mon[32438]: from='client.14226 v1:172.21.15.81:0/4048644020' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:49:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:39 smithi104 ceph-mon[32438]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T18:49:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:39 smithi104 ceph-mon[32438]: Reconfiguring daemon mgr.y on smithi081 2024-09-21T18:49:41.479 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:41.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:41.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:41.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:41 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:41 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1582698223' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea"} : dispatch 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1582698223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea"}]': finished 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1585915449' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: Standby manager daemon x started 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T18:49:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:42 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1582698223' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea"} : dispatch 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1582698223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea"}]': finished 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1585915449' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: Standby manager daemon x started 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T18:49:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1582698223' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1582698223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea"}]': finished 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1585915449' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: Standby manager daemon x started 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T18:49:42.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T18:49:42.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:42 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/4285202788' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T18:49:43.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:43 smithi081 ceph-mon[29384]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:43.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:43 smithi081 ceph-mon[29384]: mgrmap e14: y(active, since 56s), standbys: x 2024-09-21T18:49:43.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:43 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T18:49:43.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:43 smithi081 ceph-mon[39192]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:43.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:43 smithi081 ceph-mon[39192]: mgrmap e14: y(active, since 56s), standbys: x 2024-09-21T18:49:43.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:43 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T18:49:43.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:43 smithi104 ceph-mon[32438]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:43.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:43 smithi104 ceph-mon[32438]: mgrmap e14: y(active, since 56s), standbys: x 2024-09-21T18:49:43.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:43 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T18:49:45.750 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:45 smithi081 ceph-mon[29384]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:45.750 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:45 smithi081 ceph-mon[39192]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:45.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:45 smithi104 ceph-mon[32438]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:47.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:47 smithi081 ceph-mon[29384]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:47.637 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:47 smithi081 ceph-mon[39192]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:47.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:47 smithi104 ceph-mon[32438]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[29384]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T18:49:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[29384]: Deploying daemon osd.0 on smithi081 2024-09-21T18:49:49.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[39192]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:49.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T18:49:49.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:49.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:49 smithi081 ceph-mon[39192]: Deploying daemon osd.0 on smithi081 2024-09-21T18:49:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:49 smithi104 ceph-mon[32438]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:49 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T18:49:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:49 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:49 smithi104 ceph-mon[32438]: Deploying daemon osd.0 on smithi081 2024-09-21T18:49:51.697 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:51 smithi081 ceph-mon[29384]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:51.698 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:51 smithi081 ceph-mon[39192]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:51 smithi104 ceph-mon[32438]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:53.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:53 smithi104 ceph-mon[32438]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:53.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:53 smithi081 ceph-mon[29384]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:53 smithi081 ceph-mon[39192]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[29384]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[39192]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:55.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:55 smithi104 ceph-mon[32438]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:57.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:57 smithi081 ceph-mon[29384]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:57.662 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:57 smithi081 ceph-mon[39192]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:57.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:57 smithi104 ceph-mon[32438]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[29384]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:58.909 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[39192]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:58.909 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:58.909 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:58.909 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:58.909 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:58.909 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:58 smithi104 ceph-mon[32438]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:49:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:49:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:49:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.530 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 0 on host 'smithi081' 2024-09-21T18:49:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:49:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:49:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:49:59.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:49:59.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:49:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:00.137 DEBUG:teuthology.orchestra.run.smithi081:osd.0> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T18:50:00.139 INFO:tasks.cephadm:Deploying osd.1 on smithi081 with /dev/vg_nvme/lv_3... 2024-09-21T18:50:00.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T18:50:00.961 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:50:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:00 smithi104 ceph-mon[32438]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:00 smithi104 ceph-mon[32438]: overall HEALTH_OK 2024-09-21T18:50:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:00 smithi104 ceph-mon[32438]: from='osd.0 v1:172.21.15.81:6801/390647394' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T18:50:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:00 smithi104 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T18:50:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[29384]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[29384]: overall HEALTH_OK 2024-09-21T18:50:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[29384]: from='osd.0 v1:172.21.15.81:6801/390647394' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T18:50:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[29384]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T18:50:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[39192]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[39192]: overall HEALTH_OK 2024-09-21T18:50:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[39192]: from='osd.0 v1:172.21.15.81:6801/390647394' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T18:50:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:00 smithi081 ceph-mon[39192]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T18:50:01.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[39192]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T18:50:01.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[39192]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T18:50:01.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:01.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[39192]: from='osd.0 v1:172.21.15.81:6801/390647394' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:01.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[39192]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:01.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[29384]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T18:50:01.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[29384]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T18:50:01.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:01.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[29384]: from='osd.0 v1:172.21.15.81:6801/390647394' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:01.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-mon[29384]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:01.941 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0[42449]: 2024-09-21T18:50:01.676+0000 7fdff9edf640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T18:50:01.941 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:01 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0[42449]: 2024-09-21T18:50:01.678+0000 7fdff5d06640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:50:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:01 smithi104 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T18:50:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:01 smithi104 ceph-mon[32438]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T18:50:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:01 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:01 smithi104 ceph-mon[32438]: from='osd.0 v1:172.21.15.81:6801/390647394' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:01 smithi104 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:02 smithi104 ceph-mon[32438]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:02 smithi104 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:02 smithi104 ceph-mon[32438]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T18:50:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:02 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:02 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.013 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[39192]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[39192]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[39192]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[29384]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[29384]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[29384]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:02 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: osd.0 v1:172.21.15.81:6801/390647394 boot 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:03.932 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: osd.0 v1:172.21.15.81:6801/390647394 boot 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.933 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:03.934 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:03.934 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:03.934 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:03.934 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: osd.0 v1:172.21.15.81:6801/390647394 boot 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:04.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:04.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:04.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:04.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi081 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi081 to 16064M 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[29384]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[29384]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi081 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi081 to 16064M 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[39192]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:04.983 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:04 smithi081 ceph-mon[39192]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T18:50:04.986 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:50:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:04 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi081 2024-09-21T18:50:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:04 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi081 to 16064M 2024-09-21T18:50:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:04 smithi104 ceph-mon[32438]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T18:50:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:04 smithi104 ceph-mon[32438]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T18:50:05.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi081:vg_nvme/lv_3 2024-09-21T18:50:05.259 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:50:06.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[29384]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:06.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[29384]: from='client.14253 v1:172.21.15.81:0/852089530' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:06.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:06.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:06.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:06.885 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[39192]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:06.885 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[39192]: from='client.14253 v1:172.21.15.81:0/852089530' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:06.885 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:06.885 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:06.886 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:06 smithi104 ceph-mon[32438]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:06 smithi104 ceph-mon[32438]: from='client.14253 v1:172.21.15.81:0/852089530' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:08 smithi104 ceph-mon[32438]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/944142407' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5"} : dispatch 2024-09-21T18:50:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/944142407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5"}]': finished 2024-09-21T18:50:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:08 smithi104 ceph-mon[32438]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T18:50:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:08 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[39192]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/944142407' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5"} : dispatch 2024-09-21T18:50:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/944142407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5"}]': finished 2024-09-21T18:50:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[39192]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T18:50:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:09.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[29384]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:09.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/944142407' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5"} : dispatch 2024-09-21T18:50:09.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/944142407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5"}]': finished 2024-09-21T18:50:09.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[29384]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T18:50:09.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:08 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1490221108' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:50:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1490221108' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:50:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1490221108' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:50:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:11 smithi104 ceph-mon[32438]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:11.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:11 smithi081 ceph-mon[29384]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:11.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:11 smithi081 ceph-mon[39192]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:13.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:13 smithi104 ceph-mon[32438]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:13.802 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:13 smithi081 ceph-mon[29384]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:13.803 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:13 smithi081 ceph-mon[39192]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:15.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:15 smithi081 ceph-mon[39192]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:15.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T18:50:15.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:15 smithi081 ceph-mon[29384]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T18:50:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:15 smithi104 ceph-mon[32438]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T18:50:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:16.716 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:16 smithi081 ceph-mon[39192]: Deploying daemon osd.1 on smithi081 2024-09-21T18:50:16.717 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:16 smithi081 ceph-mon[29384]: Deploying daemon osd.1 on smithi081 2024-09-21T18:50:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:16 smithi104 ceph-mon[32438]: Deploying daemon osd.1 on smithi081 2024-09-21T18:50:17.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:17 smithi081 ceph-mon[29384]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:17.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:17 smithi081 ceph-mon[39192]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:17.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:17 smithi104 ceph-mon[32438]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:19.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:19 smithi104 ceph-mon[32438]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:19.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:19 smithi081 ceph-mon[29384]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:19.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:19 smithi081 ceph-mon[39192]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:21.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:21 smithi104 ceph-mon[32438]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:21 smithi081 ceph-mon[39192]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:21 smithi081 ceph-mon[29384]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:22 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:22 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:22 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:22 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:22 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:22.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:22 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:22.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:22 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:22.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:22 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:22.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:22 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:23.781 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:23 smithi081 ceph-mon[29384]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:23.782 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:23 smithi081 ceph-mon[39192]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:24.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:23 smithi104 ceph-mon[32438]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:25.437 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[29384]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[39192]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:25.438 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:25.439 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:25 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.475 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 1 on host 'smithi081' 2024-09-21T18:50:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:25 smithi104 ceph-mon[32438]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:25 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:25 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:25 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:25 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:25 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.127 DEBUG:teuthology.orchestra.run.smithi081:osd.1> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T18:50:26.129 INFO:tasks.cephadm:Deploying osd.2 on smithi081 with /dev/vg_nvme/lv_2... 2024-09-21T18:50:26.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T18:50:26.433 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:26.433 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.433 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.433 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:26.434 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.434 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:26 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:26.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:26 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:26 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:26.816 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:26 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1[48235]: 2024-09-21T18:50:26.482+0000 7fdd8b610640 -1 osd.1 0 log_to_monitors true 2024-09-21T18:50:27.364 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:50:27.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:27 smithi081 ceph-mon[29384]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:27.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:27 smithi081 ceph-mon[29384]: from='osd.1 v1:172.21.15.81:6805/3705537065' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T18:50:27.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:27 smithi081 ceph-mon[29384]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T18:50:27.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:27 smithi081 ceph-mon[39192]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:27.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:27 smithi081 ceph-mon[39192]: from='osd.1 v1:172.21.15.81:6805/3705537065' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T18:50:27.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:27 smithi081 ceph-mon[39192]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T18:50:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:27 smithi104 ceph-mon[32438]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:27 smithi104 ceph-mon[32438]: from='osd.1 v1:172.21.15.81:6805/3705537065' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T18:50:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:27 smithi104 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T18:50:28.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:28 smithi104 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T18:50:28.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:28 smithi104 ceph-mon[32438]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T18:50:28.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:28 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:28.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:28 smithi104 ceph-mon[32438]: from='osd.1 v1:172.21.15.81:6805/3705537065' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:28.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:28 smithi104 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:28.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[29384]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T18:50:28.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[29384]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[29384]: from='osd.1 v1:172.21.15.81:6805/3705537065' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[29384]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[39192]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[39192]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[39192]: from='osd.1 v1:172.21.15.81:6805/3705537065' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:28.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-mon[39192]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:29.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1[48235]: 2024-09-21T18:50:28.935+0000 7fdd87da7640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T18:50:29.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:28 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1[48235]: 2024-09-21T18:50:28.938+0000 7fdd7f3b2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:50:29.647 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:50:29.647 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:50:29.647 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:29.647 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[39192]: from='osd.1 ' entity='osd.1' 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:29 smithi081 ceph-mon[29384]: from='osd.1 ' entity='osd.1' 2024-09-21T18:50:29.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:29 smithi104 ceph-mon[32438]: from='osd.1 ' entity='osd.1' 2024-09-21T18:50:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: osd.1 v1:172.21.15.81:6805/3705537065 boot 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:30.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:30.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:30.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: osd.1 v1:172.21.15.81:6805/3705537065 boot 2024-09-21T18:50:30.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T18:50:30.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:30.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: osd.1 v1:172.21.15.81:6805/3705537065 boot 2024-09-21T18:50:30.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T18:50:30.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:50:31.092 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:50:31.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi081:vg_nvme/lv_2 2024-09-21T18:50:31.366 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:50:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:31 smithi104 ceph-mon[32438]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:31 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi081 2024-09-21T18:50:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:31 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi081 to 8032M 2024-09-21T18:50:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:31 smithi104 ceph-mon[32438]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[39192]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi081 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi081 to 8032M 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[39192]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[29384]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi081 2024-09-21T18:50:31.759 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi081 to 8032M 2024-09-21T18:50:31.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:31 smithi081 ceph-mon[29384]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T18:50:32.737 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:32 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:32.737 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:32 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:32.738 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:32 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:32.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:32 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:32.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:32 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:32.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:32 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:32.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:32 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:32.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:32 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:32.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:32 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:33.608 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:33 smithi081 ceph-mon[29384]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:33.609 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:33 smithi081 ceph-mon[29384]: from='client.34169 v1:172.21.15.81:0/122662898' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:33.609 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:33 smithi081 ceph-mon[39192]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:33.609 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:33 smithi081 ceph-mon[39192]: from='client.34169 v1:172.21.15.81:0/122662898' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:33.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:33 smithi104 ceph-mon[32438]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:33.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:33 smithi104 ceph-mon[32438]: from='client.34169 v1:172.21.15.81:0/122662898' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2728526802' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938"} : dispatch 2024-09-21T18:50:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2728526802' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938"}]': finished 2024-09-21T18:50:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[39192]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T18:50:34.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:34.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2728526802' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938"} : dispatch 2024-09-21T18:50:34.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2728526802' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938"}]': finished 2024-09-21T18:50:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[29384]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T18:50:34.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2728526802' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938"} : dispatch 2024-09-21T18:50:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2728526802' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938"}]': finished 2024-09-21T18:50:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:34 smithi104 ceph-mon[32438]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T18:50:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:35.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:35 smithi081 ceph-mon[29384]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:35.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:35 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2395679635' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:50:35.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:35 smithi081 ceph-mon[39192]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:35.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:35 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2395679635' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:50:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:35 smithi104 ceph-mon[32438]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:35 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2395679635' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:50:37.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:37 smithi081 ceph-mon[29384]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:37.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:37 smithi081 ceph-mon[39192]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:37 smithi104 ceph-mon[32438]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:39.745 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:39 smithi081 ceph-mon[29384]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:39.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:39 smithi081 ceph-mon[39192]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:39 smithi104 ceph-mon[32438]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:41.773 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:41 smithi081 ceph-mon[29384]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:41.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:41 smithi081 ceph-mon[39192]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:41 smithi104 ceph-mon[32438]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:42.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:42 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T18:50:42.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:42 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:42.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:42 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T18:50:42.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:42 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:42 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T18:50:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:42 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:43.600 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:43 smithi081 ceph-mon[29384]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:43.600 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:43 smithi081 ceph-mon[29384]: Deploying daemon osd.2 on smithi081 2024-09-21T18:50:43.600 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:43 smithi081 ceph-mon[39192]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:43.600 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:43 smithi081 ceph-mon[39192]: Deploying daemon osd.2 on smithi081 2024-09-21T18:50:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:43 smithi104 ceph-mon[32438]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:43 smithi104 ceph-mon[32438]: Deploying daemon osd.2 on smithi081 2024-09-21T18:50:45.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:45 smithi081 ceph-mon[29384]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:45.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:45 smithi081 ceph-mon[39192]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:46.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:45 smithi104 ceph-mon[32438]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:47 smithi081 ceph-mon[29384]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:47 smithi081 ceph-mon[39192]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:48.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:47 smithi104 ceph-mon[32438]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:48.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:48 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:48.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:48 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:48.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:48 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:48.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:48 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:48.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:48 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:48.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:48 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:48 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:48 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:48 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:50.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:49 smithi104 ceph-mon[32438]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:49 smithi081 ceph-mon[29384]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:49 smithi081 ceph-mon[39192]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:51.396 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 2 on host 'smithi081' 2024-09-21T18:50:51.721 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:51.721 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:51.722 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:51.723 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.723 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:51.723 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.723 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:51 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:51 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:52.015 DEBUG:teuthology.orchestra.run.smithi081:osd.2> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T18:50:52.017 INFO:tasks.cephadm:Deploying osd.3 on smithi081 with /dev/vg_nvme/lv_1... 2024-09-21T18:50:52.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T18:50:52.816 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2[54024]: 2024-09-21T18:50:52.620+0000 7ffafdd92640 -1 osd.2 0 log_to_monitors true 2024-09-21T18:50:53.671 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:50:53.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:53 smithi104 ceph-mon[32438]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:53.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:53 smithi104 ceph-mon[32438]: from='osd.2 v1:172.21.15.81:6809/486671153' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T18:50:53.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:53 smithi104 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T18:50:53.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:53 smithi081 ceph-mon[29384]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:53.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:53 smithi081 ceph-mon[29384]: from='osd.2 v1:172.21.15.81:6809/486671153' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T18:50:53.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:53 smithi081 ceph-mon[29384]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T18:50:53.792 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:53 smithi081 ceph-mon[39192]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:53.793 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:53 smithi081 ceph-mon[39192]: from='osd.2 v1:172.21.15.81:6809/486671153' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T18:50:53.793 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:53 smithi081 ceph-mon[39192]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T18:50:54.724 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[39192]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T18:50:54.724 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[39192]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[39192]: from='osd.2 v1:172.21.15.81:6809/486671153' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[39192]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[29384]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[29384]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[29384]: from='osd.2 v1:172.21.15.81:6809/486671153' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:54.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-mon[29384]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:54.726 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2[54024]: 2024-09-21T18:50:54.510+0000 7ffaf9d15640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T18:50:54.726 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:54 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2[54024]: 2024-09-21T18:50:54.513+0000 7ffaf5b3c640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:50:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:54 smithi104 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T18:50:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:54 smithi104 ceph-mon[32438]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T18:50:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:54 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:54 smithi104 ceph-mon[32438]: from='osd.2 v1:172.21.15.81:6809/486671153' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:54 smithi104 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:55.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.727 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:55.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:50:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T18:50:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:50:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi081 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi081 to 5354M 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: osd.2 v1:172.21.15.81:6809/486671153 boot 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi081 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi081 to 5354M 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: osd.2 v1:172.21.15.81:6809/486671153 boot 2024-09-21T18:50:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T18:50:56.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:56.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:56 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:50:56.839 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:50:56.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi081:vg_nvme/lv_1 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi081 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi081 to 5354M 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: osd.2 v1:172.21.15.81:6809/486671153 boot 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:50:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:56 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:50:57.115 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:50:57.747 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[29384]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:57.747 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[29384]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57611]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57614]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[39192]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T18:50:57.748 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[39192]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T18:50:57.749 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57576]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57576]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57576]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57576]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57596]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57596]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57596]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.749 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57596]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57599]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57599]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57599]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57599]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57602]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57602]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57602]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57605]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57605]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57605]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57605]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.750 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T18:50:57.751 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.751 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.751 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57608]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:57.793 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 ceph-mon[32438]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T18:50:57.793 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T18:50:57.793 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 ceph-mon[32438]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T18:50:57.794 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:50:57.794 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34390]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T18:50:57.794 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34390]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:57.794 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34390]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:57.794 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34390]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57614]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57614]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57614]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57636]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57636]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57636]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57636]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57655]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:58.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:57 smithi081 sudo[57655]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34393]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T18:50:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34393]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T18:50:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34393]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T18:50:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:57 smithi104 sudo[34393]: pam_unix(sudo:session): session closed for user root 2024-09-21T18:50:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:50:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T18:50:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[29384]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:58.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:58 smithi081 ceph-mon[39192]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:50:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T18:50:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:50:59.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:50:59.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:50:59.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T18:50:59.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:50:59.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:58 smithi104 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T18:50:59.749 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[39192]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:50:59.749 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[39192]: from='client.14313 v1:172.21.15.81:0/4068512454' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:59.749 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[39192]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T18:50:59.749 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[39192]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T18:50:59.749 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[29384]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:50:59.750 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[29384]: from='client.14313 v1:172.21.15.81:0/4068512454' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:50:59.750 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[29384]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T18:50:59.750 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:50:59 smithi081 ceph-mon[29384]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T18:51:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:59 smithi104 ceph-mon[32438]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:59 smithi104 ceph-mon[32438]: from='client.14313 v1:172.21.15.81:0/4068512454' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:59 smithi104 ceph-mon[32438]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T18:51:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:50:59 smithi104 ceph-mon[32438]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T18:51:00.671 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3980138719' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"} : dispatch 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[39192]: from='client.34211 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"} : dispatch 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[39192]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"}]': finished 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[39192]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3980138719' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"} : dispatch 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[29384]: from='client.34211 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"} : dispatch 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[29384]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"}]': finished 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[29384]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T18:51:00.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:00 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3980138719' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"} : dispatch 2024-09-21T18:51:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:00 smithi104 ceph-mon[32438]: from='client.34211 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"} : dispatch 2024-09-21T18:51:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:00 smithi104 ceph-mon[32438]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e"}]': finished 2024-09-21T18:51:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:00 smithi104 ceph-mon[32438]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T18:51:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:00 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:01 smithi081 ceph-mon[29384]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1763955626' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:01 smithi081 ceph-mon[39192]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1763955626' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:01 smithi104 ceph-mon[32438]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1763955626' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:03 smithi081 ceph-mon[29384]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:03 smithi081 ceph-mon[39192]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:03 smithi104 ceph-mon[32438]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:05 smithi081 ceph-mon[29384]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:05 smithi081 ceph-mon[39192]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:05 smithi104 ceph-mon[32438]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:07 smithi081 ceph-mon[29384]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T18:51:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:07 smithi081 ceph-mon[39192]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:07 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T18:51:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:07 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:08.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:07 smithi104 ceph-mon[32438]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:08.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:07 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T18:51:08.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:07 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:08.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:08 smithi081 ceph-mon[29384]: Deploying daemon osd.3 on smithi081 2024-09-21T18:51:08.812 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:08 smithi081 ceph-mon[39192]: Deploying daemon osd.3 on smithi081 2024-09-21T18:51:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:08 smithi104 ceph-mon[32438]: Deploying daemon osd.3 on smithi081 2024-09-21T18:51:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:09 smithi081 ceph-mon[29384]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:09.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:09 smithi081 ceph-mon[39192]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:09 smithi104 ceph-mon[32438]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:11.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:11 smithi081 ceph-mon[29384]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:11.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:11 smithi081 ceph-mon[39192]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:12.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:11 smithi104 ceph-mon[32438]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:13.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[29384]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:13.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:13.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:13.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:13.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[39192]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:13.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:13.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:13.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:13 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:13 smithi104 ceph-mon[32438]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:13 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:13 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:13 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:15.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:15 smithi081 ceph-mon[29384]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:15.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:15 smithi081 ceph-mon[39192]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:16.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:15 smithi104 ceph-mon[32438]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:16.221 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 3 on host 'smithi081' 2024-09-21T18:51:16.847 DEBUG:teuthology.orchestra.run.smithi081:osd.3> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T18:51:16.850 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_4... 2024-09-21T18:51:16.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T18:51:17.107 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:51:17.465 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:17 smithi104 ceph-mon[32438]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:17.465 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.465 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.465 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.465 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.493 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[39192]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:17.493 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.493 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.493 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.493 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[29384]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:17.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:17.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:18.066 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 18:51:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T18:51:17.831+0000 7fc731678640 -1 osd.3 0 log_to_monitors true 2024-09-21T18:51:18.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:18 smithi104 ceph-mon[32438]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T18:51:18.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:18 smithi081 ceph-mon[29384]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T18:51:18.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:18 smithi081 ceph-mon[39192]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T18:51:19.328 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:51:19.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[29384]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:19.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[29384]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T18:51:19.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[29384]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T18:51:19.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[29384]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:51:19.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:19.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-09-21T18:51:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:51:19.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[39192]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:19.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[39192]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T18:51:19.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[39192]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T18:51:19.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[39192]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:51:19.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:19.710 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:19 smithi104 ceph-mon[32438]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:19.710 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:19 smithi104 ceph-mon[32438]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T18:51:19.710 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:19 smithi104 ceph-mon[32438]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T18:51:19.711 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:19 smithi104 ceph-mon[32438]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]} : dispatch 2024-09-21T18:51:19.711 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:19 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.066 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T18:51:19.684+0000 7fc72de0f640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T18:51:20.066 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 18:51:19 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T18:51:19.689+0000 7fc72541a640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:51:20.470 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.471 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:20.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:20 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='osd.3 v1:172.21.15.81:6813/1516859335' entity='osd.3' 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:51:20.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:51:20.598 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:20.598 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:20 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:21.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:51:21.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:51:21.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='client.24238 v1:172.21.15.104:0/95760093' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: osd.3 v1:172.21.15.81:6813/1516859335 boot 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi081 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi081 to 4016M 2024-09-21T18:51:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='client.24238 v1:172.21.15.104:0/95760093' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: osd.3 v1:172.21.15.81:6813/1516859335 boot 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi081 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi081 to 4016M 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:51:21.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='client.24238 v1:172.21.15.104:0/95760093' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: osd.3 v1:172.21.15.81:6813/1516859335 boot 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi081 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi081 to 4016M 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:21.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:21.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:21.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:21.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:21.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:21.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:22.953 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:22 smithi104 ceph-mon[32438]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T18:51:22.953 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/1717098451' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"} : dispatch 2024-09-21T18:51:22.953 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:22 smithi104 ceph-mon[32438]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"} : dispatch 2024-09-21T18:51:22.953 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:22 smithi104 ceph-mon[32438]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"}]': finished 2024-09-21T18:51:22.953 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:22 smithi104 ceph-mon[32438]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T18:51:22.953 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:22 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[29384]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T18:51:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/1717098451' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"} : dispatch 2024-09-21T18:51:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[29384]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"} : dispatch 2024-09-21T18:51:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[29384]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"}]': finished 2024-09-21T18:51:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[29384]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T18:51:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[39192]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T18:51:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/1717098451' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"} : dispatch 2024-09-21T18:51:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[39192]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"} : dispatch 2024-09-21T18:51:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[39192]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85220357-38ea-4c90-b422-5d703329f106"}]': finished 2024-09-21T18:51:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[39192]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T18:51:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:22 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:24.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:23 smithi104 ceph-mon[32438]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:23 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/2999131045' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:23 smithi081 ceph-mon[39192]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/2999131045' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:23 smithi081 ceph-mon[29384]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:23 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/2999131045' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:25 smithi104 ceph-mon[32438]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:26.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:25 smithi081 ceph-mon[39192]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:25 smithi081 ceph-mon[29384]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:27 smithi104 ceph-mon[32438]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:27 smithi081 ceph-mon[39192]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:27 smithi081 ceph-mon[29384]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:29.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:29 smithi104 ceph-mon[32438]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:29.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T18:51:29.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:29 smithi081 ceph-mon[29384]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T18:51:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:29 smithi081 ceph-mon[39192]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T18:51:30.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:30.851 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:30 smithi104 ceph-mon[32438]: Deploying daemon osd.4 on smithi104 2024-09-21T18:51:30.967 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:30 smithi081 ceph-mon[39192]: Deploying daemon osd.4 on smithi104 2024-09-21T18:51:30.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:30 smithi081 ceph-mon[29384]: Deploying daemon osd.4 on smithi104 2024-09-21T18:51:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:31 smithi104 ceph-mon[32438]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:31.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:31 smithi081 ceph-mon[29384]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:31.980 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:31 smithi081 ceph-mon[39192]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:33 smithi104 ceph-mon[32438]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:33 smithi081 ceph-mon[29384]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:33 smithi081 ceph-mon[39192]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:35 smithi104 ceph-mon[32438]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:35 smithi081 ceph-mon[29384]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:36.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:35 smithi081 ceph-mon[39192]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:36.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:36 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:36.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:36 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:36.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:36 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:36 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:36 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:36 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:36 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:36 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:36 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:37.997 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:37 smithi104 ceph-mon[32438]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:37 smithi081 ceph-mon[29384]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:37 smithi081 ceph-mon[39192]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:38 smithi104 ceph-mon[32438]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:38 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[29384]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[39192]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:38 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:39.667 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2024-09-21T18:51:40.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:39 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:40.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:39 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:40.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:39 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:40.281 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T18:51:40.284 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_3... 2024-09-21T18:51:40.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T18:51:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:39 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:39 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:39 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:39 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:51:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:39 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:39 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:41.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:40 smithi104 ceph-mon[32438]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:41.221 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 18:51:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T18:51:40.988+0000 7fd1b3a05640 -1 osd.4 0 log_to_monitors true 2024-09-21T18:51:41.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:40 smithi081 ceph-mon[29384]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:41.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:40 smithi081 ceph-mon[39192]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:41.366 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:51:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:41 smithi104 ceph-mon[32438]: from='osd.4 v1:172.21.15.104:6800/1904940409' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T18:51:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:41 smithi104 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T18:51:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:41 smithi081 ceph-mon[29384]: from='osd.4 v1:172.21.15.104:6800/1904940409' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T18:51:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:41 smithi081 ceph-mon[29384]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T18:51:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:41 smithi081 ceph-mon[39192]: from='osd.4 v1:172.21.15.104:6800/1904940409' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T18:51:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:41 smithi081 ceph-mon[39192]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T18:51:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:42 smithi104 ceph-mon[32438]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:42 smithi104 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T18:51:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:42 smithi104 ceph-mon[32438]: from='osd.4 v1:172.21.15.104:6800/1904940409' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:51:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:42 smithi104 ceph-mon[32438]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T18:51:43.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:42 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:43.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:42 smithi104 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:51:43.263 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[29384]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[29384]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[29384]: from='osd.4 v1:172.21.15.104:6800/1904940409' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[29384]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[29384]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[39192]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[39192]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[39192]: from='osd.4 v1:172.21.15.104:6800/1904940409' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[39192]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:43.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:42 smithi081 ceph-mon[39192]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:51:43.689 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T18:51:43.397+0000 7fd1af988640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T18:51:43.690 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T18:51:43.401+0000 7fd1a77a7640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: from='osd.4 ' entity='osd.4' 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: osd.4 v1:172.21.15.104:6800/1904940409 boot 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T18:51:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:43 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:51:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T18:51:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: from='osd.4 ' entity='osd.4' 2024-09-21T18:51:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: osd.4 v1:172.21.15.104:6800/1904940409 boot 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: from='osd.4 ' entity='osd.4' 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: osd.4 v1:172.21.15.104:6800/1904940409 boot 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T18:51:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:43 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:51:45.138 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:51:45.141 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi104 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi104 to 17088M 2024-09-21T18:51:45.142 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.143 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:45.143 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:45.143 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.143 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:44 smithi104 ceph-mon[32438]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi104 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi104 to 17088M 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[29384]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi104 2024-09-21T18:51:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi104 to 17088M 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:51:45.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:44 smithi081 ceph-mon[39192]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T18:51:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:51:45.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-09-21T18:51:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:51:46.708 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:46 smithi104 ceph-mon[32438]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T18:51:46.708 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:46 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:51:46.708 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:46 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:51:46.709 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:46 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[29384]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[39192]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:51:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:51:46.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:46 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:47.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:47 smithi104 ceph-mon[32438]: pgmap v112: 1 pgs: 1 activating+degraded; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:47.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:47 smithi104 ceph-mon[32438]: from='client.24265 v1:172.21.15.104:0/1125509440' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:47.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:47 smithi104 ceph-mon[32438]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-21T18:51:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:47 smithi081 ceph-mon[29384]: pgmap v112: 1 pgs: 1 activating+degraded; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:47 smithi081 ceph-mon[29384]: from='client.24265 v1:172.21.15.104:0/1125509440' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:47 smithi081 ceph-mon[29384]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-21T18:51:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:47 smithi081 ceph-mon[39192]: pgmap v112: 1 pgs: 1 activating+degraded; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:47 smithi081 ceph-mon[39192]: from='client.24265 v1:172.21.15.104:0/1125509440' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:51:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:47 smithi081 ceph-mon[39192]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-09-21T18:51:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: pgmap v113: 1 pgs: 1 activating+degraded; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/2745934140' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"} : dispatch 2024-09-21T18:51:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: from='client.24271 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"} : dispatch 2024-09-21T18:51:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"}]': finished 2024-09-21T18:51:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T18:51:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:51:49.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/411864428' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: pgmap v113: 1 pgs: 1 activating+degraded; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/2745934140' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"} : dispatch 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: from='client.24271 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"} : dispatch 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"}]': finished 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/411864428' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:49.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: pgmap v113: 1 pgs: 1 activating+degraded; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:49.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/2745934140' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"} : dispatch 2024-09-21T18:51:49.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: from='client.24271 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"} : dispatch 2024-09-21T18:51:49.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c"}]': finished 2024-09-21T18:51:49.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T18:51:49.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:51:49.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/411864428' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:51:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:51 smithi104 ceph-mon[32438]: pgmap v115: 1 pgs: 1 activating+degraded; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:51.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:51 smithi081 ceph-mon[29384]: pgmap v115: 1 pgs: 1 activating+degraded; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:51.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:51 smithi081 ceph-mon[39192]: pgmap v115: 1 pgs: 1 activating+degraded; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-09-21T18:51:52.714 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:52 smithi104 ceph-mon[32438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-21T18:51:52.714 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:52 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:51:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:52 smithi081 ceph-mon[29384]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-21T18:51:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:52 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:51:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:52 smithi081 ceph-mon[39192]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-09-21T18:51:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:52 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:51:53.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:53 smithi104 ceph-mon[32438]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T18:51:53.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:53 smithi081 ceph-mon[29384]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T18:51:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:53 smithi081 ceph-mon[39192]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T18:51:54.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:54 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:51:54.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:54 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:51:54.662 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:54 smithi104 ceph-mon[32438]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T18:51:54.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:54 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[29384]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:51:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[39192]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T18:51:54.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:54 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:51:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:55 smithi104 ceph-mon[32438]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:51:55.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:55 smithi081 ceph-mon[29384]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:51:55.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:55 smithi081 ceph-mon[39192]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:51:56.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:56 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T18:51:56.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:56 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:56 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T18:51:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:56 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:56 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T18:51:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:56 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:51:57.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:57 smithi104 ceph-mon[32438]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:51:57.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:57 smithi104 ceph-mon[32438]: Deploying daemon osd.5 on smithi104 2024-09-21T18:51:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:57 smithi081 ceph-mon[29384]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:51:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:57 smithi081 ceph-mon[29384]: Deploying daemon osd.5 on smithi104 2024-09-21T18:51:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:57 smithi081 ceph-mon[39192]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:51:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:57 smithi081 ceph-mon[39192]: Deploying daemon osd.5 on smithi104 2024-09-21T18:51:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:51:59 smithi104 ceph-mon[32438]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T18:51:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:51:59 smithi081 ceph-mon[29384]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T18:51:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:51:59 smithi081 ceph-mon[39192]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T18:52:01.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:01 smithi104 ceph-mon[32438]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:01.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:01 smithi081 ceph-mon[29384]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:01.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:01 smithi081 ceph-mon[39192]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:02.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:02 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:02.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:02 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:02.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:02 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:02.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:02 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:02.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:02 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:02.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:02 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:02.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:02 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:02.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:02 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:02.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:02 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:03.716 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:03 smithi104 ceph-mon[32438]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:03.716 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:03 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T18:52:03.716 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:03 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T18:52:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:03 smithi081 ceph-mon[29384]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:03 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T18:52:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:03 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T18:52:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:03 smithi081 ceph-mon[39192]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:03 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T18:52:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:03 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T18:52:05.391 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:05 smithi104 ceph-mon[32438]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:05.391 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:05 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.391 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:05 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.391 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:05 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:05.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:05 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:05.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:05 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.438 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2024-09-21T18:52:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[29384]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[39192]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:05 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:06.055 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T18:52:06.057 INFO:tasks.cephadm:Deploying osd.6 on smithi104 with /dev/vg_nvme/lv_2... 2024-09-21T18:52:06.058 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T18:52:06.378 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:06.379 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:06.379 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:06.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:06.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:06.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:06.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:06.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:06.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:07.218 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 18:52:06 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[41800]: 2024-09-21T18:52:06.944+0000 7ff24d808640 -1 osd.5 0 log_to_monitors true 2024-09-21T18:52:07.291 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:52:07.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:07 smithi104 ceph-mon[32438]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:07.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:07 smithi104 ceph-mon[32438]: from='osd.5 v1:172.21.15.104:6804/2569144231' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T18:52:07.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:07 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T18:52:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:07 smithi081 ceph-mon[29384]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:07 smithi081 ceph-mon[29384]: from='osd.5 v1:172.21.15.104:6804/2569144231' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T18:52:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:07 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T18:52:07.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:07 smithi081 ceph-mon[39192]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:07.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:07 smithi081 ceph-mon[39192]: from='osd.5 v1:172.21.15.104:6804/2569144231' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T18:52:07.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:07 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T18:52:08.727 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[41800]: 2024-09-21T18:52:08.451+0000 7ff24978b640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T18:52:08.727 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[41800]: 2024-09-21T18:52:08.454+0000 7ff2455b2640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:52:08.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T18:52:08.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-mon[32438]: from='osd.5 v1:172.21.15.104:6804/2569144231' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:08.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-mon[32438]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T18:52:08.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:08.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:08 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T18:52:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[29384]: from='osd.5 v1:172.21.15.104:6804/2569144231' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[29384]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T18:52:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:08.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T18:52:08.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[39192]: from='osd.5 v1:172.21.15.104:6804/2569144231' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:08.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[39192]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T18:52:08.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:08.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:08 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T18:52:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: osd.5 v1:172.21.15.104:6804/2569144231 boot 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T18:52:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:09 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T18:52:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: osd.5 v1:172.21.15.104:6804/2569144231 boot 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: osd.5 v1:172.21.15.104:6804/2569144231 boot 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T18:52:09.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:09 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:52:10.713 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:10 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:10.713 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:10 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:10.713 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:10 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi104 2024-09-21T18:52:10.713 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:10 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi104 to 8544M 2024-09-21T18:52:10.713 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:10 smithi104 ceph-mon[32438]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T18:52:10.716 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:52:10.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi104 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi104 to 8544M 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[29384]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:10.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi104 2024-09-21T18:52:10.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi104 to 8544M 2024-09-21T18:52:10.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:10 smithi081 ceph-mon[39192]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T18:52:10.995 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:52:11.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:11 smithi104 ceph-mon[32438]: pgmap v129: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:11.674 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:11 smithi104 ceph-mon[32438]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T18:52:11.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:11 smithi081 ceph-mon[29384]: pgmap v129: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:11.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:11 smithi081 ceph-mon[29384]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T18:52:11.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:11 smithi081 ceph-mon[39192]: pgmap v129: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:11.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:11 smithi081 ceph-mon[39192]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T18:52:12.638 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:12 smithi104 ceph-mon[32438]: from='client.24292 v1:172.21.15.104:0/1651205175' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:52:12.639 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:12 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:52:12.639 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:12 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:52:12.639 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:12 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[29384]: from='client.24292 v1:172.21.15.104:0/1651205175' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[39192]: from='client.24292 v1:172.21.15.104:0/1651205175' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:52:12.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:52:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:12 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:13.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:13 smithi104 ceph-mon[32438]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:13 smithi081 ceph-mon[29384]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:13.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:13 smithi081 ceph-mon[39192]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/3400859747' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"} : dispatch 2024-09-21T18:52:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:14 smithi104 ceph-mon[32438]: from='client.24298 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"} : dispatch 2024-09-21T18:52:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:14 smithi104 ceph-mon[32438]: from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"}]': finished 2024-09-21T18:52:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:14 smithi104 ceph-mon[32438]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T18:52:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:14 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/1047223428' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:52:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/3400859747' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"} : dispatch 2024-09-21T18:52:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[29384]: from='client.24298 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"} : dispatch 2024-09-21T18:52:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[29384]: from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"}]': finished 2024-09-21T18:52:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[29384]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T18:52:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/1047223428' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:52:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/3400859747' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"} : dispatch 2024-09-21T18:52:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[39192]: from='client.24298 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"} : dispatch 2024-09-21T18:52:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[39192]: from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd"}]': finished 2024-09-21T18:52:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[39192]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T18:52:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/1047223428' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:52:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:15 smithi104 ceph-mon[32438]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:15.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:15 smithi081 ceph-mon[29384]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:15.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:15 smithi081 ceph-mon[39192]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:17 smithi104 ceph-mon[32438]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T18:52:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:17 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:17 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:17.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:17 smithi081 ceph-mon[29384]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T18:52:17.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:17 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:17.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:17 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:17.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:17 smithi081 ceph-mon[39192]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T18:52:17.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:17 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:17.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:17 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:18.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:18 smithi104 ceph-mon[32438]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T18:52:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:18 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:18.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:18 smithi081 ceph-mon[29384]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T18:52:18.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:18 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:18 smithi081 ceph-mon[39192]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T18:52:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:18 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:19.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:19 smithi104 ceph-mon[32438]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T18:52:19.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:19 smithi081 ceph-mon[29384]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T18:52:19.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:19 smithi081 ceph-mon[39192]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T18:52:21.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:21 smithi104 ceph-mon[32438]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:52:21.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T18:52:21.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:21 smithi081 ceph-mon[29384]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:52:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T18:52:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:21 smithi081 ceph-mon[39192]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:52:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T18:52:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:22.611 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:22 smithi104 ceph-mon[32438]: Deploying daemon osd.6 on smithi104 2024-09-21T18:52:22.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:22 smithi081 ceph-mon[29384]: Deploying daemon osd.6 on smithi104 2024-09-21T18:52:22.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:22 smithi081 ceph-mon[39192]: Deploying daemon osd.6 on smithi104 2024-09-21T18:52:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:23 smithi104 ceph-mon[32438]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:52:23.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:23 smithi081 ceph-mon[29384]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:52:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:23 smithi081 ceph-mon[39192]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T18:52:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[29384]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-21T18:52:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[29384]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T18:52:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:25.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[39192]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-21T18:52:25.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:25.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:25.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[39192]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T18:52:25.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:25 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:25.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:25 smithi104 ceph-mon[32438]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-21T18:52:25.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:25 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:25.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:25 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:25.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:25 smithi104 ceph-mon[32438]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T18:52:25.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:25 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:27 smithi104 ceph-mon[32438]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:27 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:27 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:27 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:27.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[29384]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:27.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:27.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:27.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:27.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[39192]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:27.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:27.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:27.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:27 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[29384]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:29.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:29.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:29.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[39192]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:29.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:29.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:29.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:29 smithi104 ceph-mon[32438]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:29.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:29.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:29.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:29.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:30.258 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 6 on host 'smithi104' 2024-09-21T18:52:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:30.798 DEBUG:teuthology.orchestra.run.smithi104:osd.6> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T18:52:30.800 INFO:tasks.cephadm:Deploying osd.7 on smithi104 with /dev/vg_nvme/lv_1... 2024-09-21T18:52:30.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T18:52:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:31 smithi104 ceph-mon[32438]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:31 smithi104 ceph-mon[32438]: from='osd.6 v1:172.21.15.104:6808/185975065' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T18:52:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:31 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T18:52:31.757 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 18:52:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[47075]: 2024-09-21T18:52:31.494+0000 7f0699275640 -1 osd.6 0 log_to_monitors true 2024-09-21T18:52:31.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:31 smithi081 ceph-mon[29384]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:31.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:31 smithi081 ceph-mon[29384]: from='osd.6 v1:172.21.15.104:6808/185975065' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T18:52:31.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:31 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T18:52:31.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:31 smithi081 ceph-mon[39192]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:31.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:31 smithi081 ceph-mon[39192]: from='osd.6 v1:172.21.15.104:6808/185975065' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T18:52:31.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:31 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T18:52:32.402 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:52:32.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T18:52:32.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-mon[32438]: from='osd.6 v1:172.21.15.104:6808/185975065' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:32.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-mon[32438]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T18:52:32.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:32.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:32.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:33.037 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[47075]: 2024-09-21T18:52:32.968+0000 7f06951f8640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T18:52:33.038 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 18:52:32 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[47075]: 2024-09-21T18:52:32.975+0000 7f068d017640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:52:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T18:52:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[39192]: from='osd.6 v1:172.21.15.104:6808/185975065' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[39192]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T18:52:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:33.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T18:52:33.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[29384]: from='osd.6 v1:172.21.15.104:6808/185975065' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:33.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[29384]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T18:52:33.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:33.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:33.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:32 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:33.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:33.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T18:52:33.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:33.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' 2024-09-21T18:52:33.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: osd.6 v1:172.21.15.104:6808/185975065 boot 2024-09-21T18:52:33.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T18:52:33.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:33 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: osd.6 v1:172.21.15.104:6808/185975065 boot 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T18:52:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: osd.6 v1:172.21.15.104:6808/185975065 boot 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T18:52:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:33 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi104 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi104 to 5696M 2024-09-21T18:52:35.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:35.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:35.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:34 smithi104 ceph-mon[32438]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi104 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi104 to 5696M 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[29384]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi104 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi104 to 5696M 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:35.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:34 smithi081 ceph-mon[39192]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T18:52:35.367 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T18:52:35.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-09-21T18:52:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:52:35.740 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:35 smithi104 ceph-mon[32438]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:35.740 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:35 smithi104 ceph-mon[32438]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T18:52:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:35 smithi081 ceph-mon[29384]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:35 smithi081 ceph-mon[29384]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T18:52:36.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:35 smithi081 ceph-mon[39192]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:36.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:35 smithi081 ceph-mon[39192]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T18:52:36.937 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:36 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:52:36.937 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:36 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:52:36.937 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:36 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:36 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:52:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:36 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:52:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:36 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:36 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T18:52:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:36 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T18:52:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:36 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:37.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:37 smithi104 ceph-mon[32438]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:37.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:37 smithi104 ceph-mon[32438]: from='client.24319 v1:172.21.15.104:0/534242954' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:52:37.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:37 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T18:52:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:37 smithi081 ceph-mon[29384]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:37 smithi081 ceph-mon[29384]: from='client.24319 v1:172.21.15.104:0/534242954' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:52:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:37 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T18:52:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:37 smithi081 ceph-mon[39192]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:37 smithi081 ceph-mon[39192]: from='client.24319 v1:172.21.15.104:0/534242954' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:52:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:37 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/1977859507' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"} : dispatch 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: from='client.24325 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"} : dispatch 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"}]': finished 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/3387497632' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/1977859507' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"} : dispatch 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: from='client.24325 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"} : dispatch 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"}]': finished 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/3387497632' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/1977859507' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"} : dispatch 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: from='client.24325 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"} : dispatch 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045"}]': finished 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/3387497632' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T18:52:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:40 smithi104 ceph-mon[32438]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:40 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T18:52:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:40 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:52:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:40 smithi081 ceph-mon[39192]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:40 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T18:52:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:40 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:52:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:40 smithi081 ceph-mon[29384]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:40 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T18:52:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:40 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:52:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:42 smithi104 ceph-mon[32438]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T18:52:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:42 smithi081 ceph-mon[29384]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T18:52:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:42 smithi081 ceph-mon[39192]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T18:52:44.712 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:44 smithi104 ceph-mon[32438]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T18:52:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:44 smithi081 ceph-mon[29384]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T18:52:45.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:44 smithi081 ceph-mon[39192]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T18:52:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:46 smithi104 ceph-mon[32438]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:46 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T18:52:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:46 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:46 smithi104 ceph-mon[32438]: Deploying daemon osd.7 on smithi104 2024-09-21T18:52:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[29384]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T18:52:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[29384]: Deploying daemon osd.7 on smithi104 2024-09-21T18:52:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[39192]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:47.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T18:52:47.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:47.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:46 smithi081 ceph-mon[39192]: Deploying daemon osd.7 on smithi104 2024-09-21T18:52:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:48 smithi104 ceph-mon[32438]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:48 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:48 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:48 smithi081 ceph-mon[29384]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:48 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:48 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:48 smithi081 ceph-mon[39192]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T18:52:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:48 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:48 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:50.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:49 smithi104 ceph-mon[32438]: osdmap e49: 8 total, 7 up, 8 in 2024-09-21T18:52:50.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:49 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:49 smithi081 ceph-mon[29384]: osdmap e49: 8 total, 7 up, 8 in 2024-09-21T18:52:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:49 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:49 smithi081 ceph-mon[39192]: osdmap e49: 8 total, 7 up, 8 in 2024-09-21T18:52:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:49 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:51 smithi104 ceph-mon[32438]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:51 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:51 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:51.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:51 smithi081 ceph-mon[29384]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:51.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:51 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:51.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:51 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:51.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:51 smithi081 ceph-mon[39192]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:51.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:51 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:51.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:51 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:52 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:52 smithi104 ceph-mon[32438]: osdmap e50: 8 total, 7 up, 8 in 2024-09-21T18:52:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:52 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:52 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:52 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[29384]: osdmap e50: 8 total, 7 up, 8 in 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[39192]: osdmap e50: 8 total, 7 up, 8 in 2024-09-21T18:52:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:52.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:52.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:52 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:53 smithi104 ceph-mon[32438]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:53.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:53 smithi081 ceph-mon[29384]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:53 smithi081 ceph-mon[39192]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:54.180 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 7 on host 'smithi104' 2024-09-21T18:52:54.795 DEBUG:teuthology.orchestra.run.smithi104:osd.7> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T18:52:54.797 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T18:52:54.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T18:52:55.056 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:52:55.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:55 smithi081 ceph-mon[29384]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:55.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:55.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:55 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:55.422 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:55 smithi081 ceph-mon[39192]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:55.423 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:55.423 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:55 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:55.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:55 smithi104 ceph-mon[32438]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:55 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:55.789 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:52:56.223 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 18:52:55 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T18:52:55.954+0000 7f418ebba640 -1 osd.7 0 log_to_monitors true 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/984336012' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[29384]: from='osd.7 v1:172.21.15.104:6812/385235741' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[29384]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/984336012' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[39192]: from='osd.7 v1:172.21.15.104:6812/385235741' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T18:52:56.494 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:56 smithi081 ceph-mon[39192]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T18:52:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:56 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:56 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/984336012' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T18:52:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:56 smithi104 ceph-mon[32438]: from='osd.7 v1:172.21.15.104:6812/385235741' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T18:52:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:56 smithi104 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T18:52:56.529 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":50,"num_osds":8,"num_up_osds":7,"osd_up_since":1726944753,"num_in_osds":8,"osd_in_since":1726944758,"num_remapped_pgs":0} 2024-09-21T18:52:57.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T18:52:57.330+0000 7f418ab3d640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T18:52:57.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T18:52:57.337+0000 7f4186964640 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:52:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-mon[32438]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T18:52:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-mon[32438]: from='osd.7 v1:172.21.15.104:6812/385235741' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-mon[32438]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T18:52:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:57 smithi104 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:57.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T18:52:57.775 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[29384]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[29384]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[29384]: from='osd.7 v1:172.21.15.104:6812/385235741' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[29384]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[29384]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[39192]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:57.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[39192]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T18:52:57.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[39192]: from='osd.7 v1:172.21.15.104:6812/385235741' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:57.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[39192]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T18:52:57.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:57.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:57 smithi081 ceph-mon[39192]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T18:52:58.436 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:52:58.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[29384]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:58.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[29384]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T18:52:58.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.437 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[39192]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:58.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[39192]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T18:52:58.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:58 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:58 smithi104 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T18:52:58.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:58 smithi104 ceph-mon[32438]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T18:52:58.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:58.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:58 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:59.027 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":53,"num_osds":8,"num_up_osds":8,"osd_up_since":1726944778,"num_in_osds":8,"osd_in_since":1726944758,"num_remapped_pgs":1} 2024-09-21T18:52:59.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T18:52:59.269 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: osd.7 v1:172.21.15.104:6812/385235741 boot 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/116095312' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: Detected new or changed devices on smithi104 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi104 to 4272M 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.681 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: osd.7 v1:172.21.15.104:6812/385235741 boot 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/116095312' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: Detected new or changed devices on smithi104 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi104 to 4272M 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:59.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:52:59 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: osd.7 v1:172.21.15.104:6812/385235741 boot 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/116095312' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: Detected new or changed devices on smithi104 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi104 to 4272M 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:52:59.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:52:59 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:52:59.943 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:52:59.944 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":54,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","created":"2024-09-21T18:47:59.361278+0000","modified":"2024-09-21T18:52:59.342829+0000","last_up_change":"2024-09-21T18:52:58.333331+0000","last_in_change":"2024-09-21T18:52:38.643758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:50:55.921148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd6626b0-dc43-42d0-a459-9cf3911457ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6801","nonce":390647394}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6802","nonce":390647394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6804","nonce":390647394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6803","nonce":390647394}]},"public_addr":"172.21.15.81:6801/390647394","cluster_addr":"172.21.15.81:6802/390647394","heartbeat_back_addr":"172.21.15.81:6804/390647394","heartbeat_front_addr":"172.21.15.81:6803/390647394","state":["exists","up"]},{"osd":1,"uuid":"2f541064-dc48-4f7f-b29b-065491e148b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6805","nonce":3705537065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6806","nonce":3705537065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6808","nonce":3705537065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6807","nonce":3705537065}]},"public_addr":"172.21.15.81:6805/3705537065","cluster_addr":"172.21.15.81:6806/3705537065","heartbeat_back_addr":"172.21.15.81:6808/3705537065","heartbeat_front_addr":"172.21.15.81:6807/3705537065","state":["exists","up"]},{"osd":2,"uuid":"e7050b14-f463-4de3-8a8d-e05e78507938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6809","nonce":486671153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6810","nonce":486671153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6812","nonce":486671153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6811","nonce":486671153}]},"public_addr":"172.21.15.81:6809/486671153","cluster_addr":"172.21.15.81:6810/486671153","heartbeat_back_addr":"172.21.15.81:6812/486671153","heartbeat_front_addr":"172.21.15.81:6811/486671153","state":["exists","up"]},{"osd":3,"uuid":"75b4cb80-138b-445b-a93a-3c4c598b5e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6813","nonce":1516859335}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6814","nonce":1516859335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6816","nonce":1516859335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6815","nonce":1516859335}]},"public_addr":"172.21.15.81:6813/1516859335","cluster_addr":"172.21.15.81:6814/1516859335","heartbeat_back_addr":"172.21.15.81:6816/1516859335","heartbeat_front_addr":"172.21.15.81:6815/1516859335","state":["exists","up"]},{"osd":4,"uuid":"85220357-38ea-4c90-b422-5d703329f106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6800","nonce":1904940409}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":1904940409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":1904940409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":1904940409}]},"public_addr":"172.21.15.104:6800/1904940409","cluster_addr":"172.21.15.104:6801/1904940409","heartbeat_back_addr":"172.21.15.104:6803/1904940409","heartbeat_front_addr":"172.21.15.104:6802/1904940409","state":["exists","up"]},{"osd":5,"uuid":"a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":2569144231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":2569144231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":2569144231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":2569144231}]},"public_addr":"172.21.15.104:6804/2569144231","cluster_addr":"172.21.15.104:6805/2569144231","heartbeat_back_addr":"172.21.15.104:6807/2569144231","heartbeat_front_addr":"172.21.15.104:6806/2569144231","state":["exists","up"]},{"osd":6,"uuid":"e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":185975065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":185975065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":185975065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":185975065}]},"public_addr":"172.21.15.104:6808/185975065","cluster_addr":"172.21.15.104:6809/185975065","heartbeat_back_addr":"172.21.15.104:6811/185975065","heartbeat_front_addr":"172.21.15.104:6810/185975065","state":["exists","up"]},{"osd":7,"uuid":"f15ee432-aa23-4e5a-9df4-478538af2045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":385235741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6813","nonce":385235741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6815","nonce":385235741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6814","nonce":385235741}]},"public_addr":"172.21.15.104:6812/385235741","cluster_addr":"172.21.15.104:6813/385235741","heartbeat_back_addr":"172.21.15.104:6815/385235741","heartbeat_front_addr":"172.21.15.104:6814/385235741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:47.726542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:51:52.969627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:16.480622+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:24.036573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:50.777436+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:07.986387+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:32.486003+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:56.948418+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6800/1508794297":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/806683817":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/1611463627":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/3045112983":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/2203644131":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3906790599":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/628265626":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/116812683":"2024-09-22T18:48:21.208275+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:53:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:00 smithi081 ceph-mon[29384]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T18:53:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2486195483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:00.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:00 smithi081 ceph-mon[39192]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T18:53:00.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2486195483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:00.688 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T18:53:00.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T18:53:00.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:00 smithi104 ceph-mon[32438]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T18:53:00.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2486195483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:00.929 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:01.548 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-09-21T18:53:01.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:01 smithi104 ceph-mon[32438]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:01.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:01 smithi104 ceph-mon[32438]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T18:53:01.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:01 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T18:53:01.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:01 smithi081 ceph-mon[29384]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:01.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:01 smithi081 ceph-mon[29384]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T18:53:01.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:01 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T18:53:01.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:01 smithi081 ceph-mon[39192]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:01.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:01 smithi081 ceph-mon[39192]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T18:53:01.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:01 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T18:53:02.181 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi081 2024-09-21T18:53:02.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply rgw foo.a --placement '1;smithi081=foo.a' 2024-09-21T18:53:02.442 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:02.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1061101220' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T18:53:02.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1061101220' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T18:53:02.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1061101220' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T18:53:03.074 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled rgw.foo.a update... 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T18:53:03.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T18:53:03.658 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.658 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:03 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:03.684 DEBUG:teuthology.orchestra.run.smithi081:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@rgw.foo.a.service 2024-09-21T18:53:03.686 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi104 2024-09-21T18:53:03.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool create datapool 3 3 replicated 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:03.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:03.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:03 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:03.931 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:04.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:04 smithi104 ceph-mon[32438]: from='client.24370 v1:172.21.15.104:0/1122025187' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi081=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:04.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:04 smithi104 ceph-mon[32438]: Saving service rgw.foo.a spec with placement smithi081=foo.a;count:1 2024-09-21T18:53:04.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:04 smithi104 ceph-mon[32438]: Deploying daemon rgw.foo.a on smithi081 2024-09-21T18:53:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:04 smithi081 ceph-mon[29384]: from='client.24370 v1:172.21.15.104:0/1122025187' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi081=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:04 smithi081 ceph-mon[29384]: Saving service rgw.foo.a spec with placement smithi081=foo.a;count:1 2024-09-21T18:53:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:04 smithi081 ceph-mon[29384]: Deploying daemon rgw.foo.a on smithi081 2024-09-21T18:53:04.805 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:04 smithi081 ceph-mon[39192]: from='client.24370 v1:172.21.15.104:0/1122025187' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi081=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:04.805 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:04 smithi081 ceph-mon[39192]: Saving service rgw.foo.a spec with placement smithi081=foo.a;count:1 2024-09-21T18:53:04.805 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:04 smithi081 ceph-mon[39192]: Deploying daemon rgw.foo.a on smithi081 2024-09-21T18:53:05.066 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:04 smithi081 systemd[1]: Starting Ceph rgw.foo.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:53:05.377 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'datapool' created 2024-09-21T18:53:05.455 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:05 smithi081 ceph-mon[29384]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:05.455 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/4238702959' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T18:53:05.455 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:05 smithi081 ceph-mon[39192]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:05.455 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/4238702959' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T18:53:05.455 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:05 smithi081 podman[63838]: 2024-09-21 18:53:05.092742296 +0000 UTC m=+0.015610444 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T18:53:05.455 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:05 smithi081 podman[63838]: 2024-09-21 18:53:05.199984075 +0000 UTC m=+0.122852208 container create 76aac42e501cc7fbd4ae64d5babc3c77881bb1211771d76496aa646772ce4048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T18:53:05.456 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:05 smithi081 podman[63838]: 2024-09-21 18:53:05.452912835 +0000 UTC m=+0.375780978 container init 76aac42e501cc7fbd4ae64d5babc3c77881bb1211771d76496aa646772ce4048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T18:53:05.697 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:05 smithi104 ceph-mon[32438]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:05.697 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/4238702959' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T18:53:05.816 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:05 smithi081 podman[63838]: 2024-09-21 18:53:05.455999209 +0000 UTC m=+0.378867351 container start 76aac42e501cc7fbd4ae64d5babc3c77881bb1211771d76496aa646772ce4048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731) 2024-09-21T18:53:05.816 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:05 smithi081 bash[63838]: 76aac42e501cc7fbd4ae64d5babc3c77881bb1211771d76496aa646772ce4048 2024-09-21T18:53:05.816 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 18:53:05 smithi081 systemd[1]: Started Ceph rgw.foo.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:53:05.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- rbd pool init datapool 2024-09-21T18:53:06.244 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/4238702959' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: Saving service rgw.foo.a spec with placement smithi081=foo.a;count:1 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/4238702959' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: Saving service rgw.foo.a spec with placement smithi081=foo.a;count:1 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:06 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/4238702959' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: Saving service rgw.foo.a spec with placement smithi081=foo.a;count:1 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:06.668 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:06 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: pgmap v176: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/697031715' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/1487343134' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='client.24388 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:07.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:07 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: pgmap v176: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/697031715' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/1487343134' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='client.24388 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: pgmap v176: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/697031715' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/1487343134' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='client.24388 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:07.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:07 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:08 smithi104 ceph-mon[32438]: Checking dashboard <-> RGW credentials 2024-09-21T18:53:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/697031715' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T18:53:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:08 smithi104 ceph-mon[32438]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T18:53:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:08 smithi104 ceph-mon[32438]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[29384]: Checking dashboard <-> RGW credentials 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/697031715' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[29384]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[29384]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[39192]: Checking dashboard <-> RGW credentials 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/697031715' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[39192]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T18:53:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:08 smithi081 ceph-mon[39192]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T18:53:09.704 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: pgmap v179: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T18:53:09.704 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T18:53:09.704 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:53:09.704 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:53:09.704 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.704 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.705 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.705 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T18:53:09.705 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T18:53:09.705 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:09 smithi104 ceph-mon[32438]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: pgmap v179: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[29384]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: pgmap v179: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T18:53:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:09 smithi081 ceph-mon[39192]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T18:53:09.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.104 --placement '1;smithi104=iscsi.a' 2024-09-21T18:53:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:10.918 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled iscsi.datapool update... 2024-09-21T18:53:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:11 smithi104 ceph-mon[32438]: pgmap v182: 68 pgs: 2 creating+activating, 17 creating+peering, 22 unknown, 27 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-21T18:53:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:11 smithi104 ceph-mon[32438]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T18:53:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:11 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:11 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[29384]: pgmap v182: 68 pgs: 2 creating+activating, 17 creating+peering, 22 unknown, 27 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-21T18:53:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[29384]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T18:53:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:11.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[39192]: pgmap v182: 68 pgs: 2 creating+activating, 17 creating+peering, 22 unknown, 27 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-21T18:53:11.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[39192]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T18:53:11.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T18:53:11.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:11 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:11.572 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-21T18:53:11.572 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:53:11.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T18:53:11.600 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:53:11.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T18:53:11.625 DEBUG:teuthology.orchestra.run.smithi104:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@iscsi.iscsi.a.service 2024-09-21T18:53:11.667 INFO:tasks.cephadm:Adding prometheus.a on smithi104 2024-09-21T18:53:11.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi104=a' 2024-09-21T18:53:11.949 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:12.643 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled prometheus update... 2024-09-21T18:53:12.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:12 smithi104 ceph-mon[32438]: from='client.24433 v1:172.21.15.104:0/161956722' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.104", "placement": "1;smithi104=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:12.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:12 smithi104 ceph-mon[32438]: Saving service iscsi.datapool spec with placement smithi104=iscsi.a;count:1 2024-09-21T18:53:12.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T18:53:12.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:12 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T18:53:12.644 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:12 smithi104 ceph-mon[32438]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T18:53:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[29384]: from='client.24433 v1:172.21.15.104:0/161956722' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.104", "placement": "1;smithi104=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[29384]: Saving service iscsi.datapool spec with placement smithi104=iscsi.a;count:1 2024-09-21T18:53:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T18:53:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T18:53:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[29384]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T18:53:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[39192]: from='client.24433 v1:172.21.15.104:0/161956722' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.104", "placement": "1;smithi104=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[39192]: Saving service iscsi.datapool spec with placement smithi104=iscsi.a;count:1 2024-09-21T18:53:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T18:53:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T18:53:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:12 smithi081 ceph-mon[39192]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T18:53:13.257 DEBUG:teuthology.orchestra.run.smithi104:prometheus.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T18:53:13.260 INFO:tasks.cephadm:Adding node-exporter.a on smithi081 2024-09-21T18:53:13.260 INFO:tasks.cephadm:Adding node-exporter.b on smithi104 2024-09-21T18:53:13.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi081=a;smithi104=b' 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: pgmap v185: 100 pgs: 2 creating+activating, 25 creating+peering, 19 unknown, 54 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: from='client.24439 v1:172.21.15.104:0/4219548635' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: Saving service prometheus spec with placement smithi104=a;count:1 2024-09-21T18:53:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:13 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: pgmap v185: 100 pgs: 2 creating+activating, 25 creating+peering, 19 unknown, 54 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: from='client.24439 v1:172.21.15.104:0/4219548635' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: Saving service prometheus spec with placement smithi104=a;count:1 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: pgmap v185: 100 pgs: 2 creating+activating, 25 creating+peering, 19 unknown, 54 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: from='client.24439 v1:172.21.15.104:0/4219548635' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: Saving service prometheus spec with placement smithi104=a;count:1 2024-09-21T18:53:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:13 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:14.149 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled node-exporter update... 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T18:53:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:14 smithi104 ceph-mon[32438]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T18:53:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T18:53:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T18:53:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T18:53:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[29384]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2645283795' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:14.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/604236037' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T18:53:14.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T18:53:14.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:14 smithi081 ceph-mon[39192]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T18:53:14.823 DEBUG:teuthology.orchestra.run.smithi081:node-exporter.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T18:53:14.826 DEBUG:teuthology.orchestra.run.smithi104:node-exporter.b> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T18:53:14.828 INFO:tasks.cephadm:Adding alertmanager.a on smithi081 2024-09-21T18:53:14.828 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi081=a' 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: pgmap v188: 132 pgs: 2 creating+activating, 22 creating+peering, 48 unknown, 60 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='client.24445 v1:172.21.15.104:0/692309612' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi081=a;smithi104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: Saving service node-exporter spec with placement smithi081=a;smithi104=b;count:2 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T18:53:15.943 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T18:53:15.943 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:15 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:15.971 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: pgmap v188: 132 pgs: 2 creating+activating, 22 creating+peering, 48 unknown, 60 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-21T18:53:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='client.24445 v1:172.21.15.104:0/692309612' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi081=a;smithi104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: Saving service node-exporter spec with placement smithi081=a;smithi104=b;count:2 2024-09-21T18:53:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T18:53:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T18:53:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: pgmap v188: 132 pgs: 2 creating+activating, 22 creating+peering, 48 unknown, 60 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-21T18:53:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='client.24445 v1:172.21.15.104:0/692309612' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi081=a;smithi104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: Saving service node-exporter spec with placement smithi081=a;smithi104=b;count:2 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T18:53:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:15 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:16.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:16 smithi104 systemd[1]: Starting Ceph iscsi.iscsi.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:53:16.827 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled alertmanager update... 2024-09-21T18:53:16.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:16 smithi104 podman[57245]: 2024-09-21 18:53:16.803945057 +0000 UTC m=+0.105688282 container create f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T18:53:16.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:16 smithi104 podman[57245]: 2024-09-21 18:53:16.717301609 +0000 UTC m=+0.019044856 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T18:53:16.893 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:16 smithi104 ceph-mon[32438]: Deploying daemon iscsi.iscsi.a on smithi104 2024-09-21T18:53:17.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:16 smithi081 ceph-mon[29384]: Deploying daemon iscsi.iscsi.a on smithi104 2024-09-21T18:53:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:16 smithi081 ceph-mon[39192]: Deploying daemon iscsi.iscsi.a on smithi104 2024-09-21T18:53:17.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 podman[57245]: 2024-09-21 18:53:17.201180785 +0000 UTC m=+0.502924016 container init f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T18:53:17.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 podman[57245]: 2024-09-21 18:53:17.204594344 +0000 UTC m=+0.506337577 container start f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T18:53:17.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 bash[57245]: f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 2024-09-21T18:53:17.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 systemd[1]: Started Ceph iscsi.iscsi.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:53:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-mon[32438]: pgmap v190: 132 pgs: 4 creating+peering, 9 unknown, 119 active+clean; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 2.6 KiB/s wr, 53 op/s 2024-09-21T18:53:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:17.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:17.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug Started the configuration object watcher 2024-09-21T18:53:17.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug Checking for config object changes every 1s 2024-09-21T18:53:17.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug Processing osd blocklist entries for this node 2024-09-21T18:53:17.835 DEBUG:teuthology.orchestra.run.smithi081:alertmanager.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T18:53:17.838 INFO:tasks.cephadm:Adding grafana.a on smithi104 2024-09-21T18:53:17.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi104=a' 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[29384]: pgmap v190: 132 pgs: 4 creating+peering, 9 unknown, 119 active+clean; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 2.6 KiB/s wr, 53 op/s 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[39192]: pgmap v190: 132 pgs: 4 creating+peering, 9 unknown, 119 active+clean; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 2.6 KiB/s wr, 53 op/s 2024-09-21T18:53:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:17 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:18.095 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug Reading the configuration object to update local LIO configuration 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug Configuration does not have an entry for this host(smithi104.front.sepia.ceph.com) - nothing to define to LIO 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: * Environment: production 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: Use a production WSGI server instead. 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: * Debug mode: off 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug * Running on all addresses. 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: * Running on all addresses. 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug * Running on http://[fe80::ec4:7aff:fe8f:ce89]:5000/ (Press CTRL+C to quit) 2024-09-21T18:53:18.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: * Running on http://[fe80::ec4:7aff:fe8f:ce89]:5000/ (Press CTRL+C to quit) 2024-09-21T18:53:18.338 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:18.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:18 smithi104 ceph-mon[32438]: from='client.24466 v1:172.21.15.104:0/1835903398' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi081=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:18.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:18 smithi104 ceph-mon[32438]: Saving service alertmanager spec with placement smithi081=a;count:1 2024-09-21T18:53:18.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:18 smithi104 ceph-mon[32438]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T18:53:18.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/2323728116' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T18:53:19.047 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled grafana update... 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[29384]: from='client.24466 v1:172.21.15.104:0/1835903398' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi081=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[29384]: Saving service alertmanager spec with placement smithi081=a;count:1 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[29384]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/2323728116' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[39192]: from='client.24466 v1:172.21.15.104:0/1835903398' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi081=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[39192]: Saving service alertmanager spec with placement smithi081=a;count:1 2024-09-21T18:53:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[39192]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T18:53:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/2323728116' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T18:53:19.897 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:19 smithi104 ceph-mon[32438]: Deploying daemon prometheus.a on smithi104 2024-09-21T18:53:19.897 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:19 smithi104 ceph-mon[32438]: pgmap v191: 132 pgs: 132 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.0 KiB/s wr, 81 op/s 2024-09-21T18:53:19.897 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:19 smithi104 ceph-mon[32438]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T18:53:19.897 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:19 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:19.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:19 smithi104 ceph-mon[32438]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T18:53:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[29384]: Deploying daemon prometheus.a on smithi104 2024-09-21T18:53:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[29384]: pgmap v191: 132 pgs: 132 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.0 KiB/s wr, 81 op/s 2024-09-21T18:53:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[29384]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T18:53:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[29384]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T18:53:20.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[39192]: Deploying daemon prometheus.a on smithi104 2024-09-21T18:53:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[39192]: pgmap v191: 132 pgs: 132 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.0 KiB/s wr, 81 op/s 2024-09-21T18:53:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[39192]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T18:53:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:19 smithi081 ceph-mon[39192]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T18:53:20.160 DEBUG:teuthology.orchestra.run.smithi104:grafana.a> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T18:53:20.162 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T18:53:20.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T18:53:20.417 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:20.795 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[29384]: from='client.24502 v1:172.21.15.104:0/1926474669' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:20.795 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[29384]: Saving service grafana spec with placement smithi104=a;count:1 2024-09-21T18:53:20.795 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:53:20.795 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:53:20.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[39192]: from='client.24502 v1:172.21.15.104:0/1926474669' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:20.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[39192]: Saving service grafana spec with placement smithi104=a;count:1 2024-09-21T18:53:20.796 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:53:20.796 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:20 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:53:21.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:20 smithi104 ceph-mon[32438]: from='client.24502 v1:172.21.15.104:0/1926474669' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T18:53:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:20 smithi104 ceph-mon[32438]: Saving service grafana spec with placement smithi104=a;count:1 2024-09-21T18:53:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:20 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:53:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:20 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:53:21.143 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-09-21T18:53:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAhFu9mahVbCBAAnjqSvribuN6YANAnC6Kzug== 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[39192]: pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 5.9 KiB/s wr, 181 op/s 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[39192]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3601693209' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3601693209' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[29384]: pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 5.9 KiB/s wr, 181 op/s 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[29384]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3601693209' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:21.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3601693209' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T18:53:21.877 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-21T18:53:21.877 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T18:53:21.877 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T18:53:21.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T18:53:21.939 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:21 smithi104 ceph-mon[32438]: pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 5.9 KiB/s wr, 181 op/s 2024-09-21T18:53:21.939 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:21 smithi104 ceph-mon[32438]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T18:53:21.939 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:21 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:21.939 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3601693209' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:21.939 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3601693209' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T18:53:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:53:23.192 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-09-21T18:53:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQAjFu9mpCsWCxAAYWxkdUCgU32axrFQMOfXeg== 2024-09-21T18:53:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:23 smithi104 ceph-mon[32438]: pgmap v195: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.0 KiB/s wr, 155 op/s 2024-09-21T18:53:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:23 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.104:0/4213843717' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:23 smithi104 ceph-mon[32438]: from='client.24514 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:23 smithi104 ceph-mon[32438]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T18:53:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:23 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:53:23.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:23 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[39192]: pgmap v195: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.0 KiB/s wr, 155 op/s 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.104:0/4213843717' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[39192]: from='client.24514 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[39192]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:53:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[29384]: pgmap v195: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.0 KiB/s wr, 155 op/s 2024-09-21T18:53:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.104:0/4213843717' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[29384]: from='client.24514 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T18:53:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[29384]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T18:53:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:53:24.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:23 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:53:24.144 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T18:53:24.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T18:53:24.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T18:53:24.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T18:53:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:25.127 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:25.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:25 smithi104 ceph-mon[32438]: pgmap v196: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.2 KiB/s wr, 113 op/s 2024-09-21T18:53:25.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:25 smithi104 ceph-mon[32438]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T18:53:25.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:25 smithi104 ceph-mon[32438]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T18:53:25.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:25 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:53:25.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:25 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:53:25.496 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3167386636' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T18:53:25.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[39192]: pgmap v196: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.2 KiB/s wr, 113 op/s 2024-09-21T18:53:25.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[39192]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T18:53:25.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[39192]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T18:53:25.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:53:25.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:53:25.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3167386636' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T18:53:25.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[29384]: pgmap v196: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.2 KiB/s wr, 113 op/s 2024-09-21T18:53:25.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[29384]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T18:53:25.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[29384]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T18:53:25.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:53:25.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:53:25.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3167386636' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T18:53:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:[{"version":24,"timestamp":"2024-09-21T18:53:14.625314+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"x1VJlY4uyZFU3d7DTPBnROJFkXrNbqrh8z7QUJ2r"}]}] 2024-09-21T18:53:25.800 INFO:tasks.ceph_manager:config epoch is 24 2024-09-21T18:53:25.800 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T18:53:25.801 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T18:53:25.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T18:53:26.045 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:26.703 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:26.703 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:26 smithi081 ceph-mon[29384]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T18:53:26.703 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:26 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:26.704 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:26 smithi081 ceph-mon[39192]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T18:53:26.704 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:26 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:26.745 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:26 smithi104 ceph-mon[32438]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T18:53:26.745 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:26 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:27.436 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6800","nonce":3426409743}]},"active_addr":"172.21.15.81:6800/3426409743","active_change":"2024-09-21T18:48:45.625800+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.81:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":198739293}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":188521944}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1496889754}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":3701153923}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":4266768293}]}]} 2024-09-21T18:53:27.440 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T18:53:27.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T18:53:27.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T18:53:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:27.704 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:27 smithi081 ceph-mon[39192]: pgmap v200: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-21T18:53:27.704 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:27 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3738361931' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T18:53:27.704 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:27 smithi081 ceph-mon[29384]: pgmap v200: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-21T18:53:27.705 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:27 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3738361931' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T18:53:27.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:53:27.736 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:27 smithi104 ceph-mon[32438]: pgmap v200: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-21T18:53:27.736 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:27 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3738361931' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T18:53:28.319 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:28.319 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":70,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","created":"2024-09-21T18:47:59.361278+0000","modified":"2024-09-21T18:53:25.427244+0000","last_up_change":"2024-09-21T18:52:58.333331+0000","last_in_change":"2024-09-21T18:52:38.643758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:50:55.921148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T18:53:04.563630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":60,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T18:53:05.494307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T18:53:07.470408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T18:53:09.418988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T18:53:11.462116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd6626b0-dc43-42d0-a459-9cf3911457ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6801","nonce":390647394}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6802","nonce":390647394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6804","nonce":390647394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6803","nonce":390647394}]},"public_addr":"172.21.15.81:6801/390647394","cluster_addr":"172.21.15.81:6802/390647394","heartbeat_back_addr":"172.21.15.81:6804/390647394","heartbeat_front_addr":"172.21.15.81:6803/390647394","state":["exists","up"]},{"osd":1,"uuid":"2f541064-dc48-4f7f-b29b-065491e148b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6805","nonce":3705537065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6806","nonce":3705537065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6808","nonce":3705537065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6807","nonce":3705537065}]},"public_addr":"172.21.15.81:6805/3705537065","cluster_addr":"172.21.15.81:6806/3705537065","heartbeat_back_addr":"172.21.15.81:6808/3705537065","heartbeat_front_addr":"172.21.15.81:6807/3705537065","state":["exists","up"]},{"osd":2,"uuid":"e7050b14-f463-4de3-8a8d-e05e78507938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6809","nonce":486671153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6810","nonce":486671153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6812","nonce":486671153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6811","nonce":486671153}]},"public_addr":"172.21.15.81:6809/486671153","cluster_addr":"172.21.15.81:6810/486671153","heartbeat_back_addr":"172.21.15.81:6812/486671153","heartbeat_front_addr":"172.21.15.81:6811/486671153","state":["exists","up"]},{"osd":3,"uuid":"75b4cb80-138b-445b-a93a-3c4c598b5e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6813","nonce":1516859335}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6814","nonce":1516859335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6816","nonce":1516859335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6815","nonce":1516859335}]},"public_addr":"172.21.15.81:6813/1516859335","cluster_addr":"172.21.15.81:6814/1516859335","heartbeat_back_addr":"172.21.15.81:6816/1516859335","heartbeat_front_addr":"172.21.15.81:6815/1516859335","state":["exists","up"]},{"osd":4,"uuid":"85220357-38ea-4c90-b422-5d703329f106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6800","nonce":1904940409}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":1904940409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":1904940409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":1904940409}]},"public_addr":"172.21.15.104:6800/1904940409","cluster_addr":"172.21.15.104:6801/1904940409","heartbeat_back_addr":"172.21.15.104:6803/1904940409","heartbeat_front_addr":"172.21.15.104:6802/1904940409","state":["exists","up"]},{"osd":5,"uuid":"a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":2569144231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":2569144231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":2569144231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":2569144231}]},"public_addr":"172.21.15.104:6804/2569144231","cluster_addr":"172.21.15.104:6805/2569144231","heartbeat_back_addr":"172.21.15.104:6807/2569144231","heartbeat_front_addr":"172.21.15.104:6806/2569144231","state":["exists","up"]},{"osd":6,"uuid":"e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":185975065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":185975065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":185975065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":185975065}]},"public_addr":"172.21.15.104:6808/185975065","cluster_addr":"172.21.15.104:6809/185975065","heartbeat_back_addr":"172.21.15.104:6811/185975065","heartbeat_front_addr":"172.21.15.104:6810/185975065","state":["exists","up"]},{"osd":7,"uuid":"f15ee432-aa23-4e5a-9df4-478538af2045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":385235741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6813","nonce":385235741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6815","nonce":385235741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6814","nonce":385235741}]},"public_addr":"172.21.15.104:6812/385235741","cluster_addr":"172.21.15.104:6813/385235741","heartbeat_back_addr":"172.21.15.104:6815/385235741","heartbeat_front_addr":"172.21.15.104:6814/385235741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:47.726542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:53:23.256808+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:53:20.549050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:24.036573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:50.777436+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:53:24.560715+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:32.486003+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:56.948418+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6800/1508794297":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/806683817":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/1611463627":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/3045112983":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/2203644131":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3906790599":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/628265626":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/116812683":"2024-09-22T18:48:21.208275+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:53:28.440 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 systemd[1]: Starting Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:53:28.696 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:28.731 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1859393750' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:28.732 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 podman[58181]: 2024-09-21 18:53:28.342242298 +0000 UTC m=+0.016737331 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T18:53:28.732 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 podman[58181]: 2024-09-21 18:53:28.452917146 +0000 UTC m=+0.127412199 container create 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:53:28.732 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 podman[58181]: 2024-09-21 18:53:28.67193123 +0000 UTC m=+0.346426249 container init 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:53:28.732 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 podman[58181]: 2024-09-21 18:53:28.674843469 +0000 UTC m=+0.349338490 container start 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:53:28.732 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:28.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:28 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1859393750' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:28.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:28.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:28 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1859393750' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:29.007 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.694Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.694Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.694Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi104 (none))" 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.694Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.694Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.696Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.696Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T18:53:29.008 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.698Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T18:53:29.009 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.698Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T18:53:29.009 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.699Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T18:53:29.009 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.699Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.663µs 2024-09-21T18:53:29.009 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.699Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T18:53:29.009 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.699Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-21T18:53:29.009 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.699Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.438µs wal_replay_duration=189.847µs wbl_replay_duration=111ns total_replay_duration=229.031µs 2024-09-21T18:53:29.010 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.701Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T18:53:29.010 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.701Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T18:53:29.010 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.701Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T18:53:29.010 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.716Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.474609ms db_storage=670ns remote_storage=975ns web_handler=202ns query_engine=543ns scrape=451.438µs scrape_sd=99.076µs notify=941ns notify_sd=865ns rules=14.620951ms tracing=5.557µs 2024-09-21T18:53:29.010 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.716Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T18:53:29.010 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:53:28.716Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T18:53:29.011 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 bash[58181]: 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 2024-09-21T18:53:29.011 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:53:28 smithi104 systemd[1]: Started Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:53:29.067 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T18:53:29.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T18:53:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[29384]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[29384]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[39192]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:29.693 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.694 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.694 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.694 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T18:53:29.694 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-mon[39192]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T18:53:29.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-mon[32438]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' 2024-09-21T18:53:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T18:53:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-mon[32438]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T18:53:29.991 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ignoring --setuser ceph since I am not root 2024-09-21T18:53:29.991 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:29 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ignoring --setgroup ceph since I am not root 2024-09-21T18:53:29.991 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:29.991 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":71,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","created":"2024-09-21T18:47:59.361278+0000","modified":"2024-09-21T18:53:29.424606+0000","last_up_change":"2024-09-21T18:52:58.333331+0000","last_in_change":"2024-09-21T18:52:38.643758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:50:55.921148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T18:53:04.563630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":60,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T18:53:05.494307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T18:53:07.470408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T18:53:09.418988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T18:53:11.462116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd6626b0-dc43-42d0-a459-9cf3911457ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6801","nonce":390647394}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6802","nonce":390647394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6804","nonce":390647394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6803","nonce":390647394}]},"public_addr":"172.21.15.81:6801/390647394","cluster_addr":"172.21.15.81:6802/390647394","heartbeat_back_addr":"172.21.15.81:6804/390647394","heartbeat_front_addr":"172.21.15.81:6803/390647394","state":["exists","up"]},{"osd":1,"uuid":"2f541064-dc48-4f7f-b29b-065491e148b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6805","nonce":3705537065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6806","nonce":3705537065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6808","nonce":3705537065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6807","nonce":3705537065}]},"public_addr":"172.21.15.81:6805/3705537065","cluster_addr":"172.21.15.81:6806/3705537065","heartbeat_back_addr":"172.21.15.81:6808/3705537065","heartbeat_front_addr":"172.21.15.81:6807/3705537065","state":["exists","up"]},{"osd":2,"uuid":"e7050b14-f463-4de3-8a8d-e05e78507938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6809","nonce":486671153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6810","nonce":486671153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6812","nonce":486671153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6811","nonce":486671153}]},"public_addr":"172.21.15.81:6809/486671153","cluster_addr":"172.21.15.81:6810/486671153","heartbeat_back_addr":"172.21.15.81:6812/486671153","heartbeat_front_addr":"172.21.15.81:6811/486671153","state":["exists","up"]},{"osd":3,"uuid":"75b4cb80-138b-445b-a93a-3c4c598b5e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6813","nonce":1516859335}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6814","nonce":1516859335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6816","nonce":1516859335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6815","nonce":1516859335}]},"public_addr":"172.21.15.81:6813/1516859335","cluster_addr":"172.21.15.81:6814/1516859335","heartbeat_back_addr":"172.21.15.81:6816/1516859335","heartbeat_front_addr":"172.21.15.81:6815/1516859335","state":["exists","up"]},{"osd":4,"uuid":"85220357-38ea-4c90-b422-5d703329f106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6800","nonce":1904940409}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":1904940409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":1904940409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":1904940409}]},"public_addr":"172.21.15.104:6800/1904940409","cluster_addr":"172.21.15.104:6801/1904940409","heartbeat_back_addr":"172.21.15.104:6803/1904940409","heartbeat_front_addr":"172.21.15.104:6802/1904940409","state":["exists","up"]},{"osd":5,"uuid":"a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":2569144231}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":2569144231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":2569144231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":2569144231}]},"public_addr":"172.21.15.104:6804/2569144231","cluster_addr":"172.21.15.104:6805/2569144231","heartbeat_back_addr":"172.21.15.104:6807/2569144231","heartbeat_front_addr":"172.21.15.104:6806/2569144231","state":["exists","up"]},{"osd":6,"uuid":"e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":185975065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":185975065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":185975065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":185975065}]},"public_addr":"172.21.15.104:6808/185975065","cluster_addr":"172.21.15.104:6809/185975065","heartbeat_back_addr":"172.21.15.104:6811/185975065","heartbeat_front_addr":"172.21.15.104:6810/185975065","state":["exists","up"]},{"osd":7,"uuid":"f15ee432-aa23-4e5a-9df4-478538af2045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":385235741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6813","nonce":385235741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6815","nonce":385235741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6814","nonce":385235741}]},"public_addr":"172.21.15.104:6812/385235741","cluster_addr":"172.21.15.104:6813/385235741","heartbeat_back_addr":"172.21.15.104:6815/385235741","heartbeat_front_addr":"172.21.15.104:6814/385235741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:47.726542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:53:23.256808+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:53:20.549050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:24.036573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:50.777436+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:53:24.560715+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:32.486003+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:52:56.948418+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6800/1508794297":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/806683817":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/1611463627":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/3045112983":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/2203644131":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3906790599":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/628265626":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/116812683":"2024-09-22T18:48:21.208275+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:53:30.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: ignoring --setuser ceph since I am not root 2024-09-21T18:53:30.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:29 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: ignoring --setgroup ceph since I am not root 2024-09-21T18:53:30.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T18:53:30.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T18:53:30.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T18:53:30.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T18:53:30.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T18:53:30.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T18:53:30.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T18:53:30.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T18:53:31.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T18:53:31.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T18:53:31.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[29384]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T18:53:31.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[29384]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-21T18:53:31.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2296298554' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:31.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T18:53:31.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T18:53:31.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[39192]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T18:53:31.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[39192]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-21T18:53:31.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2296298554' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:31.248 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.248 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.248 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.253 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.253 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.254 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:30 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T18:53:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:30 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T18:53:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:30 smithi104 ceph-mon[32438]: from='mgr.14152 v1:172.21.15.81:0/1239011433' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T18:53:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:30 smithi104 ceph-mon[32438]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-21T18:53:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2296298554' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T18:53:31.257 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:31.258 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:33.018 INFO:teuthology.orchestra.run.smithi081.stdout:193273528334 2024-09-21T18:53:33.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T18:53:35.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: [21/Sep/2024:18:53:34] ENGINE Bus STARTING 2024-09-21T18:53:35.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: CherryPy Checker: 2024-09-21T18:53:35.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: The Application mounted at '' has an empty config. 2024-09-21T18:53:35.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: 2024-09-21T18:53:35.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: [21/Sep/2024:18:53:34] ENGINE Serving on http://:::9283 2024-09-21T18:53:35.257 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 18:53:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x[33940]: [21/Sep/2024:18:53:34] ENGINE Bus STARTED 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: Standby manager daemon x restarted 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: Standby manager daemon x started 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T18:53:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: Active manager daemon y restarted 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: Activating manager daemon y 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: mgrmap e18: y(active, starting, since 0.00874563s), standbys: x 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:53:35.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[29384]: Manager daemon y is now available 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: Standby manager daemon x restarted 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: Standby manager daemon x started 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T18:53:35.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: Active manager daemon y restarted 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: Activating manager daemon y 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: mgrmap e18: y(active, starting, since 0.00874563s), standbys: x 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:53:35.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:53:35.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:35 smithi081 ceph-mon[39192]: Manager daemon y is now available 2024-09-21T18:53:35.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T18:53:35.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: Standby manager daemon x restarted 2024-09-21T18:53:35.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: Standby manager daemon x started 2024-09-21T18:53:35.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T18:53:35.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T18:53:35.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.? v1:172.21.15.104:0/816308859' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: Active manager daemon y restarted 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: Activating manager daemon y 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: mgrmap e18: y(active, starting, since 0.00874563s), standbys: x 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T18:53:35.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T18:53:35.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:35 smithi104 ceph-mon[32438]: Manager daemon y is now available 2024-09-21T18:53:35.989 INFO:teuthology.orchestra.run.smithi081.stdout:128849018903 2024-09-21T18:53:35.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T18:53:36.488 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:36.488 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:36.488 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:53:36.488 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:53:36.488 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[29384]: mgrmap e19: y(active, since 1.012s), standbys: x 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:53:36] ENGINE Bus STARTING 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: CherryPy Checker: 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: The Application mounted at '' has an empty config. 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:53:36.489 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:53:36] ENGINE Serving on http://:::9283 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:53:36.490 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-mon[39192]: mgrmap e19: y(active, since 1.012s), standbys: x 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T18:53:36.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:53:36.686 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T18:53:36.686 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:36 smithi104 ceph-mon[32438]: mgrmap e19: y(active, since 1.012s), standbys: x 2024-09-21T18:53:36.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:36 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:53:36] ENGINE Bus STARTED 2024-09-21T18:53:37.059 INFO:teuthology.orchestra.run.smithi081.stdout:227633266697 2024-09-21T18:53:37.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T18:53:37.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:53:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:53:37.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:37 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:53:37.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:37 smithi104 ceph-mon[32438]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:37.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:53:37.771 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-mon[29384]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-mon[39192]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:53:37.651+0000 7f9be6f71640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:18:53:37] ENGINE Error in HTTPServer.serve 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: Traceback (most recent call last): 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._connections.run(self.expiration_interval) 2024-09-21T18:53:37.772 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._run(expiration_interval) 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: s = self.context.wrap_socket( 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: return self.sslsocket_class._create( 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self.do_handshake() 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: self._sslobj.do_handshake() 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:53:37.773 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: [21/Sep/2024:18:53:37] ENGINE Bus STARTING 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: [21/Sep/2024:18:53:37] ENGINE Serving on https://172.21.15.81:7150 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: [21/Sep/2024:18:53:37] ENGINE Error in HTTPServer.serve 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T18:53:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._connections.run(self.expiration_interval) 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._run(expiration_interval) 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: s = self.context.wrap_socket( 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.sslsocket_class._create( 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.do_handshake() 2024-09-21T18:53:38.758 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:53:38.759 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._sslobj.do_handshake() 2024-09-21T18:53:38.759 INFO:journalctl@ceph.mon.b.smithi104.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:53:38.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: [21/Sep/2024:18:53:37] ENGINE Serving on http://172.21.15.81:8765 2024-09-21T18:53:38.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:38 smithi104 ceph-mon[32438]: [21/Sep/2024:18:53:37] ENGINE Bus STARTED 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: [21/Sep/2024:18:53:37] ENGINE Bus STARTING 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: [21/Sep/2024:18:53:37] ENGINE Serving on https://172.21.15.81:7150 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: [21/Sep/2024:18:53:37] ENGINE Error in HTTPServer.serve 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: Traceback (most recent call last): 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: self._connections.run(self.expiration_interval) 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: self._run(expiration_interval) 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:53:38.795 INFO:journalctl@ceph.mon.c.smithi081.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: s = self.context.wrap_socket( 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: return self.sslsocket_class._create( 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: self.do_handshake() 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: self._sslobj.do_handshake() 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: [21/Sep/2024:18:53:37] ENGINE Serving on http://172.21.15.81:8765 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[39192]: [21/Sep/2024:18:53:37] ENGINE Bus STARTED 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: [21/Sep/2024:18:53:37] ENGINE Bus STARTING 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:38.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: [21/Sep/2024:18:53:37] ENGINE Serving on https://172.21.15.81:7150 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: [21/Sep/2024:18:53:37] ENGINE Error in HTTPServer.serve 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: Traceback (most recent call last): 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._connections.run(self.expiration_interval) 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._run(expiration_interval) 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: s = self.context.wrap_socket( 2024-09-21T18:53:38.797 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout: return self.sslsocket_class._create( 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout: self.do_handshake() 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout: self._sslobj.do_handshake() 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: [21/Sep/2024:18:53:37] ENGINE Serving on http://172.21.15.81:8765 2024-09-21T18:53:38.798 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:38 smithi081 ceph-mon[29384]: [21/Sep/2024:18:53:37] ENGINE Bus STARTED 2024-09-21T18:53:38.824 INFO:teuthology.orchestra.run.smithi081.stdout:107374182428 2024-09-21T18:53:38.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T18:53:39.141 INFO:teuthology.orchestra.run.smithi081.stdout:55834574886 2024-09-21T18:53:39.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T18:53:39.602 INFO:teuthology.orchestra.run.smithi081.stdout:158913789969 2024-09-21T18:53:39.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:39 smithi104 ceph-mon[32438]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T18:53:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:53:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:53:39.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[29384]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:39.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:39.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:39 smithi081 ceph-mon[39192]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T18:53:40.062 INFO:teuthology.orchestra.run.smithi081.stdout:77309411363 2024-09-21T18:53:40.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T18:53:40.238 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:40.241 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:40.246 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:40.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:40 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi104 to 3760M 2024-09-21T18:53:40.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:40 smithi081 ceph-mon[39192]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:40.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:40 smithi081 ceph-mon[39192]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-21T18:53:40.697 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:40 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi104 to 3760M 2024-09-21T18:53:40.698 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:40 smithi081 ceph-mon[29384]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:40.698 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:40 smithi081 ceph-mon[29384]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-21T18:53:40.732 INFO:teuthology.orchestra.run.smithi081.stdout:34359738413 2024-09-21T18:53:40.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T18:53:40.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:40 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi104 to 3760M 2024-09-21T18:53:40.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:40 smithi104 ceph-mon[32438]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-21T18:53:40.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:40 smithi104 ceph-mon[32438]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-21T18:53:41.202 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:41.202 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:41.203 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:41.363 INFO:teuthology.orchestra.run.smithi081.stdout:193273528335 2024-09-21T18:53:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/63545657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T18:53:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/63545657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T18:53:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/63545657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T18:53:41.972 INFO:teuthology.orchestra.run.smithi081.stdout:107374182430 2024-09-21T18:53:42.640 INFO:teuthology.orchestra.run.smithi081.stdout:128849018905 2024-09-21T18:53:42.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:42 smithi081 ceph-mon[29384]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T18:53:42.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4285508066' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T18:53:42.641 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:42 smithi081 ceph-mon[39192]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T18:53:42.641 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4285508066' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T18:53:42.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:42 smithi104 ceph-mon[32438]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T18:53:42.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4285508066' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T18:53:43.492 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:43 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1073835068' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T18:53:43.492 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:43 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1073835068' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T18:53:43.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:43 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1073835068' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T18:53:44.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528334 got 193273528335 for osd.6 2024-09-21T18:53:44.146 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:44.455 INFO:teuthology.orchestra.run.smithi081.stdout:55834574888 2024-09-21T18:53:44.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:44 smithi104 ceph-mon[32438]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T18:53:44.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2775073685' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:44 smithi081 ceph-mon[29384]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T18:53:44.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2775073685' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T18:53:44.790 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:44 smithi081 ceph-mon[39192]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T18:53:44.791 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2775073685' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T18:53:45.117 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574886 got 55834574888 for osd.1 2024-09-21T18:53:45.118 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:45.612 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018903 got 128849018905 for osd.4 2024-09-21T18:53:45.612 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:45.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:45 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/916558344' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T18:53:45.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:45 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/916558344' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T18:53:45.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:45 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/916558344' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T18:53:45.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182428 got 107374182430 for osd.3 2024-09-21T18:53:45.991 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:46.481 INFO:teuthology.orchestra.run.smithi081.stdout:158913789971 2024-09-21T18:53:46.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:46 smithi104 ceph-mon[32438]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:46 smithi081 ceph-mon[29384]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:46 smithi081 ceph-mon[39192]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:47.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789969 got 158913789971 for osd.5 2024-09-21T18:53:47.035 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:47.282 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:47.283 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:47.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:53:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:53:47.444 INFO:teuthology.orchestra.run.smithi081.stdout:227633266699 2024-09-21T18:53:47.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3437825503' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T18:53:47.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:53:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3437825503' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T18:53:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3437825503' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T18:53:48.127 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266697 got 227633266699 for osd.7 2024-09-21T18:53:48.127 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:48.497 INFO:teuthology.orchestra.run.smithi081.stdout:34359738415 2024-09-21T18:53:48.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:48 smithi104 ceph-mon[32438]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:48.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:48.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:48 smithi081 ceph-mon[29384]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:48.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:48.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:48 smithi081 ceph-mon[39192]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:48.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:49.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738415 for osd.0 2024-09-21T18:53:49.272 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:49.661 INFO:teuthology.orchestra.run.smithi081.stdout:77309411365 2024-09-21T18:53:49.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1136860335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T18:53:49.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1136860335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T18:53:49.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1136860335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T18:53:50.427 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411365 for osd.2 2024-09-21T18:53:50.427 DEBUG:teuthology.parallel:result is None 2024-09-21T18:53:50.427 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T18:53:50.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:53:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1296440060' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1296440060' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:50.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:50.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:50.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:50 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1296440060' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T18:53:51.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:53:51.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:50 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:53:51.452 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:51.454 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:53:51.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi081 to 3760M 2024-09-21T18:53:51.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi081 to 3760M 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:51.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi081 to 3760M 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi081:/etc/ceph/ceph.conf 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:53:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: Updating smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:52.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:53:52.345 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-21T18:53:51.371436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":887,"num_read_kb":744,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240540,"kb_used_data":7676,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720676,"statfs":{"total":767960285184,"available":767713972224,"internally_reserved":0,"allocated":7860224,"data_stored":4179393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":238276148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":191,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804992+0000","last_change":"2024-09-21T18:53:07.405054+0000","last_active":"2024-09-21T18:53:39.804992+0000","last_peered":"2024-09-21T18:53:39.804992+0000","last_clean":"2024-09-21T18:53:39.804992+0000","last_became_active":"2024-09-21T18:53:07.404774+0000","last_became_peered":"2024-09-21T18:53:07.404774+0000","last_unstale":"2024-09-21T18:53:39.804992+0000","last_undegraded":"2024-09-21T18:53:39.804992+0000","last_fullsized":"2024-09-21T18:53:39.804992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.853378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365761+0000","last_change":"2024-09-21T18:53:09.412508+0000","last_active":"2024-09-21T18:53:39.365761+0000","last_peered":"2024-09-21T18:53:39.365761+0000","last_clean":"2024-09-21T18:53:39.365761+0000","last_became_active":"2024-09-21T18:53:09.412193+0000","last_became_peered":"2024-09-21T18:53:09.412193+0000","last_unstale":"2024-09-21T18:53:39.365761+0000","last_undegraded":"2024-09-21T18:53:39.365761+0000","last_fullsized":"2024-09-21T18:53:39.365761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401891+0000","last_change":"2024-09-21T18:53:11.410506+0000","last_active":"2024-09-21T18:53:39.401891+0000","last_peered":"2024-09-21T18:53:39.401891+0000","last_clean":"2024-09-21T18:53:39.401891+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:53:39.401891+0000","last_undegraded":"2024-09-21T18:53:39.401891+0000","last_fullsized":"2024-09-21T18:53:39.401891+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.829493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365897+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:53:39.365897+0000","last_peered":"2024-09-21T18:53:39.365897+0000","last_clean":"2024-09-21T18:53:39.365897+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:53:39.365897+0000","last_undegraded":"2024-09-21T18:53:39.365897+0000","last_fullsized":"2024-09-21T18:53:39.365897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366163+0000","last_change":"2024-09-21T18:53:13.658931+0000","last_active":"2024-09-21T18:53:39.366163+0000","last_peered":"2024-09-21T18:53:39.366163+0000","last_clean":"2024-09-21T18:53:39.366163+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:53:39.366163+0000","last_undegraded":"2024-09-21T18:53:39.366163+0000","last_fullsized":"2024-09-21T18:53:39.366163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.480075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366676+0000","last_change":"2024-09-21T18:53:07.390902+0000","last_active":"2024-09-21T18:53:39.366676+0000","last_peered":"2024-09-21T18:53:39.366676+0000","last_clean":"2024-09-21T18:53:39.366676+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:53:39.366676+0000","last_undegraded":"2024-09-21T18:53:39.366676+0000","last_fullsized":"2024-09-21T18:53:39.366676+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.326332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":65,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365892+0000","last_change":"2024-09-21T18:53:09.407061+0000","last_active":"2024-09-21T18:53:39.365892+0000","last_peered":"2024-09-21T18:53:39.365892+0000","last_clean":"2024-09-21T18:53:39.365892+0000","last_became_active":"2024-09-21T18:53:09.406774+0000","last_became_peered":"2024-09-21T18:53:09.406774+0000","last_unstale":"2024-09-21T18:53:39.365892+0000","last_undegraded":"2024-09-21T18:53:39.365892+0000","last_fullsized":"2024-09-21T18:53:39.365892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.239535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365756+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:53:39.365756+0000","last_peered":"2024-09-21T18:53:39.365756+0000","last_clean":"2024-09-21T18:53:39.365756+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:53:39.365756+0000","last_undegraded":"2024-09-21T18:53:39.365756+0000","last_fullsized":"2024-09-21T18:53:39.365756+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.141973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659244+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:53:39.659244+0000","last_peered":"2024-09-21T18:53:39.659244+0000","last_clean":"2024-09-21T18:53:39.659244+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:53:39.659244+0000","last_undegraded":"2024-09-21T18:53:39.659244+0000","last_fullsized":"2024-09-21T18:53:39.659244+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.801164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366078+0000","last_change":"2024-09-21T18:53:09.401868+0000","last_active":"2024-09-21T18:53:39.366078+0000","last_peered":"2024-09-21T18:53:39.366078+0000","last_clean":"2024-09-21T18:53:39.366078+0000","last_became_active":"2024-09-21T18:53:09.401547+0000","last_became_peered":"2024-09-21T18:53:09.401547+0000","last_unstale":"2024-09-21T18:53:39.366078+0000","last_undegraded":"2024-09-21T18:53:39.366078+0000","last_fullsized":"2024-09-21T18:53:39.366078+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.067009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659510+0000","last_change":"2024-09-21T18:53:11.409118+0000","last_active":"2024-09-21T18:53:39.659510+0000","last_peered":"2024-09-21T18:53:39.659510+0000","last_clean":"2024-09-21T18:53:39.659510+0000","last_became_active":"2024-09-21T18:53:11.408537+0000","last_became_peered":"2024-09-21T18:53:11.408537+0000","last_unstale":"2024-09-21T18:53:39.659510+0000","last_undegraded":"2024-09-21T18:53:39.659510+0000","last_fullsized":"2024-09-21T18:53:39.659510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.248524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805282+0000","last_change":"2024-09-21T18:53:13.429374+0000","last_active":"2024-09-21T18:53:39.805282+0000","last_peered":"2024-09-21T18:53:39.805282+0000","last_clean":"2024-09-21T18:53:39.805282+0000","last_became_active":"2024-09-21T18:53:13.429007+0000","last_became_peered":"2024-09-21T18:53:13.429007+0000","last_unstale":"2024-09-21T18:53:39.805282+0000","last_undegraded":"2024-09-21T18:53:39.805282+0000","last_fullsized":"2024-09-21T18:53:39.805282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.767711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659156+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:53:39.659156+0000","last_peered":"2024-09-21T18:53:39.659156+0000","last_clean":"2024-09-21T18:53:39.659156+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:53:39.659156+0000","last_undegraded":"2024-09-21T18:53:39.659156+0000","last_fullsized":"2024-09-21T18:53:39.659156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365939+0000","last_change":"2024-09-21T18:53:09.406568+0000","last_active":"2024-09-21T18:53:39.365939+0000","last_peered":"2024-09-21T18:53:39.365939+0000","last_clean":"2024-09-21T18:53:39.365939+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:53:39.365939+0000","last_undegraded":"2024-09-21T18:53:39.365939+0000","last_fullsized":"2024-09-21T18:53:39.365939+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805673+0000","last_change":"2024-09-21T18:53:11.412969+0000","last_active":"2024-09-21T18:53:39.805673+0000","last_peered":"2024-09-21T18:53:39.805673+0000","last_clean":"2024-09-21T18:53:39.805673+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:53:39.805673+0000","last_undegraded":"2024-09-21T18:53:39.805673+0000","last_fullsized":"2024-09-21T18:53:39.805673+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.874706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659386+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:53:39.659386+0000","last_peered":"2024-09-21T18:53:39.659386+0000","last_clean":"2024-09-21T18:53:39.659386+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:53:39.659386+0000","last_undegraded":"2024-09-21T18:53:39.659386+0000","last_fullsized":"2024-09-21T18:53:39.659386+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.945242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366572+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:53:39.366572+0000","last_peered":"2024-09-21T18:53:39.366572+0000","last_clean":"2024-09-21T18:53:39.366572+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:53:39.366572+0000","last_undegraded":"2024-09-21T18:53:39.366572+0000","last_fullsized":"2024-09-21T18:53:39.366572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.198560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805616+0000","last_change":"2024-09-21T18:53:07.405471+0000","last_active":"2024-09-21T18:53:39.805616+0000","last_peered":"2024-09-21T18:53:39.805616+0000","last_clean":"2024-09-21T18:53:39.805616+0000","last_became_active":"2024-09-21T18:53:07.405177+0000","last_became_peered":"2024-09-21T18:53:07.405177+0000","last_unstale":"2024-09-21T18:53:39.805616+0000","last_undegraded":"2024-09-21T18:53:39.805616+0000","last_fullsized":"2024-09-21T18:53:39.805616+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.483473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366610+0000","last_change":"2024-09-21T18:53:09.407164+0000","last_active":"2024-09-21T18:53:39.366610+0000","last_peered":"2024-09-21T18:53:39.366610+0000","last_clean":"2024-09-21T18:53:39.366610+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:53:39.366610+0000","last_undegraded":"2024-09-21T18:53:39.366610+0000","last_fullsized":"2024-09-21T18:53:39.366610+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.948438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402526+0000","last_change":"2024-09-21T18:53:11.411989+0000","last_active":"2024-09-21T18:53:39.402526+0000","last_peered":"2024-09-21T18:53:39.402526+0000","last_clean":"2024-09-21T18:53:39.402526+0000","last_became_active":"2024-09-21T18:53:11.411624+0000","last_became_peered":"2024-09-21T18:53:11.411624+0000","last_unstale":"2024-09-21T18:53:39.402526+0000","last_undegraded":"2024-09-21T18:53:39.402526+0000","last_fullsized":"2024-09-21T18:53:39.402526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.979898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366773+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:53:39.366773+0000","last_peered":"2024-09-21T18:53:39.366773+0000","last_clean":"2024-09-21T18:53:39.366773+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:53:39.366773+0000","last_undegraded":"2024-09-21T18:53:39.366773+0000","last_fullsized":"2024-09-21T18:53:39.366773+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.301610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366169+0000","last_change":"2024-09-21T18:53:07.397249+0000","last_active":"2024-09-21T18:53:39.366169+0000","last_peered":"2024-09-21T18:53:39.366169+0000","last_clean":"2024-09-21T18:53:39.366169+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:53:39.366169+0000","last_undegraded":"2024-09-21T18:53:39.366169+0000","last_fullsized":"2024-09-21T18:53:39.366169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.232024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":63,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366451+0000","last_change":"2024-09-21T18:53:09.406892+0000","last_active":"2024-09-21T18:53:39.366451+0000","last_peered":"2024-09-21T18:53:39.366451+0000","last_clean":"2024-09-21T18:53:39.366451+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:53:39.366451+0000","last_undegraded":"2024-09-21T18:53:39.366451+0000","last_fullsized":"2024-09-21T18:53:39.366451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.459852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366085+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:53:39.366085+0000","last_peered":"2024-09-21T18:53:39.366085+0000","last_clean":"2024-09-21T18:53:39.366085+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:53:39.366085+0000","last_undegraded":"2024-09-21T18:53:39.366085+0000","last_fullsized":"2024-09-21T18:53:39.366085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.060686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806034+0000","last_change":"2024-09-21T18:53:13.426725+0000","last_active":"2024-09-21T18:53:39.806034+0000","last_peered":"2024-09-21T18:53:39.806034+0000","last_clean":"2024-09-21T18:53:39.806034+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:53:39.806034+0000","last_undegraded":"2024-09-21T18:53:39.806034+0000","last_fullsized":"2024-09-21T18:53:39.806034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401685+0000","last_change":"2024-09-21T18:53:07.397345+0000","last_active":"2024-09-21T18:53:39.401685+0000","last_peered":"2024-09-21T18:53:39.401685+0000","last_clean":"2024-09-21T18:53:39.401685+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:53:39.401685+0000","last_undegraded":"2024-09-21T18:53:39.401685+0000","last_fullsized":"2024-09-21T18:53:39.401685+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.808378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805790+0000","last_change":"2024-09-21T18:53:09.719399+0000","last_active":"2024-09-21T18:53:39.805790+0000","last_peered":"2024-09-21T18:53:39.805790+0000","last_clean":"2024-09-21T18:53:39.805790+0000","last_became_active":"2024-09-21T18:53:09.719059+0000","last_became_peered":"2024-09-21T18:53:09.719059+0000","last_unstale":"2024-09-21T18:53:39.805790+0000","last_undegraded":"2024-09-21T18:53:39.805790+0000","last_fullsized":"2024-09-21T18:53:39.805790+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.854862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455319+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:53:42.455319+0000","last_peered":"2024-09-21T18:53:42.455319+0000","last_clean":"2024-09-21T18:53:42.455319+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:53:42.455319+0000","last_undegraded":"2024-09-21T18:53:42.455319+0000","last_fullsized":"2024-09-21T18:53:42.455319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.329971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402686+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:53:39.402686+0000","last_peered":"2024-09-21T18:53:39.402686+0000","last_clean":"2024-09-21T18:53:39.402686+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:53:39.402686+0000","last_undegraded":"2024-09-21T18:53:39.402686+0000","last_fullsized":"2024-09-21T18:53:39.402686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.373058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365495+0000","last_change":"2024-09-21T18:53:07.398833+0000","last_active":"2024-09-21T18:53:39.365495+0000","last_peered":"2024-09-21T18:53:39.365495+0000","last_clean":"2024-09-21T18:53:39.365495+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:53:39.365495+0000","last_undegraded":"2024-09-21T18:53:39.365495+0000","last_fullsized":"2024-09-21T18:53:39.365495+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455246+0000","last_change":"2024-09-21T18:53:11.411518+0000","last_active":"2024-09-21T18:53:42.455246+0000","last_peered":"2024-09-21T18:53:42.455246+0000","last_clean":"2024-09-21T18:53:42.455246+0000","last_became_active":"2024-09-21T18:53:11.411212+0000","last_became_peered":"2024-09-21T18:53:11.411212+0000","last_unstale":"2024-09-21T18:53:42.455246+0000","last_undegraded":"2024-09-21T18:53:42.455246+0000","last_fullsized":"2024-09-21T18:53:42.455246+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.137419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659601+0000","last_change":"2024-09-21T18:53:13.433767+0000","last_active":"2024-09-21T18:53:39.659601+0000","last_peered":"2024-09-21T18:53:39.659601+0000","last_clean":"2024-09-21T18:53:39.659601+0000","last_became_active":"2024-09-21T18:53:13.433575+0000","last_became_peered":"2024-09-21T18:53:13.433575+0000","last_unstale":"2024-09-21T18:53:39.659601+0000","last_undegraded":"2024-09-21T18:53:39.659601+0000","last_fullsized":"2024-09-21T18:53:39.659601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.484363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":72,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365369+0000","last_change":"2024-09-21T18:53:09.719338+0000","last_active":"2024-09-21T18:53:39.365369+0000","last_peered":"2024-09-21T18:53:39.365369+0000","last_clean":"2024-09-21T18:53:39.365369+0000","last_became_active":"2024-09-21T18:53:09.719017+0000","last_became_peered":"2024-09-21T18:53:09.719017+0000","last_unstale":"2024-09-21T18:53:39.365369+0000","last_undegraded":"2024-09-21T18:53:39.365369+0000","last_fullsized":"2024-09-21T18:53:39.365369+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.426276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366986+0000","last_change":"2024-09-21T18:53:07.400418+0000","last_active":"2024-09-21T18:53:39.366986+0000","last_peered":"2024-09-21T18:53:39.366986+0000","last_clean":"2024-09-21T18:53:39.366986+0000","last_became_active":"2024-09-21T18:53:07.400143+0000","last_became_peered":"2024-09-21T18:53:07.400143+0000","last_unstale":"2024-09-21T18:53:39.366986+0000","last_undegraded":"2024-09-21T18:53:39.366986+0000","last_fullsized":"2024-09-21T18:53:39.366986+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.737589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365822+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:53:39.365822+0000","last_peered":"2024-09-21T18:53:39.365822+0000","last_clean":"2024-09-21T18:53:39.365822+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:53:39.365822+0000","last_undegraded":"2024-09-21T18:53:39.365822+0000","last_fullsized":"2024-09-21T18:53:39.365822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.956086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366022+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:53:39.366022+0000","last_peered":"2024-09-21T18:53:39.366022+0000","last_clean":"2024-09-21T18:53:39.366022+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:53:39.366022+0000","last_undegraded":"2024-09-21T18:53:39.366022+0000","last_fullsized":"2024-09-21T18:53:39.366022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.926924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":60,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365872+0000","last_change":"2024-09-21T18:53:09.400587+0000","last_active":"2024-09-21T18:53:39.365872+0000","last_peered":"2024-09-21T18:53:39.365872+0000","last_clean":"2024-09-21T18:53:39.365872+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:53:39.365872+0000","last_undegraded":"2024-09-21T18:53:39.365872+0000","last_fullsized":"2024-09-21T18:53:39.365872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.173359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804973+0000","last_change":"2024-09-21T18:53:07.404382+0000","last_active":"2024-09-21T18:53:39.804973+0000","last_peered":"2024-09-21T18:53:39.804973+0000","last_clean":"2024-09-21T18:53:39.804973+0000","last_became_active":"2024-09-21T18:53:07.404018+0000","last_became_peered":"2024-09-21T18:53:07.404018+0000","last_unstale":"2024-09-21T18:53:39.804973+0000","last_undegraded":"2024-09-21T18:53:39.804973+0000","last_fullsized":"2024-09-21T18:53:39.804973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.245821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":53,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.454988+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:53:42.454988+0000","last_peered":"2024-09-21T18:53:42.454988+0000","last_clean":"2024-09-21T18:53:42.454988+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:53:42.454988+0000","last_undegraded":"2024-09-21T18:53:42.454988+0000","last_fullsized":"2024-09-21T18:53:42.454988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.339632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805949+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:53:39.805949+0000","last_peered":"2024-09-21T18:53:39.805949+0000","last_clean":"2024-09-21T18:53:39.805949+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:53:39.805949+0000","last_undegraded":"2024-09-21T18:53:39.805949+0000","last_fullsized":"2024-09-21T18:53:39.805949+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.973293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402122+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:53:39.402122+0000","last_peered":"2024-09-21T18:53:39.402122+0000","last_clean":"2024-09-21T18:53:39.402122+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:53:39.402122+0000","last_undegraded":"2024-09-21T18:53:39.402122+0000","last_fullsized":"2024-09-21T18:53:39.402122+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.374498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659123+0000","last_change":"2024-09-21T18:53:07.403537+0000","last_active":"2024-09-21T18:53:39.659123+0000","last_peered":"2024-09-21T18:53:39.659123+0000","last_clean":"2024-09-21T18:53:39.659123+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:53:39.659123+0000","last_undegraded":"2024-09-21T18:53:39.659123+0000","last_fullsized":"2024-09-21T18:53:39.659123+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.558691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":51,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805817+0000","last_change":"2024-09-21T18:53:11.407531+0000","last_active":"2024-09-21T18:53:39.805817+0000","last_peered":"2024-09-21T18:53:39.805817+0000","last_clean":"2024-09-21T18:53:39.805817+0000","last_became_active":"2024-09-21T18:53:11.407244+0000","last_became_peered":"2024-09-21T18:53:11.407244+0000","last_unstale":"2024-09-21T18:53:39.805817+0000","last_undegraded":"2024-09-21T18:53:39.805817+0000","last_fullsized":"2024-09-21T18:53:39.805817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.763939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805349+0000","last_change":"2024-09-21T18:53:13.435210+0000","last_active":"2024-09-21T18:53:39.805349+0000","last_peered":"2024-09-21T18:53:39.805349+0000","last_clean":"2024-09-21T18:53:39.805349+0000","last_became_active":"2024-09-21T18:53:13.434753+0000","last_became_peered":"2024-09-21T18:53:13.434753+0000","last_unstale":"2024-09-21T18:53:39.805349+0000","last_undegraded":"2024-09-21T18:53:39.805349+0000","last_fullsized":"2024-09-21T18:53:39.805349+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.794214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":75,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805838+0000","last_change":"2024-09-21T18:53:09.719454+0000","last_active":"2024-09-21T18:53:39.805838+0000","last_peered":"2024-09-21T18:53:39.805838+0000","last_clean":"2024-09-21T18:53:39.805838+0000","last_became_active":"2024-09-21T18:53:09.719114+0000","last_became_peered":"2024-09-21T18:53:09.719114+0000","last_unstale":"2024-09-21T18:53:39.805838+0000","last_undegraded":"2024-09-21T18:53:39.805838+0000","last_fullsized":"2024-09-21T18:53:39.805838+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.176379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401702+0000","last_change":"2024-09-21T18:53:07.400542+0000","last_active":"2024-09-21T18:53:39.401702+0000","last_peered":"2024-09-21T18:53:39.401702+0000","last_clean":"2024-09-21T18:53:39.401702+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:53:39.401702+0000","last_undegraded":"2024-09-21T18:53:39.401702+0000","last_fullsized":"2024-09-21T18:53:39.401702+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.007870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805118+0000","last_change":"2024-09-21T18:53:11.404992+0000","last_active":"2024-09-21T18:53:39.805118+0000","last_peered":"2024-09-21T18:53:39.805118+0000","last_clean":"2024-09-21T18:53:39.805118+0000","last_became_active":"2024-09-21T18:53:11.404577+0000","last_became_peered":"2024-09-21T18:53:11.404577+0000","last_unstale":"2024-09-21T18:53:39.805118+0000","last_undegraded":"2024-09-21T18:53:39.805118+0000","last_fullsized":"2024-09-21T18:53:39.805118+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.773972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401855+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:53:39.401855+0000","last_peered":"2024-09-21T18:53:39.401855+0000","last_clean":"2024-09-21T18:53:39.401855+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:53:39.401855+0000","last_undegraded":"2024-09-21T18:53:39.401855+0000","last_fullsized":"2024-09-21T18:53:39.401855+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.173798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":95,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402453+0000","last_change":"2024-09-21T18:53:09.410172+0000","last_active":"2024-09-21T18:53:39.402453+0000","last_peered":"2024-09-21T18:53:39.402453+0000","last_clean":"2024-09-21T18:53:39.402453+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:53:39.402453+0000","last_undegraded":"2024-09-21T18:53:39.402453+0000","last_fullsized":"2024-09-21T18:53:39.402453+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.020223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366204+0000","last_change":"2024-09-21T18:53:07.403315+0000","last_active":"2024-09-21T18:53:39.366204+0000","last_peered":"2024-09-21T18:53:39.366204+0000","last_clean":"2024-09-21T18:53:39.366204+0000","last_became_active":"2024-09-21T18:53:07.403002+0000","last_became_peered":"2024-09-21T18:53:07.403002+0000","last_unstale":"2024-09-21T18:53:39.366204+0000","last_undegraded":"2024-09-21T18:53:39.366204+0000","last_fullsized":"2024-09-21T18:53:39.366204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:08.340495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":53,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659620+0000","last_change":"2024-09-21T18:53:09.398954+0000","last_active":"2024-09-21T18:53:39.659620+0000","last_peered":"2024-09-21T18:53:39.659620+0000","last_clean":"2024-09-21T18:53:39.659620+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:53:39.659620+0000","last_undegraded":"2024-09-21T18:53:39.659620+0000","last_fullsized":"2024-09-21T18:53:39.659620+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.467230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805634+0000","last_change":"2024-09-21T18:53:11.411504+0000","last_active":"2024-09-21T18:53:39.805634+0000","last_peered":"2024-09-21T18:53:39.805634+0000","last_clean":"2024-09-21T18:53:39.805634+0000","last_became_active":"2024-09-21T18:53:11.411229+0000","last_became_peered":"2024-09-21T18:53:11.411229+0000","last_unstale":"2024-09-21T18:53:39.805634+0000","last_undegraded":"2024-09-21T18:53:39.805634+0000","last_fullsized":"2024-09-21T18:53:39.805634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.568499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366452+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:53:39.366452+0000","last_peered":"2024-09-21T18:53:39.366452+0000","last_clean":"2024-09-21T18:53:39.366452+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:53:39.366452+0000","last_undegraded":"2024-09-21T18:53:39.366452+0000","last_fullsized":"2024-09-21T18:53:39.366452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.888758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402711+0000","last_change":"2024-09-21T18:53:09.412601+0000","last_active":"2024-09-21T18:53:39.402711+0000","last_peered":"2024-09-21T18:53:39.402711+0000","last_clean":"2024-09-21T18:53:39.402711+0000","last_became_active":"2024-09-21T18:53:09.412304+0000","last_became_peered":"2024-09-21T18:53:09.412304+0000","last_unstale":"2024-09-21T18:53:39.402711+0000","last_undegraded":"2024-09-21T18:53:39.402711+0000","last_fullsized":"2024-09-21T18:53:39.402711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.434125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401670+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:53:39.401670+0000","last_peered":"2024-09-21T18:53:39.401670+0000","last_clean":"2024-09-21T18:53:39.401670+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:53:39.401670+0000","last_undegraded":"2024-09-21T18:53:39.401670+0000","last_fullsized":"2024-09-21T18:53:39.401670+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.025870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365720+0000","last_change":"2024-09-21T18:53:09.400372+0000","last_active":"2024-09-21T18:53:39.365720+0000","last_peered":"2024-09-21T18:53:39.365720+0000","last_clean":"2024-09-21T18:53:39.365720+0000","last_became_active":"2024-09-21T18:53:06.391866+0000","last_became_peered":"2024-09-21T18:53:06.391866+0000","last_unstale":"2024-09-21T18:53:39.365720+0000","last_undegraded":"2024-09-21T18:53:39.365720+0000","last_fullsized":"2024-09-21T18:53:39.365720+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366142+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:53:39.366142+0000","last_peered":"2024-09-21T18:53:39.366142+0000","last_clean":"2024-09-21T18:53:39.366142+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:53:39.366142+0000","last_undegraded":"2024-09-21T18:53:39.366142+0000","last_fullsized":"2024-09-21T18:53:39.366142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.265652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806109+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:53:39.806109+0000","last_peered":"2024-09-21T18:53:39.806109+0000","last_clean":"2024-09-21T18:53:39.806109+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:53:39.806109+0000","last_undegraded":"2024-09-21T18:53:39.806109+0000","last_fullsized":"2024-09-21T18:53:39.806109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805805+0000","last_change":"2024-09-21T18:53:09.407641+0000","last_active":"2024-09-21T18:53:39.805805+0000","last_peered":"2024-09-21T18:53:39.805805+0000","last_clean":"2024-09-21T18:53:39.805805+0000","last_became_active":"2024-09-21T18:53:09.407302+0000","last_became_peered":"2024-09-21T18:53:09.407302+0000","last_unstale":"2024-09-21T18:53:39.805805+0000","last_undegraded":"2024-09-21T18:53:39.805805+0000","last_fullsized":"2024-09-21T18:53:39.805805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.070171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804934+0000","last_change":"2024-09-21T18:53:07.404328+0000","last_active":"2024-09-21T18:53:39.804934+0000","last_peered":"2024-09-21T18:53:39.804934+0000","last_clean":"2024-09-21T18:53:39.804934+0000","last_became_active":"2024-09-21T18:53:07.404060+0000","last_became_peered":"2024-09-21T18:53:07.404060+0000","last_unstale":"2024-09-21T18:53:39.804934+0000","last_undegraded":"2024-09-21T18:53:39.804934+0000","last_fullsized":"2024-09-21T18:53:39.804934+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.843533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":92,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:49.566582+0000","last_change":"2024-09-21T18:53:09.719413+0000","last_active":"2024-09-21T18:53:49.566582+0000","last_peered":"2024-09-21T18:53:49.566582+0000","last_clean":"2024-09-21T18:53:49.566582+0000","last_became_active":"2024-09-21T18:53:06.394373+0000","last_became_peered":"2024-09-21T18:53:06.394373+0000","last_unstale":"2024-09-21T18:53:49.566582+0000","last_undegraded":"2024-09-21T18:53:49.566582+0000","last_fullsized":"2024-09-21T18:53:49.566582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.196010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659926+0000","last_change":"2024-09-21T18:53:11.409878+0000","last_active":"2024-09-21T18:53:39.659926+0000","last_peered":"2024-09-21T18:53:39.659926+0000","last_clean":"2024-09-21T18:53:39.659926+0000","last_became_active":"2024-09-21T18:53:11.409637+0000","last_became_peered":"2024-09-21T18:53:11.409637+0000","last_unstale":"2024-09-21T18:53:39.659926+0000","last_undegraded":"2024-09-21T18:53:39.659926+0000","last_fullsized":"2024-09-21T18:53:39.659926+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.893970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402098+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:53:39.402098+0000","last_peered":"2024-09-21T18:53:39.402098+0000","last_clean":"2024-09-21T18:53:39.402098+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:53:39.402098+0000","last_undegraded":"2024-09-21T18:53:39.402098+0000","last_fullsized":"2024-09-21T18:53:39.402098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.377231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455236+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:53:42.455236+0000","last_peered":"2024-09-21T18:53:42.455236+0000","last_clean":"2024-09-21T18:53:42.455236+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:53:42.455236+0000","last_undegraded":"2024-09-21T18:53:42.455236+0000","last_fullsized":"2024-09-21T18:53:42.455236+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.645863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366238+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:53:39.366238+0000","last_peered":"2024-09-21T18:53:39.366238+0000","last_clean":"2024-09-21T18:53:39.366238+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:53:39.366238+0000","last_undegraded":"2024-09-21T18:53:39.366238+0000","last_fullsized":"2024-09-21T18:53:39.366238+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.707207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":87,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806474+0000","last_change":"2024-09-21T18:53:29.766304+0000","last_active":"2024-09-21T18:53:39.806474+0000","last_peered":"2024-09-21T18:53:39.806474+0000","last_clean":"2024-09-21T18:53:39.806474+0000","last_became_active":"2024-09-21T18:53:00.358105+0000","last_became_peered":"2024-09-21T18:53:00.358105+0000","last_unstale":"2024-09-21T18:53:39.806474+0000","last_undegraded":"2024-09-21T18:53:39.806474+0000","last_fullsized":"2024-09-21T18:53:39.806474+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:50.998311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806259+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:53:39.806259+0000","last_peered":"2024-09-21T18:53:39.806259+0000","last_clean":"2024-09-21T18:53:39.806259+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:53:39.806259+0000","last_undegraded":"2024-09-21T18:53:39.806259+0000","last_fullsized":"2024-09-21T18:53:39.806259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.081383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659874+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:53:39.659874+0000","last_peered":"2024-09-21T18:53:39.659874+0000","last_clean":"2024-09-21T18:53:39.659874+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:53:39.659874+0000","last_undegraded":"2024-09-21T18:53:39.659874+0000","last_fullsized":"2024-09-21T18:53:39.659874+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.641099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366660+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:53:39.366660+0000","last_peered":"2024-09-21T18:53:39.366660+0000","last_clean":"2024-09-21T18:53:39.366660+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:53:39.366660+0000","last_undegraded":"2024-09-21T18:53:39.366660+0000","last_fullsized":"2024-09-21T18:53:39.366660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366237+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:53:39.366237+0000","last_peered":"2024-09-21T18:53:39.366237+0000","last_clean":"2024-09-21T18:53:39.366237+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:53:39.366237+0000","last_undegraded":"2024-09-21T18:53:39.366237+0000","last_fullsized":"2024-09-21T18:53:39.366237+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.491463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659836+0000","last_change":"2024-09-21T18:53:11.409004+0000","last_active":"2024-09-21T18:53:39.659836+0000","last_peered":"2024-09-21T18:53:39.659836+0000","last_clean":"2024-09-21T18:53:39.659836+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:53:39.659836+0000","last_undegraded":"2024-09-21T18:53:39.659836+0000","last_fullsized":"2024-09-21T18:53:39.659836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.557609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366142+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:53:39.366142+0000","last_peered":"2024-09-21T18:53:39.366142+0000","last_clean":"2024-09-21T18:53:39.366142+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:53:39.366142+0000","last_undegraded":"2024-09-21T18:53:39.366142+0000","last_fullsized":"2024-09-21T18:53:39.366142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.277061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":67,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366456+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:53:39.366456+0000","last_peered":"2024-09-21T18:53:39.366456+0000","last_clean":"2024-09-21T18:53:39.366456+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:53:39.366456+0000","last_undegraded":"2024-09-21T18:53:39.366456+0000","last_fullsized":"2024-09-21T18:53:39.366456+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.263644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804679+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:53:39.804679+0000","last_peered":"2024-09-21T18:53:39.804679+0000","last_clean":"2024-09-21T18:53:39.804679+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:53:39.804679+0000","last_undegraded":"2024-09-21T18:53:39.804679+0000","last_fullsized":"2024-09-21T18:53:39.804679+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402286+0000","last_change":"2024-09-21T18:53:11.409557+0000","last_active":"2024-09-21T18:53:39.402286+0000","last_peered":"2024-09-21T18:53:39.402286+0000","last_clean":"2024-09-21T18:53:39.402286+0000","last_became_active":"2024-09-21T18:53:11.409200+0000","last_became_peered":"2024-09-21T18:53:11.409200+0000","last_unstale":"2024-09-21T18:53:39.402286+0000","last_undegraded":"2024-09-21T18:53:39.402286+0000","last_fullsized":"2024-09-21T18:53:39.402286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:37.475931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366132+0000","last_change":"2024-09-21T18:53:13.430863+0000","last_active":"2024-09-21T18:53:39.366132+0000","last_peered":"2024-09-21T18:53:39.366132+0000","last_clean":"2024-09-21T18:53:39.366132+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:53:39.366132+0000","last_undegraded":"2024-09-21T18:53:39.366132+0000","last_fullsized":"2024-09-21T18:53:39.366132+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:38.403912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366137+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:53:39.366137+0000","last_peered":"2024-09-21T18:53:39.366137+0000","last_clean":"2024-09-21T18:53:39.366137+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:53:39.366137+0000","last_undegraded":"2024-09-21T18:53:39.366137+0000","last_fullsized":"2024-09-21T18:53:39.366137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.582865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":50,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366588+0000","last_change":"2024-09-21T18:53:07.400595+0000","last_active":"2024-09-21T18:53:39.366588+0000","last_peered":"2024-09-21T18:53:39.366588+0000","last_clean":"2024-09-21T18:53:39.366588+0000","last_became_active":"2024-09-21T18:53:07.399708+0000","last_became_peered":"2024-09-21T18:53:07.399708+0000","last_unstale":"2024-09-21T18:53:39.366588+0000","last_undegraded":"2024-09-21T18:53:39.366588+0000","last_fullsized":"2024-09-21T18:53:39.366588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:07.859688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.454988+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:53:42.454988+0000","last_peered":"2024-09-21T18:53:42.454988+0000","last_clean":"2024-09-21T18:53:42.454988+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:53:42.454988+0000","last_undegraded":"2024-09-21T18:53:42.454988+0000","last_fullsized":"2024-09-21T18:53:42.454988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.955877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366444+0000","last_change":"2024-09-21T18:53:13.428208+0000","last_active":"2024-09-21T18:53:39.366444+0000","last_peered":"2024-09-21T18:53:39.366444+0000","last_clean":"2024-09-21T18:53:39.366444+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:53:39.366444+0000","last_undegraded":"2024-09-21T18:53:39.366444+0000","last_fullsized":"2024-09-21T18:53:39.366444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.939232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805782+0000","last_change":"2024-09-21T18:53:09.408663+0000","last_active":"2024-09-21T18:53:39.805782+0000","last_peered":"2024-09-21T18:53:39.805782+0000","last_clean":"2024-09-21T18:53:39.805782+0000","last_became_active":"2024-09-21T18:53:09.408315+0000","last_became_peered":"2024-09-21T18:53:09.408315+0000","last_unstale":"2024-09-21T18:53:39.805782+0000","last_undegraded":"2024-09-21T18:53:39.805782+0000","last_fullsized":"2024-09-21T18:53:39.805782+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.191003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659103+0000","last_change":"2024-09-21T18:53:07.400068+0000","last_active":"2024-09-21T18:53:39.659103+0000","last_peered":"2024-09-21T18:53:39.659103+0000","last_clean":"2024-09-21T18:53:39.659103+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:53:39.659103+0000","last_undegraded":"2024-09-21T18:53:39.659103+0000","last_fullsized":"2024-09-21T18:53:39.659103+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.160743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365773+0000","last_change":"2024-09-21T18:53:11.415649+0000","last_active":"2024-09-21T18:53:39.365773+0000","last_peered":"2024-09-21T18:53:39.365773+0000","last_clean":"2024-09-21T18:53:39.365773+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:53:39.365773+0000","last_undegraded":"2024-09-21T18:53:39.365773+0000","last_fullsized":"2024-09-21T18:53:39.365773+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.547416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805243+0000","last_change":"2024-09-21T18:53:13.434903+0000","last_active":"2024-09-21T18:53:39.805243+0000","last_peered":"2024-09-21T18:53:39.805243+0000","last_clean":"2024-09-21T18:53:39.805243+0000","last_became_active":"2024-09-21T18:53:13.434569+0000","last_became_peered":"2024-09-21T18:53:13.434569+0000","last_unstale":"2024-09-21T18:53:39.805243+0000","last_undegraded":"2024-09-21T18:53:39.805243+0000","last_fullsized":"2024-09-21T18:53:39.805243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.431252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805212+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:53:39.805212+0000","last_peered":"2024-09-21T18:53:39.805212+0000","last_clean":"2024-09-21T18:53:39.805212+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:53:39.805212+0000","last_undegraded":"2024-09-21T18:53:39.805212+0000","last_fullsized":"2024-09-21T18:53:39.805212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.041097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806556+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:53:39.806556+0000","last_peered":"2024-09-21T18:53:39.806556+0000","last_clean":"2024-09-21T18:53:39.806556+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:53:39.806556+0000","last_undegraded":"2024-09-21T18:53:39.806556+0000","last_fullsized":"2024-09-21T18:53:39.806556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.852227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365800+0000","last_change":"2024-09-21T18:53:11.413440+0000","last_active":"2024-09-21T18:53:39.365800+0000","last_peered":"2024-09-21T18:53:39.365800+0000","last_clean":"2024-09-21T18:53:39.365800+0000","last_became_active":"2024-09-21T18:53:11.413142+0000","last_became_peered":"2024-09-21T18:53:11.413142+0000","last_unstale":"2024-09-21T18:53:39.365800+0000","last_undegraded":"2024-09-21T18:53:39.365800+0000","last_fullsized":"2024-09-21T18:53:39.365800+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.886861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806391+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:53:39.806391+0000","last_peered":"2024-09-21T18:53:39.806391+0000","last_clean":"2024-09-21T18:53:39.806391+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:53:39.806391+0000","last_undegraded":"2024-09-21T18:53:39.806391+0000","last_fullsized":"2024-09-21T18:53:39.806391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.894993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":62,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366120+0000","last_change":"2024-09-21T18:53:09.406428+0000","last_active":"2024-09-21T18:53:39.366120+0000","last_peered":"2024-09-21T18:53:39.366120+0000","last_clean":"2024-09-21T18:53:39.366120+0000","last_became_active":"2024-09-21T18:53:09.406030+0000","last_became_peered":"2024-09-21T18:53:09.406030+0000","last_unstale":"2024-09-21T18:53:39.366120+0000","last_undegraded":"2024-09-21T18:53:39.366120+0000","last_fullsized":"2024-09-21T18:53:39.366120+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.045459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804870+0000","last_change":"2024-09-21T18:53:07.400289+0000","last_active":"2024-09-21T18:53:39.804870+0000","last_peered":"2024-09-21T18:53:39.804870+0000","last_clean":"2024-09-21T18:53:39.804870+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:53:39.804870+0000","last_undegraded":"2024-09-21T18:53:39.804870+0000","last_fullsized":"2024-09-21T18:53:39.804870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:06.969808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365529+0000","last_change":"2024-09-21T18:53:11.417570+0000","last_active":"2024-09-21T18:53:39.365529+0000","last_peered":"2024-09-21T18:53:39.365529+0000","last_clean":"2024-09-21T18:53:39.365529+0000","last_became_active":"2024-09-21T18:53:11.417277+0000","last_became_peered":"2024-09-21T18:53:11.417277+0000","last_unstale":"2024-09-21T18:53:39.365529+0000","last_undegraded":"2024-09-21T18:53:39.365529+0000","last_fullsized":"2024-09-21T18:53:39.365529+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.958604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366256+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:53:39.366256+0000","last_peered":"2024-09-21T18:53:39.366256+0000","last_clean":"2024-09-21T18:53:39.366256+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:53:39.366256+0000","last_undegraded":"2024-09-21T18:53:39.366256+0000","last_fullsized":"2024-09-21T18:53:39.366256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.706132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659282+0000","last_change":"2024-09-21T18:53:09.717860+0000","last_active":"2024-09-21T18:53:39.659282+0000","last_peered":"2024-09-21T18:53:39.659282+0000","last_clean":"2024-09-21T18:53:39.659282+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:53:39.659282+0000","last_undegraded":"2024-09-21T18:53:39.659282+0000","last_fullsized":"2024-09-21T18:53:39.659282+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.363297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804914+0000","last_change":"2024-09-21T18:53:07.405281+0000","last_active":"2024-09-21T18:53:39.804914+0000","last_peered":"2024-09-21T18:53:39.804914+0000","last_clean":"2024-09-21T18:53:39.804914+0000","last_became_active":"2024-09-21T18:53:07.404988+0000","last_became_peered":"2024-09-21T18:53:07.404988+0000","last_unstale":"2024-09-21T18:53:39.804914+0000","last_undegraded":"2024-09-21T18:53:39.804914+0000","last_fullsized":"2024-09-21T18:53:39.804914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.141596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455248+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:53:42.455248+0000","last_peered":"2024-09-21T18:53:42.455248+0000","last_clean":"2024-09-21T18:53:42.455248+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:53:42.455248+0000","last_undegraded":"2024-09-21T18:53:42.455248+0000","last_fullsized":"2024-09-21T18:53:42.455248+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.640908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659413+0000","last_change":"2024-09-21T18:53:13.657306+0000","last_active":"2024-09-21T18:53:39.659413+0000","last_peered":"2024-09-21T18:53:39.659413+0000","last_clean":"2024-09-21T18:53:39.659413+0000","last_became_active":"2024-09-21T18:53:13.656931+0000","last_became_peered":"2024-09-21T18:53:13.656931+0000","last_unstale":"2024-09-21T18:53:39.659413+0000","last_undegraded":"2024-09-21T18:53:39.659413+0000","last_fullsized":"2024-09-21T18:53:39.659413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:42.145553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805686+0000","last_change":"2024-09-21T18:53:07.397798+0000","last_active":"2024-09-21T18:53:39.805686+0000","last_peered":"2024-09-21T18:53:39.805686+0000","last_clean":"2024-09-21T18:53:39.805686+0000","last_became_active":"2024-09-21T18:53:07.397469+0000","last_became_peered":"2024-09-21T18:53:07.397469+0000","last_unstale":"2024-09-21T18:53:39.805686+0000","last_undegraded":"2024-09-21T18:53:39.805686+0000","last_fullsized":"2024-09-21T18:53:39.805686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.024888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366907+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:53:39.366907+0000","last_peered":"2024-09-21T18:53:39.366907+0000","last_clean":"2024-09-21T18:53:39.366907+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:53:39.366907+0000","last_undegraded":"2024-09-21T18:53:39.366907+0000","last_fullsized":"2024-09-21T18:53:39.366907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.137675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659807+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:53:39.659807+0000","last_peered":"2024-09-21T18:53:39.659807+0000","last_clean":"2024-09-21T18:53:39.659807+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:53:39.659807+0000","last_undegraded":"2024-09-21T18:53:39.659807+0000","last_fullsized":"2024-09-21T18:53:39.659807+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.109044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806481+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:53:39.806481+0000","last_peered":"2024-09-21T18:53:39.806481+0000","last_clean":"2024-09-21T18:53:39.806481+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:53:39.806481+0000","last_undegraded":"2024-09-21T18:53:39.806481+0000","last_fullsized":"2024-09-21T18:53:39.806481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805862+0000","last_change":"2024-09-21T18:53:09.720374+0000","last_active":"2024-09-21T18:53:39.805862+0000","last_peered":"2024-09-21T18:53:39.805862+0000","last_clean":"2024-09-21T18:53:39.805862+0000","last_became_active":"2024-09-21T18:53:09.720056+0000","last_became_peered":"2024-09-21T18:53:09.720056+0000","last_unstale":"2024-09-21T18:53:39.805862+0000","last_undegraded":"2024-09-21T18:53:39.805862+0000","last_fullsized":"2024-09-21T18:53:39.805862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.445280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804758+0000","last_change":"2024-09-21T18:53:07.406957+0000","last_active":"2024-09-21T18:53:39.804758+0000","last_peered":"2024-09-21T18:53:39.804758+0000","last_clean":"2024-09-21T18:53:39.804758+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:53:39.804758+0000","last_undegraded":"2024-09-21T18:53:39.804758+0000","last_fullsized":"2024-09-21T18:53:39.804758+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.565944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366803+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:53:39.366803+0000","last_peered":"2024-09-21T18:53:39.366803+0000","last_clean":"2024-09-21T18:53:39.366803+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:53:39.366803+0000","last_undegraded":"2024-09-21T18:53:39.366803+0000","last_fullsized":"2024-09-21T18:53:39.366803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.484261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366750+0000","last_change":"2024-09-21T18:53:13.423168+0000","last_active":"2024-09-21T18:53:39.366750+0000","last_peered":"2024-09-21T18:53:39.366750+0000","last_clean":"2024-09-21T18:53:39.366750+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:53:39.366750+0000","last_undegraded":"2024-09-21T18:53:39.366750+0000","last_fullsized":"2024-09-21T18:53:39.366750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.401268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659787+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:53:39.659787+0000","last_peered":"2024-09-21T18:53:39.659787+0000","last_clean":"2024-09-21T18:53:39.659787+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:53:39.659787+0000","last_undegraded":"2024-09-21T18:53:39.659787+0000","last_fullsized":"2024-09-21T18:53:39.659787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.011518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804933+0000","last_change":"2024-09-21T18:53:07.405516+0000","last_active":"2024-09-21T18:53:39.804933+0000","last_peered":"2024-09-21T18:53:39.804933+0000","last_clean":"2024-09-21T18:53:39.804933+0000","last_became_active":"2024-09-21T18:53:07.405223+0000","last_became_peered":"2024-09-21T18:53:07.405223+0000","last_unstale":"2024-09-21T18:53:39.804933+0000","last_undegraded":"2024-09-21T18:53:39.804933+0000","last_fullsized":"2024-09-21T18:53:39.804933+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.984977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":53,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455298+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:53:42.455298+0000","last_peered":"2024-09-21T18:53:42.455298+0000","last_clean":"2024-09-21T18:53:42.455298+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:53:42.455298+0000","last_undegraded":"2024-09-21T18:53:42.455298+0000","last_fullsized":"2024-09-21T18:53:42.455298+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.168997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366608+0000","last_change":"2024-09-21T18:53:13.423826+0000","last_active":"2024-09-21T18:53:39.366608+0000","last_peered":"2024-09-21T18:53:39.366608+0000","last_clean":"2024-09-21T18:53:39.366608+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:53:39.366608+0000","last_undegraded":"2024-09-21T18:53:39.366608+0000","last_fullsized":"2024-09-21T18:53:39.366608+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.967749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366987+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:53:39.366987+0000","last_peered":"2024-09-21T18:53:39.366987+0000","last_clean":"2024-09-21T18:53:39.366987+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:53:39.366987+0000","last_undegraded":"2024-09-21T18:53:39.366987+0000","last_fullsized":"2024-09-21T18:53:39.366987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.620456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804788+0000","last_change":"2024-09-21T18:53:07.403003+0000","last_active":"2024-09-21T18:53:39.804788+0000","last_peered":"2024-09-21T18:53:39.804788+0000","last_clean":"2024-09-21T18:53:39.804788+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:53:39.804788+0000","last_undegraded":"2024-09-21T18:53:39.804788+0000","last_fullsized":"2024-09-21T18:53:39.804788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.651586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455317+0000","last_change":"2024-09-21T18:53:11.411052+0000","last_active":"2024-09-21T18:53:42.455317+0000","last_peered":"2024-09-21T18:53:42.455317+0000","last_clean":"2024-09-21T18:53:42.455317+0000","last_became_active":"2024-09-21T18:53:11.410769+0000","last_became_peered":"2024-09-21T18:53:11.410769+0000","last_unstale":"2024-09-21T18:53:42.455317+0000","last_undegraded":"2024-09-21T18:53:42.455317+0000","last_fullsized":"2024-09-21T18:53:42.455317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805631+0000","last_change":"2024-09-21T18:53:13.431033+0000","last_active":"2024-09-21T18:53:39.805631+0000","last_peered":"2024-09-21T18:53:39.805631+0000","last_clean":"2024-09-21T18:53:39.805631+0000","last_became_active":"2024-09-21T18:53:13.430667+0000","last_became_peered":"2024-09-21T18:53:13.430667+0000","last_unstale":"2024-09-21T18:53:39.805631+0000","last_undegraded":"2024-09-21T18:53:39.805631+0000","last_fullsized":"2024-09-21T18:53:39.805631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.681228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365948+0000","last_change":"2024-09-21T18:53:09.407481+0000","last_active":"2024-09-21T18:53:39.365948+0000","last_peered":"2024-09-21T18:53:39.365948+0000","last_clean":"2024-09-21T18:53:39.365948+0000","last_became_active":"2024-09-21T18:53:09.407184+0000","last_became_peered":"2024-09-21T18:53:09.407184+0000","last_unstale":"2024-09-21T18:53:39.365948+0000","last_undegraded":"2024-09-21T18:53:39.365948+0000","last_fullsized":"2024-09-21T18:53:39.365948+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.470465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366358+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:53:39.366358+0000","last_peered":"2024-09-21T18:53:39.366358+0000","last_clean":"2024-09-21T18:53:39.366358+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:53:39.366358+0000","last_undegraded":"2024-09-21T18:53:39.366358+0000","last_fullsized":"2024-09-21T18:53:39.366358+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.908635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402053+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:53:39.402053+0000","last_peered":"2024-09-21T18:53:39.402053+0000","last_clean":"2024-09-21T18:53:39.402053+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:53:39.402053+0000","last_undegraded":"2024-09-21T18:53:39.402053+0000","last_fullsized":"2024-09-21T18:53:39.402053+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.492927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365399+0000","last_change":"2024-09-21T18:53:13.433363+0000","last_active":"2024-09-21T18:53:39.365399+0000","last_peered":"2024-09-21T18:53:39.365399+0000","last_clean":"2024-09-21T18:53:39.365399+0000","last_became_active":"2024-09-21T18:53:13.433057+0000","last_became_peered":"2024-09-21T18:53:13.433057+0000","last_unstale":"2024-09-21T18:53:39.365399+0000","last_undegraded":"2024-09-21T18:53:39.365399+0000","last_fullsized":"2024-09-21T18:53:39.365399+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.215327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402167+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:53:39.402167+0000","last_peered":"2024-09-21T18:53:39.402167+0000","last_clean":"2024-09-21T18:53:39.402167+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:53:39.402167+0000","last_undegraded":"2024-09-21T18:53:39.402167+0000","last_fullsized":"2024-09-21T18:53:39.402167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.126928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804798+0000","last_change":"2024-09-21T18:53:07.406445+0000","last_active":"2024-09-21T18:53:39.804798+0000","last_peered":"2024-09-21T18:53:39.804798+0000","last_clean":"2024-09-21T18:53:39.804798+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:53:39.804798+0000","last_undegraded":"2024-09-21T18:53:39.804798+0000","last_fullsized":"2024-09-21T18:53:39.804798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.278365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366344+0000","last_change":"2024-09-21T18:53:11.411055+0000","last_active":"2024-09-21T18:53:39.366344+0000","last_peered":"2024-09-21T18:53:39.366344+0000","last_clean":"2024-09-21T18:53:39.366344+0000","last_became_active":"2024-09-21T18:53:11.410775+0000","last_became_peered":"2024-09-21T18:53:11.410775+0000","last_unstale":"2024-09-21T18:53:39.366344+0000","last_undegraded":"2024-09-21T18:53:39.366344+0000","last_fullsized":"2024-09-21T18:53:39.366344+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.614499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805130+0000","last_change":"2024-09-21T18:53:13.433872+0000","last_active":"2024-09-21T18:53:39.805130+0000","last_peered":"2024-09-21T18:53:39.805130+0000","last_clean":"2024-09-21T18:53:39.805130+0000","last_became_active":"2024-09-21T18:53:13.433534+0000","last_became_peered":"2024-09-21T18:53:13.433534+0000","last_unstale":"2024-09-21T18:53:39.805130+0000","last_undegraded":"2024-09-21T18:53:39.805130+0000","last_fullsized":"2024-09-21T18:53:39.805130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.158748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":59,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365648+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:53:39.365648+0000","last_peered":"2024-09-21T18:53:39.365648+0000","last_clean":"2024-09-21T18:53:39.365648+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:53:39.365648+0000","last_undegraded":"2024-09-21T18:53:39.365648+0000","last_fullsized":"2024-09-21T18:53:39.365648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.024213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659141+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:53:39.659141+0000","last_peered":"2024-09-21T18:53:39.659141+0000","last_clean":"2024-09-21T18:53:39.659141+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:53:39.659141+0000","last_undegraded":"2024-09-21T18:53:39.659141+0000","last_fullsized":"2024-09-21T18:53:39.659141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.249689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804970+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:53:39.804970+0000","last_peered":"2024-09-21T18:53:39.804970+0000","last_clean":"2024-09-21T18:53:39.804970+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:53:39.804970+0000","last_undegraded":"2024-09-21T18:53:39.804970+0000","last_fullsized":"2024-09-21T18:53:39.804970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.908774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366496+0000","last_change":"2024-09-21T18:53:13.658591+0000","last_active":"2024-09-21T18:53:39.366496+0000","last_peered":"2024-09-21T18:53:39.366496+0000","last_clean":"2024-09-21T18:53:39.366496+0000","last_became_active":"2024-09-21T18:53:13.657938+0000","last_became_peered":"2024-09-21T18:53:13.657938+0000","last_unstale":"2024-09-21T18:53:39.366496+0000","last_undegraded":"2024-09-21T18:53:39.366496+0000","last_fullsized":"2024-09-21T18:53:39.366496+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.163794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":46,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365325+0000","last_change":"2024-09-21T18:53:09.409585+0000","last_active":"2024-09-21T18:53:39.365325+0000","last_peered":"2024-09-21T18:53:39.365325+0000","last_clean":"2024-09-21T18:53:39.365325+0000","last_became_active":"2024-09-21T18:53:09.409216+0000","last_became_peered":"2024-09-21T18:53:09.409216+0000","last_unstale":"2024-09-21T18:53:39.365325+0000","last_undegraded":"2024-09-21T18:53:39.365325+0000","last_fullsized":"2024-09-21T18:53:39.365325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804988+0000","last_change":"2024-09-21T18:53:07.401813+0000","last_active":"2024-09-21T18:53:39.804988+0000","last_peered":"2024-09-21T18:53:39.804988+0000","last_clean":"2024-09-21T18:53:39.804988+0000","last_became_active":"2024-09-21T18:53:07.401545+0000","last_became_peered":"2024-09-21T18:53:07.401545+0000","last_unstale":"2024-09-21T18:53:39.804988+0000","last_undegraded":"2024-09-21T18:53:39.804988+0000","last_fullsized":"2024-09-21T18:53:39.804988+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.528471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805547+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:53:39.805547+0000","last_peered":"2024-09-21T18:53:39.805547+0000","last_clean":"2024-09-21T18:53:39.805547+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:53:39.805547+0000","last_undegraded":"2024-09-21T18:53:39.805547+0000","last_fullsized":"2024-09-21T18:53:39.805547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.190087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805573+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:53:39.805573+0000","last_peered":"2024-09-21T18:53:39.805573+0000","last_clean":"2024-09-21T18:53:39.805573+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:53:39.805573+0000","last_undegraded":"2024-09-21T18:53:39.805573+0000","last_fullsized":"2024-09-21T18:53:39.805573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402586+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:53:39.402586+0000","last_peered":"2024-09-21T18:53:39.402586+0000","last_clean":"2024-09-21T18:53:39.402586+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:53:39.402586+0000","last_undegraded":"2024-09-21T18:53:39.402586+0000","last_fullsized":"2024-09-21T18:53:39.402586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.931032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365475+0000","last_change":"2024-09-21T18:53:07.400561+0000","last_active":"2024-09-21T18:53:39.365475+0000","last_peered":"2024-09-21T18:53:39.365475+0000","last_clean":"2024-09-21T18:53:39.365475+0000","last_became_active":"2024-09-21T18:53:07.400254+0000","last_became_peered":"2024-09-21T18:53:07.400254+0000","last_unstale":"2024-09-21T18:53:39.365475+0000","last_undegraded":"2024-09-21T18:53:39.365475+0000","last_fullsized":"2024-09-21T18:53:39.365475+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.077186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805851+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:53:39.805851+0000","last_peered":"2024-09-21T18:53:39.805851+0000","last_clean":"2024-09-21T18:53:39.805851+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:53:39.805851+0000","last_undegraded":"2024-09-21T18:53:39.805851+0000","last_fullsized":"2024-09-21T18:53:39.805851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.130619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805737+0000","last_change":"2024-09-21T18:53:11.415531+0000","last_active":"2024-09-21T18:53:39.805737+0000","last_peered":"2024-09-21T18:53:39.805737+0000","last_clean":"2024-09-21T18:53:39.805737+0000","last_became_active":"2024-09-21T18:53:11.415209+0000","last_became_peered":"2024-09-21T18:53:11.415209+0000","last_unstale":"2024-09-21T18:53:39.805737+0000","last_undegraded":"2024-09-21T18:53:39.805737+0000","last_fullsized":"2024-09-21T18:53:39.805737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.845440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266701,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1351680,"data_stored":891253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528338,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1335296,"data_stored":890604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789973,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":737280,"data_stored":300146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018907,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31936,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713216,"statfs":{"total":95995035648,"available":95962333184,"internally_reserved":0,"allocated":765952,"data_stored":301385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182432,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":778240,"data_stored":301684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":733184,"data_stored":300371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":802816,"data_stored":302008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1355776,"data_stored":891942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:53:52.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:53:52.939 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:53.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:52 smithi104 ceph-mon[32438]: Deploying daemon alertmanager.a on smithi081 2024-09-21T18:53:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:52 smithi104 ceph-mon[32438]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T18:53:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:52 smithi104 ceph-mon[32438]: from='client.24670 v1:172.21.15.81:0/322886019' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:53:53.061 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:52 smithi081 ceph-mon[39192]: Deploying daemon alertmanager.a on smithi081 2024-09-21T18:53:53.062 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:52 smithi081 ceph-mon[39192]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T18:53:53.062 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:52 smithi081 ceph-mon[39192]: from='client.24670 v1:172.21.15.81:0/322886019' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:53:53.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:52 smithi081 ceph-mon[29384]: Deploying daemon alertmanager.a on smithi081 2024-09-21T18:53:53.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:52 smithi081 ceph-mon[29384]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T18:53:53.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:52 smithi081 ceph-mon[29384]: from='client.24670 v1:172.21.15.81:0/322886019' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:53:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:53.799 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[29384]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[29384]: from='client.14787 v1:172.21.15.81:0/1253529373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[39192]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[39192]: from='client.14787 v1:172.21.15.81:0/1253529373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:53:54.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:54 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:53:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:54 smithi104 ceph-mon[32438]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:54 smithi104 ceph-mon[32438]: from='client.14787 v1:172.21.15.81:0/1253529373' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:53:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:54 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:53:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:54 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:53:55.281 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-21T18:53:53.371953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":887,"num_read_kb":744,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240540,"kb_used_data":7676,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720676,"statfs":{"total":767960285184,"available":767713972224,"internally_reserved":0,"allocated":7860224,"data_stored":4179393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":238276148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003079"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804992+0000","last_change":"2024-09-21T18:53:07.405054+0000","last_active":"2024-09-21T18:53:39.804992+0000","last_peered":"2024-09-21T18:53:39.804992+0000","last_clean":"2024-09-21T18:53:39.804992+0000","last_became_active":"2024-09-21T18:53:07.404774+0000","last_became_peered":"2024-09-21T18:53:07.404774+0000","last_unstale":"2024-09-21T18:53:39.804992+0000","last_undegraded":"2024-09-21T18:53:39.804992+0000","last_fullsized":"2024-09-21T18:53:39.804992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.853378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365761+0000","last_change":"2024-09-21T18:53:09.412508+0000","last_active":"2024-09-21T18:53:39.365761+0000","last_peered":"2024-09-21T18:53:39.365761+0000","last_clean":"2024-09-21T18:53:39.365761+0000","last_became_active":"2024-09-21T18:53:09.412193+0000","last_became_peered":"2024-09-21T18:53:09.412193+0000","last_unstale":"2024-09-21T18:53:39.365761+0000","last_undegraded":"2024-09-21T18:53:39.365761+0000","last_fullsized":"2024-09-21T18:53:39.365761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401891+0000","last_change":"2024-09-21T18:53:11.410506+0000","last_active":"2024-09-21T18:53:39.401891+0000","last_peered":"2024-09-21T18:53:39.401891+0000","last_clean":"2024-09-21T18:53:39.401891+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:53:39.401891+0000","last_undegraded":"2024-09-21T18:53:39.401891+0000","last_fullsized":"2024-09-21T18:53:39.401891+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.829493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365897+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:53:39.365897+0000","last_peered":"2024-09-21T18:53:39.365897+0000","last_clean":"2024-09-21T18:53:39.365897+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:53:39.365897+0000","last_undegraded":"2024-09-21T18:53:39.365897+0000","last_fullsized":"2024-09-21T18:53:39.365897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366163+0000","last_change":"2024-09-21T18:53:13.658931+0000","last_active":"2024-09-21T18:53:39.366163+0000","last_peered":"2024-09-21T18:53:39.366163+0000","last_clean":"2024-09-21T18:53:39.366163+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:53:39.366163+0000","last_undegraded":"2024-09-21T18:53:39.366163+0000","last_fullsized":"2024-09-21T18:53:39.366163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.480075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366676+0000","last_change":"2024-09-21T18:53:07.390902+0000","last_active":"2024-09-21T18:53:39.366676+0000","last_peered":"2024-09-21T18:53:39.366676+0000","last_clean":"2024-09-21T18:53:39.366676+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:53:39.366676+0000","last_undegraded":"2024-09-21T18:53:39.366676+0000","last_fullsized":"2024-09-21T18:53:39.366676+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.326332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":65,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365892+0000","last_change":"2024-09-21T18:53:09.407061+0000","last_active":"2024-09-21T18:53:39.365892+0000","last_peered":"2024-09-21T18:53:39.365892+0000","last_clean":"2024-09-21T18:53:39.365892+0000","last_became_active":"2024-09-21T18:53:09.406774+0000","last_became_peered":"2024-09-21T18:53:09.406774+0000","last_unstale":"2024-09-21T18:53:39.365892+0000","last_undegraded":"2024-09-21T18:53:39.365892+0000","last_fullsized":"2024-09-21T18:53:39.365892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.239535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365756+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:53:39.365756+0000","last_peered":"2024-09-21T18:53:39.365756+0000","last_clean":"2024-09-21T18:53:39.365756+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:53:39.365756+0000","last_undegraded":"2024-09-21T18:53:39.365756+0000","last_fullsized":"2024-09-21T18:53:39.365756+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.141973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659244+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:53:39.659244+0000","last_peered":"2024-09-21T18:53:39.659244+0000","last_clean":"2024-09-21T18:53:39.659244+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:53:39.659244+0000","last_undegraded":"2024-09-21T18:53:39.659244+0000","last_fullsized":"2024-09-21T18:53:39.659244+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.801164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366078+0000","last_change":"2024-09-21T18:53:09.401868+0000","last_active":"2024-09-21T18:53:39.366078+0000","last_peered":"2024-09-21T18:53:39.366078+0000","last_clean":"2024-09-21T18:53:39.366078+0000","last_became_active":"2024-09-21T18:53:09.401547+0000","last_became_peered":"2024-09-21T18:53:09.401547+0000","last_unstale":"2024-09-21T18:53:39.366078+0000","last_undegraded":"2024-09-21T18:53:39.366078+0000","last_fullsized":"2024-09-21T18:53:39.366078+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.067009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659510+0000","last_change":"2024-09-21T18:53:11.409118+0000","last_active":"2024-09-21T18:53:39.659510+0000","last_peered":"2024-09-21T18:53:39.659510+0000","last_clean":"2024-09-21T18:53:39.659510+0000","last_became_active":"2024-09-21T18:53:11.408537+0000","last_became_peered":"2024-09-21T18:53:11.408537+0000","last_unstale":"2024-09-21T18:53:39.659510+0000","last_undegraded":"2024-09-21T18:53:39.659510+0000","last_fullsized":"2024-09-21T18:53:39.659510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.248524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805282+0000","last_change":"2024-09-21T18:53:13.429374+0000","last_active":"2024-09-21T18:53:39.805282+0000","last_peered":"2024-09-21T18:53:39.805282+0000","last_clean":"2024-09-21T18:53:39.805282+0000","last_became_active":"2024-09-21T18:53:13.429007+0000","last_became_peered":"2024-09-21T18:53:13.429007+0000","last_unstale":"2024-09-21T18:53:39.805282+0000","last_undegraded":"2024-09-21T18:53:39.805282+0000","last_fullsized":"2024-09-21T18:53:39.805282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.767711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659156+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:53:39.659156+0000","last_peered":"2024-09-21T18:53:39.659156+0000","last_clean":"2024-09-21T18:53:39.659156+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:53:39.659156+0000","last_undegraded":"2024-09-21T18:53:39.659156+0000","last_fullsized":"2024-09-21T18:53:39.659156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365939+0000","last_change":"2024-09-21T18:53:09.406568+0000","last_active":"2024-09-21T18:53:39.365939+0000","last_peered":"2024-09-21T18:53:39.365939+0000","last_clean":"2024-09-21T18:53:39.365939+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:53:39.365939+0000","last_undegraded":"2024-09-21T18:53:39.365939+0000","last_fullsized":"2024-09-21T18:53:39.365939+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805673+0000","last_change":"2024-09-21T18:53:11.412969+0000","last_active":"2024-09-21T18:53:39.805673+0000","last_peered":"2024-09-21T18:53:39.805673+0000","last_clean":"2024-09-21T18:53:39.805673+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:53:39.805673+0000","last_undegraded":"2024-09-21T18:53:39.805673+0000","last_fullsized":"2024-09-21T18:53:39.805673+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.874706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659386+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:53:39.659386+0000","last_peered":"2024-09-21T18:53:39.659386+0000","last_clean":"2024-09-21T18:53:39.659386+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:53:39.659386+0000","last_undegraded":"2024-09-21T18:53:39.659386+0000","last_fullsized":"2024-09-21T18:53:39.659386+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.945242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366572+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:53:39.366572+0000","last_peered":"2024-09-21T18:53:39.366572+0000","last_clean":"2024-09-21T18:53:39.366572+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:53:39.366572+0000","last_undegraded":"2024-09-21T18:53:39.366572+0000","last_fullsized":"2024-09-21T18:53:39.366572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.198560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805616+0000","last_change":"2024-09-21T18:53:07.405471+0000","last_active":"2024-09-21T18:53:39.805616+0000","last_peered":"2024-09-21T18:53:39.805616+0000","last_clean":"2024-09-21T18:53:39.805616+0000","last_became_active":"2024-09-21T18:53:07.405177+0000","last_became_peered":"2024-09-21T18:53:07.405177+0000","last_unstale":"2024-09-21T18:53:39.805616+0000","last_undegraded":"2024-09-21T18:53:39.805616+0000","last_fullsized":"2024-09-21T18:53:39.805616+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.483473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366610+0000","last_change":"2024-09-21T18:53:09.407164+0000","last_active":"2024-09-21T18:53:39.366610+0000","last_peered":"2024-09-21T18:53:39.366610+0000","last_clean":"2024-09-21T18:53:39.366610+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:53:39.366610+0000","last_undegraded":"2024-09-21T18:53:39.366610+0000","last_fullsized":"2024-09-21T18:53:39.366610+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.948438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402526+0000","last_change":"2024-09-21T18:53:11.411989+0000","last_active":"2024-09-21T18:53:39.402526+0000","last_peered":"2024-09-21T18:53:39.402526+0000","last_clean":"2024-09-21T18:53:39.402526+0000","last_became_active":"2024-09-21T18:53:11.411624+0000","last_became_peered":"2024-09-21T18:53:11.411624+0000","last_unstale":"2024-09-21T18:53:39.402526+0000","last_undegraded":"2024-09-21T18:53:39.402526+0000","last_fullsized":"2024-09-21T18:53:39.402526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.979898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366773+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:53:39.366773+0000","last_peered":"2024-09-21T18:53:39.366773+0000","last_clean":"2024-09-21T18:53:39.366773+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:53:39.366773+0000","last_undegraded":"2024-09-21T18:53:39.366773+0000","last_fullsized":"2024-09-21T18:53:39.366773+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.301610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366169+0000","last_change":"2024-09-21T18:53:07.397249+0000","last_active":"2024-09-21T18:53:39.366169+0000","last_peered":"2024-09-21T18:53:39.366169+0000","last_clean":"2024-09-21T18:53:39.366169+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:53:39.366169+0000","last_undegraded":"2024-09-21T18:53:39.366169+0000","last_fullsized":"2024-09-21T18:53:39.366169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.232024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":63,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366451+0000","last_change":"2024-09-21T18:53:09.406892+0000","last_active":"2024-09-21T18:53:39.366451+0000","last_peered":"2024-09-21T18:53:39.366451+0000","last_clean":"2024-09-21T18:53:39.366451+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:53:39.366451+0000","last_undegraded":"2024-09-21T18:53:39.366451+0000","last_fullsized":"2024-09-21T18:53:39.366451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.459852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366085+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:53:39.366085+0000","last_peered":"2024-09-21T18:53:39.366085+0000","last_clean":"2024-09-21T18:53:39.366085+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:53:39.366085+0000","last_undegraded":"2024-09-21T18:53:39.366085+0000","last_fullsized":"2024-09-21T18:53:39.366085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.060686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806034+0000","last_change":"2024-09-21T18:53:13.426725+0000","last_active":"2024-09-21T18:53:39.806034+0000","last_peered":"2024-09-21T18:53:39.806034+0000","last_clean":"2024-09-21T18:53:39.806034+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:53:39.806034+0000","last_undegraded":"2024-09-21T18:53:39.806034+0000","last_fullsized":"2024-09-21T18:53:39.806034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401685+0000","last_change":"2024-09-21T18:53:07.397345+0000","last_active":"2024-09-21T18:53:39.401685+0000","last_peered":"2024-09-21T18:53:39.401685+0000","last_clean":"2024-09-21T18:53:39.401685+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:53:39.401685+0000","last_undegraded":"2024-09-21T18:53:39.401685+0000","last_fullsized":"2024-09-21T18:53:39.401685+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.808378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805790+0000","last_change":"2024-09-21T18:53:09.719399+0000","last_active":"2024-09-21T18:53:39.805790+0000","last_peered":"2024-09-21T18:53:39.805790+0000","last_clean":"2024-09-21T18:53:39.805790+0000","last_became_active":"2024-09-21T18:53:09.719059+0000","last_became_peered":"2024-09-21T18:53:09.719059+0000","last_unstale":"2024-09-21T18:53:39.805790+0000","last_undegraded":"2024-09-21T18:53:39.805790+0000","last_fullsized":"2024-09-21T18:53:39.805790+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.854862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455319+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:53:42.455319+0000","last_peered":"2024-09-21T18:53:42.455319+0000","last_clean":"2024-09-21T18:53:42.455319+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:53:42.455319+0000","last_undegraded":"2024-09-21T18:53:42.455319+0000","last_fullsized":"2024-09-21T18:53:42.455319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.329971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402686+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:53:39.402686+0000","last_peered":"2024-09-21T18:53:39.402686+0000","last_clean":"2024-09-21T18:53:39.402686+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:53:39.402686+0000","last_undegraded":"2024-09-21T18:53:39.402686+0000","last_fullsized":"2024-09-21T18:53:39.402686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.373058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365495+0000","last_change":"2024-09-21T18:53:07.398833+0000","last_active":"2024-09-21T18:53:39.365495+0000","last_peered":"2024-09-21T18:53:39.365495+0000","last_clean":"2024-09-21T18:53:39.365495+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:53:39.365495+0000","last_undegraded":"2024-09-21T18:53:39.365495+0000","last_fullsized":"2024-09-21T18:53:39.365495+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455246+0000","last_change":"2024-09-21T18:53:11.411518+0000","last_active":"2024-09-21T18:53:42.455246+0000","last_peered":"2024-09-21T18:53:42.455246+0000","last_clean":"2024-09-21T18:53:42.455246+0000","last_became_active":"2024-09-21T18:53:11.411212+0000","last_became_peered":"2024-09-21T18:53:11.411212+0000","last_unstale":"2024-09-21T18:53:42.455246+0000","last_undegraded":"2024-09-21T18:53:42.455246+0000","last_fullsized":"2024-09-21T18:53:42.455246+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.137419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659601+0000","last_change":"2024-09-21T18:53:13.433767+0000","last_active":"2024-09-21T18:53:39.659601+0000","last_peered":"2024-09-21T18:53:39.659601+0000","last_clean":"2024-09-21T18:53:39.659601+0000","last_became_active":"2024-09-21T18:53:13.433575+0000","last_became_peered":"2024-09-21T18:53:13.433575+0000","last_unstale":"2024-09-21T18:53:39.659601+0000","last_undegraded":"2024-09-21T18:53:39.659601+0000","last_fullsized":"2024-09-21T18:53:39.659601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.484363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":72,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365369+0000","last_change":"2024-09-21T18:53:09.719338+0000","last_active":"2024-09-21T18:53:39.365369+0000","last_peered":"2024-09-21T18:53:39.365369+0000","last_clean":"2024-09-21T18:53:39.365369+0000","last_became_active":"2024-09-21T18:53:09.719017+0000","last_became_peered":"2024-09-21T18:53:09.719017+0000","last_unstale":"2024-09-21T18:53:39.365369+0000","last_undegraded":"2024-09-21T18:53:39.365369+0000","last_fullsized":"2024-09-21T18:53:39.365369+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.426276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366986+0000","last_change":"2024-09-21T18:53:07.400418+0000","last_active":"2024-09-21T18:53:39.366986+0000","last_peered":"2024-09-21T18:53:39.366986+0000","last_clean":"2024-09-21T18:53:39.366986+0000","last_became_active":"2024-09-21T18:53:07.400143+0000","last_became_peered":"2024-09-21T18:53:07.400143+0000","last_unstale":"2024-09-21T18:53:39.366986+0000","last_undegraded":"2024-09-21T18:53:39.366986+0000","last_fullsized":"2024-09-21T18:53:39.366986+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.737589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365822+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:53:39.365822+0000","last_peered":"2024-09-21T18:53:39.365822+0000","last_clean":"2024-09-21T18:53:39.365822+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:53:39.365822+0000","last_undegraded":"2024-09-21T18:53:39.365822+0000","last_fullsized":"2024-09-21T18:53:39.365822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.956086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366022+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:53:39.366022+0000","last_peered":"2024-09-21T18:53:39.366022+0000","last_clean":"2024-09-21T18:53:39.366022+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:53:39.366022+0000","last_undegraded":"2024-09-21T18:53:39.366022+0000","last_fullsized":"2024-09-21T18:53:39.366022+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.926924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":60,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365872+0000","last_change":"2024-09-21T18:53:09.400587+0000","last_active":"2024-09-21T18:53:39.365872+0000","last_peered":"2024-09-21T18:53:39.365872+0000","last_clean":"2024-09-21T18:53:39.365872+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:53:39.365872+0000","last_undegraded":"2024-09-21T18:53:39.365872+0000","last_fullsized":"2024-09-21T18:53:39.365872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.173359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804973+0000","last_change":"2024-09-21T18:53:07.404382+0000","last_active":"2024-09-21T18:53:39.804973+0000","last_peered":"2024-09-21T18:53:39.804973+0000","last_clean":"2024-09-21T18:53:39.804973+0000","last_became_active":"2024-09-21T18:53:07.404018+0000","last_became_peered":"2024-09-21T18:53:07.404018+0000","last_unstale":"2024-09-21T18:53:39.804973+0000","last_undegraded":"2024-09-21T18:53:39.804973+0000","last_fullsized":"2024-09-21T18:53:39.804973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.245821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":53,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.454988+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:53:42.454988+0000","last_peered":"2024-09-21T18:53:42.454988+0000","last_clean":"2024-09-21T18:53:42.454988+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:53:42.454988+0000","last_undegraded":"2024-09-21T18:53:42.454988+0000","last_fullsized":"2024-09-21T18:53:42.454988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.339632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805949+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:53:39.805949+0000","last_peered":"2024-09-21T18:53:39.805949+0000","last_clean":"2024-09-21T18:53:39.805949+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:53:39.805949+0000","last_undegraded":"2024-09-21T18:53:39.805949+0000","last_fullsized":"2024-09-21T18:53:39.805949+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.973293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402122+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:53:39.402122+0000","last_peered":"2024-09-21T18:53:39.402122+0000","last_clean":"2024-09-21T18:53:39.402122+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:53:39.402122+0000","last_undegraded":"2024-09-21T18:53:39.402122+0000","last_fullsized":"2024-09-21T18:53:39.402122+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.374498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659123+0000","last_change":"2024-09-21T18:53:07.403537+0000","last_active":"2024-09-21T18:53:39.659123+0000","last_peered":"2024-09-21T18:53:39.659123+0000","last_clean":"2024-09-21T18:53:39.659123+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:53:39.659123+0000","last_undegraded":"2024-09-21T18:53:39.659123+0000","last_fullsized":"2024-09-21T18:53:39.659123+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.558691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455006+0000","last_change":"2024-09-21T18:53:11.407531+0000","last_active":"2024-09-21T18:53:42.455006+0000","last_peered":"2024-09-21T18:53:42.455006+0000","last_clean":"2024-09-21T18:53:42.455006+0000","last_became_active":"2024-09-21T18:53:11.407244+0000","last_became_peered":"2024-09-21T18:53:11.407244+0000","last_unstale":"2024-09-21T18:53:42.455006+0000","last_undegraded":"2024-09-21T18:53:42.455006+0000","last_fullsized":"2024-09-21T18:53:42.455006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.763939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805349+0000","last_change":"2024-09-21T18:53:13.435210+0000","last_active":"2024-09-21T18:53:39.805349+0000","last_peered":"2024-09-21T18:53:39.805349+0000","last_clean":"2024-09-21T18:53:39.805349+0000","last_became_active":"2024-09-21T18:53:13.434753+0000","last_became_peered":"2024-09-21T18:53:13.434753+0000","last_unstale":"2024-09-21T18:53:39.805349+0000","last_undegraded":"2024-09-21T18:53:39.805349+0000","last_fullsized":"2024-09-21T18:53:39.805349+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.794214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":75,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805838+0000","last_change":"2024-09-21T18:53:09.719454+0000","last_active":"2024-09-21T18:53:39.805838+0000","last_peered":"2024-09-21T18:53:39.805838+0000","last_clean":"2024-09-21T18:53:39.805838+0000","last_became_active":"2024-09-21T18:53:09.719114+0000","last_became_peered":"2024-09-21T18:53:09.719114+0000","last_unstale":"2024-09-21T18:53:39.805838+0000","last_undegraded":"2024-09-21T18:53:39.805838+0000","last_fullsized":"2024-09-21T18:53:39.805838+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.176379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401702+0000","last_change":"2024-09-21T18:53:07.400542+0000","last_active":"2024-09-21T18:53:39.401702+0000","last_peered":"2024-09-21T18:53:39.401702+0000","last_clean":"2024-09-21T18:53:39.401702+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:53:39.401702+0000","last_undegraded":"2024-09-21T18:53:39.401702+0000","last_fullsized":"2024-09-21T18:53:39.401702+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.007870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805118+0000","last_change":"2024-09-21T18:53:11.404992+0000","last_active":"2024-09-21T18:53:39.805118+0000","last_peered":"2024-09-21T18:53:39.805118+0000","last_clean":"2024-09-21T18:53:39.805118+0000","last_became_active":"2024-09-21T18:53:11.404577+0000","last_became_peered":"2024-09-21T18:53:11.404577+0000","last_unstale":"2024-09-21T18:53:39.805118+0000","last_undegraded":"2024-09-21T18:53:39.805118+0000","last_fullsized":"2024-09-21T18:53:39.805118+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.773972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401855+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:53:39.401855+0000","last_peered":"2024-09-21T18:53:39.401855+0000","last_clean":"2024-09-21T18:53:39.401855+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:53:39.401855+0000","last_undegraded":"2024-09-21T18:53:39.401855+0000","last_fullsized":"2024-09-21T18:53:39.401855+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.173798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":96,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455020+0000","last_change":"2024-09-21T18:53:09.410172+0000","last_active":"2024-09-21T18:53:42.455020+0000","last_peered":"2024-09-21T18:53:42.455020+0000","last_clean":"2024-09-21T18:53:42.455020+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:53:42.455020+0000","last_undegraded":"2024-09-21T18:53:42.455020+0000","last_fullsized":"2024-09-21T18:53:42.455020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.020223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366204+0000","last_change":"2024-09-21T18:53:07.403315+0000","last_active":"2024-09-21T18:53:39.366204+0000","last_peered":"2024-09-21T18:53:39.366204+0000","last_clean":"2024-09-21T18:53:39.366204+0000","last_became_active":"2024-09-21T18:53:07.403002+0000","last_became_peered":"2024-09-21T18:53:07.403002+0000","last_unstale":"2024-09-21T18:53:39.366204+0000","last_undegraded":"2024-09-21T18:53:39.366204+0000","last_fullsized":"2024-09-21T18:53:39.366204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:08.340495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":53,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659620+0000","last_change":"2024-09-21T18:53:09.398954+0000","last_active":"2024-09-21T18:53:39.659620+0000","last_peered":"2024-09-21T18:53:39.659620+0000","last_clean":"2024-09-21T18:53:39.659620+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:53:39.659620+0000","last_undegraded":"2024-09-21T18:53:39.659620+0000","last_fullsized":"2024-09-21T18:53:39.659620+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.467230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805634+0000","last_change":"2024-09-21T18:53:11.411504+0000","last_active":"2024-09-21T18:53:39.805634+0000","last_peered":"2024-09-21T18:53:39.805634+0000","last_clean":"2024-09-21T18:53:39.805634+0000","last_became_active":"2024-09-21T18:53:11.411229+0000","last_became_peered":"2024-09-21T18:53:11.411229+0000","last_unstale":"2024-09-21T18:53:39.805634+0000","last_undegraded":"2024-09-21T18:53:39.805634+0000","last_fullsized":"2024-09-21T18:53:39.805634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.568499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366452+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:53:39.366452+0000","last_peered":"2024-09-21T18:53:39.366452+0000","last_clean":"2024-09-21T18:53:39.366452+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:53:39.366452+0000","last_undegraded":"2024-09-21T18:53:39.366452+0000","last_fullsized":"2024-09-21T18:53:39.366452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.888758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402711+0000","last_change":"2024-09-21T18:53:09.412601+0000","last_active":"2024-09-21T18:53:39.402711+0000","last_peered":"2024-09-21T18:53:39.402711+0000","last_clean":"2024-09-21T18:53:39.402711+0000","last_became_active":"2024-09-21T18:53:09.412304+0000","last_became_peered":"2024-09-21T18:53:09.412304+0000","last_unstale":"2024-09-21T18:53:39.402711+0000","last_undegraded":"2024-09-21T18:53:39.402711+0000","last_fullsized":"2024-09-21T18:53:39.402711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.434125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.401670+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:53:39.401670+0000","last_peered":"2024-09-21T18:53:39.401670+0000","last_clean":"2024-09-21T18:53:39.401670+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:53:39.401670+0000","last_undegraded":"2024-09-21T18:53:39.401670+0000","last_fullsized":"2024-09-21T18:53:39.401670+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.025870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365720+0000","last_change":"2024-09-21T18:53:09.400372+0000","last_active":"2024-09-21T18:53:39.365720+0000","last_peered":"2024-09-21T18:53:39.365720+0000","last_clean":"2024-09-21T18:53:39.365720+0000","last_became_active":"2024-09-21T18:53:06.391866+0000","last_became_peered":"2024-09-21T18:53:06.391866+0000","last_unstale":"2024-09-21T18:53:39.365720+0000","last_undegraded":"2024-09-21T18:53:39.365720+0000","last_fullsized":"2024-09-21T18:53:39.365720+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366142+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:53:39.366142+0000","last_peered":"2024-09-21T18:53:39.366142+0000","last_clean":"2024-09-21T18:53:39.366142+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:53:39.366142+0000","last_undegraded":"2024-09-21T18:53:39.366142+0000","last_fullsized":"2024-09-21T18:53:39.366142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.265652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806109+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:53:39.806109+0000","last_peered":"2024-09-21T18:53:39.806109+0000","last_clean":"2024-09-21T18:53:39.806109+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:53:39.806109+0000","last_undegraded":"2024-09-21T18:53:39.806109+0000","last_fullsized":"2024-09-21T18:53:39.806109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805805+0000","last_change":"2024-09-21T18:53:09.407641+0000","last_active":"2024-09-21T18:53:39.805805+0000","last_peered":"2024-09-21T18:53:39.805805+0000","last_clean":"2024-09-21T18:53:39.805805+0000","last_became_active":"2024-09-21T18:53:09.407302+0000","last_became_peered":"2024-09-21T18:53:09.407302+0000","last_unstale":"2024-09-21T18:53:39.805805+0000","last_undegraded":"2024-09-21T18:53:39.805805+0000","last_fullsized":"2024-09-21T18:53:39.805805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.070171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804934+0000","last_change":"2024-09-21T18:53:07.404328+0000","last_active":"2024-09-21T18:53:39.804934+0000","last_peered":"2024-09-21T18:53:39.804934+0000","last_clean":"2024-09-21T18:53:39.804934+0000","last_became_active":"2024-09-21T18:53:07.404060+0000","last_became_peered":"2024-09-21T18:53:07.404060+0000","last_unstale":"2024-09-21T18:53:39.804934+0000","last_undegraded":"2024-09-21T18:53:39.804934+0000","last_fullsized":"2024-09-21T18:53:39.804934+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.843533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":92,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:49.566582+0000","last_change":"2024-09-21T18:53:09.719413+0000","last_active":"2024-09-21T18:53:49.566582+0000","last_peered":"2024-09-21T18:53:49.566582+0000","last_clean":"2024-09-21T18:53:49.566582+0000","last_became_active":"2024-09-21T18:53:06.394373+0000","last_became_peered":"2024-09-21T18:53:06.394373+0000","last_unstale":"2024-09-21T18:53:49.566582+0000","last_undegraded":"2024-09-21T18:53:49.566582+0000","last_fullsized":"2024-09-21T18:53:49.566582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.196010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659926+0000","last_change":"2024-09-21T18:53:11.409878+0000","last_active":"2024-09-21T18:53:39.659926+0000","last_peered":"2024-09-21T18:53:39.659926+0000","last_clean":"2024-09-21T18:53:39.659926+0000","last_became_active":"2024-09-21T18:53:11.409637+0000","last_became_peered":"2024-09-21T18:53:11.409637+0000","last_unstale":"2024-09-21T18:53:39.659926+0000","last_undegraded":"2024-09-21T18:53:39.659926+0000","last_fullsized":"2024-09-21T18:53:39.659926+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.893970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402098+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:53:39.402098+0000","last_peered":"2024-09-21T18:53:39.402098+0000","last_clean":"2024-09-21T18:53:39.402098+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:53:39.402098+0000","last_undegraded":"2024-09-21T18:53:39.402098+0000","last_fullsized":"2024-09-21T18:53:39.402098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.377231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455236+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:53:42.455236+0000","last_peered":"2024-09-21T18:53:42.455236+0000","last_clean":"2024-09-21T18:53:42.455236+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:53:42.455236+0000","last_undegraded":"2024-09-21T18:53:42.455236+0000","last_fullsized":"2024-09-21T18:53:42.455236+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.645863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366238+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:53:39.366238+0000","last_peered":"2024-09-21T18:53:39.366238+0000","last_clean":"2024-09-21T18:53:39.366238+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:53:39.366238+0000","last_undegraded":"2024-09-21T18:53:39.366238+0000","last_fullsized":"2024-09-21T18:53:39.366238+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.707207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":87,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806474+0000","last_change":"2024-09-21T18:53:29.766304+0000","last_active":"2024-09-21T18:53:39.806474+0000","last_peered":"2024-09-21T18:53:39.806474+0000","last_clean":"2024-09-21T18:53:39.806474+0000","last_became_active":"2024-09-21T18:53:00.358105+0000","last_became_peered":"2024-09-21T18:53:00.358105+0000","last_unstale":"2024-09-21T18:53:39.806474+0000","last_undegraded":"2024-09-21T18:53:39.806474+0000","last_fullsized":"2024-09-21T18:53:39.806474+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:50.998311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806259+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:53:39.806259+0000","last_peered":"2024-09-21T18:53:39.806259+0000","last_clean":"2024-09-21T18:53:39.806259+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:53:39.806259+0000","last_undegraded":"2024-09-21T18:53:39.806259+0000","last_fullsized":"2024-09-21T18:53:39.806259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.081383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659874+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:53:39.659874+0000","last_peered":"2024-09-21T18:53:39.659874+0000","last_clean":"2024-09-21T18:53:39.659874+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:53:39.659874+0000","last_undegraded":"2024-09-21T18:53:39.659874+0000","last_fullsized":"2024-09-21T18:53:39.659874+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.641099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366660+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:53:39.366660+0000","last_peered":"2024-09-21T18:53:39.366660+0000","last_clean":"2024-09-21T18:53:39.366660+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:53:39.366660+0000","last_undegraded":"2024-09-21T18:53:39.366660+0000","last_fullsized":"2024-09-21T18:53:39.366660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366237+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:53:39.366237+0000","last_peered":"2024-09-21T18:53:39.366237+0000","last_clean":"2024-09-21T18:53:39.366237+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:53:39.366237+0000","last_undegraded":"2024-09-21T18:53:39.366237+0000","last_fullsized":"2024-09-21T18:53:39.366237+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.491463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659836+0000","last_change":"2024-09-21T18:53:11.409004+0000","last_active":"2024-09-21T18:53:39.659836+0000","last_peered":"2024-09-21T18:53:39.659836+0000","last_clean":"2024-09-21T18:53:39.659836+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:53:39.659836+0000","last_undegraded":"2024-09-21T18:53:39.659836+0000","last_fullsized":"2024-09-21T18:53:39.659836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.557609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366142+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:53:39.366142+0000","last_peered":"2024-09-21T18:53:39.366142+0000","last_clean":"2024-09-21T18:53:39.366142+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:53:39.366142+0000","last_undegraded":"2024-09-21T18:53:39.366142+0000","last_fullsized":"2024-09-21T18:53:39.366142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.277061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":67,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366456+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:53:39.366456+0000","last_peered":"2024-09-21T18:53:39.366456+0000","last_clean":"2024-09-21T18:53:39.366456+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:53:39.366456+0000","last_undegraded":"2024-09-21T18:53:39.366456+0000","last_fullsized":"2024-09-21T18:53:39.366456+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.263644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804679+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:53:39.804679+0000","last_peered":"2024-09-21T18:53:39.804679+0000","last_clean":"2024-09-21T18:53:39.804679+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:53:39.804679+0000","last_undegraded":"2024-09-21T18:53:39.804679+0000","last_fullsized":"2024-09-21T18:53:39.804679+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402286+0000","last_change":"2024-09-21T18:53:11.409557+0000","last_active":"2024-09-21T18:53:39.402286+0000","last_peered":"2024-09-21T18:53:39.402286+0000","last_clean":"2024-09-21T18:53:39.402286+0000","last_became_active":"2024-09-21T18:53:11.409200+0000","last_became_peered":"2024-09-21T18:53:11.409200+0000","last_unstale":"2024-09-21T18:53:39.402286+0000","last_undegraded":"2024-09-21T18:53:39.402286+0000","last_fullsized":"2024-09-21T18:53:39.402286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:37.475931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366132+0000","last_change":"2024-09-21T18:53:13.430863+0000","last_active":"2024-09-21T18:53:39.366132+0000","last_peered":"2024-09-21T18:53:39.366132+0000","last_clean":"2024-09-21T18:53:39.366132+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:53:39.366132+0000","last_undegraded":"2024-09-21T18:53:39.366132+0000","last_fullsized":"2024-09-21T18:53:39.366132+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:38.403912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366137+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:53:39.366137+0000","last_peered":"2024-09-21T18:53:39.366137+0000","last_clean":"2024-09-21T18:53:39.366137+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:53:39.366137+0000","last_undegraded":"2024-09-21T18:53:39.366137+0000","last_fullsized":"2024-09-21T18:53:39.366137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.582865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":50,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366588+0000","last_change":"2024-09-21T18:53:07.400595+0000","last_active":"2024-09-21T18:53:39.366588+0000","last_peered":"2024-09-21T18:53:39.366588+0000","last_clean":"2024-09-21T18:53:39.366588+0000","last_became_active":"2024-09-21T18:53:07.399708+0000","last_became_peered":"2024-09-21T18:53:07.399708+0000","last_unstale":"2024-09-21T18:53:39.366588+0000","last_undegraded":"2024-09-21T18:53:39.366588+0000","last_fullsized":"2024-09-21T18:53:39.366588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:07.859688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.454988+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:53:42.454988+0000","last_peered":"2024-09-21T18:53:42.454988+0000","last_clean":"2024-09-21T18:53:42.454988+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:53:42.454988+0000","last_undegraded":"2024-09-21T18:53:42.454988+0000","last_fullsized":"2024-09-21T18:53:42.454988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.955877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366444+0000","last_change":"2024-09-21T18:53:13.428208+0000","last_active":"2024-09-21T18:53:39.366444+0000","last_peered":"2024-09-21T18:53:39.366444+0000","last_clean":"2024-09-21T18:53:39.366444+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:53:39.366444+0000","last_undegraded":"2024-09-21T18:53:39.366444+0000","last_fullsized":"2024-09-21T18:53:39.366444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.939232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805782+0000","last_change":"2024-09-21T18:53:09.408663+0000","last_active":"2024-09-21T18:53:39.805782+0000","last_peered":"2024-09-21T18:53:39.805782+0000","last_clean":"2024-09-21T18:53:39.805782+0000","last_became_active":"2024-09-21T18:53:09.408315+0000","last_became_peered":"2024-09-21T18:53:09.408315+0000","last_unstale":"2024-09-21T18:53:39.805782+0000","last_undegraded":"2024-09-21T18:53:39.805782+0000","last_fullsized":"2024-09-21T18:53:39.805782+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.191003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659103+0000","last_change":"2024-09-21T18:53:07.400068+0000","last_active":"2024-09-21T18:53:39.659103+0000","last_peered":"2024-09-21T18:53:39.659103+0000","last_clean":"2024-09-21T18:53:39.659103+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:53:39.659103+0000","last_undegraded":"2024-09-21T18:53:39.659103+0000","last_fullsized":"2024-09-21T18:53:39.659103+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.160743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365773+0000","last_change":"2024-09-21T18:53:11.415649+0000","last_active":"2024-09-21T18:53:39.365773+0000","last_peered":"2024-09-21T18:53:39.365773+0000","last_clean":"2024-09-21T18:53:39.365773+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:53:39.365773+0000","last_undegraded":"2024-09-21T18:53:39.365773+0000","last_fullsized":"2024-09-21T18:53:39.365773+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.547416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805243+0000","last_change":"2024-09-21T18:53:13.434903+0000","last_active":"2024-09-21T18:53:39.805243+0000","last_peered":"2024-09-21T18:53:39.805243+0000","last_clean":"2024-09-21T18:53:39.805243+0000","last_became_active":"2024-09-21T18:53:13.434569+0000","last_became_peered":"2024-09-21T18:53:13.434569+0000","last_unstale":"2024-09-21T18:53:39.805243+0000","last_undegraded":"2024-09-21T18:53:39.805243+0000","last_fullsized":"2024-09-21T18:53:39.805243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.431252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805212+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:53:39.805212+0000","last_peered":"2024-09-21T18:53:39.805212+0000","last_clean":"2024-09-21T18:53:39.805212+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:53:39.805212+0000","last_undegraded":"2024-09-21T18:53:39.805212+0000","last_fullsized":"2024-09-21T18:53:39.805212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.041097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806556+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:53:39.806556+0000","last_peered":"2024-09-21T18:53:39.806556+0000","last_clean":"2024-09-21T18:53:39.806556+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:53:39.806556+0000","last_undegraded":"2024-09-21T18:53:39.806556+0000","last_fullsized":"2024-09-21T18:53:39.806556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.852227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365800+0000","last_change":"2024-09-21T18:53:11.413440+0000","last_active":"2024-09-21T18:53:39.365800+0000","last_peered":"2024-09-21T18:53:39.365800+0000","last_clean":"2024-09-21T18:53:39.365800+0000","last_became_active":"2024-09-21T18:53:11.413142+0000","last_became_peered":"2024-09-21T18:53:11.413142+0000","last_unstale":"2024-09-21T18:53:39.365800+0000","last_undegraded":"2024-09-21T18:53:39.365800+0000","last_fullsized":"2024-09-21T18:53:39.365800+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.886861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806391+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:53:39.806391+0000","last_peered":"2024-09-21T18:53:39.806391+0000","last_clean":"2024-09-21T18:53:39.806391+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:53:39.806391+0000","last_undegraded":"2024-09-21T18:53:39.806391+0000","last_fullsized":"2024-09-21T18:53:39.806391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.894993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":62,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366120+0000","last_change":"2024-09-21T18:53:09.406428+0000","last_active":"2024-09-21T18:53:39.366120+0000","last_peered":"2024-09-21T18:53:39.366120+0000","last_clean":"2024-09-21T18:53:39.366120+0000","last_became_active":"2024-09-21T18:53:09.406030+0000","last_became_peered":"2024-09-21T18:53:09.406030+0000","last_unstale":"2024-09-21T18:53:39.366120+0000","last_undegraded":"2024-09-21T18:53:39.366120+0000","last_fullsized":"2024-09-21T18:53:39.366120+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.045459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804870+0000","last_change":"2024-09-21T18:53:07.400289+0000","last_active":"2024-09-21T18:53:39.804870+0000","last_peered":"2024-09-21T18:53:39.804870+0000","last_clean":"2024-09-21T18:53:39.804870+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:53:39.804870+0000","last_undegraded":"2024-09-21T18:53:39.804870+0000","last_fullsized":"2024-09-21T18:53:39.804870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:06.969808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365529+0000","last_change":"2024-09-21T18:53:11.417570+0000","last_active":"2024-09-21T18:53:39.365529+0000","last_peered":"2024-09-21T18:53:39.365529+0000","last_clean":"2024-09-21T18:53:39.365529+0000","last_became_active":"2024-09-21T18:53:11.417277+0000","last_became_peered":"2024-09-21T18:53:11.417277+0000","last_unstale":"2024-09-21T18:53:39.365529+0000","last_undegraded":"2024-09-21T18:53:39.365529+0000","last_fullsized":"2024-09-21T18:53:39.365529+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.958604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366256+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:53:39.366256+0000","last_peered":"2024-09-21T18:53:39.366256+0000","last_clean":"2024-09-21T18:53:39.366256+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:53:39.366256+0000","last_undegraded":"2024-09-21T18:53:39.366256+0000","last_fullsized":"2024-09-21T18:53:39.366256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.706132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659282+0000","last_change":"2024-09-21T18:53:09.717860+0000","last_active":"2024-09-21T18:53:39.659282+0000","last_peered":"2024-09-21T18:53:39.659282+0000","last_clean":"2024-09-21T18:53:39.659282+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:53:39.659282+0000","last_undegraded":"2024-09-21T18:53:39.659282+0000","last_fullsized":"2024-09-21T18:53:39.659282+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.363297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804914+0000","last_change":"2024-09-21T18:53:07.405281+0000","last_active":"2024-09-21T18:53:39.804914+0000","last_peered":"2024-09-21T18:53:39.804914+0000","last_clean":"2024-09-21T18:53:39.804914+0000","last_became_active":"2024-09-21T18:53:07.404988+0000","last_became_peered":"2024-09-21T18:53:07.404988+0000","last_unstale":"2024-09-21T18:53:39.804914+0000","last_undegraded":"2024-09-21T18:53:39.804914+0000","last_fullsized":"2024-09-21T18:53:39.804914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.141596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455248+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:53:42.455248+0000","last_peered":"2024-09-21T18:53:42.455248+0000","last_clean":"2024-09-21T18:53:42.455248+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:53:42.455248+0000","last_undegraded":"2024-09-21T18:53:42.455248+0000","last_fullsized":"2024-09-21T18:53:42.455248+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.640908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659413+0000","last_change":"2024-09-21T18:53:13.657306+0000","last_active":"2024-09-21T18:53:39.659413+0000","last_peered":"2024-09-21T18:53:39.659413+0000","last_clean":"2024-09-21T18:53:39.659413+0000","last_became_active":"2024-09-21T18:53:13.656931+0000","last_became_peered":"2024-09-21T18:53:13.656931+0000","last_unstale":"2024-09-21T18:53:39.659413+0000","last_undegraded":"2024-09-21T18:53:39.659413+0000","last_fullsized":"2024-09-21T18:53:39.659413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:42.145553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805686+0000","last_change":"2024-09-21T18:53:07.397798+0000","last_active":"2024-09-21T18:53:39.805686+0000","last_peered":"2024-09-21T18:53:39.805686+0000","last_clean":"2024-09-21T18:53:39.805686+0000","last_became_active":"2024-09-21T18:53:07.397469+0000","last_became_peered":"2024-09-21T18:53:07.397469+0000","last_unstale":"2024-09-21T18:53:39.805686+0000","last_undegraded":"2024-09-21T18:53:39.805686+0000","last_fullsized":"2024-09-21T18:53:39.805686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.024888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366907+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:53:39.366907+0000","last_peered":"2024-09-21T18:53:39.366907+0000","last_clean":"2024-09-21T18:53:39.366907+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:53:39.366907+0000","last_undegraded":"2024-09-21T18:53:39.366907+0000","last_fullsized":"2024-09-21T18:53:39.366907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.137675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659807+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:53:39.659807+0000","last_peered":"2024-09-21T18:53:39.659807+0000","last_clean":"2024-09-21T18:53:39.659807+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:53:39.659807+0000","last_undegraded":"2024-09-21T18:53:39.659807+0000","last_fullsized":"2024-09-21T18:53:39.659807+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.109044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.806481+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:53:39.806481+0000","last_peered":"2024-09-21T18:53:39.806481+0000","last_clean":"2024-09-21T18:53:39.806481+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:53:39.806481+0000","last_undegraded":"2024-09-21T18:53:39.806481+0000","last_fullsized":"2024-09-21T18:53:39.806481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805862+0000","last_change":"2024-09-21T18:53:09.720374+0000","last_active":"2024-09-21T18:53:39.805862+0000","last_peered":"2024-09-21T18:53:39.805862+0000","last_clean":"2024-09-21T18:53:39.805862+0000","last_became_active":"2024-09-21T18:53:09.720056+0000","last_became_peered":"2024-09-21T18:53:09.720056+0000","last_unstale":"2024-09-21T18:53:39.805862+0000","last_undegraded":"2024-09-21T18:53:39.805862+0000","last_fullsized":"2024-09-21T18:53:39.805862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.445280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804758+0000","last_change":"2024-09-21T18:53:07.406957+0000","last_active":"2024-09-21T18:53:39.804758+0000","last_peered":"2024-09-21T18:53:39.804758+0000","last_clean":"2024-09-21T18:53:39.804758+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:53:39.804758+0000","last_undegraded":"2024-09-21T18:53:39.804758+0000","last_fullsized":"2024-09-21T18:53:39.804758+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.565944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366803+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:53:39.366803+0000","last_peered":"2024-09-21T18:53:39.366803+0000","last_clean":"2024-09-21T18:53:39.366803+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:53:39.366803+0000","last_undegraded":"2024-09-21T18:53:39.366803+0000","last_fullsized":"2024-09-21T18:53:39.366803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.484261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366750+0000","last_change":"2024-09-21T18:53:13.423168+0000","last_active":"2024-09-21T18:53:39.366750+0000","last_peered":"2024-09-21T18:53:39.366750+0000","last_clean":"2024-09-21T18:53:39.366750+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:53:39.366750+0000","last_undegraded":"2024-09-21T18:53:39.366750+0000","last_fullsized":"2024-09-21T18:53:39.366750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.401268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659787+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:53:39.659787+0000","last_peered":"2024-09-21T18:53:39.659787+0000","last_clean":"2024-09-21T18:53:39.659787+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:53:39.659787+0000","last_undegraded":"2024-09-21T18:53:39.659787+0000","last_fullsized":"2024-09-21T18:53:39.659787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.011518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804933+0000","last_change":"2024-09-21T18:53:07.405516+0000","last_active":"2024-09-21T18:53:39.804933+0000","last_peered":"2024-09-21T18:53:39.804933+0000","last_clean":"2024-09-21T18:53:39.804933+0000","last_became_active":"2024-09-21T18:53:07.405223+0000","last_became_peered":"2024-09-21T18:53:07.405223+0000","last_unstale":"2024-09-21T18:53:39.804933+0000","last_undegraded":"2024-09-21T18:53:39.804933+0000","last_fullsized":"2024-09-21T18:53:39.804933+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.984977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":53,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455298+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:53:42.455298+0000","last_peered":"2024-09-21T18:53:42.455298+0000","last_clean":"2024-09-21T18:53:42.455298+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:53:42.455298+0000","last_undegraded":"2024-09-21T18:53:42.455298+0000","last_fullsized":"2024-09-21T18:53:42.455298+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.168997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366608+0000","last_change":"2024-09-21T18:53:13.423826+0000","last_active":"2024-09-21T18:53:39.366608+0000","last_peered":"2024-09-21T18:53:39.366608+0000","last_clean":"2024-09-21T18:53:39.366608+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:53:39.366608+0000","last_undegraded":"2024-09-21T18:53:39.366608+0000","last_fullsized":"2024-09-21T18:53:39.366608+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.967749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366987+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:53:39.366987+0000","last_peered":"2024-09-21T18:53:39.366987+0000","last_clean":"2024-09-21T18:53:39.366987+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:53:39.366987+0000","last_undegraded":"2024-09-21T18:53:39.366987+0000","last_fullsized":"2024-09-21T18:53:39.366987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.620456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804788+0000","last_change":"2024-09-21T18:53:07.403003+0000","last_active":"2024-09-21T18:53:39.804788+0000","last_peered":"2024-09-21T18:53:39.804788+0000","last_clean":"2024-09-21T18:53:39.804788+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:53:39.804788+0000","last_undegraded":"2024-09-21T18:53:39.804788+0000","last_fullsized":"2024-09-21T18:53:39.804788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.651586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:42.455317+0000","last_change":"2024-09-21T18:53:11.411052+0000","last_active":"2024-09-21T18:53:42.455317+0000","last_peered":"2024-09-21T18:53:42.455317+0000","last_clean":"2024-09-21T18:53:42.455317+0000","last_became_active":"2024-09-21T18:53:11.410769+0000","last_became_peered":"2024-09-21T18:53:11.410769+0000","last_unstale":"2024-09-21T18:53:42.455317+0000","last_undegraded":"2024-09-21T18:53:42.455317+0000","last_fullsized":"2024-09-21T18:53:42.455317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805631+0000","last_change":"2024-09-21T18:53:13.431033+0000","last_active":"2024-09-21T18:53:39.805631+0000","last_peered":"2024-09-21T18:53:39.805631+0000","last_clean":"2024-09-21T18:53:39.805631+0000","last_became_active":"2024-09-21T18:53:13.430667+0000","last_became_peered":"2024-09-21T18:53:13.430667+0000","last_unstale":"2024-09-21T18:53:39.805631+0000","last_undegraded":"2024-09-21T18:53:39.805631+0000","last_fullsized":"2024-09-21T18:53:39.805631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.681228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365948+0000","last_change":"2024-09-21T18:53:09.407481+0000","last_active":"2024-09-21T18:53:39.365948+0000","last_peered":"2024-09-21T18:53:39.365948+0000","last_clean":"2024-09-21T18:53:39.365948+0000","last_became_active":"2024-09-21T18:53:09.407184+0000","last_became_peered":"2024-09-21T18:53:09.407184+0000","last_unstale":"2024-09-21T18:53:39.365948+0000","last_undegraded":"2024-09-21T18:53:39.365948+0000","last_fullsized":"2024-09-21T18:53:39.365948+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.470465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366358+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:53:39.366358+0000","last_peered":"2024-09-21T18:53:39.366358+0000","last_clean":"2024-09-21T18:53:39.366358+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:53:39.366358+0000","last_undegraded":"2024-09-21T18:53:39.366358+0000","last_fullsized":"2024-09-21T18:53:39.366358+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.908635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402053+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:53:39.402053+0000","last_peered":"2024-09-21T18:53:39.402053+0000","last_clean":"2024-09-21T18:53:39.402053+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:53:39.402053+0000","last_undegraded":"2024-09-21T18:53:39.402053+0000","last_fullsized":"2024-09-21T18:53:39.402053+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.492927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365399+0000","last_change":"2024-09-21T18:53:13.433363+0000","last_active":"2024-09-21T18:53:39.365399+0000","last_peered":"2024-09-21T18:53:39.365399+0000","last_clean":"2024-09-21T18:53:39.365399+0000","last_became_active":"2024-09-21T18:53:13.433057+0000","last_became_peered":"2024-09-21T18:53:13.433057+0000","last_unstale":"2024-09-21T18:53:39.365399+0000","last_undegraded":"2024-09-21T18:53:39.365399+0000","last_fullsized":"2024-09-21T18:53:39.365399+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.215327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":55,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402167+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:53:39.402167+0000","last_peered":"2024-09-21T18:53:39.402167+0000","last_clean":"2024-09-21T18:53:39.402167+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:53:39.402167+0000","last_undegraded":"2024-09-21T18:53:39.402167+0000","last_fullsized":"2024-09-21T18:53:39.402167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.126928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804798+0000","last_change":"2024-09-21T18:53:07.406445+0000","last_active":"2024-09-21T18:53:39.804798+0000","last_peered":"2024-09-21T18:53:39.804798+0000","last_clean":"2024-09-21T18:53:39.804798+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:53:39.804798+0000","last_undegraded":"2024-09-21T18:53:39.804798+0000","last_fullsized":"2024-09-21T18:53:39.804798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.278365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366344+0000","last_change":"2024-09-21T18:53:11.411055+0000","last_active":"2024-09-21T18:53:39.366344+0000","last_peered":"2024-09-21T18:53:39.366344+0000","last_clean":"2024-09-21T18:53:39.366344+0000","last_became_active":"2024-09-21T18:53:11.410775+0000","last_became_peered":"2024-09-21T18:53:11.410775+0000","last_unstale":"2024-09-21T18:53:39.366344+0000","last_undegraded":"2024-09-21T18:53:39.366344+0000","last_fullsized":"2024-09-21T18:53:39.366344+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.614499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805130+0000","last_change":"2024-09-21T18:53:13.433872+0000","last_active":"2024-09-21T18:53:39.805130+0000","last_peered":"2024-09-21T18:53:39.805130+0000","last_clean":"2024-09-21T18:53:39.805130+0000","last_became_active":"2024-09-21T18:53:13.433534+0000","last_became_peered":"2024-09-21T18:53:13.433534+0000","last_unstale":"2024-09-21T18:53:39.805130+0000","last_undegraded":"2024-09-21T18:53:39.805130+0000","last_fullsized":"2024-09-21T18:53:39.805130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.158748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":59,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365648+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:53:39.365648+0000","last_peered":"2024-09-21T18:53:39.365648+0000","last_clean":"2024-09-21T18:53:39.365648+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:53:39.365648+0000","last_undegraded":"2024-09-21T18:53:39.365648+0000","last_fullsized":"2024-09-21T18:53:39.365648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.024213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.659141+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:53:39.659141+0000","last_peered":"2024-09-21T18:53:39.659141+0000","last_clean":"2024-09-21T18:53:39.659141+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:53:39.659141+0000","last_undegraded":"2024-09-21T18:53:39.659141+0000","last_fullsized":"2024-09-21T18:53:39.659141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.249689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804970+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:53:39.804970+0000","last_peered":"2024-09-21T18:53:39.804970+0000","last_clean":"2024-09-21T18:53:39.804970+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:53:39.804970+0000","last_undegraded":"2024-09-21T18:53:39.804970+0000","last_fullsized":"2024-09-21T18:53:39.804970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.908774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.366496+0000","last_change":"2024-09-21T18:53:13.658591+0000","last_active":"2024-09-21T18:53:39.366496+0000","last_peered":"2024-09-21T18:53:39.366496+0000","last_clean":"2024-09-21T18:53:39.366496+0000","last_became_active":"2024-09-21T18:53:13.657938+0000","last_became_peered":"2024-09-21T18:53:13.657938+0000","last_unstale":"2024-09-21T18:53:39.366496+0000","last_undegraded":"2024-09-21T18:53:39.366496+0000","last_fullsized":"2024-09-21T18:53:39.366496+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.163794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":46,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365325+0000","last_change":"2024-09-21T18:53:09.409585+0000","last_active":"2024-09-21T18:53:39.365325+0000","last_peered":"2024-09-21T18:53:39.365325+0000","last_clean":"2024-09-21T18:53:39.365325+0000","last_became_active":"2024-09-21T18:53:09.409216+0000","last_became_peered":"2024-09-21T18:53:09.409216+0000","last_unstale":"2024-09-21T18:53:39.365325+0000","last_undegraded":"2024-09-21T18:53:39.365325+0000","last_fullsized":"2024-09-21T18:53:39.365325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.804988+0000","last_change":"2024-09-21T18:53:07.401813+0000","last_active":"2024-09-21T18:53:39.804988+0000","last_peered":"2024-09-21T18:53:39.804988+0000","last_clean":"2024-09-21T18:53:39.804988+0000","last_became_active":"2024-09-21T18:53:07.401545+0000","last_became_peered":"2024-09-21T18:53:07.401545+0000","last_unstale":"2024-09-21T18:53:39.804988+0000","last_undegraded":"2024-09-21T18:53:39.804988+0000","last_fullsized":"2024-09-21T18:53:39.804988+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.528471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805547+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:53:39.805547+0000","last_peered":"2024-09-21T18:53:39.805547+0000","last_clean":"2024-09-21T18:53:39.805547+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:53:39.805547+0000","last_undegraded":"2024-09-21T18:53:39.805547+0000","last_fullsized":"2024-09-21T18:53:39.805547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.190087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805573+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:53:39.805573+0000","last_peered":"2024-09-21T18:53:39.805573+0000","last_clean":"2024-09-21T18:53:39.805573+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:53:39.805573+0000","last_undegraded":"2024-09-21T18:53:39.805573+0000","last_fullsized":"2024-09-21T18:53:39.805573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":31,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.402586+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:53:39.402586+0000","last_peered":"2024-09-21T18:53:39.402586+0000","last_clean":"2024-09-21T18:53:39.402586+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:53:39.402586+0000","last_undegraded":"2024-09-21T18:53:39.402586+0000","last_fullsized":"2024-09-21T18:53:39.402586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.931032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.365475+0000","last_change":"2024-09-21T18:53:07.400561+0000","last_active":"2024-09-21T18:53:39.365475+0000","last_peered":"2024-09-21T18:53:39.365475+0000","last_clean":"2024-09-21T18:53:39.365475+0000","last_became_active":"2024-09-21T18:53:07.400254+0000","last_became_peered":"2024-09-21T18:53:07.400254+0000","last_unstale":"2024-09-21T18:53:39.365475+0000","last_undegraded":"2024-09-21T18:53:39.365475+0000","last_fullsized":"2024-09-21T18:53:39.365475+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.077186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805851+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:53:39.805851+0000","last_peered":"2024-09-21T18:53:39.805851+0000","last_clean":"2024-09-21T18:53:39.805851+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:53:39.805851+0000","last_undegraded":"2024-09-21T18:53:39.805851+0000","last_fullsized":"2024-09-21T18:53:39.805851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.130619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-09-21T18:53:39.805737+0000","last_change":"2024-09-21T18:53:11.415531+0000","last_active":"2024-09-21T18:53:39.805737+0000","last_peered":"2024-09-21T18:53:39.805737+0000","last_clean":"2024-09-21T18:53:39.805737+0000","last_became_active":"2024-09-21T18:53:11.415209+0000","last_became_peered":"2024-09-21T18:53:11.415209+0000","last_unstale":"2024-09-21T18:53:39.805737+0000","last_undegraded":"2024-09-21T18:53:39.805737+0000","last_fullsized":"2024-09-21T18:53:39.805737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.845440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266701,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1351680,"data_stored":891253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528338,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1335296,"data_stored":890604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789973,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":737280,"data_stored":300146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018907,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31936,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713216,"statfs":{"total":95995035648,"available":95962333184,"internally_reserved":0,"allocated":765952,"data_stored":301385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182432,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":778240,"data_stored":301684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":733184,"data_stored":300371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":802816,"data_stored":302008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738416,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1355776,"data_stored":891942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:53:55.287 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T18:53:55.287 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T18:53:55.287 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T18:53:55.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T18:53:56.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:56 smithi104 ceph-mon[32438]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:56.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:56 smithi104 ceph-mon[32438]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T18:53:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:56 smithi081 ceph-mon[29384]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:56 smithi081 ceph-mon[29384]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T18:53:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:56 smithi081 ceph-mon[39192]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:53:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:56 smithi081 ceph-mon[39192]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T18:53:56.831 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:53:57.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:53:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:53:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:53:58.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:53:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:53:58.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:58 smithi081 ceph-mon[29384]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T18:53:58.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:58.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:58 smithi081 ceph-mon[39192]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T18:53:58.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:58.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:58 smithi104 ceph-mon[32438]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T18:53:58.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:53:58.938 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:53:58.938 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T18:53:59.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2187907466' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T18:53:59.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:53:59 smithi081 ceph-mon[29384]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T18:53:59.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2187907466' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T18:53:59.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:53:59 smithi081 ceph-mon[39192]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T18:53:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2187907466' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T18:53:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:53:59 smithi104 ceph-mon[32438]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T18:54:00.372 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T18:54:00.373 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T18:54:00.373 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T18:54:00.387 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T18:54:00.387 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-adk-testing-2024-09-21-1014'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-09-21-1014', 'sha1': '0ecbb435437226bc091a0c2d4549ba25bf724efe'}} 2024-09-21T18:54:00.387 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T18:54:00.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T18:54:00.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:00 smithi081 ceph-mon[29384]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T18:54:00.595 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:00 smithi081 ceph-mon[39192]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T18:54:00.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:00 smithi104 ceph-mon[32438]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T18:54:01.195 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:01.316 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:54:01 smithi081 systemd[1]: Started Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:54:01.897 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:01 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:01] ENGINE Bus STOPPING 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T18:54:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T18:54:02.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: Deploying daemon grafana.a on smithi104 2024-09-21T18:54:02.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:02 smithi104 ceph-mon[32438]: from='client.34667 ' entity='client.admin' 2024-09-21T18:54:02.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T18:54:02.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STOPPED 2024-09-21T18:54:02.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STARTING 2024-09-21T18:54:02.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Serving on http://:::9283 2024-09-21T18:54:02.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STARTED 2024-09-21T18:54:02.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STOPPING 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: Deploying daemon grafana.a on smithi104 2024-09-21T18:54:02.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[29384]: from='client.34667 ' entity='client.admin' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:02.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: Deploying daemon grafana.a on smithi104 2024-09-21T18:54:02.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-mon[39192]: from='client.34667 ' entity='client.admin' 2024-09-21T18:54:02.634 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T18:54:02.634 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T18:54:02.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:54:02.881 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:02.943 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T18:54:02.943 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STOPPED 2024-09-21T18:54:02.944 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STARTING 2024-09-21T18:54:02.944 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Serving on http://:::9283 2024-09-21T18:54:02.944 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:02 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:54:02] ENGINE Bus STARTED 2024-09-21T18:54:03.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:54:03.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:54:03.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:54:03.279 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:54:03.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:54:03.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:54:03.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:54:03.280 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:54:03.280 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:54:03 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[70807]: ts=2024-09-21T18:54:03.011Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000170561s 2024-09-21T18:54:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:03 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:54:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:03 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:54:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:03 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:54:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:03 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:54:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:54:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:54:04.231 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-21T18:54:03.374317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":897,"num_read_kb":754,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240732,"kb_used_data":7868,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720484,"statfs":{"total":767960285184,"available":767713775616,"internally_reserved":0,"allocated":8056832,"data_stored":4281745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":238276148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001914"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012542+0000","last_change":"2024-09-21T18:53:07.405054+0000","last_active":"2024-09-21T18:54:00.012542+0000","last_peered":"2024-09-21T18:54:00.012542+0000","last_clean":"2024-09-21T18:54:00.012542+0000","last_became_active":"2024-09-21T18:53:07.404774+0000","last_became_peered":"2024-09-21T18:53:07.404774+0000","last_unstale":"2024-09-21T18:54:00.012542+0000","last_undegraded":"2024-09-21T18:54:00.012542+0000","last_fullsized":"2024-09-21T18:54:00.012542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.853378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434676+0000","last_change":"2024-09-21T18:53:09.412508+0000","last_active":"2024-09-21T18:53:59.434676+0000","last_peered":"2024-09-21T18:53:59.434676+0000","last_clean":"2024-09-21T18:53:59.434676+0000","last_became_active":"2024-09-21T18:53:09.412193+0000","last_became_peered":"2024-09-21T18:53:09.412193+0000","last_unstale":"2024-09-21T18:53:59.434676+0000","last_undegraded":"2024-09-21T18:53:59.434676+0000","last_fullsized":"2024-09-21T18:53:59.434676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435852+0000","last_change":"2024-09-21T18:53:11.410506+0000","last_active":"2024-09-21T18:53:59.435852+0000","last_peered":"2024-09-21T18:53:59.435852+0000","last_clean":"2024-09-21T18:53:59.435852+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:53:59.435852+0000","last_undegraded":"2024-09-21T18:53:59.435852+0000","last_fullsized":"2024-09-21T18:53:59.435852+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.829493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434784+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:53:59.434784+0000","last_peered":"2024-09-21T18:53:59.434784+0000","last_clean":"2024-09-21T18:53:59.434784+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:53:59.434784+0000","last_undegraded":"2024-09-21T18:53:59.434784+0000","last_fullsized":"2024-09-21T18:53:59.434784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435518+0000","last_change":"2024-09-21T18:53:13.658931+0000","last_active":"2024-09-21T18:53:59.435518+0000","last_peered":"2024-09-21T18:53:59.435518+0000","last_clean":"2024-09-21T18:53:59.435518+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:53:59.435518+0000","last_undegraded":"2024-09-21T18:53:59.435518+0000","last_fullsized":"2024-09-21T18:53:59.435518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.480075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437152+0000","last_change":"2024-09-21T18:53:07.390902+0000","last_active":"2024-09-21T18:53:59.437152+0000","last_peered":"2024-09-21T18:53:59.437152+0000","last_clean":"2024-09-21T18:53:59.437152+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:53:59.437152+0000","last_undegraded":"2024-09-21T18:53:59.437152+0000","last_fullsized":"2024-09-21T18:53:59.437152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.326332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":69,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436300+0000","last_change":"2024-09-21T18:53:09.407061+0000","last_active":"2024-09-21T18:53:59.436300+0000","last_peered":"2024-09-21T18:53:59.436300+0000","last_clean":"2024-09-21T18:53:59.436300+0000","last_became_active":"2024-09-21T18:53:09.406774+0000","last_became_peered":"2024-09-21T18:53:09.406774+0000","last_unstale":"2024-09-21T18:53:59.436300+0000","last_undegraded":"2024-09-21T18:53:59.436300+0000","last_fullsized":"2024-09-21T18:53:59.436300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.239535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434654+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:53:59.434654+0000","last_peered":"2024-09-21T18:53:59.434654+0000","last_clean":"2024-09-21T18:53:59.434654+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:53:59.434654+0000","last_undegraded":"2024-09-21T18:53:59.434654+0000","last_fullsized":"2024-09-21T18:53:59.434654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.141973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012916+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:54:00.012916+0000","last_peered":"2024-09-21T18:54:00.012916+0000","last_clean":"2024-09-21T18:54:00.012916+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:54:00.012916+0000","last_undegraded":"2024-09-21T18:54:00.012916+0000","last_fullsized":"2024-09-21T18:54:00.012916+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.801164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435060+0000","last_change":"2024-09-21T18:53:09.401868+0000","last_active":"2024-09-21T18:53:59.435060+0000","last_peered":"2024-09-21T18:53:59.435060+0000","last_clean":"2024-09-21T18:53:59.435060+0000","last_became_active":"2024-09-21T18:53:09.401547+0000","last_became_peered":"2024-09-21T18:53:09.401547+0000","last_unstale":"2024-09-21T18:53:59.435060+0000","last_undegraded":"2024-09-21T18:53:59.435060+0000","last_fullsized":"2024-09-21T18:53:59.435060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.067009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012314+0000","last_change":"2024-09-21T18:53:11.409118+0000","last_active":"2024-09-21T18:54:00.012314+0000","last_peered":"2024-09-21T18:54:00.012314+0000","last_clean":"2024-09-21T18:54:00.012314+0000","last_became_active":"2024-09-21T18:53:11.408537+0000","last_became_peered":"2024-09-21T18:53:11.408537+0000","last_unstale":"2024-09-21T18:54:00.012314+0000","last_undegraded":"2024-09-21T18:54:00.012314+0000","last_fullsized":"2024-09-21T18:54:00.012314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.248524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011582+0000","last_change":"2024-09-21T18:53:13.429374+0000","last_active":"2024-09-21T18:54:00.011582+0000","last_peered":"2024-09-21T18:54:00.011582+0000","last_clean":"2024-09-21T18:54:00.011582+0000","last_became_active":"2024-09-21T18:53:13.429007+0000","last_became_peered":"2024-09-21T18:53:13.429007+0000","last_unstale":"2024-09-21T18:54:00.011582+0000","last_undegraded":"2024-09-21T18:54:00.011582+0000","last_fullsized":"2024-09-21T18:54:00.011582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.767711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012641+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:54:00.012641+0000","last_peered":"2024-09-21T18:54:00.012641+0000","last_clean":"2024-09-21T18:54:00.012641+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:54:00.012641+0000","last_undegraded":"2024-09-21T18:54:00.012641+0000","last_fullsized":"2024-09-21T18:54:00.012641+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435065+0000","last_change":"2024-09-21T18:53:09.406568+0000","last_active":"2024-09-21T18:53:59.435065+0000","last_peered":"2024-09-21T18:53:59.435065+0000","last_clean":"2024-09-21T18:53:59.435065+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:53:59.435065+0000","last_undegraded":"2024-09-21T18:53:59.435065+0000","last_fullsized":"2024-09-21T18:53:59.435065+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865775+0000","last_change":"2024-09-21T18:53:11.412969+0000","last_active":"2024-09-21T18:53:59.865775+0000","last_peered":"2024-09-21T18:53:59.865775+0000","last_clean":"2024-09-21T18:53:59.865775+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:53:59.865775+0000","last_undegraded":"2024-09-21T18:53:59.865775+0000","last_fullsized":"2024-09-21T18:53:59.865775+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.874706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012170+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:54:00.012170+0000","last_peered":"2024-09-21T18:54:00.012170+0000","last_clean":"2024-09-21T18:54:00.012170+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:54:00.012170+0000","last_undegraded":"2024-09-21T18:54:00.012170+0000","last_fullsized":"2024-09-21T18:54:00.012170+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.945242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435317+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:53:59.435317+0000","last_peered":"2024-09-21T18:53:59.435317+0000","last_clean":"2024-09-21T18:53:59.435317+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:53:59.435317+0000","last_undegraded":"2024-09-21T18:53:59.435317+0000","last_fullsized":"2024-09-21T18:53:59.435317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.198560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012651+0000","last_change":"2024-09-21T18:53:07.405471+0000","last_active":"2024-09-21T18:54:00.012651+0000","last_peered":"2024-09-21T18:54:00.012651+0000","last_clean":"2024-09-21T18:54:00.012651+0000","last_became_active":"2024-09-21T18:53:07.405177+0000","last_became_peered":"2024-09-21T18:53:07.405177+0000","last_unstale":"2024-09-21T18:54:00.012651+0000","last_undegraded":"2024-09-21T18:54:00.012651+0000","last_fullsized":"2024-09-21T18:54:00.012651+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.483473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012086+0000","last_change":"2024-09-21T18:53:09.407164+0000","last_active":"2024-09-21T18:54:00.012086+0000","last_peered":"2024-09-21T18:54:00.012086+0000","last_clean":"2024-09-21T18:54:00.012086+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:54:00.012086+0000","last_undegraded":"2024-09-21T18:54:00.012086+0000","last_fullsized":"2024-09-21T18:54:00.012086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.948438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436967+0000","last_change":"2024-09-21T18:53:11.411989+0000","last_active":"2024-09-21T18:53:59.436967+0000","last_peered":"2024-09-21T18:53:59.436967+0000","last_clean":"2024-09-21T18:53:59.436967+0000","last_became_active":"2024-09-21T18:53:11.411624+0000","last_became_peered":"2024-09-21T18:53:11.411624+0000","last_unstale":"2024-09-21T18:53:59.436967+0000","last_undegraded":"2024-09-21T18:53:59.436967+0000","last_fullsized":"2024-09-21T18:53:59.436967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.979898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437258+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:53:59.437258+0000","last_peered":"2024-09-21T18:53:59.437258+0000","last_clean":"2024-09-21T18:53:59.437258+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:53:59.437258+0000","last_undegraded":"2024-09-21T18:53:59.437258+0000","last_fullsized":"2024-09-21T18:53:59.437258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.301610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435473+0000","last_change":"2024-09-21T18:53:07.397249+0000","last_active":"2024-09-21T18:53:59.435473+0000","last_peered":"2024-09-21T18:53:59.435473+0000","last_clean":"2024-09-21T18:53:59.435473+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:53:59.435473+0000","last_undegraded":"2024-09-21T18:53:59.435473+0000","last_fullsized":"2024-09-21T18:53:59.435473+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.232024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436768+0000","last_change":"2024-09-21T18:53:09.406892+0000","last_active":"2024-09-21T18:53:59.436768+0000","last_peered":"2024-09-21T18:53:59.436768+0000","last_clean":"2024-09-21T18:53:59.436768+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:53:59.436768+0000","last_undegraded":"2024-09-21T18:53:59.436768+0000","last_fullsized":"2024-09-21T18:53:59.436768+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.459852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435171+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:53:59.435171+0000","last_peered":"2024-09-21T18:53:59.435171+0000","last_clean":"2024-09-21T18:53:59.435171+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:53:59.435171+0000","last_undegraded":"2024-09-21T18:53:59.435171+0000","last_fullsized":"2024-09-21T18:53:59.435171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.060686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866021+0000","last_change":"2024-09-21T18:53:13.426725+0000","last_active":"2024-09-21T18:53:59.866021+0000","last_peered":"2024-09-21T18:53:59.866021+0000","last_clean":"2024-09-21T18:53:59.866021+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:53:59.866021+0000","last_undegraded":"2024-09-21T18:53:59.866021+0000","last_fullsized":"2024-09-21T18:53:59.866021+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436567+0000","last_change":"2024-09-21T18:53:07.397345+0000","last_active":"2024-09-21T18:53:59.436567+0000","last_peered":"2024-09-21T18:53:59.436567+0000","last_clean":"2024-09-21T18:53:59.436567+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:53:59.436567+0000","last_undegraded":"2024-09-21T18:53:59.436567+0000","last_fullsized":"2024-09-21T18:53:59.436567+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.808378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012446+0000","last_change":"2024-09-21T18:53:09.719399+0000","last_active":"2024-09-21T18:54:00.012446+0000","last_peered":"2024-09-21T18:54:00.012446+0000","last_clean":"2024-09-21T18:54:00.012446+0000","last_became_active":"2024-09-21T18:53:09.719059+0000","last_became_peered":"2024-09-21T18:53:09.719059+0000","last_unstale":"2024-09-21T18:54:00.012446+0000","last_undegraded":"2024-09-21T18:54:00.012446+0000","last_fullsized":"2024-09-21T18:54:00.012446+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.854862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013562+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:54:00.013562+0000","last_peered":"2024-09-21T18:54:00.013562+0000","last_clean":"2024-09-21T18:54:00.013562+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:54:00.013562+0000","last_undegraded":"2024-09-21T18:54:00.013562+0000","last_fullsized":"2024-09-21T18:54:00.013562+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.329971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437030+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:53:59.437030+0000","last_peered":"2024-09-21T18:53:59.437030+0000","last_clean":"2024-09-21T18:53:59.437030+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:53:59.437030+0000","last_undegraded":"2024-09-21T18:53:59.437030+0000","last_fullsized":"2024-09-21T18:53:59.437030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.373058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436699+0000","last_change":"2024-09-21T18:53:07.398833+0000","last_active":"2024-09-21T18:53:59.436699+0000","last_peered":"2024-09-21T18:53:59.436699+0000","last_clean":"2024-09-21T18:53:59.436699+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:53:59.436699+0000","last_undegraded":"2024-09-21T18:53:59.436699+0000","last_fullsized":"2024-09-21T18:53:59.436699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435434+0000","last_change":"2024-09-21T18:53:11.411518+0000","last_active":"2024-09-21T18:53:59.435434+0000","last_peered":"2024-09-21T18:53:59.435434+0000","last_clean":"2024-09-21T18:53:59.435434+0000","last_became_active":"2024-09-21T18:53:11.411212+0000","last_became_peered":"2024-09-21T18:53:11.411212+0000","last_unstale":"2024-09-21T18:53:59.435434+0000","last_undegraded":"2024-09-21T18:53:59.435434+0000","last_fullsized":"2024-09-21T18:53:59.435434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.137419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012670+0000","last_change":"2024-09-21T18:53:13.433767+0000","last_active":"2024-09-21T18:54:00.012670+0000","last_peered":"2024-09-21T18:54:00.012670+0000","last_clean":"2024-09-21T18:54:00.012670+0000","last_became_active":"2024-09-21T18:53:13.433575+0000","last_became_peered":"2024-09-21T18:53:13.433575+0000","last_unstale":"2024-09-21T18:54:00.012670+0000","last_undegraded":"2024-09-21T18:54:00.012670+0000","last_fullsized":"2024-09-21T18:54:00.012670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.484363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":76,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436514+0000","last_change":"2024-09-21T18:53:09.719338+0000","last_active":"2024-09-21T18:53:59.436514+0000","last_peered":"2024-09-21T18:53:59.436514+0000","last_clean":"2024-09-21T18:53:59.436514+0000","last_became_active":"2024-09-21T18:53:09.719017+0000","last_became_peered":"2024-09-21T18:53:09.719017+0000","last_unstale":"2024-09-21T18:53:59.436514+0000","last_undegraded":"2024-09-21T18:53:59.436514+0000","last_fullsized":"2024-09-21T18:53:59.436514+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.426276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437469+0000","last_change":"2024-09-21T18:53:07.400418+0000","last_active":"2024-09-21T18:53:59.437469+0000","last_peered":"2024-09-21T18:53:59.437469+0000","last_clean":"2024-09-21T18:53:59.437469+0000","last_became_active":"2024-09-21T18:53:07.400143+0000","last_became_peered":"2024-09-21T18:53:07.400143+0000","last_unstale":"2024-09-21T18:53:59.437469+0000","last_undegraded":"2024-09-21T18:53:59.437469+0000","last_fullsized":"2024-09-21T18:53:59.437469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.737589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434684+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:53:59.434684+0000","last_peered":"2024-09-21T18:53:59.434684+0000","last_clean":"2024-09-21T18:53:59.434684+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:53:59.434684+0000","last_undegraded":"2024-09-21T18:53:59.434684+0000","last_fullsized":"2024-09-21T18:53:59.434684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.956086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434991+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:53:59.434991+0000","last_peered":"2024-09-21T18:53:59.434991+0000","last_clean":"2024-09-21T18:53:59.434991+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:53:59.434991+0000","last_undegraded":"2024-09-21T18:53:59.434991+0000","last_fullsized":"2024-09-21T18:53:59.434991+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.926924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":64,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434775+0000","last_change":"2024-09-21T18:53:09.400587+0000","last_active":"2024-09-21T18:53:59.434775+0000","last_peered":"2024-09-21T18:53:59.434775+0000","last_clean":"2024-09-21T18:53:59.434775+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:53:59.434775+0000","last_undegraded":"2024-09-21T18:53:59.434775+0000","last_fullsized":"2024-09-21T18:53:59.434775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.173359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012612+0000","last_change":"2024-09-21T18:53:07.404382+0000","last_active":"2024-09-21T18:54:00.012612+0000","last_peered":"2024-09-21T18:54:00.012612+0000","last_clean":"2024-09-21T18:54:00.012612+0000","last_became_active":"2024-09-21T18:53:07.404018+0000","last_became_peered":"2024-09-21T18:53:07.404018+0000","last_unstale":"2024-09-21T18:54:00.012612+0000","last_undegraded":"2024-09-21T18:54:00.012612+0000","last_fullsized":"2024-09-21T18:54:00.012612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.245821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435360+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:53:59.435360+0000","last_peered":"2024-09-21T18:53:59.435360+0000","last_clean":"2024-09-21T18:53:59.435360+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:53:59.435360+0000","last_undegraded":"2024-09-21T18:53:59.435360+0000","last_fullsized":"2024-09-21T18:53:59.435360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.339632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865864+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:53:59.865864+0000","last_peered":"2024-09-21T18:53:59.865864+0000","last_clean":"2024-09-21T18:53:59.865864+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:53:59.865864+0000","last_undegraded":"2024-09-21T18:53:59.865864+0000","last_fullsized":"2024-09-21T18:53:59.865864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.973293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436625+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:53:59.436625+0000","last_peered":"2024-09-21T18:53:59.436625+0000","last_clean":"2024-09-21T18:53:59.436625+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:53:59.436625+0000","last_undegraded":"2024-09-21T18:53:59.436625+0000","last_fullsized":"2024-09-21T18:53:59.436625+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.374498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013013+0000","last_change":"2024-09-21T18:53:07.403537+0000","last_active":"2024-09-21T18:54:00.013013+0000","last_peered":"2024-09-21T18:54:00.013013+0000","last_clean":"2024-09-21T18:54:00.013013+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:54:00.013013+0000","last_undegraded":"2024-09-21T18:54:00.013013+0000","last_fullsized":"2024-09-21T18:54:00.013013+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.558691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012374+0000","last_change":"2024-09-21T18:53:11.407531+0000","last_active":"2024-09-21T18:54:00.012374+0000","last_peered":"2024-09-21T18:54:00.012374+0000","last_clean":"2024-09-21T18:54:00.012374+0000","last_became_active":"2024-09-21T18:53:11.407244+0000","last_became_peered":"2024-09-21T18:53:11.407244+0000","last_unstale":"2024-09-21T18:54:00.012374+0000","last_undegraded":"2024-09-21T18:54:00.012374+0000","last_fullsized":"2024-09-21T18:54:00.012374+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.763939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011738+0000","last_change":"2024-09-21T18:53:13.435210+0000","last_active":"2024-09-21T18:54:00.011738+0000","last_peered":"2024-09-21T18:54:00.011738+0000","last_clean":"2024-09-21T18:54:00.011738+0000","last_became_active":"2024-09-21T18:53:13.434753+0000","last_became_peered":"2024-09-21T18:53:13.434753+0000","last_unstale":"2024-09-21T18:54:00.011738+0000","last_undegraded":"2024-09-21T18:54:00.011738+0000","last_fullsized":"2024-09-21T18:54:00.011738+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.794214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":79,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012487+0000","last_change":"2024-09-21T18:53:09.719454+0000","last_active":"2024-09-21T18:54:00.012487+0000","last_peered":"2024-09-21T18:54:00.012487+0000","last_clean":"2024-09-21T18:54:00.012487+0000","last_became_active":"2024-09-21T18:53:09.719114+0000","last_became_peered":"2024-09-21T18:53:09.719114+0000","last_unstale":"2024-09-21T18:54:00.012487+0000","last_undegraded":"2024-09-21T18:54:00.012487+0000","last_fullsized":"2024-09-21T18:54:00.012487+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.176379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436502+0000","last_change":"2024-09-21T18:53:07.400542+0000","last_active":"2024-09-21T18:53:59.436502+0000","last_peered":"2024-09-21T18:53:59.436502+0000","last_clean":"2024-09-21T18:53:59.436502+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:53:59.436502+0000","last_undegraded":"2024-09-21T18:53:59.436502+0000","last_fullsized":"2024-09-21T18:53:59.436502+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.007870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012485+0000","last_change":"2024-09-21T18:53:11.404992+0000","last_active":"2024-09-21T18:54:00.012485+0000","last_peered":"2024-09-21T18:54:00.012485+0000","last_clean":"2024-09-21T18:54:00.012485+0000","last_became_active":"2024-09-21T18:53:11.404577+0000","last_became_peered":"2024-09-21T18:53:11.404577+0000","last_unstale":"2024-09-21T18:54:00.012485+0000","last_undegraded":"2024-09-21T18:54:00.012485+0000","last_fullsized":"2024-09-21T18:54:00.012485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.773972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435754+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:53:59.435754+0000","last_peered":"2024-09-21T18:53:59.435754+0000","last_clean":"2024-09-21T18:53:59.435754+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:53:59.435754+0000","last_undegraded":"2024-09-21T18:53:59.435754+0000","last_fullsized":"2024-09-21T18:53:59.435754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.173798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":102,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436352+0000","last_change":"2024-09-21T18:53:09.410172+0000","last_active":"2024-09-21T18:53:59.436352+0000","last_peered":"2024-09-21T18:53:59.436352+0000","last_clean":"2024-09-21T18:53:59.436352+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:53:59.436352+0000","last_undegraded":"2024-09-21T18:53:59.436352+0000","last_fullsized":"2024-09-21T18:53:59.436352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.020223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435363+0000","last_change":"2024-09-21T18:53:07.403315+0000","last_active":"2024-09-21T18:53:59.435363+0000","last_peered":"2024-09-21T18:53:59.435363+0000","last_clean":"2024-09-21T18:53:59.435363+0000","last_became_active":"2024-09-21T18:53:07.403002+0000","last_became_peered":"2024-09-21T18:53:07.403002+0000","last_unstale":"2024-09-21T18:53:59.435363+0000","last_undegraded":"2024-09-21T18:53:59.435363+0000","last_fullsized":"2024-09-21T18:53:59.435363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:08.340495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012430+0000","last_change":"2024-09-21T18:53:09.398954+0000","last_active":"2024-09-21T18:54:00.012430+0000","last_peered":"2024-09-21T18:54:00.012430+0000","last_clean":"2024-09-21T18:54:00.012430+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:54:00.012430+0000","last_undegraded":"2024-09-21T18:54:00.012430+0000","last_fullsized":"2024-09-21T18:54:00.012430+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.467230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012383+0000","last_change":"2024-09-21T18:53:11.411504+0000","last_active":"2024-09-21T18:54:00.012383+0000","last_peered":"2024-09-21T18:54:00.012383+0000","last_clean":"2024-09-21T18:54:00.012383+0000","last_became_active":"2024-09-21T18:53:11.411229+0000","last_became_peered":"2024-09-21T18:53:11.411229+0000","last_unstale":"2024-09-21T18:54:00.012383+0000","last_undegraded":"2024-09-21T18:54:00.012383+0000","last_fullsized":"2024-09-21T18:54:00.012383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.568499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436995+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:53:59.436995+0000","last_peered":"2024-09-21T18:53:59.436995+0000","last_clean":"2024-09-21T18:53:59.436995+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:53:59.436995+0000","last_undegraded":"2024-09-21T18:53:59.436995+0000","last_fullsized":"2024-09-21T18:53:59.436995+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.888758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437318+0000","last_change":"2024-09-21T18:53:09.412601+0000","last_active":"2024-09-21T18:53:59.437318+0000","last_peered":"2024-09-21T18:53:59.437318+0000","last_clean":"2024-09-21T18:53:59.437318+0000","last_became_active":"2024-09-21T18:53:09.412304+0000","last_became_peered":"2024-09-21T18:53:09.412304+0000","last_unstale":"2024-09-21T18:53:59.437318+0000","last_undegraded":"2024-09-21T18:53:59.437318+0000","last_fullsized":"2024-09-21T18:53:59.437318+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.434125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436187+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:53:59.436187+0000","last_peered":"2024-09-21T18:53:59.436187+0000","last_clean":"2024-09-21T18:53:59.436187+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:53:59.436187+0000","last_undegraded":"2024-09-21T18:53:59.436187+0000","last_fullsized":"2024-09-21T18:53:59.436187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.025870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011742+0000","last_change":"2024-09-21T18:53:09.400372+0000","last_active":"2024-09-21T18:54:00.011742+0000","last_peered":"2024-09-21T18:54:00.011742+0000","last_clean":"2024-09-21T18:54:00.011742+0000","last_became_active":"2024-09-21T18:53:06.391866+0000","last_became_peered":"2024-09-21T18:53:06.391866+0000","last_unstale":"2024-09-21T18:54:00.011742+0000","last_undegraded":"2024-09-21T18:54:00.011742+0000","last_fullsized":"2024-09-21T18:54:00.011742+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011837+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:54:00.011837+0000","last_peered":"2024-09-21T18:54:00.011837+0000","last_clean":"2024-09-21T18:54:00.011837+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:54:00.011837+0000","last_undegraded":"2024-09-21T18:54:00.011837+0000","last_fullsized":"2024-09-21T18:54:00.011837+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.265652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866098+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:53:59.866098+0000","last_peered":"2024-09-21T18:53:59.866098+0000","last_clean":"2024-09-21T18:53:59.866098+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:53:59.866098+0000","last_undegraded":"2024-09-21T18:53:59.866098+0000","last_fullsized":"2024-09-21T18:53:59.866098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012495+0000","last_change":"2024-09-21T18:53:09.407641+0000","last_active":"2024-09-21T18:54:00.012495+0000","last_peered":"2024-09-21T18:54:00.012495+0000","last_clean":"2024-09-21T18:54:00.012495+0000","last_became_active":"2024-09-21T18:53:09.407302+0000","last_became_peered":"2024-09-21T18:53:09.407302+0000","last_unstale":"2024-09-21T18:54:00.012495+0000","last_undegraded":"2024-09-21T18:54:00.012495+0000","last_fullsized":"2024-09-21T18:54:00.012495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.070171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012047+0000","last_change":"2024-09-21T18:53:07.404328+0000","last_active":"2024-09-21T18:54:00.012047+0000","last_peered":"2024-09-21T18:54:00.012047+0000","last_clean":"2024-09-21T18:54:00.012047+0000","last_became_active":"2024-09-21T18:53:07.404060+0000","last_became_peered":"2024-09-21T18:53:07.404060+0000","last_unstale":"2024-09-21T18:54:00.012047+0000","last_undegraded":"2024-09-21T18:54:00.012047+0000","last_fullsized":"2024-09-21T18:54:00.012047+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.843533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":106,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865811+0000","last_change":"2024-09-21T18:53:09.719413+0000","last_active":"2024-09-21T18:53:59.865811+0000","last_peered":"2024-09-21T18:53:59.865811+0000","last_clean":"2024-09-21T18:53:59.865811+0000","last_became_active":"2024-09-21T18:53:06.394373+0000","last_became_peered":"2024-09-21T18:53:06.394373+0000","last_unstale":"2024-09-21T18:53:59.865811+0000","last_undegraded":"2024-09-21T18:53:59.865811+0000","last_fullsized":"2024-09-21T18:53:59.865811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.196010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013416+0000","last_change":"2024-09-21T18:53:11.409878+0000","last_active":"2024-09-21T18:54:00.013416+0000","last_peered":"2024-09-21T18:54:00.013416+0000","last_clean":"2024-09-21T18:54:00.013416+0000","last_became_active":"2024-09-21T18:53:11.409637+0000","last_became_peered":"2024-09-21T18:53:11.409637+0000","last_unstale":"2024-09-21T18:54:00.013416+0000","last_undegraded":"2024-09-21T18:54:00.013416+0000","last_fullsized":"2024-09-21T18:54:00.013416+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.893970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436044+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:53:59.436044+0000","last_peered":"2024-09-21T18:53:59.436044+0000","last_clean":"2024-09-21T18:53:59.436044+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:53:59.436044+0000","last_undegraded":"2024-09-21T18:53:59.436044+0000","last_fullsized":"2024-09-21T18:53:59.436044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.377231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":75,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013301+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:54:00.013301+0000","last_peered":"2024-09-21T18:54:00.013301+0000","last_clean":"2024-09-21T18:54:00.013301+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:54:00.013301+0000","last_undegraded":"2024-09-21T18:54:00.013301+0000","last_fullsized":"2024-09-21T18:54:00.013301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.645863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435646+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:53:59.435646+0000","last_peered":"2024-09-21T18:53:59.435646+0000","last_clean":"2024-09-21T18:53:59.435646+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:53:59.435646+0000","last_undegraded":"2024-09-21T18:53:59.435646+0000","last_fullsized":"2024-09-21T18:53:59.435646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.707207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":91,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865887+0000","last_change":"2024-09-21T18:53:29.766304+0000","last_active":"2024-09-21T18:53:59.865887+0000","last_peered":"2024-09-21T18:53:59.865887+0000","last_clean":"2024-09-21T18:53:59.865887+0000","last_became_active":"2024-09-21T18:53:00.358105+0000","last_became_peered":"2024-09-21T18:53:00.358105+0000","last_unstale":"2024-09-21T18:53:59.865887+0000","last_undegraded":"2024-09-21T18:53:59.865887+0000","last_fullsized":"2024-09-21T18:53:59.865887+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:50.998311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866190+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:53:59.866190+0000","last_peered":"2024-09-21T18:53:59.866190+0000","last_clean":"2024-09-21T18:53:59.866190+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:53:59.866190+0000","last_undegraded":"2024-09-21T18:53:59.866190+0000","last_fullsized":"2024-09-21T18:53:59.866190+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.081383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013319+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:54:00.013319+0000","last_peered":"2024-09-21T18:54:00.013319+0000","last_clean":"2024-09-21T18:54:00.013319+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:54:00.013319+0000","last_undegraded":"2024-09-21T18:54:00.013319+0000","last_fullsized":"2024-09-21T18:54:00.013319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.641099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":62,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435553+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:53:59.435553+0000","last_peered":"2024-09-21T18:53:59.435553+0000","last_clean":"2024-09-21T18:53:59.435553+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:53:59.435553+0000","last_undegraded":"2024-09-21T18:53:59.435553+0000","last_fullsized":"2024-09-21T18:53:59.435553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435045+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:53:59.435045+0000","last_peered":"2024-09-21T18:53:59.435045+0000","last_clean":"2024-09-21T18:53:59.435045+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:53:59.435045+0000","last_undegraded":"2024-09-21T18:53:59.435045+0000","last_fullsized":"2024-09-21T18:53:59.435045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.491463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013256+0000","last_change":"2024-09-21T18:53:11.409004+0000","last_active":"2024-09-21T18:54:00.013256+0000","last_peered":"2024-09-21T18:54:00.013256+0000","last_clean":"2024-09-21T18:54:00.013256+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:54:00.013256+0000","last_undegraded":"2024-09-21T18:54:00.013256+0000","last_fullsized":"2024-09-21T18:54:00.013256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.557609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435394+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:53:59.435394+0000","last_peered":"2024-09-21T18:53:59.435394+0000","last_clean":"2024-09-21T18:53:59.435394+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:53:59.435394+0000","last_undegraded":"2024-09-21T18:53:59.435394+0000","last_fullsized":"2024-09-21T18:53:59.435394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.277061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":71,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435409+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:53:59.435409+0000","last_peered":"2024-09-21T18:53:59.435409+0000","last_clean":"2024-09-21T18:53:59.435409+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:53:59.435409+0000","last_undegraded":"2024-09-21T18:53:59.435409+0000","last_fullsized":"2024-09-21T18:53:59.435409+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.263644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012920+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:54:00.012920+0000","last_peered":"2024-09-21T18:54:00.012920+0000","last_clean":"2024-09-21T18:54:00.012920+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:54:00.012920+0000","last_undegraded":"2024-09-21T18:54:00.012920+0000","last_fullsized":"2024-09-21T18:54:00.012920+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436139+0000","last_change":"2024-09-21T18:53:11.409557+0000","last_active":"2024-09-21T18:53:59.436139+0000","last_peered":"2024-09-21T18:53:59.436139+0000","last_clean":"2024-09-21T18:53:59.436139+0000","last_became_active":"2024-09-21T18:53:11.409200+0000","last_became_peered":"2024-09-21T18:53:11.409200+0000","last_unstale":"2024-09-21T18:53:59.436139+0000","last_undegraded":"2024-09-21T18:53:59.436139+0000","last_fullsized":"2024-09-21T18:53:59.436139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:37.475931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012158+0000","last_change":"2024-09-21T18:53:13.430863+0000","last_active":"2024-09-21T18:54:00.012158+0000","last_peered":"2024-09-21T18:54:00.012158+0000","last_clean":"2024-09-21T18:54:00.012158+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:54:00.012158+0000","last_undegraded":"2024-09-21T18:54:00.012158+0000","last_fullsized":"2024-09-21T18:54:00.012158+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:38.403912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435160+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:53:59.435160+0000","last_peered":"2024-09-21T18:53:59.435160+0000","last_clean":"2024-09-21T18:53:59.435160+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:53:59.435160+0000","last_undegraded":"2024-09-21T18:53:59.435160+0000","last_fullsized":"2024-09-21T18:53:59.435160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.582865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435904+0000","last_change":"2024-09-21T18:53:07.400595+0000","last_active":"2024-09-21T18:53:59.435904+0000","last_peered":"2024-09-21T18:53:59.435904+0000","last_clean":"2024-09-21T18:53:59.435904+0000","last_became_active":"2024-09-21T18:53:07.399708+0000","last_became_peered":"2024-09-21T18:53:07.399708+0000","last_unstale":"2024-09-21T18:53:59.435904+0000","last_undegraded":"2024-09-21T18:53:59.435904+0000","last_fullsized":"2024-09-21T18:53:59.435904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:07.859688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011967+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:54:00.011967+0000","last_peered":"2024-09-21T18:54:00.011967+0000","last_clean":"2024-09-21T18:54:00.011967+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:54:00.011967+0000","last_undegraded":"2024-09-21T18:54:00.011967+0000","last_fullsized":"2024-09-21T18:54:00.011967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.955877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435155+0000","last_change":"2024-09-21T18:53:13.428208+0000","last_active":"2024-09-21T18:53:59.435155+0000","last_peered":"2024-09-21T18:53:59.435155+0000","last_clean":"2024-09-21T18:53:59.435155+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:53:59.435155+0000","last_undegraded":"2024-09-21T18:53:59.435155+0000","last_fullsized":"2024-09-21T18:53:59.435155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.939232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012441+0000","last_change":"2024-09-21T18:53:09.408663+0000","last_active":"2024-09-21T18:54:00.012441+0000","last_peered":"2024-09-21T18:54:00.012441+0000","last_clean":"2024-09-21T18:54:00.012441+0000","last_became_active":"2024-09-21T18:53:09.408315+0000","last_became_peered":"2024-09-21T18:53:09.408315+0000","last_unstale":"2024-09-21T18:54:00.012441+0000","last_undegraded":"2024-09-21T18:54:00.012441+0000","last_fullsized":"2024-09-21T18:54:00.012441+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.191003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012708+0000","last_change":"2024-09-21T18:53:07.400068+0000","last_active":"2024-09-21T18:54:00.012708+0000","last_peered":"2024-09-21T18:54:00.012708+0000","last_clean":"2024-09-21T18:54:00.012708+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:54:00.012708+0000","last_undegraded":"2024-09-21T18:54:00.012708+0000","last_fullsized":"2024-09-21T18:54:00.012708+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.160743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011811+0000","last_change":"2024-09-21T18:53:11.415649+0000","last_active":"2024-09-21T18:54:00.011811+0000","last_peered":"2024-09-21T18:54:00.011811+0000","last_clean":"2024-09-21T18:54:00.011811+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:54:00.011811+0000","last_undegraded":"2024-09-21T18:54:00.011811+0000","last_fullsized":"2024-09-21T18:54:00.011811+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.547416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011953+0000","last_change":"2024-09-21T18:53:13.434903+0000","last_active":"2024-09-21T18:54:00.011953+0000","last_peered":"2024-09-21T18:54:00.011953+0000","last_clean":"2024-09-21T18:54:00.011953+0000","last_became_active":"2024-09-21T18:53:13.434569+0000","last_became_peered":"2024-09-21T18:53:13.434569+0000","last_unstale":"2024-09-21T18:54:00.011953+0000","last_undegraded":"2024-09-21T18:54:00.011953+0000","last_fullsized":"2024-09-21T18:54:00.011953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.431252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":73,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012809+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:54:00.012809+0000","last_peered":"2024-09-21T18:54:00.012809+0000","last_clean":"2024-09-21T18:54:00.012809+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:54:00.012809+0000","last_undegraded":"2024-09-21T18:54:00.012809+0000","last_fullsized":"2024-09-21T18:54:00.012809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.041097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866461+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:53:59.866461+0000","last_peered":"2024-09-21T18:53:59.866461+0000","last_clean":"2024-09-21T18:53:59.866461+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:53:59.866461+0000","last_undegraded":"2024-09-21T18:53:59.866461+0000","last_fullsized":"2024-09-21T18:53:59.866461+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.852227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011703+0000","last_change":"2024-09-21T18:53:11.413440+0000","last_active":"2024-09-21T18:54:00.011703+0000","last_peered":"2024-09-21T18:54:00.011703+0000","last_clean":"2024-09-21T18:54:00.011703+0000","last_became_active":"2024-09-21T18:53:11.413142+0000","last_became_peered":"2024-09-21T18:53:11.413142+0000","last_unstale":"2024-09-21T18:54:00.011703+0000","last_undegraded":"2024-09-21T18:54:00.011703+0000","last_fullsized":"2024-09-21T18:54:00.011703+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.886861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865822+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:53:59.865822+0000","last_peered":"2024-09-21T18:53:59.865822+0000","last_clean":"2024-09-21T18:53:59.865822+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:53:59.865822+0000","last_undegraded":"2024-09-21T18:53:59.865822+0000","last_fullsized":"2024-09-21T18:53:59.865822+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.894993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":66,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434887+0000","last_change":"2024-09-21T18:53:09.406428+0000","last_active":"2024-09-21T18:53:59.434887+0000","last_peered":"2024-09-21T18:53:59.434887+0000","last_clean":"2024-09-21T18:53:59.434887+0000","last_became_active":"2024-09-21T18:53:09.406030+0000","last_became_peered":"2024-09-21T18:53:09.406030+0000","last_unstale":"2024-09-21T18:53:59.434887+0000","last_undegraded":"2024-09-21T18:53:59.434887+0000","last_fullsized":"2024-09-21T18:53:59.434887+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.045459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866571+0000","last_change":"2024-09-21T18:53:07.400289+0000","last_active":"2024-09-21T18:53:59.866571+0000","last_peered":"2024-09-21T18:53:59.866571+0000","last_clean":"2024-09-21T18:53:59.866571+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:53:59.866571+0000","last_undegraded":"2024-09-21T18:53:59.866571+0000","last_fullsized":"2024-09-21T18:53:59.866571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:06.969808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011546+0000","last_change":"2024-09-21T18:53:11.417570+0000","last_active":"2024-09-21T18:54:00.011546+0000","last_peered":"2024-09-21T18:54:00.011546+0000","last_clean":"2024-09-21T18:54:00.011546+0000","last_became_active":"2024-09-21T18:53:11.417277+0000","last_became_peered":"2024-09-21T18:53:11.417277+0000","last_unstale":"2024-09-21T18:54:00.011546+0000","last_undegraded":"2024-09-21T18:54:00.011546+0000","last_fullsized":"2024-09-21T18:54:00.011546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.958604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435606+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:53:59.435606+0000","last_peered":"2024-09-21T18:53:59.435606+0000","last_clean":"2024-09-21T18:53:59.435606+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:53:59.435606+0000","last_undegraded":"2024-09-21T18:53:59.435606+0000","last_fullsized":"2024-09-21T18:53:59.435606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.706132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011989+0000","last_change":"2024-09-21T18:53:09.717860+0000","last_active":"2024-09-21T18:54:00.011989+0000","last_peered":"2024-09-21T18:54:00.011989+0000","last_clean":"2024-09-21T18:54:00.011989+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:54:00.011989+0000","last_undegraded":"2024-09-21T18:54:00.011989+0000","last_fullsized":"2024-09-21T18:54:00.011989+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.363297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866634+0000","last_change":"2024-09-21T18:53:07.405281+0000","last_active":"2024-09-21T18:53:59.866634+0000","last_peered":"2024-09-21T18:53:59.866634+0000","last_clean":"2024-09-21T18:53:59.866634+0000","last_became_active":"2024-09-21T18:53:07.404988+0000","last_became_peered":"2024-09-21T18:53:07.404988+0000","last_unstale":"2024-09-21T18:53:59.866634+0000","last_undegraded":"2024-09-21T18:53:59.866634+0000","last_fullsized":"2024-09-21T18:53:59.866634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.141596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865814+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:53:59.865814+0000","last_peered":"2024-09-21T18:53:59.865814+0000","last_clean":"2024-09-21T18:53:59.865814+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:53:59.865814+0000","last_undegraded":"2024-09-21T18:53:59.865814+0000","last_fullsized":"2024-09-21T18:53:59.865814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.640908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012110+0000","last_change":"2024-09-21T18:53:13.657306+0000","last_active":"2024-09-21T18:54:00.012110+0000","last_peered":"2024-09-21T18:54:00.012110+0000","last_clean":"2024-09-21T18:54:00.012110+0000","last_became_active":"2024-09-21T18:53:13.656931+0000","last_became_peered":"2024-09-21T18:53:13.656931+0000","last_unstale":"2024-09-21T18:54:00.012110+0000","last_undegraded":"2024-09-21T18:54:00.012110+0000","last_fullsized":"2024-09-21T18:54:00.012110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:42.145553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012274+0000","last_change":"2024-09-21T18:53:07.397798+0000","last_active":"2024-09-21T18:54:00.012274+0000","last_peered":"2024-09-21T18:54:00.012274+0000","last_clean":"2024-09-21T18:54:00.012274+0000","last_became_active":"2024-09-21T18:53:07.397469+0000","last_became_peered":"2024-09-21T18:53:07.397469+0000","last_unstale":"2024-09-21T18:54:00.012274+0000","last_undegraded":"2024-09-21T18:54:00.012274+0000","last_fullsized":"2024-09-21T18:54:00.012274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.024888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437363+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:53:59.437363+0000","last_peered":"2024-09-21T18:53:59.437363+0000","last_clean":"2024-09-21T18:53:59.437363+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:53:59.437363+0000","last_undegraded":"2024-09-21T18:53:59.437363+0000","last_fullsized":"2024-09-21T18:53:59.437363+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.137675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013137+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:54:00.013137+0000","last_peered":"2024-09-21T18:54:00.013137+0000","last_clean":"2024-09-21T18:54:00.013137+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:54:00.013137+0000","last_undegraded":"2024-09-21T18:54:00.013137+0000","last_fullsized":"2024-09-21T18:54:00.013137+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.109044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866097+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:53:59.866097+0000","last_peered":"2024-09-21T18:53:59.866097+0000","last_clean":"2024-09-21T18:53:59.866097+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:53:59.866097+0000","last_undegraded":"2024-09-21T18:53:59.866097+0000","last_fullsized":"2024-09-21T18:53:59.866097+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012565+0000","last_change":"2024-09-21T18:53:09.720374+0000","last_active":"2024-09-21T18:54:00.012565+0000","last_peered":"2024-09-21T18:54:00.012565+0000","last_clean":"2024-09-21T18:54:00.012565+0000","last_became_active":"2024-09-21T18:53:09.720056+0000","last_became_peered":"2024-09-21T18:53:09.720056+0000","last_unstale":"2024-09-21T18:54:00.012565+0000","last_undegraded":"2024-09-21T18:54:00.012565+0000","last_fullsized":"2024-09-21T18:54:00.012565+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.445280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866014+0000","last_change":"2024-09-21T18:53:07.406957+0000","last_active":"2024-09-21T18:53:59.866014+0000","last_peered":"2024-09-21T18:53:59.866014+0000","last_clean":"2024-09-21T18:53:59.866014+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:53:59.866014+0000","last_undegraded":"2024-09-21T18:53:59.866014+0000","last_fullsized":"2024-09-21T18:53:59.866014+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.565944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437277+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:53:59.437277+0000","last_peered":"2024-09-21T18:53:59.437277+0000","last_clean":"2024-09-21T18:53:59.437277+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:53:59.437277+0000","last_undegraded":"2024-09-21T18:53:59.437277+0000","last_fullsized":"2024-09-21T18:53:59.437277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.484261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011873+0000","last_change":"2024-09-21T18:53:13.423168+0000","last_active":"2024-09-21T18:54:00.011873+0000","last_peered":"2024-09-21T18:54:00.011873+0000","last_clean":"2024-09-21T18:54:00.011873+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:54:00.011873+0000","last_undegraded":"2024-09-21T18:54:00.011873+0000","last_fullsized":"2024-09-21T18:54:00.011873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.401268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012910+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:54:00.012910+0000","last_peered":"2024-09-21T18:54:00.012910+0000","last_clean":"2024-09-21T18:54:00.012910+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:54:00.012910+0000","last_undegraded":"2024-09-21T18:54:00.012910+0000","last_fullsized":"2024-09-21T18:54:00.012910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.011518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012269+0000","last_change":"2024-09-21T18:53:07.405516+0000","last_active":"2024-09-21T18:54:00.012269+0000","last_peered":"2024-09-21T18:54:00.012269+0000","last_clean":"2024-09-21T18:54:00.012269+0000","last_became_active":"2024-09-21T18:53:07.405223+0000","last_became_peered":"2024-09-21T18:53:07.405223+0000","last_unstale":"2024-09-21T18:54:00.012269+0000","last_undegraded":"2024-09-21T18:54:00.012269+0000","last_fullsized":"2024-09-21T18:54:00.012269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.984977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435507+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:53:59.435507+0000","last_peered":"2024-09-21T18:53:59.435507+0000","last_clean":"2024-09-21T18:53:59.435507+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:53:59.435507+0000","last_undegraded":"2024-09-21T18:53:59.435507+0000","last_fullsized":"2024-09-21T18:53:59.435507+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.168997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435722+0000","last_change":"2024-09-21T18:53:13.423826+0000","last_active":"2024-09-21T18:53:59.435722+0000","last_peered":"2024-09-21T18:53:59.435722+0000","last_clean":"2024-09-21T18:53:59.435722+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:53:59.435722+0000","last_undegraded":"2024-09-21T18:53:59.435722+0000","last_fullsized":"2024-09-21T18:53:59.435722+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.967749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435766+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:53:59.435766+0000","last_peered":"2024-09-21T18:53:59.435766+0000","last_clean":"2024-09-21T18:53:59.435766+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:53:59.435766+0000","last_undegraded":"2024-09-21T18:53:59.435766+0000","last_fullsized":"2024-09-21T18:53:59.435766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.620456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866242+0000","last_change":"2024-09-21T18:53:07.403003+0000","last_active":"2024-09-21T18:53:59.866242+0000","last_peered":"2024-09-21T18:53:59.866242+0000","last_clean":"2024-09-21T18:53:59.866242+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:53:59.866242+0000","last_undegraded":"2024-09-21T18:53:59.866242+0000","last_fullsized":"2024-09-21T18:53:59.866242+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.651586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012774+0000","last_change":"2024-09-21T18:53:11.411052+0000","last_active":"2024-09-21T18:54:00.012774+0000","last_peered":"2024-09-21T18:54:00.012774+0000","last_clean":"2024-09-21T18:54:00.012774+0000","last_became_active":"2024-09-21T18:53:11.410769+0000","last_became_peered":"2024-09-21T18:53:11.410769+0000","last_unstale":"2024-09-21T18:54:00.012774+0000","last_undegraded":"2024-09-21T18:54:00.012774+0000","last_fullsized":"2024-09-21T18:54:00.012774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012174+0000","last_change":"2024-09-21T18:53:13.431033+0000","last_active":"2024-09-21T18:54:00.012174+0000","last_peered":"2024-09-21T18:54:00.012174+0000","last_clean":"2024-09-21T18:54:00.012174+0000","last_became_active":"2024-09-21T18:53:13.430667+0000","last_became_peered":"2024-09-21T18:53:13.430667+0000","last_unstale":"2024-09-21T18:54:00.012174+0000","last_undegraded":"2024-09-21T18:54:00.012174+0000","last_fullsized":"2024-09-21T18:54:00.012174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.681228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":62,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435007+0000","last_change":"2024-09-21T18:53:09.407481+0000","last_active":"2024-09-21T18:53:59.435007+0000","last_peered":"2024-09-21T18:53:59.435007+0000","last_clean":"2024-09-21T18:53:59.435007+0000","last_became_active":"2024-09-21T18:53:09.407184+0000","last_became_peered":"2024-09-21T18:53:09.407184+0000","last_unstale":"2024-09-21T18:53:59.435007+0000","last_undegraded":"2024-09-21T18:53:59.435007+0000","last_fullsized":"2024-09-21T18:53:59.435007+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.470465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435463+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:53:59.435463+0000","last_peered":"2024-09-21T18:53:59.435463+0000","last_clean":"2024-09-21T18:53:59.435463+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:53:59.435463+0000","last_undegraded":"2024-09-21T18:53:59.435463+0000","last_fullsized":"2024-09-21T18:53:59.435463+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.908635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436495+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:53:59.436495+0000","last_peered":"2024-09-21T18:53:59.436495+0000","last_clean":"2024-09-21T18:53:59.436495+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:53:59.436495+0000","last_undegraded":"2024-09-21T18:53:59.436495+0000","last_fullsized":"2024-09-21T18:53:59.436495+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.492927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435472+0000","last_change":"2024-09-21T18:53:13.433363+0000","last_active":"2024-09-21T18:53:59.435472+0000","last_peered":"2024-09-21T18:53:59.435472+0000","last_clean":"2024-09-21T18:53:59.435472+0000","last_became_active":"2024-09-21T18:53:13.433057+0000","last_became_peered":"2024-09-21T18:53:13.433057+0000","last_unstale":"2024-09-21T18:53:59.435472+0000","last_undegraded":"2024-09-21T18:53:59.435472+0000","last_fullsized":"2024-09-21T18:53:59.435472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.215327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437612+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:53:59.437612+0000","last_peered":"2024-09-21T18:53:59.437612+0000","last_clean":"2024-09-21T18:53:59.437612+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:53:59.437612+0000","last_undegraded":"2024-09-21T18:53:59.437612+0000","last_fullsized":"2024-09-21T18:53:59.437612+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.126928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866331+0000","last_change":"2024-09-21T18:53:07.406445+0000","last_active":"2024-09-21T18:53:59.866331+0000","last_peered":"2024-09-21T18:53:59.866331+0000","last_clean":"2024-09-21T18:53:59.866331+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:53:59.866331+0000","last_undegraded":"2024-09-21T18:53:59.866331+0000","last_fullsized":"2024-09-21T18:53:59.866331+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.278365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435693+0000","last_change":"2024-09-21T18:53:11.411055+0000","last_active":"2024-09-21T18:53:59.435693+0000","last_peered":"2024-09-21T18:53:59.435693+0000","last_clean":"2024-09-21T18:53:59.435693+0000","last_became_active":"2024-09-21T18:53:11.410775+0000","last_became_peered":"2024-09-21T18:53:11.410775+0000","last_unstale":"2024-09-21T18:53:59.435693+0000","last_undegraded":"2024-09-21T18:53:59.435693+0000","last_fullsized":"2024-09-21T18:53:59.435693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.614499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011507+0000","last_change":"2024-09-21T18:53:13.433872+0000","last_active":"2024-09-21T18:54:00.011507+0000","last_peered":"2024-09-21T18:54:00.011507+0000","last_clean":"2024-09-21T18:54:00.011507+0000","last_became_active":"2024-09-21T18:53:13.433534+0000","last_became_peered":"2024-09-21T18:53:13.433534+0000","last_unstale":"2024-09-21T18:54:00.011507+0000","last_undegraded":"2024-09-21T18:54:00.011507+0000","last_fullsized":"2024-09-21T18:54:00.011507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.158748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436094+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:53:59.436094+0000","last_peered":"2024-09-21T18:53:59.436094+0000","last_clean":"2024-09-21T18:53:59.436094+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:53:59.436094+0000","last_undegraded":"2024-09-21T18:53:59.436094+0000","last_fullsized":"2024-09-21T18:53:59.436094+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.024213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013223+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:54:00.013223+0000","last_peered":"2024-09-21T18:54:00.013223+0000","last_clean":"2024-09-21T18:54:00.013223+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:54:00.013223+0000","last_undegraded":"2024-09-21T18:54:00.013223+0000","last_fullsized":"2024-09-21T18:54:00.013223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.249689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865569+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:53:59.865569+0000","last_peered":"2024-09-21T18:53:59.865569+0000","last_clean":"2024-09-21T18:53:59.865569+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:53:59.865569+0000","last_undegraded":"2024-09-21T18:53:59.865569+0000","last_fullsized":"2024-09-21T18:53:59.865569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.908774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435826+0000","last_change":"2024-09-21T18:53:13.658591+0000","last_active":"2024-09-21T18:53:59.435826+0000","last_peered":"2024-09-21T18:53:59.435826+0000","last_clean":"2024-09-21T18:53:59.435826+0000","last_became_active":"2024-09-21T18:53:13.657938+0000","last_became_peered":"2024-09-21T18:53:13.657938+0000","last_unstale":"2024-09-21T18:53:59.435826+0000","last_undegraded":"2024-09-21T18:53:59.435826+0000","last_fullsized":"2024-09-21T18:53:59.435826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.163794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436433+0000","last_change":"2024-09-21T18:53:09.409585+0000","last_active":"2024-09-21T18:53:59.436433+0000","last_peered":"2024-09-21T18:53:59.436433+0000","last_clean":"2024-09-21T18:53:59.436433+0000","last_became_active":"2024-09-21T18:53:09.409216+0000","last_became_peered":"2024-09-21T18:53:09.409216+0000","last_unstale":"2024-09-21T18:53:59.436433+0000","last_undegraded":"2024-09-21T18:53:59.436433+0000","last_fullsized":"2024-09-21T18:53:59.436433+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012633+0000","last_change":"2024-09-21T18:53:07.401813+0000","last_active":"2024-09-21T18:54:00.012633+0000","last_peered":"2024-09-21T18:54:00.012633+0000","last_clean":"2024-09-21T18:54:00.012633+0000","last_became_active":"2024-09-21T18:53:07.401545+0000","last_became_peered":"2024-09-21T18:53:07.401545+0000","last_unstale":"2024-09-21T18:54:00.012633+0000","last_undegraded":"2024-09-21T18:54:00.012633+0000","last_fullsized":"2024-09-21T18:54:00.012633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.528471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012467+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:54:00.012467+0000","last_peered":"2024-09-21T18:54:00.012467+0000","last_clean":"2024-09-21T18:54:00.012467+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:54:00.012467+0000","last_undegraded":"2024-09-21T18:54:00.012467+0000","last_fullsized":"2024-09-21T18:54:00.012467+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.190087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865778+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:53:59.865778+0000","last_peered":"2024-09-21T18:53:59.865778+0000","last_clean":"2024-09-21T18:53:59.865778+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:53:59.865778+0000","last_undegraded":"2024-09-21T18:53:59.865778+0000","last_fullsized":"2024-09-21T18:53:59.865778+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436308+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:53:59.436308+0000","last_peered":"2024-09-21T18:53:59.436308+0000","last_clean":"2024-09-21T18:53:59.436308+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:53:59.436308+0000","last_undegraded":"2024-09-21T18:53:59.436308+0000","last_fullsized":"2024-09-21T18:53:59.436308+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.931032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436630+0000","last_change":"2024-09-21T18:53:07.400561+0000","last_active":"2024-09-21T18:53:59.436630+0000","last_peered":"2024-09-21T18:53:59.436630+0000","last_clean":"2024-09-21T18:53:59.436630+0000","last_became_active":"2024-09-21T18:53:07.400254+0000","last_became_peered":"2024-09-21T18:53:07.400254+0000","last_unstale":"2024-09-21T18:53:59.436630+0000","last_undegraded":"2024-09-21T18:53:59.436630+0000","last_fullsized":"2024-09-21T18:53:59.436630+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.077186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013566+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:54:00.013566+0000","last_peered":"2024-09-21T18:54:00.013566+0000","last_clean":"2024-09-21T18:54:00.013566+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:54:00.013566+0000","last_undegraded":"2024-09-21T18:54:00.013566+0000","last_fullsized":"2024-09-21T18:54:00.013566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.130619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012401+0000","last_change":"2024-09-21T18:53:11.415531+0000","last_active":"2024-09-21T18:54:00.012401+0000","last_peered":"2024-09-21T18:54:00.012401+0000","last_clean":"2024-09-21T18:54:00.012401+0000","last_became_active":"2024-09-21T18:53:11.415209+0000","last_became_peered":"2024-09-21T18:53:11.415209+0000","last_unstale":"2024-09-21T18:54:00.012401+0000","last_undegraded":"2024-09-21T18:54:00.012401+0000","last_fullsized":"2024-09-21T18:54:00.012401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.845440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266703,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1376256,"data_stored":904047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528340,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712636,"statfs":{"total":95995035648,"available":95961739264,"internally_reserved":0,"allocated":1359872,"data_stored":903398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789975,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":761856,"data_stored":312940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018909,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":790528,"data_stored":314179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":802816,"data_stored":314478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":757760,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":827392,"data_stored":314802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1380352,"data_stored":904736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:54:04.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:54:04.489 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:04.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:04 smithi104 ceph-mon[32438]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:04 smithi104 ceph-mon[32438]: from='client.14805 v1:172.21.15.81:0/3797548799' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:04 smithi081 ceph-mon[39192]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:04.646 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:04 smithi081 ceph-mon[39192]: from='client.14805 v1:172.21.15.81:0/3797548799' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:04.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:04 smithi081 ceph-mon[29384]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:04.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:04 smithi081 ceph-mon[29384]: from='client.14805 v1:172.21.15.81:0/3797548799' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:05.148 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:54:05.149 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:54:05.456 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:05 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:05.456 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:05 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:05.456 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:05 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:05.456 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:05 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:05.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:05 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:05.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:05 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:05.920 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-21T18:54:03.374317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":897,"num_read_kb":754,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240732,"kb_used_data":7868,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720484,"statfs":{"total":767960285184,"available":767713775616,"internally_reserved":0,"allocated":8056832,"data_stored":4281745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":238276148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001914"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012542+0000","last_change":"2024-09-21T18:53:07.405054+0000","last_active":"2024-09-21T18:54:00.012542+0000","last_peered":"2024-09-21T18:54:00.012542+0000","last_clean":"2024-09-21T18:54:00.012542+0000","last_became_active":"2024-09-21T18:53:07.404774+0000","last_became_peered":"2024-09-21T18:53:07.404774+0000","last_unstale":"2024-09-21T18:54:00.012542+0000","last_undegraded":"2024-09-21T18:54:00.012542+0000","last_fullsized":"2024-09-21T18:54:00.012542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.853378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434676+0000","last_change":"2024-09-21T18:53:09.412508+0000","last_active":"2024-09-21T18:53:59.434676+0000","last_peered":"2024-09-21T18:53:59.434676+0000","last_clean":"2024-09-21T18:53:59.434676+0000","last_became_active":"2024-09-21T18:53:09.412193+0000","last_became_peered":"2024-09-21T18:53:09.412193+0000","last_unstale":"2024-09-21T18:53:59.434676+0000","last_undegraded":"2024-09-21T18:53:59.434676+0000","last_fullsized":"2024-09-21T18:53:59.434676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435852+0000","last_change":"2024-09-21T18:53:11.410506+0000","last_active":"2024-09-21T18:53:59.435852+0000","last_peered":"2024-09-21T18:53:59.435852+0000","last_clean":"2024-09-21T18:53:59.435852+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:53:59.435852+0000","last_undegraded":"2024-09-21T18:53:59.435852+0000","last_fullsized":"2024-09-21T18:53:59.435852+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.829493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434784+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:53:59.434784+0000","last_peered":"2024-09-21T18:53:59.434784+0000","last_clean":"2024-09-21T18:53:59.434784+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:53:59.434784+0000","last_undegraded":"2024-09-21T18:53:59.434784+0000","last_fullsized":"2024-09-21T18:53:59.434784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435518+0000","last_change":"2024-09-21T18:53:13.658931+0000","last_active":"2024-09-21T18:53:59.435518+0000","last_peered":"2024-09-21T18:53:59.435518+0000","last_clean":"2024-09-21T18:53:59.435518+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:53:59.435518+0000","last_undegraded":"2024-09-21T18:53:59.435518+0000","last_fullsized":"2024-09-21T18:53:59.435518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.480075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437152+0000","last_change":"2024-09-21T18:53:07.390902+0000","last_active":"2024-09-21T18:53:59.437152+0000","last_peered":"2024-09-21T18:53:59.437152+0000","last_clean":"2024-09-21T18:53:59.437152+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:53:59.437152+0000","last_undegraded":"2024-09-21T18:53:59.437152+0000","last_fullsized":"2024-09-21T18:53:59.437152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.326332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":69,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436300+0000","last_change":"2024-09-21T18:53:09.407061+0000","last_active":"2024-09-21T18:53:59.436300+0000","last_peered":"2024-09-21T18:53:59.436300+0000","last_clean":"2024-09-21T18:53:59.436300+0000","last_became_active":"2024-09-21T18:53:09.406774+0000","last_became_peered":"2024-09-21T18:53:09.406774+0000","last_unstale":"2024-09-21T18:53:59.436300+0000","last_undegraded":"2024-09-21T18:53:59.436300+0000","last_fullsized":"2024-09-21T18:53:59.436300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.239535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434654+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:53:59.434654+0000","last_peered":"2024-09-21T18:53:59.434654+0000","last_clean":"2024-09-21T18:53:59.434654+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:53:59.434654+0000","last_undegraded":"2024-09-21T18:53:59.434654+0000","last_fullsized":"2024-09-21T18:53:59.434654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.141973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012916+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:54:00.012916+0000","last_peered":"2024-09-21T18:54:00.012916+0000","last_clean":"2024-09-21T18:54:00.012916+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:54:00.012916+0000","last_undegraded":"2024-09-21T18:54:00.012916+0000","last_fullsized":"2024-09-21T18:54:00.012916+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.801164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435060+0000","last_change":"2024-09-21T18:53:09.401868+0000","last_active":"2024-09-21T18:53:59.435060+0000","last_peered":"2024-09-21T18:53:59.435060+0000","last_clean":"2024-09-21T18:53:59.435060+0000","last_became_active":"2024-09-21T18:53:09.401547+0000","last_became_peered":"2024-09-21T18:53:09.401547+0000","last_unstale":"2024-09-21T18:53:59.435060+0000","last_undegraded":"2024-09-21T18:53:59.435060+0000","last_fullsized":"2024-09-21T18:53:59.435060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.067009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012314+0000","last_change":"2024-09-21T18:53:11.409118+0000","last_active":"2024-09-21T18:54:00.012314+0000","last_peered":"2024-09-21T18:54:00.012314+0000","last_clean":"2024-09-21T18:54:00.012314+0000","last_became_active":"2024-09-21T18:53:11.408537+0000","last_became_peered":"2024-09-21T18:53:11.408537+0000","last_unstale":"2024-09-21T18:54:00.012314+0000","last_undegraded":"2024-09-21T18:54:00.012314+0000","last_fullsized":"2024-09-21T18:54:00.012314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.248524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011582+0000","last_change":"2024-09-21T18:53:13.429374+0000","last_active":"2024-09-21T18:54:00.011582+0000","last_peered":"2024-09-21T18:54:00.011582+0000","last_clean":"2024-09-21T18:54:00.011582+0000","last_became_active":"2024-09-21T18:53:13.429007+0000","last_became_peered":"2024-09-21T18:53:13.429007+0000","last_unstale":"2024-09-21T18:54:00.011582+0000","last_undegraded":"2024-09-21T18:54:00.011582+0000","last_fullsized":"2024-09-21T18:54:00.011582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.767711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012641+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:54:00.012641+0000","last_peered":"2024-09-21T18:54:00.012641+0000","last_clean":"2024-09-21T18:54:00.012641+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:54:00.012641+0000","last_undegraded":"2024-09-21T18:54:00.012641+0000","last_fullsized":"2024-09-21T18:54:00.012641+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435065+0000","last_change":"2024-09-21T18:53:09.406568+0000","last_active":"2024-09-21T18:53:59.435065+0000","last_peered":"2024-09-21T18:53:59.435065+0000","last_clean":"2024-09-21T18:53:59.435065+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:53:59.435065+0000","last_undegraded":"2024-09-21T18:53:59.435065+0000","last_fullsized":"2024-09-21T18:53:59.435065+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865775+0000","last_change":"2024-09-21T18:53:11.412969+0000","last_active":"2024-09-21T18:53:59.865775+0000","last_peered":"2024-09-21T18:53:59.865775+0000","last_clean":"2024-09-21T18:53:59.865775+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:53:59.865775+0000","last_undegraded":"2024-09-21T18:53:59.865775+0000","last_fullsized":"2024-09-21T18:53:59.865775+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.874706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012170+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:54:00.012170+0000","last_peered":"2024-09-21T18:54:00.012170+0000","last_clean":"2024-09-21T18:54:00.012170+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:54:00.012170+0000","last_undegraded":"2024-09-21T18:54:00.012170+0000","last_fullsized":"2024-09-21T18:54:00.012170+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.945242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435317+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:53:59.435317+0000","last_peered":"2024-09-21T18:53:59.435317+0000","last_clean":"2024-09-21T18:53:59.435317+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:53:59.435317+0000","last_undegraded":"2024-09-21T18:53:59.435317+0000","last_fullsized":"2024-09-21T18:53:59.435317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.198560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012651+0000","last_change":"2024-09-21T18:53:07.405471+0000","last_active":"2024-09-21T18:54:00.012651+0000","last_peered":"2024-09-21T18:54:00.012651+0000","last_clean":"2024-09-21T18:54:00.012651+0000","last_became_active":"2024-09-21T18:53:07.405177+0000","last_became_peered":"2024-09-21T18:53:07.405177+0000","last_unstale":"2024-09-21T18:54:00.012651+0000","last_undegraded":"2024-09-21T18:54:00.012651+0000","last_fullsized":"2024-09-21T18:54:00.012651+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.483473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012086+0000","last_change":"2024-09-21T18:53:09.407164+0000","last_active":"2024-09-21T18:54:00.012086+0000","last_peered":"2024-09-21T18:54:00.012086+0000","last_clean":"2024-09-21T18:54:00.012086+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:54:00.012086+0000","last_undegraded":"2024-09-21T18:54:00.012086+0000","last_fullsized":"2024-09-21T18:54:00.012086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.948438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436967+0000","last_change":"2024-09-21T18:53:11.411989+0000","last_active":"2024-09-21T18:53:59.436967+0000","last_peered":"2024-09-21T18:53:59.436967+0000","last_clean":"2024-09-21T18:53:59.436967+0000","last_became_active":"2024-09-21T18:53:11.411624+0000","last_became_peered":"2024-09-21T18:53:11.411624+0000","last_unstale":"2024-09-21T18:53:59.436967+0000","last_undegraded":"2024-09-21T18:53:59.436967+0000","last_fullsized":"2024-09-21T18:53:59.436967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.979898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437258+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:53:59.437258+0000","last_peered":"2024-09-21T18:53:59.437258+0000","last_clean":"2024-09-21T18:53:59.437258+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:53:59.437258+0000","last_undegraded":"2024-09-21T18:53:59.437258+0000","last_fullsized":"2024-09-21T18:53:59.437258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.301610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435473+0000","last_change":"2024-09-21T18:53:07.397249+0000","last_active":"2024-09-21T18:53:59.435473+0000","last_peered":"2024-09-21T18:53:59.435473+0000","last_clean":"2024-09-21T18:53:59.435473+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:53:59.435473+0000","last_undegraded":"2024-09-21T18:53:59.435473+0000","last_fullsized":"2024-09-21T18:53:59.435473+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.232024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436768+0000","last_change":"2024-09-21T18:53:09.406892+0000","last_active":"2024-09-21T18:53:59.436768+0000","last_peered":"2024-09-21T18:53:59.436768+0000","last_clean":"2024-09-21T18:53:59.436768+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:53:59.436768+0000","last_undegraded":"2024-09-21T18:53:59.436768+0000","last_fullsized":"2024-09-21T18:53:59.436768+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.459852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435171+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:53:59.435171+0000","last_peered":"2024-09-21T18:53:59.435171+0000","last_clean":"2024-09-21T18:53:59.435171+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:53:59.435171+0000","last_undegraded":"2024-09-21T18:53:59.435171+0000","last_fullsized":"2024-09-21T18:53:59.435171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.060686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866021+0000","last_change":"2024-09-21T18:53:13.426725+0000","last_active":"2024-09-21T18:53:59.866021+0000","last_peered":"2024-09-21T18:53:59.866021+0000","last_clean":"2024-09-21T18:53:59.866021+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:53:59.866021+0000","last_undegraded":"2024-09-21T18:53:59.866021+0000","last_fullsized":"2024-09-21T18:53:59.866021+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436567+0000","last_change":"2024-09-21T18:53:07.397345+0000","last_active":"2024-09-21T18:53:59.436567+0000","last_peered":"2024-09-21T18:53:59.436567+0000","last_clean":"2024-09-21T18:53:59.436567+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:53:59.436567+0000","last_undegraded":"2024-09-21T18:53:59.436567+0000","last_fullsized":"2024-09-21T18:53:59.436567+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.808378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012446+0000","last_change":"2024-09-21T18:53:09.719399+0000","last_active":"2024-09-21T18:54:00.012446+0000","last_peered":"2024-09-21T18:54:00.012446+0000","last_clean":"2024-09-21T18:54:00.012446+0000","last_became_active":"2024-09-21T18:53:09.719059+0000","last_became_peered":"2024-09-21T18:53:09.719059+0000","last_unstale":"2024-09-21T18:54:00.012446+0000","last_undegraded":"2024-09-21T18:54:00.012446+0000","last_fullsized":"2024-09-21T18:54:00.012446+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.854862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013562+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:54:00.013562+0000","last_peered":"2024-09-21T18:54:00.013562+0000","last_clean":"2024-09-21T18:54:00.013562+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:54:00.013562+0000","last_undegraded":"2024-09-21T18:54:00.013562+0000","last_fullsized":"2024-09-21T18:54:00.013562+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.329971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437030+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:53:59.437030+0000","last_peered":"2024-09-21T18:53:59.437030+0000","last_clean":"2024-09-21T18:53:59.437030+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:53:59.437030+0000","last_undegraded":"2024-09-21T18:53:59.437030+0000","last_fullsized":"2024-09-21T18:53:59.437030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.373058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436699+0000","last_change":"2024-09-21T18:53:07.398833+0000","last_active":"2024-09-21T18:53:59.436699+0000","last_peered":"2024-09-21T18:53:59.436699+0000","last_clean":"2024-09-21T18:53:59.436699+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:53:59.436699+0000","last_undegraded":"2024-09-21T18:53:59.436699+0000","last_fullsized":"2024-09-21T18:53:59.436699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435434+0000","last_change":"2024-09-21T18:53:11.411518+0000","last_active":"2024-09-21T18:53:59.435434+0000","last_peered":"2024-09-21T18:53:59.435434+0000","last_clean":"2024-09-21T18:53:59.435434+0000","last_became_active":"2024-09-21T18:53:11.411212+0000","last_became_peered":"2024-09-21T18:53:11.411212+0000","last_unstale":"2024-09-21T18:53:59.435434+0000","last_undegraded":"2024-09-21T18:53:59.435434+0000","last_fullsized":"2024-09-21T18:53:59.435434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.137419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012670+0000","last_change":"2024-09-21T18:53:13.433767+0000","last_active":"2024-09-21T18:54:00.012670+0000","last_peered":"2024-09-21T18:54:00.012670+0000","last_clean":"2024-09-21T18:54:00.012670+0000","last_became_active":"2024-09-21T18:53:13.433575+0000","last_became_peered":"2024-09-21T18:53:13.433575+0000","last_unstale":"2024-09-21T18:54:00.012670+0000","last_undegraded":"2024-09-21T18:54:00.012670+0000","last_fullsized":"2024-09-21T18:54:00.012670+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.484363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":76,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436514+0000","last_change":"2024-09-21T18:53:09.719338+0000","last_active":"2024-09-21T18:53:59.436514+0000","last_peered":"2024-09-21T18:53:59.436514+0000","last_clean":"2024-09-21T18:53:59.436514+0000","last_became_active":"2024-09-21T18:53:09.719017+0000","last_became_peered":"2024-09-21T18:53:09.719017+0000","last_unstale":"2024-09-21T18:53:59.436514+0000","last_undegraded":"2024-09-21T18:53:59.436514+0000","last_fullsized":"2024-09-21T18:53:59.436514+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.426276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437469+0000","last_change":"2024-09-21T18:53:07.400418+0000","last_active":"2024-09-21T18:53:59.437469+0000","last_peered":"2024-09-21T18:53:59.437469+0000","last_clean":"2024-09-21T18:53:59.437469+0000","last_became_active":"2024-09-21T18:53:07.400143+0000","last_became_peered":"2024-09-21T18:53:07.400143+0000","last_unstale":"2024-09-21T18:53:59.437469+0000","last_undegraded":"2024-09-21T18:53:59.437469+0000","last_fullsized":"2024-09-21T18:53:59.437469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.737589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434684+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:53:59.434684+0000","last_peered":"2024-09-21T18:53:59.434684+0000","last_clean":"2024-09-21T18:53:59.434684+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:53:59.434684+0000","last_undegraded":"2024-09-21T18:53:59.434684+0000","last_fullsized":"2024-09-21T18:53:59.434684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.956086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434991+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:53:59.434991+0000","last_peered":"2024-09-21T18:53:59.434991+0000","last_clean":"2024-09-21T18:53:59.434991+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:53:59.434991+0000","last_undegraded":"2024-09-21T18:53:59.434991+0000","last_fullsized":"2024-09-21T18:53:59.434991+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.926924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":64,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434775+0000","last_change":"2024-09-21T18:53:09.400587+0000","last_active":"2024-09-21T18:53:59.434775+0000","last_peered":"2024-09-21T18:53:59.434775+0000","last_clean":"2024-09-21T18:53:59.434775+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:53:59.434775+0000","last_undegraded":"2024-09-21T18:53:59.434775+0000","last_fullsized":"2024-09-21T18:53:59.434775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.173359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012612+0000","last_change":"2024-09-21T18:53:07.404382+0000","last_active":"2024-09-21T18:54:00.012612+0000","last_peered":"2024-09-21T18:54:00.012612+0000","last_clean":"2024-09-21T18:54:00.012612+0000","last_became_active":"2024-09-21T18:53:07.404018+0000","last_became_peered":"2024-09-21T18:53:07.404018+0000","last_unstale":"2024-09-21T18:54:00.012612+0000","last_undegraded":"2024-09-21T18:54:00.012612+0000","last_fullsized":"2024-09-21T18:54:00.012612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.245821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435360+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:53:59.435360+0000","last_peered":"2024-09-21T18:53:59.435360+0000","last_clean":"2024-09-21T18:53:59.435360+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:53:59.435360+0000","last_undegraded":"2024-09-21T18:53:59.435360+0000","last_fullsized":"2024-09-21T18:53:59.435360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.339632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865864+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:53:59.865864+0000","last_peered":"2024-09-21T18:53:59.865864+0000","last_clean":"2024-09-21T18:53:59.865864+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:53:59.865864+0000","last_undegraded":"2024-09-21T18:53:59.865864+0000","last_fullsized":"2024-09-21T18:53:59.865864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.973293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436625+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:53:59.436625+0000","last_peered":"2024-09-21T18:53:59.436625+0000","last_clean":"2024-09-21T18:53:59.436625+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:53:59.436625+0000","last_undegraded":"2024-09-21T18:53:59.436625+0000","last_fullsized":"2024-09-21T18:53:59.436625+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.374498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013013+0000","last_change":"2024-09-21T18:53:07.403537+0000","last_active":"2024-09-21T18:54:00.013013+0000","last_peered":"2024-09-21T18:54:00.013013+0000","last_clean":"2024-09-21T18:54:00.013013+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:54:00.013013+0000","last_undegraded":"2024-09-21T18:54:00.013013+0000","last_fullsized":"2024-09-21T18:54:00.013013+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.558691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012374+0000","last_change":"2024-09-21T18:53:11.407531+0000","last_active":"2024-09-21T18:54:00.012374+0000","last_peered":"2024-09-21T18:54:00.012374+0000","last_clean":"2024-09-21T18:54:00.012374+0000","last_became_active":"2024-09-21T18:53:11.407244+0000","last_became_peered":"2024-09-21T18:53:11.407244+0000","last_unstale":"2024-09-21T18:54:00.012374+0000","last_undegraded":"2024-09-21T18:54:00.012374+0000","last_fullsized":"2024-09-21T18:54:00.012374+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.763939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011738+0000","last_change":"2024-09-21T18:53:13.435210+0000","last_active":"2024-09-21T18:54:00.011738+0000","last_peered":"2024-09-21T18:54:00.011738+0000","last_clean":"2024-09-21T18:54:00.011738+0000","last_became_active":"2024-09-21T18:53:13.434753+0000","last_became_peered":"2024-09-21T18:53:13.434753+0000","last_unstale":"2024-09-21T18:54:00.011738+0000","last_undegraded":"2024-09-21T18:54:00.011738+0000","last_fullsized":"2024-09-21T18:54:00.011738+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.794214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":79,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012487+0000","last_change":"2024-09-21T18:53:09.719454+0000","last_active":"2024-09-21T18:54:00.012487+0000","last_peered":"2024-09-21T18:54:00.012487+0000","last_clean":"2024-09-21T18:54:00.012487+0000","last_became_active":"2024-09-21T18:53:09.719114+0000","last_became_peered":"2024-09-21T18:53:09.719114+0000","last_unstale":"2024-09-21T18:54:00.012487+0000","last_undegraded":"2024-09-21T18:54:00.012487+0000","last_fullsized":"2024-09-21T18:54:00.012487+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.176379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436502+0000","last_change":"2024-09-21T18:53:07.400542+0000","last_active":"2024-09-21T18:53:59.436502+0000","last_peered":"2024-09-21T18:53:59.436502+0000","last_clean":"2024-09-21T18:53:59.436502+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:53:59.436502+0000","last_undegraded":"2024-09-21T18:53:59.436502+0000","last_fullsized":"2024-09-21T18:53:59.436502+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.007870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012485+0000","last_change":"2024-09-21T18:53:11.404992+0000","last_active":"2024-09-21T18:54:00.012485+0000","last_peered":"2024-09-21T18:54:00.012485+0000","last_clean":"2024-09-21T18:54:00.012485+0000","last_became_active":"2024-09-21T18:53:11.404577+0000","last_became_peered":"2024-09-21T18:53:11.404577+0000","last_unstale":"2024-09-21T18:54:00.012485+0000","last_undegraded":"2024-09-21T18:54:00.012485+0000","last_fullsized":"2024-09-21T18:54:00.012485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.773972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435754+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:53:59.435754+0000","last_peered":"2024-09-21T18:53:59.435754+0000","last_clean":"2024-09-21T18:53:59.435754+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:53:59.435754+0000","last_undegraded":"2024-09-21T18:53:59.435754+0000","last_fullsized":"2024-09-21T18:53:59.435754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.173798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":102,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436352+0000","last_change":"2024-09-21T18:53:09.410172+0000","last_active":"2024-09-21T18:53:59.436352+0000","last_peered":"2024-09-21T18:53:59.436352+0000","last_clean":"2024-09-21T18:53:59.436352+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:53:59.436352+0000","last_undegraded":"2024-09-21T18:53:59.436352+0000","last_fullsized":"2024-09-21T18:53:59.436352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:30.020223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435363+0000","last_change":"2024-09-21T18:53:07.403315+0000","last_active":"2024-09-21T18:53:59.435363+0000","last_peered":"2024-09-21T18:53:59.435363+0000","last_clean":"2024-09-21T18:53:59.435363+0000","last_became_active":"2024-09-21T18:53:07.403002+0000","last_became_peered":"2024-09-21T18:53:07.403002+0000","last_unstale":"2024-09-21T18:53:59.435363+0000","last_undegraded":"2024-09-21T18:53:59.435363+0000","last_fullsized":"2024-09-21T18:53:59.435363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:08.340495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012430+0000","last_change":"2024-09-21T18:53:09.398954+0000","last_active":"2024-09-21T18:54:00.012430+0000","last_peered":"2024-09-21T18:54:00.012430+0000","last_clean":"2024-09-21T18:54:00.012430+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:54:00.012430+0000","last_undegraded":"2024-09-21T18:54:00.012430+0000","last_fullsized":"2024-09-21T18:54:00.012430+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.467230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012383+0000","last_change":"2024-09-21T18:53:11.411504+0000","last_active":"2024-09-21T18:54:00.012383+0000","last_peered":"2024-09-21T18:54:00.012383+0000","last_clean":"2024-09-21T18:54:00.012383+0000","last_became_active":"2024-09-21T18:53:11.411229+0000","last_became_peered":"2024-09-21T18:53:11.411229+0000","last_unstale":"2024-09-21T18:54:00.012383+0000","last_undegraded":"2024-09-21T18:54:00.012383+0000","last_fullsized":"2024-09-21T18:54:00.012383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.568499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436995+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:53:59.436995+0000","last_peered":"2024-09-21T18:53:59.436995+0000","last_clean":"2024-09-21T18:53:59.436995+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:53:59.436995+0000","last_undegraded":"2024-09-21T18:53:59.436995+0000","last_fullsized":"2024-09-21T18:53:59.436995+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.888758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437318+0000","last_change":"2024-09-21T18:53:09.412601+0000","last_active":"2024-09-21T18:53:59.437318+0000","last_peered":"2024-09-21T18:53:59.437318+0000","last_clean":"2024-09-21T18:53:59.437318+0000","last_became_active":"2024-09-21T18:53:09.412304+0000","last_became_peered":"2024-09-21T18:53:09.412304+0000","last_unstale":"2024-09-21T18:53:59.437318+0000","last_undegraded":"2024-09-21T18:53:59.437318+0000","last_fullsized":"2024-09-21T18:53:59.437318+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.434125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436187+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:53:59.436187+0000","last_peered":"2024-09-21T18:53:59.436187+0000","last_clean":"2024-09-21T18:53:59.436187+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:53:59.436187+0000","last_undegraded":"2024-09-21T18:53:59.436187+0000","last_fullsized":"2024-09-21T18:53:59.436187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.025870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011742+0000","last_change":"2024-09-21T18:53:09.400372+0000","last_active":"2024-09-21T18:54:00.011742+0000","last_peered":"2024-09-21T18:54:00.011742+0000","last_clean":"2024-09-21T18:54:00.011742+0000","last_became_active":"2024-09-21T18:53:06.391866+0000","last_became_peered":"2024-09-21T18:53:06.391866+0000","last_unstale":"2024-09-21T18:54:00.011742+0000","last_undegraded":"2024-09-21T18:54:00.011742+0000","last_fullsized":"2024-09-21T18:54:00.011742+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.711363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011837+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:54:00.011837+0000","last_peered":"2024-09-21T18:54:00.011837+0000","last_clean":"2024-09-21T18:54:00.011837+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:54:00.011837+0000","last_undegraded":"2024-09-21T18:54:00.011837+0000","last_fullsized":"2024-09-21T18:54:00.011837+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.265652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866098+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:53:59.866098+0000","last_peered":"2024-09-21T18:53:59.866098+0000","last_clean":"2024-09-21T18:53:59.866098+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:53:59.866098+0000","last_undegraded":"2024-09-21T18:53:59.866098+0000","last_fullsized":"2024-09-21T18:53:59.866098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012495+0000","last_change":"2024-09-21T18:53:09.407641+0000","last_active":"2024-09-21T18:54:00.012495+0000","last_peered":"2024-09-21T18:54:00.012495+0000","last_clean":"2024-09-21T18:54:00.012495+0000","last_became_active":"2024-09-21T18:53:09.407302+0000","last_became_peered":"2024-09-21T18:53:09.407302+0000","last_unstale":"2024-09-21T18:54:00.012495+0000","last_undegraded":"2024-09-21T18:54:00.012495+0000","last_fullsized":"2024-09-21T18:54:00.012495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.070171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012047+0000","last_change":"2024-09-21T18:53:07.404328+0000","last_active":"2024-09-21T18:54:00.012047+0000","last_peered":"2024-09-21T18:54:00.012047+0000","last_clean":"2024-09-21T18:54:00.012047+0000","last_became_active":"2024-09-21T18:53:07.404060+0000","last_became_peered":"2024-09-21T18:53:07.404060+0000","last_unstale":"2024-09-21T18:54:00.012047+0000","last_undegraded":"2024-09-21T18:54:00.012047+0000","last_fullsized":"2024-09-21T18:54:00.012047+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.843533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":106,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865811+0000","last_change":"2024-09-21T18:53:09.719413+0000","last_active":"2024-09-21T18:53:59.865811+0000","last_peered":"2024-09-21T18:53:59.865811+0000","last_clean":"2024-09-21T18:53:59.865811+0000","last_became_active":"2024-09-21T18:53:06.394373+0000","last_became_peered":"2024-09-21T18:53:06.394373+0000","last_unstale":"2024-09-21T18:53:59.865811+0000","last_undegraded":"2024-09-21T18:53:59.865811+0000","last_fullsized":"2024-09-21T18:53:59.865811+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.196010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013416+0000","last_change":"2024-09-21T18:53:11.409878+0000","last_active":"2024-09-21T18:54:00.013416+0000","last_peered":"2024-09-21T18:54:00.013416+0000","last_clean":"2024-09-21T18:54:00.013416+0000","last_became_active":"2024-09-21T18:53:11.409637+0000","last_became_peered":"2024-09-21T18:53:11.409637+0000","last_unstale":"2024-09-21T18:54:00.013416+0000","last_undegraded":"2024-09-21T18:54:00.013416+0000","last_fullsized":"2024-09-21T18:54:00.013416+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.893970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436044+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:53:59.436044+0000","last_peered":"2024-09-21T18:53:59.436044+0000","last_clean":"2024-09-21T18:53:59.436044+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:53:59.436044+0000","last_undegraded":"2024-09-21T18:53:59.436044+0000","last_fullsized":"2024-09-21T18:53:59.436044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.377231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":75,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013301+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:54:00.013301+0000","last_peered":"2024-09-21T18:54:00.013301+0000","last_clean":"2024-09-21T18:54:00.013301+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:54:00.013301+0000","last_undegraded":"2024-09-21T18:54:00.013301+0000","last_fullsized":"2024-09-21T18:54:00.013301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.645863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435646+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:53:59.435646+0000","last_peered":"2024-09-21T18:53:59.435646+0000","last_clean":"2024-09-21T18:53:59.435646+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:53:59.435646+0000","last_undegraded":"2024-09-21T18:53:59.435646+0000","last_fullsized":"2024-09-21T18:53:59.435646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.707207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":91,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865887+0000","last_change":"2024-09-21T18:53:29.766304+0000","last_active":"2024-09-21T18:53:59.865887+0000","last_peered":"2024-09-21T18:53:59.865887+0000","last_clean":"2024-09-21T18:53:59.865887+0000","last_became_active":"2024-09-21T18:53:00.358105+0000","last_became_peered":"2024-09-21T18:53:00.358105+0000","last_unstale":"2024-09-21T18:53:59.865887+0000","last_undegraded":"2024-09-21T18:53:59.865887+0000","last_fullsized":"2024-09-21T18:53:59.865887+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:50.998311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866190+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:53:59.866190+0000","last_peered":"2024-09-21T18:53:59.866190+0000","last_clean":"2024-09-21T18:53:59.866190+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:53:59.866190+0000","last_undegraded":"2024-09-21T18:53:59.866190+0000","last_fullsized":"2024-09-21T18:53:59.866190+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.081383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013319+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:54:00.013319+0000","last_peered":"2024-09-21T18:54:00.013319+0000","last_clean":"2024-09-21T18:54:00.013319+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:54:00.013319+0000","last_undegraded":"2024-09-21T18:54:00.013319+0000","last_fullsized":"2024-09-21T18:54:00.013319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.641099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":62,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435553+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:53:59.435553+0000","last_peered":"2024-09-21T18:53:59.435553+0000","last_clean":"2024-09-21T18:53:59.435553+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:53:59.435553+0000","last_undegraded":"2024-09-21T18:53:59.435553+0000","last_fullsized":"2024-09-21T18:53:59.435553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.003282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435045+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:53:59.435045+0000","last_peered":"2024-09-21T18:53:59.435045+0000","last_clean":"2024-09-21T18:53:59.435045+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:53:59.435045+0000","last_undegraded":"2024-09-21T18:53:59.435045+0000","last_fullsized":"2024-09-21T18:53:59.435045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:21.491463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013256+0000","last_change":"2024-09-21T18:53:11.409004+0000","last_active":"2024-09-21T18:54:00.013256+0000","last_peered":"2024-09-21T18:54:00.013256+0000","last_clean":"2024-09-21T18:54:00.013256+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:54:00.013256+0000","last_undegraded":"2024-09-21T18:54:00.013256+0000","last_fullsized":"2024-09-21T18:54:00.013256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.557609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435394+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:53:59.435394+0000","last_peered":"2024-09-21T18:53:59.435394+0000","last_clean":"2024-09-21T18:53:59.435394+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:53:59.435394+0000","last_undegraded":"2024-09-21T18:53:59.435394+0000","last_fullsized":"2024-09-21T18:53:59.435394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.277061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":71,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435409+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:53:59.435409+0000","last_peered":"2024-09-21T18:53:59.435409+0000","last_clean":"2024-09-21T18:53:59.435409+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:53:59.435409+0000","last_undegraded":"2024-09-21T18:53:59.435409+0000","last_fullsized":"2024-09-21T18:53:59.435409+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.263644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012920+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:54:00.012920+0000","last_peered":"2024-09-21T18:54:00.012920+0000","last_clean":"2024-09-21T18:54:00.012920+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:54:00.012920+0000","last_undegraded":"2024-09-21T18:54:00.012920+0000","last_fullsized":"2024-09-21T18:54:00.012920+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.551877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436139+0000","last_change":"2024-09-21T18:53:11.409557+0000","last_active":"2024-09-21T18:53:59.436139+0000","last_peered":"2024-09-21T18:53:59.436139+0000","last_clean":"2024-09-21T18:53:59.436139+0000","last_became_active":"2024-09-21T18:53:11.409200+0000","last_became_peered":"2024-09-21T18:53:11.409200+0000","last_unstale":"2024-09-21T18:53:59.436139+0000","last_undegraded":"2024-09-21T18:53:59.436139+0000","last_fullsized":"2024-09-21T18:53:59.436139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:37.475931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012158+0000","last_change":"2024-09-21T18:53:13.430863+0000","last_active":"2024-09-21T18:54:00.012158+0000","last_peered":"2024-09-21T18:54:00.012158+0000","last_clean":"2024-09-21T18:54:00.012158+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:54:00.012158+0000","last_undegraded":"2024-09-21T18:54:00.012158+0000","last_fullsized":"2024-09-21T18:54:00.012158+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:38.403912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435160+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:53:59.435160+0000","last_peered":"2024-09-21T18:53:59.435160+0000","last_clean":"2024-09-21T18:53:59.435160+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:53:59.435160+0000","last_undegraded":"2024-09-21T18:53:59.435160+0000","last_fullsized":"2024-09-21T18:53:59.435160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.582865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435904+0000","last_change":"2024-09-21T18:53:07.400595+0000","last_active":"2024-09-21T18:53:59.435904+0000","last_peered":"2024-09-21T18:53:59.435904+0000","last_clean":"2024-09-21T18:53:59.435904+0000","last_became_active":"2024-09-21T18:53:07.399708+0000","last_became_peered":"2024-09-21T18:53:07.399708+0000","last_unstale":"2024-09-21T18:53:59.435904+0000","last_undegraded":"2024-09-21T18:53:59.435904+0000","last_fullsized":"2024-09-21T18:53:59.435904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:07.859688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011967+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:54:00.011967+0000","last_peered":"2024-09-21T18:54:00.011967+0000","last_clean":"2024-09-21T18:54:00.011967+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:54:00.011967+0000","last_undegraded":"2024-09-21T18:54:00.011967+0000","last_fullsized":"2024-09-21T18:54:00.011967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.955877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435155+0000","last_change":"2024-09-21T18:53:13.428208+0000","last_active":"2024-09-21T18:53:59.435155+0000","last_peered":"2024-09-21T18:53:59.435155+0000","last_clean":"2024-09-21T18:53:59.435155+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:53:59.435155+0000","last_undegraded":"2024-09-21T18:53:59.435155+0000","last_fullsized":"2024-09-21T18:53:59.435155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.939232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012441+0000","last_change":"2024-09-21T18:53:09.408663+0000","last_active":"2024-09-21T18:54:00.012441+0000","last_peered":"2024-09-21T18:54:00.012441+0000","last_clean":"2024-09-21T18:54:00.012441+0000","last_became_active":"2024-09-21T18:53:09.408315+0000","last_became_peered":"2024-09-21T18:53:09.408315+0000","last_unstale":"2024-09-21T18:54:00.012441+0000","last_undegraded":"2024-09-21T18:54:00.012441+0000","last_fullsized":"2024-09-21T18:54:00.012441+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.191003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012708+0000","last_change":"2024-09-21T18:53:07.400068+0000","last_active":"2024-09-21T18:54:00.012708+0000","last_peered":"2024-09-21T18:54:00.012708+0000","last_clean":"2024-09-21T18:54:00.012708+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:54:00.012708+0000","last_undegraded":"2024-09-21T18:54:00.012708+0000","last_fullsized":"2024-09-21T18:54:00.012708+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.160743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011811+0000","last_change":"2024-09-21T18:53:11.415649+0000","last_active":"2024-09-21T18:54:00.011811+0000","last_peered":"2024-09-21T18:54:00.011811+0000","last_clean":"2024-09-21T18:54:00.011811+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:54:00.011811+0000","last_undegraded":"2024-09-21T18:54:00.011811+0000","last_fullsized":"2024-09-21T18:54:00.011811+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.547416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011953+0000","last_change":"2024-09-21T18:53:13.434903+0000","last_active":"2024-09-21T18:54:00.011953+0000","last_peered":"2024-09-21T18:54:00.011953+0000","last_clean":"2024-09-21T18:54:00.011953+0000","last_became_active":"2024-09-21T18:53:13.434569+0000","last_became_peered":"2024-09-21T18:53:13.434569+0000","last_unstale":"2024-09-21T18:54:00.011953+0000","last_undegraded":"2024-09-21T18:54:00.011953+0000","last_fullsized":"2024-09-21T18:54:00.011953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.431252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":73,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012809+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:54:00.012809+0000","last_peered":"2024-09-21T18:54:00.012809+0000","last_clean":"2024-09-21T18:54:00.012809+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:54:00.012809+0000","last_undegraded":"2024-09-21T18:54:00.012809+0000","last_fullsized":"2024-09-21T18:54:00.012809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.041097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866461+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:53:59.866461+0000","last_peered":"2024-09-21T18:53:59.866461+0000","last_clean":"2024-09-21T18:53:59.866461+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:53:59.866461+0000","last_undegraded":"2024-09-21T18:53:59.866461+0000","last_fullsized":"2024-09-21T18:53:59.866461+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.852227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011703+0000","last_change":"2024-09-21T18:53:11.413440+0000","last_active":"2024-09-21T18:54:00.011703+0000","last_peered":"2024-09-21T18:54:00.011703+0000","last_clean":"2024-09-21T18:54:00.011703+0000","last_became_active":"2024-09-21T18:53:11.413142+0000","last_became_peered":"2024-09-21T18:53:11.413142+0000","last_unstale":"2024-09-21T18:54:00.011703+0000","last_undegraded":"2024-09-21T18:54:00.011703+0000","last_fullsized":"2024-09-21T18:54:00.011703+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:26.886861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865822+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:53:59.865822+0000","last_peered":"2024-09-21T18:53:59.865822+0000","last_clean":"2024-09-21T18:53:59.865822+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:53:59.865822+0000","last_undegraded":"2024-09-21T18:53:59.865822+0000","last_fullsized":"2024-09-21T18:53:59.865822+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.894993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":66,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.434887+0000","last_change":"2024-09-21T18:53:09.406428+0000","last_active":"2024-09-21T18:53:59.434887+0000","last_peered":"2024-09-21T18:53:59.434887+0000","last_clean":"2024-09-21T18:53:59.434887+0000","last_became_active":"2024-09-21T18:53:09.406030+0000","last_became_peered":"2024-09-21T18:53:09.406030+0000","last_unstale":"2024-09-21T18:53:59.434887+0000","last_undegraded":"2024-09-21T18:53:59.434887+0000","last_fullsized":"2024-09-21T18:53:59.434887+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:32.045459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866571+0000","last_change":"2024-09-21T18:53:07.400289+0000","last_active":"2024-09-21T18:53:59.866571+0000","last_peered":"2024-09-21T18:53:59.866571+0000","last_clean":"2024-09-21T18:53:59.866571+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:53:59.866571+0000","last_undegraded":"2024-09-21T18:53:59.866571+0000","last_fullsized":"2024-09-21T18:53:59.866571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:06.969808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011546+0000","last_change":"2024-09-21T18:53:11.417570+0000","last_active":"2024-09-21T18:54:00.011546+0000","last_peered":"2024-09-21T18:54:00.011546+0000","last_clean":"2024-09-21T18:54:00.011546+0000","last_became_active":"2024-09-21T18:53:11.417277+0000","last_became_peered":"2024-09-21T18:53:11.417277+0000","last_unstale":"2024-09-21T18:54:00.011546+0000","last_undegraded":"2024-09-21T18:54:00.011546+0000","last_fullsized":"2024-09-21T18:54:00.011546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:11.958604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435606+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:53:59.435606+0000","last_peered":"2024-09-21T18:53:59.435606+0000","last_clean":"2024-09-21T18:53:59.435606+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:53:59.435606+0000","last_undegraded":"2024-09-21T18:53:59.435606+0000","last_fullsized":"2024-09-21T18:53:59.435606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.706132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011989+0000","last_change":"2024-09-21T18:53:09.717860+0000","last_active":"2024-09-21T18:54:00.011989+0000","last_peered":"2024-09-21T18:54:00.011989+0000","last_clean":"2024-09-21T18:54:00.011989+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:54:00.011989+0000","last_undegraded":"2024-09-21T18:54:00.011989+0000","last_fullsized":"2024-09-21T18:54:00.011989+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.363297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866634+0000","last_change":"2024-09-21T18:53:07.405281+0000","last_active":"2024-09-21T18:53:59.866634+0000","last_peered":"2024-09-21T18:53:59.866634+0000","last_clean":"2024-09-21T18:53:59.866634+0000","last_became_active":"2024-09-21T18:53:07.404988+0000","last_became_peered":"2024-09-21T18:53:07.404988+0000","last_unstale":"2024-09-21T18:53:59.866634+0000","last_undegraded":"2024-09-21T18:53:59.866634+0000","last_fullsized":"2024-09-21T18:53:59.866634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.141596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":60,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865814+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:53:59.865814+0000","last_peered":"2024-09-21T18:53:59.865814+0000","last_clean":"2024-09-21T18:53:59.865814+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:53:59.865814+0000","last_undegraded":"2024-09-21T18:53:59.865814+0000","last_fullsized":"2024-09-21T18:53:59.865814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.640908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012110+0000","last_change":"2024-09-21T18:53:13.657306+0000","last_active":"2024-09-21T18:54:00.012110+0000","last_peered":"2024-09-21T18:54:00.012110+0000","last_clean":"2024-09-21T18:54:00.012110+0000","last_became_active":"2024-09-21T18:53:13.656931+0000","last_became_peered":"2024-09-21T18:53:13.656931+0000","last_unstale":"2024-09-21T18:54:00.012110+0000","last_undegraded":"2024-09-21T18:54:00.012110+0000","last_fullsized":"2024-09-21T18:54:00.012110+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:42.145553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012274+0000","last_change":"2024-09-21T18:53:07.397798+0000","last_active":"2024-09-21T18:54:00.012274+0000","last_peered":"2024-09-21T18:54:00.012274+0000","last_clean":"2024-09-21T18:54:00.012274+0000","last_became_active":"2024-09-21T18:53:07.397469+0000","last_became_peered":"2024-09-21T18:53:07.397469+0000","last_unstale":"2024-09-21T18:54:00.012274+0000","last_undegraded":"2024-09-21T18:54:00.012274+0000","last_fullsized":"2024-09-21T18:54:00.012274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:31.024888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437363+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:53:59.437363+0000","last_peered":"2024-09-21T18:53:59.437363+0000","last_clean":"2024-09-21T18:53:59.437363+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:53:59.437363+0000","last_undegraded":"2024-09-21T18:53:59.437363+0000","last_fullsized":"2024-09-21T18:53:59.437363+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.137675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013137+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:54:00.013137+0000","last_peered":"2024-09-21T18:54:00.013137+0000","last_clean":"2024-09-21T18:54:00.013137+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:54:00.013137+0000","last_undegraded":"2024-09-21T18:54:00.013137+0000","last_fullsized":"2024-09-21T18:54:00.013137+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.109044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866097+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:53:59.866097+0000","last_peered":"2024-09-21T18:53:59.866097+0000","last_clean":"2024-09-21T18:53:59.866097+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:53:59.866097+0000","last_undegraded":"2024-09-21T18:53:59.866097+0000","last_fullsized":"2024-09-21T18:53:59.866097+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.165915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012565+0000","last_change":"2024-09-21T18:53:09.720374+0000","last_active":"2024-09-21T18:54:00.012565+0000","last_peered":"2024-09-21T18:54:00.012565+0000","last_clean":"2024-09-21T18:54:00.012565+0000","last_became_active":"2024-09-21T18:53:09.720056+0000","last_became_peered":"2024-09-21T18:53:09.720056+0000","last_unstale":"2024-09-21T18:54:00.012565+0000","last_undegraded":"2024-09-21T18:54:00.012565+0000","last_fullsized":"2024-09-21T18:54:00.012565+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.445280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866014+0000","last_change":"2024-09-21T18:53:07.406957+0000","last_active":"2024-09-21T18:53:59.866014+0000","last_peered":"2024-09-21T18:53:59.866014+0000","last_clean":"2024-09-21T18:53:59.866014+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:53:59.866014+0000","last_undegraded":"2024-09-21T18:53:59.866014+0000","last_fullsized":"2024-09-21T18:53:59.866014+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.565944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437277+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:53:59.437277+0000","last_peered":"2024-09-21T18:53:59.437277+0000","last_clean":"2024-09-21T18:53:59.437277+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:53:59.437277+0000","last_undegraded":"2024-09-21T18:53:59.437277+0000","last_fullsized":"2024-09-21T18:53:59.437277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.484261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011873+0000","last_change":"2024-09-21T18:53:13.423168+0000","last_active":"2024-09-21T18:54:00.011873+0000","last_peered":"2024-09-21T18:54:00.011873+0000","last_clean":"2024-09-21T18:54:00.011873+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:54:00.011873+0000","last_undegraded":"2024-09-21T18:54:00.011873+0000","last_fullsized":"2024-09-21T18:54:00.011873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.401268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012910+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:54:00.012910+0000","last_peered":"2024-09-21T18:54:00.012910+0000","last_clean":"2024-09-21T18:54:00.012910+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:54:00.012910+0000","last_undegraded":"2024-09-21T18:54:00.012910+0000","last_fullsized":"2024-09-21T18:54:00.012910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.011518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012269+0000","last_change":"2024-09-21T18:53:07.405516+0000","last_active":"2024-09-21T18:54:00.012269+0000","last_peered":"2024-09-21T18:54:00.012269+0000","last_clean":"2024-09-21T18:54:00.012269+0000","last_became_active":"2024-09-21T18:53:07.405223+0000","last_became_peered":"2024-09-21T18:53:07.405223+0000","last_unstale":"2024-09-21T18:54:00.012269+0000","last_undegraded":"2024-09-21T18:54:00.012269+0000","last_fullsized":"2024-09-21T18:54:00.012269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:34.984977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435507+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:53:59.435507+0000","last_peered":"2024-09-21T18:53:59.435507+0000","last_clean":"2024-09-21T18:53:59.435507+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:53:59.435507+0000","last_undegraded":"2024-09-21T18:53:59.435507+0000","last_fullsized":"2024-09-21T18:53:59.435507+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.168997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435722+0000","last_change":"2024-09-21T18:53:13.423826+0000","last_active":"2024-09-21T18:53:59.435722+0000","last_peered":"2024-09-21T18:53:59.435722+0000","last_clean":"2024-09-21T18:53:59.435722+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:53:59.435722+0000","last_undegraded":"2024-09-21T18:53:59.435722+0000","last_fullsized":"2024-09-21T18:53:59.435722+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:25.967749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435766+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:53:59.435766+0000","last_peered":"2024-09-21T18:53:59.435766+0000","last_clean":"2024-09-21T18:53:59.435766+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:53:59.435766+0000","last_undegraded":"2024-09-21T18:53:59.435766+0000","last_fullsized":"2024-09-21T18:53:59.435766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.620456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866242+0000","last_change":"2024-09-21T18:53:07.403003+0000","last_active":"2024-09-21T18:53:59.866242+0000","last_peered":"2024-09-21T18:53:59.866242+0000","last_clean":"2024-09-21T18:53:59.866242+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:53:59.866242+0000","last_undegraded":"2024-09-21T18:53:59.866242+0000","last_fullsized":"2024-09-21T18:53:59.866242+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.651586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012774+0000","last_change":"2024-09-21T18:53:11.411052+0000","last_active":"2024-09-21T18:54:00.012774+0000","last_peered":"2024-09-21T18:54:00.012774+0000","last_clean":"2024-09-21T18:54:00.012774+0000","last_became_active":"2024-09-21T18:53:11.410769+0000","last_became_peered":"2024-09-21T18:53:11.410769+0000","last_unstale":"2024-09-21T18:54:00.012774+0000","last_undegraded":"2024-09-21T18:54:00.012774+0000","last_fullsized":"2024-09-21T18:54:00.012774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:29.670450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012174+0000","last_change":"2024-09-21T18:53:13.431033+0000","last_active":"2024-09-21T18:54:00.012174+0000","last_peered":"2024-09-21T18:54:00.012174+0000","last_clean":"2024-09-21T18:54:00.012174+0000","last_became_active":"2024-09-21T18:53:13.430667+0000","last_became_peered":"2024-09-21T18:53:13.430667+0000","last_unstale":"2024-09-21T18:54:00.012174+0000","last_undegraded":"2024-09-21T18:54:00.012174+0000","last_fullsized":"2024-09-21T18:54:00.012174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:16.681228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":62,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435007+0000","last_change":"2024-09-21T18:53:09.407481+0000","last_active":"2024-09-21T18:53:59.435007+0000","last_peered":"2024-09-21T18:53:59.435007+0000","last_clean":"2024-09-21T18:53:59.435007+0000","last_became_active":"2024-09-21T18:53:09.407184+0000","last_became_peered":"2024-09-21T18:53:09.407184+0000","last_unstale":"2024-09-21T18:53:59.435007+0000","last_undegraded":"2024-09-21T18:53:59.435007+0000","last_fullsized":"2024-09-21T18:53:59.435007+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.470465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435463+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:53:59.435463+0000","last_peered":"2024-09-21T18:53:59.435463+0000","last_clean":"2024-09-21T18:53:59.435463+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:53:59.435463+0000","last_undegraded":"2024-09-21T18:53:59.435463+0000","last_fullsized":"2024-09-21T18:53:59.435463+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:12.908635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436495+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:53:59.436495+0000","last_peered":"2024-09-21T18:53:59.436495+0000","last_clean":"2024-09-21T18:53:59.436495+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:53:59.436495+0000","last_undegraded":"2024-09-21T18:53:59.436495+0000","last_fullsized":"2024-09-21T18:53:59.436495+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.492927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435472+0000","last_change":"2024-09-21T18:53:13.433363+0000","last_active":"2024-09-21T18:53:59.435472+0000","last_peered":"2024-09-21T18:53:59.435472+0000","last_clean":"2024-09-21T18:53:59.435472+0000","last_became_active":"2024-09-21T18:53:13.433057+0000","last_became_peered":"2024-09-21T18:53:13.433057+0000","last_unstale":"2024-09-21T18:53:59.435472+0000","last_undegraded":"2024-09-21T18:53:59.435472+0000","last_fullsized":"2024-09-21T18:53:59.435472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.215327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.437612+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:53:59.437612+0000","last_peered":"2024-09-21T18:53:59.437612+0000","last_clean":"2024-09-21T18:53:59.437612+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:53:59.437612+0000","last_undegraded":"2024-09-21T18:53:59.437612+0000","last_fullsized":"2024-09-21T18:53:59.437612+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:18.126928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.866331+0000","last_change":"2024-09-21T18:53:07.406445+0000","last_active":"2024-09-21T18:53:59.866331+0000","last_peered":"2024-09-21T18:53:59.866331+0000","last_clean":"2024-09-21T18:53:59.866331+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:53:59.866331+0000","last_undegraded":"2024-09-21T18:53:59.866331+0000","last_fullsized":"2024-09-21T18:53:59.866331+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:28.278365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435693+0000","last_change":"2024-09-21T18:53:11.411055+0000","last_active":"2024-09-21T18:53:59.435693+0000","last_peered":"2024-09-21T18:53:59.435693+0000","last_clean":"2024-09-21T18:53:59.435693+0000","last_became_active":"2024-09-21T18:53:11.410775+0000","last_became_peered":"2024-09-21T18:53:11.410775+0000","last_unstale":"2024-09-21T18:53:59.435693+0000","last_undegraded":"2024-09-21T18:53:59.435693+0000","last_fullsized":"2024-09-21T18:53:59.435693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:23.614499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.011507+0000","last_change":"2024-09-21T18:53:13.433872+0000","last_active":"2024-09-21T18:54:00.011507+0000","last_peered":"2024-09-21T18:54:00.011507+0000","last_clean":"2024-09-21T18:54:00.011507+0000","last_became_active":"2024-09-21T18:53:13.433534+0000","last_became_peered":"2024-09-21T18:53:13.433534+0000","last_unstale":"2024-09-21T18:54:00.011507+0000","last_undegraded":"2024-09-21T18:54:00.011507+0000","last_fullsized":"2024-09-21T18:54:00.011507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:20.158748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436094+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:53:59.436094+0000","last_peered":"2024-09-21T18:53:59.436094+0000","last_clean":"2024-09-21T18:53:59.436094+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:53:59.436094+0000","last_undegraded":"2024-09-21T18:53:59.436094+0000","last_fullsized":"2024-09-21T18:53:59.436094+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:22.024213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013223+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:54:00.013223+0000","last_peered":"2024-09-21T18:54:00.013223+0000","last_clean":"2024-09-21T18:54:00.013223+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:54:00.013223+0000","last_undegraded":"2024-09-21T18:54:00.013223+0000","last_fullsized":"2024-09-21T18:54:00.013223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:17.249689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865569+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:53:59.865569+0000","last_peered":"2024-09-21T18:53:59.865569+0000","last_clean":"2024-09-21T18:53:59.865569+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:53:59.865569+0000","last_undegraded":"2024-09-21T18:53:59.865569+0000","last_fullsized":"2024-09-21T18:53:59.865569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:15.908774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.435826+0000","last_change":"2024-09-21T18:53:13.658591+0000","last_active":"2024-09-21T18:53:59.435826+0000","last_peered":"2024-09-21T18:53:59.435826+0000","last_clean":"2024-09-21T18:53:59.435826+0000","last_became_active":"2024-09-21T18:53:13.657938+0000","last_became_peered":"2024-09-21T18:53:13.657938+0000","last_unstale":"2024-09-21T18:53:59.435826+0000","last_undegraded":"2024-09-21T18:53:59.435826+0000","last_fullsized":"2024-09-21T18:53:59.435826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.163794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436433+0000","last_change":"2024-09-21T18:53:09.409585+0000","last_active":"2024-09-21T18:53:59.436433+0000","last_peered":"2024-09-21T18:53:59.436433+0000","last_clean":"2024-09-21T18:53:59.436433+0000","last_became_active":"2024-09-21T18:53:09.409216+0000","last_became_peered":"2024-09-21T18:53:09.409216+0000","last_unstale":"2024-09-21T18:53:59.436433+0000","last_undegraded":"2024-09-21T18:53:59.436433+0000","last_fullsized":"2024-09-21T18:53:59.436433+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:24.123559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012633+0000","last_change":"2024-09-21T18:53:07.401813+0000","last_active":"2024-09-21T18:54:00.012633+0000","last_peered":"2024-09-21T18:54:00.012633+0000","last_clean":"2024-09-21T18:54:00.012633+0000","last_became_active":"2024-09-21T18:53:07.401545+0000","last_became_peered":"2024-09-21T18:53:07.401545+0000","last_unstale":"2024-09-21T18:54:00.012633+0000","last_undegraded":"2024-09-21T18:54:00.012633+0000","last_fullsized":"2024-09-21T18:54:00.012633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:19.528471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012467+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:54:00.012467+0000","last_peered":"2024-09-21T18:54:00.012467+0000","last_clean":"2024-09-21T18:54:00.012467+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:54:00.012467+0000","last_undegraded":"2024-09-21T18:54:00.012467+0000","last_fullsized":"2024-09-21T18:54:00.012467+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:13.190087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.865778+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:53:59.865778+0000","last_peered":"2024-09-21T18:53:59.865778+0000","last_clean":"2024-09-21T18:53:59.865778+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:53:59.865778+0000","last_undegraded":"2024-09-21T18:53:59.865778+0000","last_fullsized":"2024-09-21T18:53:59.865778+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:33.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436308+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:53:59.436308+0000","last_peered":"2024-09-21T18:53:59.436308+0000","last_clean":"2024-09-21T18:53:59.436308+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:53:59.436308+0000","last_undegraded":"2024-09-21T18:53:59.436308+0000","last_fullsized":"2024-09-21T18:53:59.436308+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:14.931032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:53:59.436630+0000","last_change":"2024-09-21T18:53:07.400561+0000","last_active":"2024-09-21T18:53:59.436630+0000","last_peered":"2024-09-21T18:53:59.436630+0000","last_clean":"2024-09-21T18:53:59.436630+0000","last_became_active":"2024-09-21T18:53:07.400254+0000","last_became_peered":"2024-09-21T18:53:07.400254+0000","last_unstale":"2024-09-21T18:53:59.436630+0000","last_undegraded":"2024-09-21T18:53:59.436630+0000","last_fullsized":"2024-09-21T18:53:59.436630+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.077186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.013566+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:54:00.013566+0000","last_peered":"2024-09-21T18:54:00.013566+0000","last_clean":"2024-09-21T18:54:00.013566+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:54:00.013566+0000","last_undegraded":"2024-09-21T18:54:00.013566+0000","last_fullsized":"2024-09-21T18:54:00.013566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:10.130619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T18:54:00.012401+0000","last_change":"2024-09-21T18:53:11.415531+0000","last_active":"2024-09-21T18:54:00.012401+0000","last_peered":"2024-09-21T18:54:00.012401+0000","last_clean":"2024-09-21T18:54:00.012401+0000","last_became_active":"2024-09-21T18:53:11.415209+0000","last_became_peered":"2024-09-21T18:53:11.415209+0000","last_unstale":"2024-09-21T18:54:00.012401+0000","last_undegraded":"2024-09-21T18:54:00.012401+0000","last_fullsized":"2024-09-21T18:54:00.012401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:27.845440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266703,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1376256,"data_stored":904047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528340,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712636,"statfs":{"total":95995035648,"available":95961739264,"internally_reserved":0,"allocated":1359872,"data_stored":903398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789975,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":761856,"data_stored":312940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018909,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":790528,"data_stored":314179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":802816,"data_stored":314478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":757760,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":827392,"data_stored":314802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1380352,"data_stored":904736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:54:05.926 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T18:54:05.926 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T18:54:06.169 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[29384]: from='client.14811 v1:172.21.15.81:0/3600517391' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[29384]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[29384]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[39192]: from='client.14811 v1:172.21.15.81:0/3600517391' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[39192]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:54:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[39192]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T18:54:06.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:06.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:06 smithi104 ceph-mon[32438]: from='client.14811 v1:172.21.15.81:0/3600517391' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:06 smithi104 ceph-mon[32438]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:54:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:06 smithi104 ceph-mon[32438]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T18:54:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:06.797 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 77 2024-09-21T18:54:06.797 INFO:teuthology.orchestra.run.smithi081.stdout:fsid d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T18:54:06.797 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-09-21T18:47:59.361278+0000 2024-09-21T18:54:06.797 INFO:teuthology.orchestra.run.smithi081.stdout:modified 2024-09-21T18:54:05.434562+0000 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:crush_version 18 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:full_ratio 0.95 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:backfillfull_ratio 0.9 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:nearfull_ratio 0.85 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:require_min_compat_client luminous 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:min_compat_client jewel 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:require_osd_release squid 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:stretch_mode_enabled false 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 60 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 61 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-21T18:54:06.798 INFO:teuthology.orchestra.run.smithi081.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:max_osd 8 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6801/390647394 v1:172.21.15.81:6802/390647394 exists,up bd6626b0-dc43-42d0-a459-9cf3911457ea 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 up in weight 1 up_from 13 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6805/3705537065 v1:172.21.15.81:6806/3705537065 exists,up 2f541064-dc48-4f7f-b29b-065491e148b5 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 up in weight 1 up_from 18 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6809/486671153 v1:172.21.15.81:6810/486671153 exists,up e7050b14-f463-4de3-8a8d-e05e78507938 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 up in weight 1 up_from 25 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6813/1516859335 v1:172.21.15.81:6814/1516859335 exists,up 75b4cb80-138b-445b-a93a-3c4c598b5e0e 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 up in weight 1 up_from 30 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6800/1904940409 v1:172.21.15.104:6801/1904940409 exists,up 85220357-38ea-4c90-b422-5d703329f106 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 up in weight 1 up_from 37 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6804/2569144231 v1:172.21.15.104:6805/2569144231 exists,up a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 up in weight 1 up_from 45 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6808/185975065 v1:172.21.15.104:6809/185975065 exists,up e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 up in weight 1 up_from 53 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6812/385235741 v1:172.21.15.104:6813/385235741 exists,up f15ee432-aa23-4e5a-9df4-478538af2045 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/188521944 expires 2024-09-22T18:53:35.352265+0000 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/116812683 expires 2024-09-22T18:48:21.208275+0000 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/628265626 expires 2024-09-22T18:48:21.208275+0000 2024-09-21T18:54:06.799 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/1496889754 expires 2024-09-22T18:53:35.352265+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/2203644131 expires 2024-09-22T18:48:21.208275+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/3906790599 expires 2024-09-22T18:48:21.208275+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/3045112983 expires 2024-09-22T18:48:45.625700+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/806683817 expires 2024-09-22T18:48:45.625700+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/3701153923 expires 2024-09-22T18:53:35.352265+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/1611463627 expires 2024-09-22T18:48:45.625700+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/1508794297 expires 2024-09-22T18:48:45.625700+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/3426409743 expires 2024-09-22T18:53:35.352265+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/198739293 expires 2024-09-22T18:53:35.352265+0000 2024-09-21T18:54:06.800 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/4266768293 expires 2024-09-22T18:53:35.352265+0000 2024-09-21T18:54:07.445 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:54:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:54:07.639 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6801/390647394 v1:172.21.15.81:6802/390647394 exists,up bd6626b0-dc43-42d0-a459-9cf3911457ea', 'osd.1 up in weight 1 up_from 13 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6805/3705537065 v1:172.21.15.81:6806/3705537065 exists,up 2f541064-dc48-4f7f-b29b-065491e148b5', 'osd.2 up in weight 1 up_from 18 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6809/486671153 v1:172.21.15.81:6810/486671153 exists,up e7050b14-f463-4de3-8a8d-e05e78507938', 'osd.3 up in weight 1 up_from 25 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.81:6813/1516859335 v1:172.21.15.81:6814/1516859335 exists,up 75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'osd.4 up in weight 1 up_from 30 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6800/1904940409 v1:172.21.15.104:6801/1904940409 exists,up 85220357-38ea-4c90-b422-5d703329f106', 'osd.5 up in weight 1 up_from 37 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6804/2569144231 v1:172.21.15.104:6805/2569144231 exists,up a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'osd.6 up in weight 1 up_from 45 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6808/185975065 v1:172.21.15.104:6809/185975065 exists,up e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'osd.7 up in weight 1 up_from 53 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.104:6812/385235741 v1:172.21.15.104:6813/385235741 exists,up f15ee432-aa23-4e5a-9df4-478538af2045'] 2024-09-21T18:54:07.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T18:54:07.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2849794733' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T18:54:07.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:54:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:54:07.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2849794733' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T18:54:07.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2849794733' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T18:54:07.883 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "mon.a", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster": "ceph", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "admin_socket_mode": "", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_client_required": "cephx, none", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_cluster_required": "cephx", 2024-09-21T18:54:08.415 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_debug": "true", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_service_required": "cephx", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_supported": "", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio": "true", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio_reap_max": "16", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_async_discard_threads": "0", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_block_size": "4096", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_aio": "false", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T18:54:08.416 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_enable_discard": "false", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_flock_retry": "3", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_inject_crash": "0", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_ioring": "false", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_ioring_hipri": "false", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T18:54:08.417 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_type": "", 2024-09-21T18:54:08.418 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T18:54:08.418 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_allocator": "hybrid", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_buffered_io": "true", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T18:54:08.419 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_replay_recovery": "false", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_sync_write": "false", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_allocator": "hybrid", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T18:54:08.420 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_create": "true", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_db_create": "false", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_db_path": "", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_db_size": "0", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_path": "", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_size": "107374182400", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_wal_create": "false", 2024-09-21T18:54:08.421 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_wal_path": "", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs": "true", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_autotune": "true", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T18:54:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_size": "0", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_type": "2q", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_clone_cow": "true", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T18:54:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_mode": "none", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_freelist": "false", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T18:54:08.424 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T18:54:08.425 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fail_eio": "false", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T18:54:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T18:54:08.427 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_blob_size": "0", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T18:54:08.428 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T18:54:08.429 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_mem": "512", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_tracing": "false", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T18:54:08.430 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_write_v2": "false", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_write_v2_random": "false", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T18:54:08.431 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_cluster_require_version": "2", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_require_signatures": "false", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_require_version": "2", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_service_require_signatures": "false", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_service_require_version": "2", 2024-09-21T18:54:08.432 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_sign_messages": "true", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "chdir": "", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_acl_type": "", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_asio_thread_count": "2", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_cache_mid": "0.750000", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_cache_size": "16384", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_caps_release_delay": "5", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_check_pool_perm": "true", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_force_sync_read": "false", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_getattr_caps": "false", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_inject_features": "", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T18:54:08.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_die_on_failed_remount": "false", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_dirsize_rbytes": "true", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_force_lazyio": "false", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_fs": "", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_inject_release_failure": "false", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_max_inline_size": "4096", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mds_namespace": "", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_metadata": "", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mount_gid": "-1", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mount_timeout": "300", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mount_uid": "-1", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mountpoint": "/", 2024-09-21T18:54:08.434 INFO:teuthology.orchestra.run.smithi081.stdout: "client_notify_timeout": "10", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc": "true", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_max_objects": "1000", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_size": "209715200", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_permissions": "true", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_quota": "true", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_quota_df": "true", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_readahead_max_bytes": "0", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_readahead_max_periods": "4", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_readahead_min": "131072", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_reconnect_stale": "false", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_shutdown_timeout": "30", 2024-09-21T18:54:08.435 INFO:teuthology.orchestra.run.smithi081.stdout: "client_snapdir": ".snap", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "client_tick_interval": "1", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "client_trace": "", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "client_use_faked_inos": "false", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "client_use_random_mds": "false", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_graylog": "false", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_graylog_port": "12201", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_monitors": "default=true", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_syslog": "false", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_syslog_level": "info", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "-", 2024-09-21T18:54:08.436 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_network": "", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_network_interface": "", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zlib_isal": "false", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zlib_level": "5", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zstd_level": "1", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T18:54:08.437 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_location": "", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_location_hook": "", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_location_hook_timeout": "10", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "daemonize": "false", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "daos_pool": "tank", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_alienstore": "0/5", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_asok": "1/5", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_asok_assert_abort": "false", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T18:54:08.438 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_auth": "1/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_bdev": "1/3", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_bluefs": "1/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_bluestore": "1/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_buffer": "0/1", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_cephsqlite": "0/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_client": "0/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_compressor": "1/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_context": "0/1", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_crush": "1/1", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_crypto": "1/5", 2024-09-21T18:54:08.439 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_cyanstore": "0/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_dpdk": "1/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_eventtrace": "1/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_filer": "0/1", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_filestore": "1/3", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_finisher": "1/1", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_fuse": "1/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_javaclient": "1/5", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_journal": "1/3", 2024-09-21T18:54:08.440 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_journaler": "0/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_kstore": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_lockdep": "0/1", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mclock": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_balancer": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_locker": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_log": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_migrator": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_memstore": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mgr": "2/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mgrc": "1/5", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mon": "20/20", 2024-09-21T18:54:08.441 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_monc": "0/10", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_ms": "1/1", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_none": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_objclass": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_objectcacher": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_objecter": "0/1", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_optracker": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_osd": "1/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_paxos": "20/20", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_perfcounter": "1/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_prioritycache": "1/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rados": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T18:54:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd_replay": "0/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_refs": "0/0", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_reserver": "1/1", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_access": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_flight": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_notification": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_sync": "1/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rocksdb": "4/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore": "0/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_backref": "0/5", 2024-09-21T18:54:08.443 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_cache": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_device": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_epm": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_journal": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_lba": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_odata": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_omap": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_onode": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_t": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_tm": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_striper": "0/1", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_test": "0/5", 2024-09-21T18:54:08.444 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_throttle": "1/1", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_timer": "0/1", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_tp": "0/5", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_trace": "1/5", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "device_failure_prediction_mode": "none", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_graylog": "false", 2024-09-21T18:54:08.445 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_journald": "false", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_stderr": "true", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_syslog": "false", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event_tracing": "false", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_cert_file": "", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_http_port": "9926", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_key_file": "", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_prio_limit": "5", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_sort_metrics": "true", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_stats_period": "5", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "fake_statfs_for_testing": "0", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "fatal_signal_handlers": "true", 2024-09-21T18:54:08.446 INFO:teuthology.orchestra.run.smithi081.stdout: "filer_max_purge_ops": "10", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filer_max_truncate_ops": "128", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_blackhole": "false", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_btrfs_snap": "true", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_caller_concurrency": "10", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_omap_check": "false", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_verify_split": "false", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_dump_file": "", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T18:54:08.447 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fadvise": "true", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fail_eio": "true", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fd_cache_size": "128", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fiemap": "false", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_inject_stall": "0", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_journal_parallel": "false", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_journal_trailing": "false", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_journal_writeahead": "false", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_kill_at": "0", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T18:54:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_merge_threshold": "-10", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_odsync_write": "false", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T18:54:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_omap_backend_path": "", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_op_threads": "2", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_punch_hole": "false", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T18:54:08.450 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_ops": "50", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_seek_data_hole": "false", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_sloppy_crc": "false", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_splice": "false", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_split_multiple": "2", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_split_rand_factor": "20", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_update_to": "1000", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T18:54:08.451 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_xfs_extsize": "false", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_zfs_snap": "false", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "fio_dir": "/tmp/fio", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_allow_other": "true", 2024-09-21T18:54:08.452 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_big_writes": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_debug": "false", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_default_permissions": "false", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_disable_pagecache": "false", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_max_write": "0", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_multithreaded": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_require_active_mds": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_set_user_groups": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_splice_move": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_splice_read": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_splice_write": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "gss_target_name": "ceph", 2024-09-21T18:54:08.453 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_file": "", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_inject_failure": "0", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_interval": "5", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "host": "smithi081", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T18:54:08.454 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "inject_early_sigterm": "false", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "jaeger_agent_port": "6799", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "jaeger_tracing_enable": "false", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_aio": "true", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_align_min_size": "65536", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_block_align": "true", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_block_size": "4096", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_dio": "true", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_discard": "false", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_force_aio": "false", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_ignore_corruption": "false", 2024-09-21T18:54:08.455 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_max_write_entries": "100", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_replay_from": "0", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_write_header_frequency": "0", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_zero_on_create": "false", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journaler_prefetch_periods": "10", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journaler_prezero_periods": "5", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "journaler_write_head_interval": "15", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "keyfile": "", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_backend": "rocksdb", 2024-09-21T18:54:08.456 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_max_bytes": "67108864", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_max_ops": "512", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_onode_map_size": "1024", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_sync_transaction": "false", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "librados_thread_count": "2", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "lockdep": "false", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "lockdep_force_backtrace": "false", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "log_coarse_timestamps": "true", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "log_flush_on_exit": "false", 2024-09-21T18:54:08.457 INFO:teuthology.orchestra.run.smithi081.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_graylog_port": "12201", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_max_new": "1000", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_max_recent": "10000", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_stderr_prefix": "", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_file": "true", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_graylog": "false", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_journald": "false", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_stderr": "false", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_syslog": "false", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_action_on_write_error": "1", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_alternate_name_max": "8192", 2024-09-21T18:54:08.458 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_asio_thread_count": "2", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_export_pin": "true", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_interval": "10", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_max": "-1", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_max_until": "-1", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_merge_size": "50", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_mode": "0", 2024-09-21T18:54:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_bits": "3", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_size": "10000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T18:54:08.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_mid": "0.700000", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_trim_interval": "1", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T18:54:08.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_auth_pins": "false", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_frag": "false", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_scatterstat": "false", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_subtrees": "false", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_default_dir_hash": "2", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_defer_session_stale": "true", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T18:54:08.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_max_entries": "0", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_prefetch": "true", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_early_reply": "true", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_enable_op_tracker": "true", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_enforce_unique_name": "true", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T18:54:08.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_health_dummy": "false", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_join_fs": "", 2024-09-21T18:54:08.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_journal_format": "1", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_create_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_export_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_import_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_journal_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_link_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_openc_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_rename_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T18:54:08.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_max_events": "-1", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_max_segments": "128", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_pause": "false", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_segment_size": "0", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_trim_threshold": "128", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_completed_requests": "100000", 2024-09-21T18:54:08.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_export_size": "20971520", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_file_recover": "32", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_purge_files": "64", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_purge_ops": "8192", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_metrics_update_interval": "2", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_min_caps_per_client": "100", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_numa_node": "-1", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_duration": "600", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_size": "20", 2024-09-21T18:54:08.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_log_threshold": "5", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_ping_grace": "15", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_ping_interval": "5", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_max_caps": "30000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_replay_interval": "1.000000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_root_ino_gid": "0", 2024-09-21T18:54:08.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_root_ino_uid": "0", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_shutdown_check": "0", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_skip_ino": "0", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T18:54:08.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T18:54:08.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_snap_min_uid": "0", 2024-09-21T18:54:08.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_snap_rstat": "false", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_symlink_recovery": "true", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_thrash_exports": "0", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_thrash_fragments": "0", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_tick_interval": "5.000000", 2024-09-21T18:54:08.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_valgrind_exit": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_verify_backtrace": "1", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_verify_scatter": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_wipe_sessions": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mempool_debug": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_page_set": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_page_size": "65536", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_client_bytes": "134217728", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_client_messages": "512", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T18:54:08.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_disabled_modules": "", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mds_messages": "128", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mon_messages": "128", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_duration": "600", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_size": "20", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T18:54:08.473 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_log_threshold": "5", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_osd_messages": "8192", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_pool": "true", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_standby_modules": "true", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_stats_period": "5", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_stats_threshold": "5", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_tick_period": "2", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_allow_pool_delete": "true", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T18:54:08.474 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_bytes": "104857600", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_target_rank": "-1", 2024-09-21T18:54:08.475 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_level": "debug", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_compact_on_start": "false", 2024-09-21T18:54:08.476 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_compact_on_trim": "true", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cpu_threads": "4", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data_avail_crit": "5", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data_avail_warn": "5", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_dump_json": "false", 2024-09-21T18:54:08.477 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_extra_checks": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_down_uptime_grace": "60", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_election_default_strategy": "1", 2024-09-21T18:54:08.478 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_election_timeout": "5.000000", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_enable_op_tracker": "true", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_fake_pool_delete": "false", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_force_quorum_join": "false", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_log_update_period": "5", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_max_detail": "50", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_to_clog": "true", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T18:54:08.479 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_host": "172.21.15.81:6789/0", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_host_override": "", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_initial_members": "", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_lease": "5.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_log_full_interval": "50", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_log_max": "10000", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_log_max_summary": "50", 2024-09-21T18:54:08.480 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_log_epochs": "500", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_osd": "10000", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_memory_autotune": "true", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_memory_target": "2147483648", 2024-09-21T18:54:08.481 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_digest_period": "5", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_min_osdmap_epochs": "50", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_complaint_time": "30", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_duration": "600", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_size": "20", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T18:54:08.482 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_log_threshold": "5", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_cache_size": "500", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T18:54:08.483 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T18:54:08.484 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_report_timeout": "900", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T18:54:08.485 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_max_osds": "4", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T18:54:08.486 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_interval": "86400", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_max_keys": "100", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_timeout": "300", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_session_timeout": "300", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_smart_report_timeout": "5", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_pool_size": "4", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_debug": "false", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T18:54:08.487 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_tick_interval": "5", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T18:54:08.488 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_older_version": "true", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "monmap": "", 2024-09-21T18:54:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_tracing_enabled": "false", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_op_threads": "3", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_cm": "false", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_device_name": "", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T18:54:08.490 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_sl": "3", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_type": "ib", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_reap_threshold": "5", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_before_connect": "false", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_ipv4": "true", 2024-09-21T18:54:08.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_ipv6": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_msgr1": "true", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_msgr2": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_port_max": "7568", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_port_min": "6800", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_retry_count": "3", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_retry_delay": "5", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_client": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_mds": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_mgr": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_mon": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_osd": "false", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_client_mode": "crc secure", 2024-09-21T18:54:08.492 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_cluster_type": "", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_compress_secure": "false", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_crc_data": "true", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_crc_header": "true", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_bug": "true", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_old_message": "true", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T18:54:08.493 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_hugepages": "", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_lro": "true", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_pmd": "", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_port_id": "0", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T18:54:08.494 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dump_on_send": "false", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_delay_type": "", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_network_congestion": "0", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_socket_failures": "0", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_max_accept_failures": "4", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_max_backoff": "15.000000", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T18:54:08.495 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_osd_compress_mode": "none", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_pq_min_cost": "65536", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_public_type": "", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_service_mode": "crc secure", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_nodelay": "true", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_type": "async+posix", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "no_config_file": "false", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T18:54:08.496 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_inflight_ops": "1024", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_timeout": "10.000000", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objectstore_blackhole": "false", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "openssl_engine_opts": "", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_max_ops": "4", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T18:54:08.497 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_slop": "0.020000", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_asio_thread_count": "2", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backfill_scan_max": "512", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backfill_scan_min": "64", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backoff_on_peering": "false", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_beacon_report_interval": "300", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_duration": "30", 2024-09-21T18:54:08.498 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_blkin_trace_all": "false", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_update_on_start": "true", 2024-09-21T18:54:08.499 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_message_cap": "256", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_op_priority": "63", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_watch_timeout": "30", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_command_max_records": "256", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_compact_on_start": "false", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_update_on_start": "true", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T18:54:08.500 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_op_order": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_shutdown": "false", 2024-09-21T18:54:08.501 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_trim_objects": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T18:54:08.502 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_default_notify_timeout": "30", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ec_partial_reads": "true", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_enable_op_tracker": "true", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_info": "true", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_shutdown": "true", 2024-09-21T18:54:08.503 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_function_tracing": "false", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_grace": "20", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_interval": "6", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_stale": "600", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T18:54:08.504 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_journal_size": "5120", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_kill_backfill_at": "0", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_cache_size": "50", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_dedup": "true", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_message_max": "40", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T18:54:08.505 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_attr_name_len": "100", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_attr_size": "0", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_backfills": "1", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_markdown_count": "5", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_markdown_period": "600", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_object_name_len": "2048", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_object_size": "134217728", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pgls": "1024", 2024-09-21T18:54:08.506 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_push_cost": "8388608", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_push_objects": "10", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_scrubs": "3", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_write_size": "90", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T18:54:08.507 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_profile": "balanced", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_base": "805306368", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T18:54:08.508 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_target": "4294967296", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_target_autotune": "false", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_min_recovery_priority": "0", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_report_interval": "5", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_num_cache_shards": "32", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_numa_node": "-1", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T18:54:08.509 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objecter_finishers": "1", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore": "bluestore", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore_fuse": "false", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore_tracing": "false", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_duration": "600", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_size": "20", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_log_threshold": "5", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_shards": "0", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T18:54:08.510 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_thread_timeout": "15", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_open_classes_on_start": "true", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_os_flags": "0", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_peering_op_priority": "255", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_delete_priority": "5", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T18:54:08.511 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_crimson": "false", 2024-09-21T18:54:08.512 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flags": "0", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_min_size": "0", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T18:54:08.513 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_size": "3", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_type": "replicated", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_cost": "20971520", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_active": "0", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T18:54:08.514 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_op_priority": "3", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_priority": "5", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_repair_during_recovery": "false", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T18:54:08.515 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_cost": "52428800", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_end_hour": "0", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T18:54:08.516 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_priority": "5", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T18:54:08.517 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_skip_data_digest": "false", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_smart_report_timeout": "5", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_priority": "5", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T18:54:08.518 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_target_transaction_size": "30", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tracing": "false", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_use_stale_snap": "false", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T18:54:08.519 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_kill_at": "0", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_max_join_drift": "10", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_min": "500", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_min_wait": "0.050000", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_service_trim_max": "500", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_service_trim_min": "10", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_stash_full_interval": "25", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_trim_max": "500", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_trim_min": "250", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "perf": "true", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "pid_file": "", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "-", 2024-09-21T18:54:08.520 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrv": "", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "public_bind_addr": "-", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "public_network": "", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "public_network_interface": "", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "qat_compressor_enabled": "false", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rados_mon_op_timeout": "0", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rados_osd_op_timeout": "0", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rados_tracing": "false", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_atime_update_interval": "60", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T18:54:08.521 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache": "true", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_policy": "writearound", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_size": "33554432", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_compression_hint": "none", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T18:54:08.522 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_clone_format": "auto", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_data_pool": "", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_features": "61", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_format": "2", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_map_options": "", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_order": "22", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_pool": "rbd", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_stripe_count": "0", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T18:54:08.523 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_io_scheduler": "simple", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_order": "24", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_pool": "", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_splay_width": "4", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T18:54:08.524 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T18:54:08.525 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T18:54:08.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_op_threads": "1", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_plugins": "", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T18:54:08.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T18:54:08.528 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_tracing": "false", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_validate_names": "true", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_validate_pool": "true", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "restapi_base_url": "", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "restapi_log_level": "", 2024-09-21T18:54:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_admin_entry": "admin", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_backend_store": "rados", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_barbican_url": "", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_beast_enable_async": "true", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T18:54:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cache_enabled": "true", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_config_store": "rados", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_content_length_compat": "false", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cross_domain_policy": "", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T18:54:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T18:54:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T18:54:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T18:54:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T18:54:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T18:54:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T18:54:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T18:54:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T18:54:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T18:54:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T18:54:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T18:54:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T18:54:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_window": "30", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T18:54:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_disable_s3select": "false", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T18:54:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dns_name": "", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dns_s3website_name": "", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_ops_log": "false", 2024-09-21T18:54:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_static_website": "false", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_usage_log": "false", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_expose_bucket": "false", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_extended_http_attrs": "", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_filter": "none", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_objs": "32", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T18:54:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_init_timeout": "300", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T18:54:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_password": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_project": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_token": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_user": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T18:54:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_url": "", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_objs": "32", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_worker": "3", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T18:54:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_uri": "", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_http_headers": "", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T18:54:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_attr_size": "0", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_listing_results": "5000", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_notify_retries": "10", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T18:54:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T18:54:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_num_control_oids": "8", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_numa_node": "-1", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T18:54:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_op_tracing": "false", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_opa_token": "", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_opa_url": "", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_rados": "false", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T18:54:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_print_continue": "true", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T18:54:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rados_tracing": "false", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_realm": "", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_realm_id": "", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_region": "", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_request_uri": "", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T18:54:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_resolve_cname": "false", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_run_sync_thread": "true", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T18:54:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_script_uri": "", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_service_provider_name": "", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_client_id": "", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_client_secret": "", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_entry": "sts", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_key": "sts", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T18:54:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_auth_url": "", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_custom_header": "", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_need_stats": "true", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_tenant_name": "", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_url": "", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_lease_period": "120", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T18:54:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_thread_pool_size": "512", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_comment": "", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_createby": "", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_encoding": "", 2024-09-21T18:54:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_flag": "false", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_origin": "", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_tracker": "", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_max_shards": "32", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_use_opa_authz": "false", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_counters_cache": "false", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T18:54:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_unique_email": "true", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_verify_ssl": "true", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zone": "", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zone_id": "", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zonegroup": "", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zonegroup_id": "", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T18:54:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_block_size": "4096", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T18:54:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_partition_filters": "false", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_perf": "false", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "run_dir": "/var/run/ceph", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_block_create": "true", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T18:54:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_device_size": "53687091200", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_full_integrity_check": "false", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_main_device_type": "SSD", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T18:54:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_segment_size": "67108864", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "set_keepcaps": "false", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "setgroup": "ceph", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "setuser": "ceph", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "setuser_match_path": "", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "thp": "false", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "threadpool_default_timeout": "60", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "throttler_perf_counter": "true", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "tmp_dir": "/tmp", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "uadk_compressor_enabled": "false", 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T18:54:08.560 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:54:08.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:08 smithi081 ceph-mon[29384]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:08.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:08.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:08 smithi081 ceph-mon[39192]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:08.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:08.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:08 smithi104 ceph-mon[32438]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:08.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:09.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T18:54:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:09 smithi104 ceph-mon[32438]: 3.b scrub starts 2024-09-21T18:54:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:09 smithi104 ceph-mon[32438]: 3.b scrub ok 2024-09-21T18:54:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:09 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T18:54:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:09 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T18:54:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:09 smithi104 ceph-mon[32438]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T18:54:09.763 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[29384]: 3.b scrub starts 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[29384]: 3.b scrub ok 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[29384]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[39192]: 3.b scrub starts 2024-09-21T18:54:09.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[39192]: 3.b scrub ok 2024-09-21T18:54:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T18:54:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T18:54:09.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:09 smithi081 ceph-mon[39192]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T18:54:10.218 INFO:teuthology.orchestra.run.smithi081.stdout:mon.a: {} 2024-09-21T18:54:10.218 INFO:teuthology.orchestra.run.smithi081.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:54:10.227 INFO:teuthology.orchestra.run.smithi081.stdout:mon.b: {} 2024-09-21T18:54:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:54:10.235 INFO:teuthology.orchestra.run.smithi081.stdout:mon.c: {} 2024-09-21T18:54:10.235 INFO:teuthology.orchestra.run.smithi081.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:54:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:10 smithi104 ceph-mon[32438]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:54:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:10 smithi104 ceph-mon[32438]: 3.3 scrub starts 2024-09-21T18:54:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:10 smithi104 ceph-mon[32438]: 3.3 scrub ok 2024-09-21T18:54:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:10 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/915288783' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:54:10.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[29384]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:54:10.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[29384]: 3.3 scrub starts 2024-09-21T18:54:10.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[29384]: 3.3 scrub ok 2024-09-21T18:54:10.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/915288783' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:54:10.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[39192]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:54:10.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[39192]: 3.3 scrub starts 2024-09-21T18:54:10.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[39192]: 3.3 scrub ok 2024-09-21T18:54:10.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:10 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/915288783' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T18:54:10.973 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-21T18:54:10.990 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T18:54:11.000 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T18:54:11.000 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T18:54:11.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T18:54:11.001 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T18:54:11.001 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:54:11.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:54:11.001 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:54:11.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd out 0 2024-09-21T18:54:11.002 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T18:54:11.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:11.002 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T18:54:11.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:54:11.003 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T18:54:11.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T18:54:11.004 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T18:54:11.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:54:11.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:11.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:11.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:11.281 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:11.293 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:11.301 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:11.315 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:54:11 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[70807]: ts=2024-09-21T18:54:11.013Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001986906s 2024-09-21T18:54:11.383 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:11.408 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:11.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:11.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:11.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:11.654 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:11.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:11 smithi081 ceph-mon[29384]: 4.1b scrub starts 2024-09-21T18:54:11.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:11 smithi081 ceph-mon[29384]: 4.1b scrub ok 2024-09-21T18:54:11.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:11 smithi081 ceph-mon[39192]: 4.1b scrub starts 2024-09-21T18:54:11.689 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:11 smithi081 ceph-mon[39192]: 4.1b scrub ok 2024-09-21T18:54:11.717 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-mon[32438]: 4.1b scrub starts 2024-09-21T18:54:11.717 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-mon[32438]: 4.1b scrub ok 2024-09-21T18:54:11.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:11.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:11.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:11.986 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 systemd[1]: Started Ceph grafana.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:54:12.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:12.062 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:12.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:12.187 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:12.197 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:54:12.197 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:54:12.217 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:54:12.217 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:54:12.233 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:54:12.233 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.984665091Z level=info msg="Starting Grafana" version=10.4.8 commit=8c384b71bde142a25ddc01ed5a183ca572b2b7f7 branch=v10.4.x compiled=2024-09-21T18:54:11Z 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985040917Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985055264Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985061513Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985066808Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985076736Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985082178Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985087103Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985093043Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T18:54:12.239 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985098594Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985103486Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985108205Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985117897Z level=info msg=Target target=[all] 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985130046Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985134775Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985139185Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985143366Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T18:54:12.240 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985147807Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=settings t=2024-09-21T18:54:11.985152024Z level=info msg="App mode production" 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore t=2024-09-21T18:54:11.98573697Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore t=2024-09-21T18:54:11.985795907Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.986896846Z level=info msg="Starting DB migrations" 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.988057081Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.988883747Z level=info msg="Migration successfully executed" id="create migration_log table" duration=826.388µs 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.989343579Z level=info msg="Executing migration" id="create user table" 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.989884152Z level=info msg="Migration successfully executed" id="create user table" duration=539.186µs 2024-09-21T18:54:12.241 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.99292554Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.993568037Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=644.574µs 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.994027864Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.994554003Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=525.789µs 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.994954954Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.99557885Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=623.037µs 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.996169648Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.99678695Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=617.21µs 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.997247406Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.999320119Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=2.072269ms 2024-09-21T18:54:12.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:11 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:11.999758936Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.000391014Z level=info msg="Migration successfully executed" id="create user table v2" duration=631.307µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.00080357Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.001411117Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=607.238µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.003281233Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.003768155Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=485.573µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.00416938Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.004443272Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=272.436µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.004784278Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.005150085Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=365.608µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.00547254Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.006178833Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=706.034µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.006507532Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.006524576Z level=info msg="Migration successfully executed" id="Update user table charset" duration=16.325µs 2024-09-21T18:54:12.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.006916038Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.007595622Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=680.573µs 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.007930598Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.008065773Z level=info msg="Migration successfully executed" id="Add missing user data" duration=134.485µs 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.008451191Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.009147469Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=695.892µs 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.009484414Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.010008036Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=523.252µs 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.010338724Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.011052205Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=713.32µs 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.011405814Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.016583556Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.177366ms 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.016931894Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.017652498Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=719.544µs 2024-09-21T18:54:12.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.017986201Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.018115181Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=129.129µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.01848308Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.018937912Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=453.965µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.019332856Z level=info msg="Executing migration" id="update login field with orgid to allow for multiple service accounts with same name across orgs" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.019526575Z level=info msg="Migration successfully executed" id="update login field with orgid to allow for multiple service accounts with same name across orgs" duration=193.992µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.019942223Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.020491251Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=548.797µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.020880024Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.021316992Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=437.756µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.021719982Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.022242529Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=524.149µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.022603827Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.02306004Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=456.195µs 2024-09-21T18:54:12.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.023410255Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.023903064Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=491.512µs 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.024244909Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.024260282Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=15.853µs 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.024663593Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.025116132Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=452.368µs 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.025453253Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.025918434Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=465.393µs 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.026262521Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.026686935Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=424.237µs 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.027013553Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.027433355Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=418.551µs 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.027775949Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.029734083Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=1.957922ms 2024-09-21T18:54:12.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.030628185Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.031148127Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=519.893µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.031488804Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.031944972Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=455.911µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.032264968Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.032711587Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=446.241µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.033070393Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.033517441Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=446.728µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.03586107Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.036358372Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=497.099µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.036732627Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.036991583Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=257.297µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.037317834Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.037678446Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=360.55µs 2024-09-21T18:54:12.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.038010134Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T18:54:12.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.038254751Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=244.494µs 2024-09-21T18:54:12.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.038637968Z level=info msg="Executing migration" id="create star table" 2024-09-21T18:54:12.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.039073807Z level=info msg="Migration successfully executed" id="create star table" duration=435.606µs 2024-09-21T18:54:12.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.039423873Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T18:54:12.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.039936755Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=512.762µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.040286383Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.040770958Z level=info msg="Migration successfully executed" id="create org table v1" duration=484.273µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.041166071Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.041621831Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=455.558µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.041954804Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.042392793Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=436.937µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.042749037Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.043253975Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=505.02µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.043612259Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.044111616Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=499.229µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.046404586Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.046894569Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=489.912µs 2024-09-21T18:54:12.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.049129662Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.049163177Z level=info msg="Migration successfully executed" id="Update org table charset" duration=33.93µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.051973884Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.052005861Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=32.595µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.05238678Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.052487002Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=100.281µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.055292817Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.055823882Z level=info msg="Migration successfully executed" id="create dashboard table" duration=530.815µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.058203256Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.058801593Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=600.422µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.061002504Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.06159558Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=594.557µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.061937927Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.062351393Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=413.379µs 2024-09-21T18:54:12.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.064795518Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.06526908Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=473.402µs 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.067717045Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.06815316Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=435.981µs 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.070713026Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.074048747Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.338083ms 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.074419051Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.074933451Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=513.396µs 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.077157155Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.077633385Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=475.99µs 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.080054859Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.080643744Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=590.396µs 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.082644239Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T18:54:12.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.082864712Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=220.532µs 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.083193963Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.083780574Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=586.482µs 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.085977998Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.086017275Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=38.702µs 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.08865795Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.08974768Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.089459ms 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.091516437Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.09266717Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.152858ms 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.094119003Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.095271952Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.155075ms 2024-09-21T18:54:12.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.09651841Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.097030391Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=511.922µs 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.099178052Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.100261922Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.082838ms 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.101699723Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.102195751Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=495.722µs 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.102567902Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.103042709Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=458.81µs 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.103409296Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.103424Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=15.294µs 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.104252288Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.104285511Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=33.288µs 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.105100763Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T18:54:12.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.106253779Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.152727ms 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.106620582Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.107737113Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.116439ms 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.108097165Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.109256456Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.159176ms 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.109627715Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.110781952Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.153982ms 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.111119536Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.111256352Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=137.56µs 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.111700703Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.112159907Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=459.038µs 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.112561263Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.113006966Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=445.517µs 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.113454859Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T18:54:12.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.113469459Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=15.089µs 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.114480785Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.115099417Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=619.13µs 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.11543678Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.115892005Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=454.024µs 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.116269681Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.119228441Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=2.959633ms 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.119623568Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.120069788Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=446.093µs 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.120478657Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.121013874Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=533.797µs 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.121405229Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.121916592Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=510.9µs 2024-09-21T18:54:12.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.124204021Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.124427845Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=223.606µs 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.127005604Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.127401921Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=397.079µs 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.129898226Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.131140318Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.240915ms 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.131507433Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.131985228Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=478.495µs 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.134267857Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.134401505Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=133.824µs 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.137277375Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.137390962Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=113.439µs 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.140032163Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T18:54:12.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.140675596Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=645.231µs 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.141026929Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.142281643Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.254544ms 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.144015172Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.144768226Z level=info msg="Migration successfully executed" id="create data_source table" duration=755.071µs 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.146598225Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.147097104Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=499.28µs 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.149445688Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.1499539Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=508.073µs 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.150333136Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.150822375Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=487.908µs 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.153027209Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.153511057Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=483.534µs 2024-09-21T18:54:12.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.155784562Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.159153338Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.370169ms 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.159569197Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.160149017Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=579.753µs 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.160577705Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.16110695Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=529.137µs 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.163467473Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.164007089Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=539.372µs 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.166305274Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.166688529Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=381.891µs 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.169107661Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.170516466Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.408519ms 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.170913637Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.172318768Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.404881ms 2024-09-21T18:54:12.259 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.173649694Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.173668469Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=18.929µs 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.176369822Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.176504501Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=134.575µs 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.179292452Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.180691922Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.401134ms 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.181081177Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.181218216Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=137.108µs 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.183908439Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.184050766Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=142.301µs 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.186627453Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.188041146Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.412236ms 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.189475033Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.189697452Z level=info msg="Migration successfully executed" id="Update uid value" duration=226.9µs 2024-09-21T18:54:12.260 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.190100685Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.190625128Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=524.088µs 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.193085199Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.193690729Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=607.317µs 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.195911593Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.196442594Z level=info msg="Migration successfully executed" id="create api_key table" duration=530.002µs 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.198514778Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.199047028Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=532.058µs 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.199411657Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.199950378Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=538.615µs 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.20217028Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.202718753Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=548.248µs 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.205180433Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T18:54:12.261 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.205766935Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=588.407µs 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.20768646Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.208167389Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=481.004µs 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.208546261Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.20902101Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=474.652µs 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.209397226Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.213107268Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.709088ms 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.213471156Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.213994709Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=523.52µs 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.214357533Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.214852332Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=495.687µs 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.215213329Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.215707308Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=493.781µs 2024-09-21T18:54:12.262 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.21606249Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.216562712Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=499.933µs 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.216927649Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.217174114Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=246.303µs 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.217731123Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.218105616Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=374.32µs 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.218557828Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.218576298Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=18.767µs 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.219594574Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.221195044Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.602351ms 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.221576936Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.223058201Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.481242ms 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.223410703Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T18:54:12.263 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.223521463Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=110.82µs 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.22442562Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.22612437Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.70069ms 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.226491062Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.227983286Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.492114ms 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.22835406Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.228876959Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=522.709µs 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.229280802Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.229664626Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=383.723µs 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.230099962Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.230678243Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=578.037µs 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.231053854Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T18:54:12.264 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.231590298Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=536.187µs 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.231949109Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.232517456Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=566.543µs 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.232894354Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.233393196Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=498.655µs 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.233755303Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.233803772Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=48.718µs 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.234627733Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.234648154Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=20.536µs 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.235483302Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.237123888Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.640486ms 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.237589426Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.239930127Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=2.340429ms 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.240401957Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T18:54:12.265 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.240456341Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=54.822µs 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.241244927Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.241904178Z level=info msg="Migration successfully executed" id="create quota table v1" duration=659.136µs 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.242347226Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.243077866Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=730.339µs 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.243499565Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.243521348Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=22.733µs 2024-09-21T18:54:12.266 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.244391112Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T18:54:12.266 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:54:12.267 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:54:12.267 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:54:12.267 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:54:12.280 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:54:12.281 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:54:12.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:12.299 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:54:12.299 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:54:12.310 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:12.316 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:54:12.316 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:54:12.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:12.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1696420637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:12.489 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.245105109Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=713.863µs 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.245559649Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.246274242Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=714.341µs 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.246699362Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.249566541Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=2.866888ms 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.250000624Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.250021865Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=21.986µs 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.250852127Z level=info msg="Executing migration" id="create session table" 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.251665511Z level=info msg="Migration successfully executed" id="create session table" duration=813.372µs 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.252131914Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.252212093Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=80.572µs 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.252962623Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.25304451Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=82.196µs 2024-09-21T18:54:12.490 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.255911936Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T18:54:12.491 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.256563155Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=651.019µs 2024-09-21T18:54:12.491 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.258775123Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T18:54:12.491 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.259444459Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=669.273µs 2024-09-21T18:54:12.491 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.261675267Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T18:54:12.491 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.26169747Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=22.921µs 2024-09-21T18:54:12.491 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.264523117Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.26455284Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=30.571µs 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.267448706Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.269235135Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.786399ms 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.270494545Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.272364666Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.872081ms 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.273153077Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.27322778Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=74.85µs 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.276214989Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.276322744Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=110.033µs 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.278836628Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T18:54:12.492 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.27933745Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=500.592µs 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.281613435Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.281628253Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=15.618µs 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.284662645Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.286579425Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.918841ms 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.287615116Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.287709844Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=94.533µs 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.290469454Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.292384442Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.916138ms 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.293348804Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.295219511Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.870436ms 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.296195656Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.296236223Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=39.953µs 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.298916475Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T18:54:12.493 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.299578479Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=662.091µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.301828443Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.302501792Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=675.418µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.304424852Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.305198184Z level=info msg="Migration successfully executed" id="create alert table v1" duration=773.269µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.307044982Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.307695838Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=650.519µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.309978329Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.310647967Z level=info msg="Migration successfully executed" id="add index alert state" duration=671.694µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.312567345Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.313136844Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=569.652µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.315286583Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.315814583Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=527.664µs 2024-09-21T18:54:12.494 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.318174093Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.318774649Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=600.301µs 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.320823717Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.321359505Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=534.63µs 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.323487042Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.329176166Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.688921ms 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.329563147Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.330047381Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=484.291µs 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.332449066Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.333149044Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=701.712µs 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.335142398Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.335485833Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=345.782µs 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.337670997Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T18:54:12.495 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.338053008Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=381.912µs 2024-09-21T18:54:12.496 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.340367547Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.340877794Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=509.971µs 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.343074305Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.345173756Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.099163ms 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.345760553Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.348006984Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.246265ms 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.34837449Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.350653988Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.279425ms 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.351249933Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.353285574Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.03539ms 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.353993758Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.354588246Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=595.319µs 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.354957441Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.354972373Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=15.709µs 2024-09-21T18:54:12.497 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.355965551Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.356005181Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=43.593µs 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.356650564Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.35715181Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=501.805µs 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.357540838Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.358094319Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=553.846µs 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.358476671Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.359037202Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=535.438µs 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.359435586Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.35998155Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=545.773µs 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.362236646Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.362809825Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=557.441µs 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.36497335Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.367169564Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.195984ms 2024-09-21T18:54:12.498 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.368008496Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.370444535Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.438154ms 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.37082775Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.370962988Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=135.2µs 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.37370759Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.374369856Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=664.083µs 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.376253215Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.376801779Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=548.617µs 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.379036171Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.381168476Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.133997ms 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.381626658Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.381664356Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=38.109µs 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.384349821Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.384857262Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=507.286µs 2024-09-21T18:54:12.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.387122571Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.387709481Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=586.71µs 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.38981253Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.389870289Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=57.937µs 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.392606352Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.393164988Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=558.597µs 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.395278678Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.395913069Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=635.22µs 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.39768098Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.398222249Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=539.677µs 2024-09-21T18:54:12.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.400523216Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.401082895Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=559.419µs 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.403063033Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.403687345Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=623.276µs 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.405839703Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.40647637Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=634.994µs 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.408604804Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.408620064Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=15.81µs 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.411329808Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.413824532Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.494688ms 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.414177466Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.414723641Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=546.43µs 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.415063468Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.417426367Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.363014ms 2024-09-21T18:54:12.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.417831385Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.418338935Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=507.773µs 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.41874791Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.419329194Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=581.228µs 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.419709529Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.420254563Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=544.89µs 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.420643519Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.427119737Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.475074ms 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.427561408Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.428056776Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=495.391µs 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.428437934Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.429009084Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=570.986µs 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.429387809Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.429607966Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=220.121µs 2024-09-21T18:54:12.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.43023389Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.43064222Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=408.159µs 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.431084902Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.431215935Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=132.536µs 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.431937164Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.43426449Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.326844ms 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.434652708Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.436962401Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.309659ms 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.437323907Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.437884254Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=561.934µs 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.438237358Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T18:54:12.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.438776476Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=538.773µs 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.439124362Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.439273332Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=149.464µs 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.439968125Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.442310107Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.342144ms 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.442675995Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.44322029Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=544.18µs 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.443587087Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.443693491Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=106.529µs 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.444436594Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.444668311Z level=info msg="Migration successfully executed" id="Move region to single row" duration=232.439µs 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.445277826Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.445837979Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=560.629µs 2024-09-21T18:54:12.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.446186834Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.44673264Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=546.128µs 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.447072377Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.447635336Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=563.715µs 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.447978491Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.448525261Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=546.564µs 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.448899475Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.449445087Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=545.874µs 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.449822919Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.450461096Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=654.72µs 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.450825293Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.45088553Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=76.129µs 2024-09-21T18:54:12.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.451633711Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.452153855Z level=info msg="Migration successfully executed" id="create test_data table" duration=520.458µs 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.452502184Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.453051759Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=549.638µs 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.453417893Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.453965955Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=547.897µs 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.454328095Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T18:54:12.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.454903572Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=575.44µs 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.455265054Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.455379905Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=114.382µs 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.456132831Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.456360204Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=228.131µs 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.457090104Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.457201923Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=112.502µs 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.457887156Z level=info msg="Executing migration" id="create team table" 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.458350312Z level=info msg="Migration successfully executed" id="create team table" duration=462.894µs 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.458855346Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.45949022Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=635.818µs 2024-09-21T18:54:12.507 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.459901991Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.460595051Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=693.443µs 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.460973575Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.463616984Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.643042ms 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.463961414Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.464075372Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=113.907µs 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.466845251Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.467427298Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=581.936µs 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.469673381Z level=info msg="Executing migration" id="create team member table" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.47017883Z level=info msg="Migration successfully executed" id="create team member table" duration=505.95µs 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.472630934Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.473179622Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=549.13µs 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.475384724Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.475980559Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=596.837µs 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.478328696Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T18:54:12.508 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.47890674Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=577.68µs 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.481053484Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.483836857Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.782847ms 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.4841971Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.487046665Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.849357ms 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.487407551Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.490107458Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.699689ms 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.490510991Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.491143459Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=632.565µs 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.493384016Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.493974848Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=591.103µs 2024-09-21T18:54:12.509 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.494336426Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T18:54:12.510 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.495008502Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=671.831µs 2024-09-21T18:54:12.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:12.558 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1696420637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T18:54:12.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:12.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1696420637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:12.598 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:54:12.598 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:54:12.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:12.681 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.497117985Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.497723964Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=604.607µs 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.500046911Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.500629432Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=582.306µs 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.500973877Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.501522446Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=548.411µs 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.50402688Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.504721259Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=697.287µs 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.50693852Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.507589371Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=652.619µs 2024-09-21T18:54:12.741 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.509517637Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T18:54:12.742 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.509891848Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=374.584µs 2024-09-21T18:54:12.742 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.510265561Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T18:54:12.742 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.510411397Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=144.535µs 2024-09-21T18:54:12.742 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.513033501Z level=info msg="Executing migration" id="create tag table" 2024-09-21T18:54:12.742 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.513559637Z level=info msg="Migration successfully executed" id="create tag table" duration=526.395µs 2024-09-21T18:54:12.742 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.515712785Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.516296597Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=584.133µs 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.518473246Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.518966103Z level=info msg="Migration successfully executed" id="create login attempt table" duration=491.455µs 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.519340697Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.519916605Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=576.316µs 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.522067299Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.522652382Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=584.817µs 2024-09-21T18:54:12.743 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.524794071Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.532941567Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=8.149579ms 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.533314349Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.533783059Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=468.654µs 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.534153126Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.534728335Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=573.409µs 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.537007088Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.537374467Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=369.393µs 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.539551591Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.540103153Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=553.85µs 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.542206019Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.542681897Z level=info msg="Migration successfully executed" id="create user auth table" duration=475.635µs 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.543048882Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T18:54:12.744 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.543618646Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=569.617µs 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.545815895Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.54586156Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=46.363µs 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.548525338Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.551670131Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.149094ms 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.552049016Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.55496755Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.918386ms 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.555321708Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.558188919Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.867289ms 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.558561213Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.561457611Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.896251ms 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.561824367Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.562376754Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=552.331µs 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.562736256Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.56572551Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=2.989734ms 2024-09-21T18:54:12.745 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.566071525Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.566615319Z level=info msg="Migration successfully executed" id="create server_lock table" duration=543.511µs 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.566966592Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.567521819Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=554.978µs 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.567875765Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.568428095Z level=info msg="Migration successfully executed" id="create user auth token table" duration=552.245µs 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.568813161Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.56938943Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=577.267µs 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.569754502Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.570333598Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=579.1µs 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.570690518Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.571317737Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=627.787µs 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.571667006Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.57480724Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.14014ms 2024-09-21T18:54:12.746 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.575185116Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.575782283Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=597.044µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.576159304Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.576741401Z level=info msg="Migration successfully executed" id="create cache_data table" duration=582.04µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.577117725Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.57769525Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=577.392µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.578071351Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.578661691Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=590.056µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.579022841Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.579631282Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=608.235µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.579966886Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.580014764Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=47.811µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.580407752Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.580472174Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=65.122µs 2024-09-21T18:54:12.747 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.580857656Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.581413753Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=555.023µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.581784416Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.582381734Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=597.141µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.582765668Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.583368815Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=602.835µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.583708227Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.583756408Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=48.418µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.584144187Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.584716806Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=572.498µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.585052752Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.585628136Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=576.239µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.585970188Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.586595862Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=625.457µs 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.588687397Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T18:54:12.748 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.589291468Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=603.915µs 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.591318057Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.594653927Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.335696ms 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.595013195Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.595691487Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=678.062µs 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.596104787Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.596172957Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=68.401µs 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.598970501Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.599710145Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=741.403µs 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.601465857Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.602137441Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=671.518µs 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.604182895Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.604824604Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=641.593µs 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.606888946Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T18:54:12.749 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.606963235Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=75.554µs 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.609689031Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.61038533Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=694.616µs 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.612519607Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.61317131Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=652.268µs 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.615219429Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.615991607Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=773.891µs 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.61634947Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.616962272Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=612.899µs 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.619181978Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.622682507Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.501501ms 2024-09-21T18:54:12.750 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.623012104Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T18:54:12.751 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.623574155Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=560.747µs 2024-09-21T18:54:12.751 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.625744044Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T18:54:12.751 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.626316005Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=570.819µs 2024-09-21T18:54:12.751 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.626650364Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T18:54:12.751 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.644169092Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.519683ms 2024-09-21T18:54:12.751 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.644573865Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.661282202Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.708144ms 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.661654889Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.662249926Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=594.817µs 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.662607914Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.663186198Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=578.158µs 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.663545471Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.666755114Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.210068ms 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.66712119Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.670356593Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.235125ms 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.670731476Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.671334202Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=604.566µs 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.671691954Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T18:54:12.752 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.672295912Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=603.657µs 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.672662574Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.673270936Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=606.604µs 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.673624805Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.674291071Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=665.848µs 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.674646143Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.674684376Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=38.514µs 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.675402482Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.678865686Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.462921ms 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.679212491Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T18:54:12.753 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.682573191Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.338462ms 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.682918598Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.686282354Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.363571ms 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.68665321Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.6872244Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=571.119µs 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.687588227Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.688187505Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=599.091µs 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.688513254Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.691918836Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.405448ms 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.692273375Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.69565702Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.383389ms 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.696002944Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.696608626Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=606.556µs 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.696951442Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T18:54:12.754 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.700298925Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.347325ms 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.700650058Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.703988526Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.338383ms 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.704340403Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.704377145Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=37.107µs 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.705198172Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.705933652Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=735.11µs 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.706300894Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.706960233Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=657.73µs 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.707306044Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.708002911Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=696.678µs 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.708373023Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.708409901Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=37.296µs 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.709387005Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T18:54:12.755 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.713120035Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.734958ms 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.71348051Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.71706445Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.583756ms 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.717420414Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.720929534Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.509013ms 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.721296687Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.724796737Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.499781ms 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.725207278Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.728781516Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.574285ms 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.72915958Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.729198081Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=38.876µs 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.729599487Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.730083737Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=484.081µs 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.730487497Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.734193088Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.705661ms 2024-09-21T18:54:12.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.734551543Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T18:54:12.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.73458965Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=38.459µs 2024-09-21T18:54:12.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.735402862Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T18:54:12.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.739076298Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.671988ms 2024-09-21T18:54:12.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.739501512Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T18:54:12.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.740166183Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=664.647µs 2024-09-21T18:54:12.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.740566991Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T18:54:12.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:12.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:12.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:12.991 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.744414038Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.846677ms 2024-09-21T18:54:12.991 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.74479283Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T18:54:12.991 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.74532813Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=535.106µs 2024-09-21T18:54:12.991 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.747694563Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T18:54:12.991 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.7483183Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=623.622µs 2024-09-21T18:54:12.991 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.748695509Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.752438Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.742178ms 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.752822321Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.753340937Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=518.612µs 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.755603257Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.756231338Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=627.939µs 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.758419127Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.758986334Z level=info msg="Migration successfully executed" id="create alert_image table" duration=566.983µs 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.759372072Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.759998018Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=626.615µs 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.762122277Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T18:54:12.992 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.762182425Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=61.433µs 2024-09-21T18:54:12.993 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.76488693Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T18:54:12.994 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.765691911Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=804.348µs 2024-09-21T18:54:12.994 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.767804615Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T18:54:12.994 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.76858331Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=779.795µs 2024-09-21T18:54:12.994 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.768950523Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T18:54:12.994 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.769199251Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T18:54:12.994 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.771879418Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.772194605Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=315.068µs 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.774560438Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.775209193Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=648.529µs 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.777439544Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.781149093Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.709317ms 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.781570536Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.782273731Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=703.083µs 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.784328027Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.785010286Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=682.02µs 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.787164933Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.787740807Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=575.433µs 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.789769741Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.790455856Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=685.94µs 2024-09-21T18:54:12.995 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.790845466Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.791493714Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=648.294µs 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.793601399Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.793621767Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=21.184µs 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.796310635Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.796373684Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=61.006µs 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.799058295Z level=info msg="Executing migration" id="add library_element folder uid" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.803397876Z level=info msg="Migration successfully executed" id="add library_element folder uid" duration=4.338798ms 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.803827784Z level=info msg="Executing migration" id="populate library_element folder_uid" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.804103491Z level=info msg="Migration successfully executed" id="populate library_element folder_uid" duration=275.675µs 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.806651457Z level=info msg="Executing migration" id="add index library_element org_id-folder_uid-name-kind" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.807356941Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_uid-name-kind" duration=705.352µs 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.809488412Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.809844078Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=358.107µs 2024-09-21T18:54:12.996 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.812203744Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.812957712Z level=info msg="Migration successfully executed" id="create data_keys table" duration=755.99µs 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.813358326Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.813970528Z level=info msg="Migration successfully executed" id="create secrets table" duration=612.142µs 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.816079419Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.838865092Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=22.787089ms 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.839243243Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.843375658Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.131707ms 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.843749607Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.843843368Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=95.855µs 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.844219325Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.864834939Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=20.615417ms 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.865207206Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T18:54:12.997 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.886296397Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=21.088933ms 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.88667488Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.88722143Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=544.984µs 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.88933394Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.890048988Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=713.656µs 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.890406876Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.890551023Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=144.152µs 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.891415345Z level=info msg="Executing migration" id="create permission table" 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.892002606Z level=info msg="Migration successfully executed" id="create permission table" duration=589.013µs 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.892359995Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.892970768Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=612.034µs 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.893356291Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.894004361Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=647.914µs 2024-09-21T18:54:12.998 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.89435905Z level=info msg="Executing migration" id="create role table" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.89492837Z level=info msg="Migration successfully executed" id="create role table" duration=569.272µs 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.895283267Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.899674113Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.390651ms 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.900028996Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.904408419Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.378817ms 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.904794589Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.905421152Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=626.519µs 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.905778874Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.90645988Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=681.14µs 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.906819793Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.907501764Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=681.799µs 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.908033872Z level=info msg="Executing migration" id="create team role table" 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.908670958Z level=info msg="Migration successfully executed" id="create team role table" duration=637.151µs 2024-09-21T18:54:12.999 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.909040718Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.909718818Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=677.692µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.910055702Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.910729542Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=671.897µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.911074057Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.911737607Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=663.173µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.912076021Z level=info msg="Executing migration" id="create user role table" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.912666121Z level=info msg="Migration successfully executed" id="create user role table" duration=588.42µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.913033178Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.913709085Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=675.649µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.91407607Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.914762015Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=685.658µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.915127035Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.915802135Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=674.93µs 2024-09-21T18:54:13.000 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.916168209Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.916800953Z level=info msg="Migration successfully executed" id="create builtin role table" duration=633.073µs 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.917165876Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.917864649Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=698.696µs 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.918224238Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.918928979Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=706.563µs 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.919288824Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.923923086Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.633457ms 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.924331946Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.925020078Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=688.286µs 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.925388543Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.926094437Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=705.76µs 2024-09-21T18:54:13.001 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.926467484Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.927134017Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=666.406µs 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.927496031Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.92815358Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=657.368µs 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.92851587Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.929042475Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=526.497µs 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.929412972Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.930125456Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=712.393µs 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.930497445Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.935152927Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.655227ms 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.93552494Z level=info msg="Executing migration" id="permission kind migration" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.94026394Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.737739ms 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.940657355Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.945296127Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.640369ms 2024-09-21T18:54:13.002 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.945644121Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.950170914Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.526587ms 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.950523663Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.951235511Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=707.252µs 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.951647995Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.952312Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=663.643µs 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.952661Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.953299984Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=639.476µs 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.953691596Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.954260794Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=569.108µs 2024-09-21T18:54:13.003 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.954622168Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.955287272Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=665.352µs 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.957513795Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.957601597Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=87.631µs 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.960249122Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.960270539Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=21.933µs 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.963049703Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.963309557Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=260.543µs 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.963687071Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.963999611Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=313.069µs 2024-09-21T18:54:13.004 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.966618016Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.967037697Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=419.654µs 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.969337276Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.969479552Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=142.201µs 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.972263814Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.972563373Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=299.48µs 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.97296332Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.973469296Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=507.006µs 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.973811807Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.974509723Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=695.941µs 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.974860798Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.979332515Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.469813ms 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.97968054Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.979717848Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=37.467µs 2024-09-21T18:54:13.005 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.98009939Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.980712607Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=612.784µs 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.981041865Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.981731576Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=689.74µs 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.982069733Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.982720427Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=650.464µs 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.9830419Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.987909841Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.840226ms 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.98826558Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.988925126Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=659.419µs 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.989263857Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.989937785Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=674.549µs 2024-09-21T18:54:13.006 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:12 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:12.990297288Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-21T18:54:13.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:13.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:13.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:13.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:13.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.007838965Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=17.541084ms 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.008199543Z level=info msg="Executing migration" id="create correlation v2" 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.008954783Z level=info msg="Migration successfully executed" id="create correlation v2" duration=754.474µs 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.009332713Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.010012003Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=679.178µs 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.010371938Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.011033941Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=661.701µs 2024-09-21T18:54:13.242 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.011375436Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.012070212Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=693.612µs 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.012412395Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.012577657Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=165.126µs 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.01317582Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.013721598Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=545.649µs 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.014087127Z level=info msg="Executing migration" id="add provisioning column" 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.018798542Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.711272ms 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.019151481Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.019667853Z level=info msg="Migration successfully executed" id="create entity_events table" duration=516.106µs 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.02004663Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.020687346Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=640.496µs 2024-09-21T18:54:13.243 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.021070875Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.021323929Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.021967409Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.022241958Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.022639794Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.023166685Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=526.824µs 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.023527455Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.024166182Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=640.047µs 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.024511841Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.025173772Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=661.795µs 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.025645498Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.026308187Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=662.538µs 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.028749102Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T18:54:13.244 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.029458626Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=711.636µs 2024-09-21T18:54:13.245 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.031541306Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.032210157Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=668.711µs 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.032565486Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.033101349Z level=info msg="Migration successfully executed" id="Drop public config table" duration=535.746µs 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.035702019Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.036500149Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=800.265µs 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.038601898Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.039285744Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=685.596µs 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.04139329Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.042121071Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=727.647µs 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.042461903Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.043126288Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=666.058µs 2024-09-21T18:54:13.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.045350347Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.059619833Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=14.284615ms 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.059983151Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.065240179Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.256868ms 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.065606164Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.070403321Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.797108ms 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.070773388Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.070909958Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=136.62µs 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.073853952Z level=info msg="Executing migration" id="add share column" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.07881357Z level=info msg="Migration successfully executed" id="add share column" duration=4.959393ms 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.079162716Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.07929054Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=127.894µs 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.082294333Z level=info msg="Executing migration" id="create file table" 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.083057817Z level=info msg="Migration successfully executed" id="create file table" duration=765.681µs 2024-09-21T18:54:13.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.083412429Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.084096365Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=683.765µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.084444738Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.085160946Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=715.86µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.087412417Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.088044805Z level=info msg="Migration successfully executed" id="create file_meta table" duration=630.665µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.090287134Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.091039312Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=751.897µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.091396685Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.091434371Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=38.089µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.09407382Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.094133316Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=59.949µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.096820772Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.09716958Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=348.585µs 2024-09-21T18:54:13.248 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.09955571Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.099693571Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=137.865µs 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.1001946Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.101056044Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=861.851µs 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.102891332Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.108107978Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=5.216357ms 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.108517994Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.108633701Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=113.955µs 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.111337274Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.112064272Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=727.722µs 2024-09-21T18:54:13.249 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.112425117Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.11267155Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=246.608µs 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.115224251Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.115410058Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=189.755µs 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.117803353Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.118100927Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=298.07µs 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.120603042Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.125417753Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.818202ms 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.125787641Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.13130799Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=5.520682ms 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.131660837Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.132330981Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=670.566µs 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.134418169Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T18:54:13.250 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.191326919Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=56.908314ms 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.191721467Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.192430094Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=709.285µs 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.192799913Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.193512796Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=711.491µs 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.195420769Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.213215357Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=17.792207ms 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.213698001Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.21890316Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.205021ms 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.219278032Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.219468796Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=190.803µs 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.21987122Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.219979015Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=107.947µs 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.220718998Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T18:54:13.251 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.220855372Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=136.162µs 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.221574704Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.221706096Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=133.665µs 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.222586079Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.222786876Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=202.743µs 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.223177219Z level=info msg="Executing migration" id="create folder table" 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.223840076Z level=info msg="Migration successfully executed" id="create folder table" duration=663.108µs 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.22422498Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.224950213Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=725.387µs 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.225302646Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T18:54:13.252 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.226042206Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=739.845µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.226396318Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.226413499Z level=info msg="Migration successfully executed" id="Update folder title length" duration=17.839µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.226837465Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.227629131Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=791.234µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.228091176Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.228769508Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=678.308µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.22914122Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.22985432Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=714.438µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.230206388Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.230496834Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=290.214µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.230855381Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.231032847Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=177.632µs 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.23170447Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-21T18:54:13.253 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.232387074Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=683.901µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.232748287Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.23343646Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=688.215µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.233785262Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.234498022Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=713.303µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.23487863Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.235625421Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=746.666µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.235975396Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.236697361Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=722.362µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.237035288Z level=info msg="Executing migration" id="create anon_device table" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.237616381Z level=info msg="Migration successfully executed" id="create anon_device table" duration=581.407µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.237955153Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.238712756Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=757.419µs 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.239070014Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-21T18:54:13.254 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.239758801Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=688.309µs 2024-09-21T18:54:13.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.240150494Z level=info msg="Executing migration" id="create signing_key table" 2024-09-21T18:54:13.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.240794217Z level=info msg="Migration successfully executed" id="create signing_key table" duration=644.062µs 2024-09-21T18:54:13.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.241134601Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-21T18:54:13.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.241808774Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=674.002µs 2024-09-21T18:54:13.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.242178744Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-21T18:54:13.255 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.243024752Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=845.465µs 2024-09-21T18:54:13.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.295 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:13.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:13.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:13.422 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_0' created 2024-09-21T18:54:13.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T18:54:13.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T18:54:13.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-mon[32438]: Deploying daemon node-exporter.a on smithi081 2024-09-21T18:54:13.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3658087320' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T18:54:13.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-mon[32438]: 3.1a scrub starts 2024-09-21T18:54:13.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-mon[32438]: 3.1a scrub ok 2024-09-21T18:54:13.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.243397193Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-21T18:54:13.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.243608727Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=216.736µs 2024-09-21T18:54:13.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.244229594Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-21T18:54:13.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.251334604Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=7.103387ms 2024-09-21T18:54:13.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.251689143Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-21T18:54:13.499 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.252146891Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=460.375µs 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.252566964Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.252577806Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=11.512µs 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.253392063Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.254084807Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=692.611µs 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.254428834Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.254438383Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=10.226µs 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.255280817Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.25602218Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=741.01µs 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.256378557Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.257101967Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=723.21µs 2024-09-21T18:54:13.500 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.25746102Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.258137046Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=675.924µs 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.25847987Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.25894418Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=464.884µs 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.259320223Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.259481983Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=162.303µs 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.259859106Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.259896086Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=37.192µs 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.262865957Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.26806763Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.201425ms 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.268431539Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.273906026Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.47338ms 2024-09-21T18:54:13.501 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.274335132Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-21T18:54:13.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.274599556Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=265.112µs 2024-09-21T18:54:13.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=migrator t=2024-09-21T18:54:13.274968968Z level=info msg="migrations completed" performed=551 skipped=0 duration=1.28695506s 2024-09-21T18:54:13.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore t=2024-09-21T18:54:13.275839516Z level=info msg="Created default organization" 2024-09-21T18:54:13.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=secrets t=2024-09-21T18:54:13.278100541Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T18:54:13.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=plugin.store t=2024-09-21T18:54:13.292727901Z level=info msg="Loading plugins..." 2024-09-21T18:54:13.502 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=local.finder t=2024-09-21T18:54:13.337159093Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=plugin.store t=2024-09-21T18:54:13.337178713Z level=info msg="Plugins loaded" count=55 duration=44.451517ms 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=query_data t=2024-09-21T18:54:13.339271021Z level=info msg="Query Service initialization" 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=live.push_http t=2024-09-21T18:54:13.34164276Z level=info msg="Live Push Gateway initialization" 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.migration t=2024-09-21T18:54:13.342650518Z level=info msg=Starting 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.migration t=2024-09-21T18:54:13.342963587Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.migration orgID=1 t=2024-09-21T18:54:13.343225154Z level=info msg="Migrating alerts for organisation" 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.migration orgID=1 t=2024-09-21T18:54:13.343652161Z level=info msg="Alerts found to migrate" alerts=0 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.migration t=2024-09-21T18:54:13.344888117Z level=info msg="Completed alerting migration" 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.state.manager t=2024-09-21T18:54:13.355320298Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=infra.usagestats.collector t=2024-09-21T18:54:13.356683535Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T18:54:13.503 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=provisioning.datasources t=2024-09-21T18:54:13.358369024Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=provisioning.datasources t=2024-09-21T18:54:13.371121774Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=provisioning.alerting t=2024-09-21T18:54:13.384273286Z level=info msg="starting to provision alerting" 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=provisioning.alerting t=2024-09-21T18:54:13.384288728Z level=info msg="finished to provision alerting" 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=grafanaStorageLogger t=2024-09-21T18:54:13.384411034Z level=info msg="Storage starting" 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.state.manager t=2024-09-21T18:54:13.384611802Z level=info msg="Warming state cache for startup" 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.state.manager t=2024-09-21T18:54:13.385018832Z level=info msg="State cache has been initialized" states=0 duration=406.262µs 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.multiorg.alertmanager t=2024-09-21T18:54:13.385188116Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ngalert.scheduler t=2024-09-21T18:54:13.385208211Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ticker t=2024-09-21T18:54:13.385239858Z level=info msg=starting first_tick=2024-09-21T18:54:20Z 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=http.server t=2024-09-21T18:54:13.386684015Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-21T18:54:13.504 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=http.server t=2024-09-21T18:54:13.386915647Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T18:54:13.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=provisioning.dashboard t=2024-09-21T18:54:13.42524287Z level=info msg="starting to provision dashboards" 2024-09-21T18:54:13.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore.transactions t=2024-09-21T18:54:13.446793406Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T18:54:13.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=plugins.update.checker t=2024-09-21T18:54:13.471211804Z level=info msg="Update check succeeded" duration=86.61989ms 2024-09-21T18:54:13.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=grafana.update.checker t=2024-09-21T18:54:13.471478302Z level=info msg="Update check succeeded" duration=86.211774ms 2024-09-21T18:54:13.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore.transactions t=2024-09-21T18:54:13.486168247Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T18:54:13.505 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore.transactions t=2024-09-21T18:54:13.496595744Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T18:54:13.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.548 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:13.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:13.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:13.670 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:13.756 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore.transactions t=2024-09-21T18:54:13.50720507Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-21T18:54:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=provisioning.dashboard t=2024-09-21T18:54:13.576899088Z level=info msg="finished to provision dashboards" 2024-09-21T18:54:13.772 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:13.793 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[29384]: Deploying daemon node-exporter.a on smithi081 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3658087320' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[29384]: 3.1a scrub starts 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[29384]: 3.1a scrub ok 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[39192]: Deploying daemon node-exporter.a on smithi081 2024-09-21T18:54:13.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3658087320' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T18:54:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[39192]: 3.1a scrub starts 2024-09-21T18:54:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:13 smithi081 ceph-mon[39192]: 3.1a scrub ok 2024-09-21T18:54:13.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:13.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:13.961 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T18:54:14.019 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:14.039 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:14.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:14.172 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:14.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=grafana-apiserver t=2024-09-21T18:54:13.940851968Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-21T18:54:14.258 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:54:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=grafana-apiserver t=2024-09-21T18:54:13.942081744Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-21T18:54:14.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:14.285 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:14.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:14.410 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:14.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:14.534 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: 3.18 scrub starts 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: 3.18 scrub ok 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: 3.c scrub starts 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: 3.c scrub ok 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3658087320' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T18:54:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[39192]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: 3.18 scrub starts 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: 3.18 scrub ok 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: 3.c scrub starts 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: 3.c scrub ok 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3658087320' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T18:54:14.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:14 smithi081 ceph-mon[29384]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T18:54:14.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:14.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: 3.18 scrub starts 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: 3.18 scrub ok 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: 3.c scrub starts 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: 3.c scrub ok 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3658087320' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T18:54:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:14 smithi104 ceph-mon[32438]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T18:54:14.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:14.779 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:14.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:14.901 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:14.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:15.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:15.024 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:15.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:15.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:15.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T18:54:15.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:15.270 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:15.372 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:15.394 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:15.440 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:54:15.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:15.516 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:15.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:54:15.618 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:15.639 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:15.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:15.755 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: 6.9 scrub starts 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: 6.9 scrub ok 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: 4.1a scrub starts 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: 4.1a scrub ok 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1135622341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1135622341' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[39192]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: 6.9 scrub starts 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: 6.9 scrub ok 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: 4.1a scrub starts 2024-09-21T18:54:15.756 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: 4.1a scrub ok 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1135622341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1135622341' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-mon[29384]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: 6.9 scrub starts 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: 6.9 scrub ok 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: 4.1a scrub starts 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: 4.1a scrub ok 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1135622341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1135622341' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:54:15.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:15 smithi104 ceph-mon[32438]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T18:54:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:15.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:15.900 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:15.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:16.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:16.024 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:16.064 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:15 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T18:54:15.752+0000 7ff1af1de640 -1 mon.a@0(leader).osd e81 definitely_dead 0 2024-09-21T18:54:16.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:16.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:16.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:16.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:16.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:16.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:16.426 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:16.427 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:16.427 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:16.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:16.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:16.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:16.649 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:16.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:16.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:16.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.11 scrub starts 2024-09-21T18:54:16.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.11 scrub ok 2024-09-21T18:54:16.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.5 scrub starts 2024-09-21T18:54:16.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.5 scrub ok 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: pgmap v30: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/863605377' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: from='client.34721 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:16.696 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.11 scrub starts 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.11 scrub ok 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.5 scrub starts 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.5 scrub ok 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: pgmap v30: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/863605377' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: from='client.34721 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T18:54:16.697 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:16 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T18:54:16.697 INFO:teuthology.orchestra.run.smithi081.stderr:marked out osd.0. 2024-09-21T18:54:16.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:16.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.11 scrub starts 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.11 scrub ok 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.5 scrub starts 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.5 scrub ok 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: pgmap v30: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/863605377' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: from='client.34721 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-21T18:54:16.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T18:54:16.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:16 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T18:54:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:16.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:16.934 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:16.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:17.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.059 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:17.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:17.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:17.181 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:17.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:17.304 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:17.315 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:54:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:54:17.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:17.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:17.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:17.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:17.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:17.682 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:17.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:17.763 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:54:17.763 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:54:17.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:17.805 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:17.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:17.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:17.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:17 smithi104 ceph-mon[32438]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T18:54:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:17 smithi104 ceph-mon[32438]: osdmap e82: 8 total, 8 up, 7 in 2024-09-21T18:54:18.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:54:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:54:18.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:18.053 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:18.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:17 smithi081 ceph-mon[29384]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T18:54:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:17 smithi081 ceph-mon[29384]: osdmap e82: 8 total, 8 up, 7 in 2024-09-21T18:54:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:17 smithi081 ceph-mon[39192]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T18:54:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:17 smithi081 ceph-mon[39192]: osdmap e82: 8 total, 8 up, 7 in 2024-09-21T18:54:18.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:18.176 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:18.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:18.298 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:18.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:18.424 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:18.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:54:18.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:18.551 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:18.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:18.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:18.700 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T18:54:18.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:18.802 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:18.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T18:54:18.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:18.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[39192]: pgmap v33: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 583 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[39192]: osdmap e83: 8 total, 8 up, 7 in 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3704714234' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[29384]: pgmap v33: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 583 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[29384]: osdmap e83: 8 total, 8 up, 7 in 2024-09-21T18:54:18.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3704714234' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:54:19.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:18 smithi104 ceph-mon[32438]: pgmap v33: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 583 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:18 smithi104 ceph-mon[32438]: osdmap e83: 8 total, 8 up, 7 in 2024-09-21T18:54:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3704714234' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:54:19.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:19.091 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:54:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:54:19.109 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:54:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:54:19.129 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:54:19.129 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:54:19.146 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:54:19.146 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:54:19.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:19.164 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:54:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:54:19.179 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:19.181 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:54:19.181 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:54:19.198 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:54:19.198 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:54:19.215 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:54:19.215 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:54:19.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:19.306 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:19.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:19.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:19.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:19.669 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:19 smithi081 bash[73064]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T18:54:19.681 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:19.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:19.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:19.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3704714234' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:54:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:19 smithi104 ceph-mon[32438]: osdmap e84: 8 total, 8 up, 7 in 2024-09-21T18:54:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2926887767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:20.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:20.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3704714234' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:54:20.046 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:19 smithi081 ceph-mon[29384]: osdmap e84: 8 total, 8 up, 7 in 2024-09-21T18:54:20.046 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2926887767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:20.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3704714234' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:54:20.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:19 smithi081 ceph-mon[39192]: osdmap e84: 8 total, 8 up, 7 in 2024-09-21T18:54:20.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2926887767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:20.128 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:20.129 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:20.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:20.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:20.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:20.309 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:20.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:20.432 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:20.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:20.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:20.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:20.681 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:20.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:20.807 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:20.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:20.931 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:20.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:20 smithi104 ceph-mon[32438]: pgmap v36: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 583 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:20 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:20 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:20 smithi104 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:54:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:20 smithi104 ceph-mon[32438]: osdmap e85: 8 total, 8 up, 7 in 2024-09-21T18:54:21.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:21.055 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:21.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[39192]: pgmap v36: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 583 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[39192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[39192]: osdmap e85: 8 total, 8 up, 7 in 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[29384]: pgmap v36: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 583 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[29384]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:54:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:20 smithi081 ceph-mon[29384]: osdmap e85: 8 total, 8 up, 7 in 2024-09-21T18:54:21.076 DEBUG:teuthology.orchestra.run.smithi081:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T18:54:21.078 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T18:54:21.078 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T18:54:21.142 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:hints = 1 2024-09-21T18:54:21.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T18:54:21.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object prefix: benchmark_data_smithi081_73391 2024-09-21T18:54:21.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:54:21.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 0 0 0 0 0 0 - 0 2024-09-21T18:54:21.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:21.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:21.297 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:21.335 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:21.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:21.474 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:21.576 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:21.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:21.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:21.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:21.816 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:21 smithi081 bash[73064]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T18:54:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:21.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:21.887 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:21.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:21.988 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:22.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:22.016 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:22.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:22.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 1 16 3903 3887 242.923 242.938 0.0038047 0.00410682 2024-09-21T18:54:22.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:22.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:22.254 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:22.292 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:22.316 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:22 smithi081 bash[73064]: Writing manifest to image destination 2024-09-21T18:54:22.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:22.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.14158.0:66 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'44, mlcod=20'40)", 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:50:57.803223+0000", 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 204.71239493100001, 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001142005, 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.521 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803223+0000", 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803223+0000", 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803224+0000", 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.359e-06 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803251+0000", 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6669000000000002e-05 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803359+0000", 2024-09-21T18:54:22.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107624 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803367+0000", 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2989999999999997e-06 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803480+0000", 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112834 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.803499+0000", 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9337000000000001e-05 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.804327+0000", 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082786399999999999 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:50:57.804365+0000", 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8019000000000003e-05 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.525 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34376.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:06.958110+0000", 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 75.557507745999999, 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.44773363700000002, 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34376", 2024-09-21T18:54:22.526 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/449352202", 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 1 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958110+0000", 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958110+0000", 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958119+0000", 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1530000000000003e-06 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958127+0000", 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3969999999999997e-06 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958138+0000", 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1051e-05 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.528 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958148+0000", 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9539999999999999e-06 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958204+0000", 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6134999999999997e-05 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958212+0000", 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8120000000000008e-06 2024-09-21T18:54:22.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.405799+0000", 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.44758725500000002 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.405844+0000", 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4879999999999997e-05 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14481.0:2 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:07.396931+0000", 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 75.118686953999998, 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0088250559999999995, 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14481", 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/697031715", 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 2 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.396931+0000", 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.396931+0000", 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.396933+0000", 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9429999999999999e-06 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.396938+0000", 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5419999999999996e-06 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.396949+0000", 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0859000000000001e-05 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.396960+0000", 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0848e-05 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.400834+0000", 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003873886 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.400842+0000", 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6899999999999992e-06 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.405674+0000", 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048320500000000001 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.535 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.405756+0000", 2024-09-21T18:54:22.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2237999999999994e-05 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:56 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T18:54:22.536 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:11.421212+0000", 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 71.094405666, 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011523149999999999, 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 56 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421212+0000", 2024-09-21T18:54:22.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421212+0000", 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421214+0000", 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.962e-06 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421219+0000", 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1279999999999999e-06 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421232+0000", 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2598e-05 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421239+0000", 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6069999999999996e-06 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421284+0000", 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5132000000000001e-05 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421490+0000", 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020636999999999999 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421604+0000", 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114002 2024-09-21T18:54:22.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.421963+0000", 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035879500000000002 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422260+0000", 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029688799999999998 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422302+0000", 2024-09-21T18:54:22.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2024999999999998e-05 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422328+0000", 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6542000000000001e-05 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422365+0000", 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6266000000000001e-05 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 94695648321408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:11.422449+0000", 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 71.093169380000006, 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013005499999999999, 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 57 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.543 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422449+0000", 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422449+0000", 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422450+0000", 2024-09-21T18:54:22.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3400000000000001e-06 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422456+0000", 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7610000000000004e-06 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422475+0000", 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9440999999999999e-05 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422482+0000", 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8789999999999997e-06 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422538+0000", 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5965999999999999e-05 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422611+0000", 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2906999999999998e-05 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.422748+0000", 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013671199999999999 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.423186+0000", 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00043834400000000002 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.423362+0000", 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017656400000000001 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.423623+0000", 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026052699999999998 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.423679+0000", 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.643e-05 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:11.423749+0000", 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9678999999999995e-05 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.548 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:207 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.424435+0000", 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.091183297000001, 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016924539999999999, 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207 2024-09-21T18:54:22.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424435+0000", 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424435+0000", 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424437+0000", 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5459999999999998e-06 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424447+0000", 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0249e-05 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424474+0000", 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6928999999999999e-05 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424482+0000", 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8320000000000006e-06 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424543+0000", 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0392e-05 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424709+0000", 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016688599999999999 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.424994+0000", 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028419399999999998 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.425601+0000", 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00060758899999999996 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.425733+0000", 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000131273 2024-09-21T18:54:22.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.426064+0000", 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00033156400000000001 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.426099+0000", 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5102999999999999e-05 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.426127+0000", 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7897000000000001e-05 2024-09-21T18:54:22.554 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:209 4.3 4:c60da066:gc::gc.4:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.427616+0000", 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.088001996000003, 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015665709999999999, 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 209 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427616+0000", 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427616+0000", 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427618+0000", 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.012e-06 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427625+0000", 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2439999999999996e-06 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427640+0000", 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4634000000000001e-05 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427649+0000", 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9670000000000007e-06 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427698+0000", 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9611000000000003e-05 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.427804+0000", 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105559 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.428110+0000", 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030605699999999999 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.428599+0000", 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00048877000000000002 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.429029+0000", 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00043011200000000002 2024-09-21T18:54:22.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.429079+0000", 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9718000000000001e-05 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.429134+0000", 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5711999999999998e-05 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.429183+0000", 2024-09-21T18:54:22.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8174999999999999e-05 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34409.0:213 5.3 5:c609908c:::notify.5:head [watch unwatch cookie 94905314017264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.435433+0000", 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.080184725999999, 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002716956, 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34409", 2024-09-21T18:54:22.561 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2645283795", 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 213 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435433+0000", 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435433+0000", 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435437+0000", 2024-09-21T18:54:22.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4809999999999998e-06 2024-09-21T18:54:22.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435445+0000", 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4579999999999993e-06 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435472+0000", 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6925000000000001e-05 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435483+0000", 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0689e-05 2024-09-21T18:54:22.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435556+0000", 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3028000000000006e-05 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435648+0000", 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2070000000000004e-05 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.435825+0000", 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017673600000000001 2024-09-21T18:54:22.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.436373+0000", 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00054859899999999998 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.437986+0000", 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016131489999999999 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.438029+0000", 2024-09-21T18:54:22.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2622e-05 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.438083+0000", 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4221000000000001e-05 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.438150+0000", 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6977999999999995e-05 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:366 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.482402+0000", 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.033216241000005, 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012527040000000001, 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 366 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.568 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482402+0000", 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482402+0000", 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482403+0000", 2024-09-21T18:54:22.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3349999999999999e-06 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482409+0000", 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.682e-06 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482424+0000", 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5311e-05 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482432+0000", 2024-09-21T18:54:22.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7389999999999999e-06 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482491+0000", 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9197000000000003e-05 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482616+0000", 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012528499999999999 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T18:54:22.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.482820+0000", 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000203628 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.483202+0000", 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038221400000000002 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.483485+0000", 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028241600000000001 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.483555+0000", 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0217999999999998e-05 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.483612+0000", 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7698000000000002e-05 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.483654+0000", 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1980999999999997e-05 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:386 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.486387+0000", 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.029230987000005, 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019418910000000001, 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 386 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.574 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486387+0000", 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486387+0000", 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486388+0000", 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.403e-06 2024-09-21T18:54:22.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486394+0000", 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9159999999999998e-06 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486410+0000", 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5571000000000001e-05 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486418+0000", 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1419999999999993e-06 2024-09-21T18:54:22.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486487+0000", 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8808999999999998e-05 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486600+0000", 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113398 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.486785+0000", 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000184363 2024-09-21T18:54:22.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.487213+0000", 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00042810199999999997 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.487655+0000", 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044221899999999999 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.488241+0000", 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00058626700000000002 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.488288+0000", 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7228999999999998e-05 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.488329+0000", 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0472000000000001e-05 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:402 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.490315+0000", 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.025302685, 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013529, 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 402 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.580 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490315+0000", 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490315+0000", 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490317+0000", 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3829999999999999e-06 2024-09-21T18:54:22.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490322+0000", 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2299999999999999e-06 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490334+0000", 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2206e-05 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490341+0000", 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9619999999999996e-06 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490390+0000", 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9048999999999997e-05 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490514+0000", 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012413499999999999 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:54:22.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.490699+0000", 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018511100000000001 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.491059+0000", 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035929799999999997 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.491439+0000", 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038045899999999998 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.491581+0000", 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014169599999999999 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.491632+0000", 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0748999999999998e-05 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.491668+0000", 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6622000000000003e-05 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.585 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:409 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.493136+0000", 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.022481756000005, 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00118078, 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 409 2024-09-21T18:54:22.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493136+0000", 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493136+0000", 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493138+0000", 2024-09-21T18:54:22.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.77e-06 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493146+0000", 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5110000000000003e-06 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493163+0000", 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6934000000000001e-05 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493183+0000", 2024-09-21T18:54:22.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.931e-05 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493238+0000", 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5473999999999998e-05 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493317+0000", 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8877999999999999e-05 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:54:22.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493476+0000", 2024-09-21T18:54:22.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015897900000000001 2024-09-21T18:54:22.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.493857+0000", 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038049299999999998 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494026+0000", 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016945400000000001 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494211+0000", 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018537800000000001 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494277+0000", 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5983999999999997e-05 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494317+0000", 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9614999999999999e-05 2024-09-21T18:54:22.592 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:412 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.494706+0000", 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.020911752000004, 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012553880000000001, 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.593 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 412 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494706+0000", 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494706+0000", 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494708+0000", 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.44e-06 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494714+0000", 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.178e-06 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494726+0000", 2024-09-21T18:54:22.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1800999999999999e-05 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494733+0000", 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6389999999999992e-06 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494798+0000", 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5011000000000005e-05 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.494925+0000", 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012668999999999999 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.495154+0000", 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000229003 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.495578+0000", 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00042413599999999998 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.495707+0000", 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012861700000000001 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.495876+0000", 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000168796 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.495924+0000", 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8754000000000001e-05 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.495962+0000", 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7323000000000002e-05 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:437 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.501579+0000", 2024-09-21T18:54:22.599 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.014039163000007, 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015308159999999999, 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 437 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501579+0000", 2024-09-21T18:54:22.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501579+0000", 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501580+0000", 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.237e-06 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501585+0000", 2024-09-21T18:54:22.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1460000000000004e-06 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501596+0000", 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0540000000000001e-05 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501608+0000", 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2143999999999999e-05 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501658+0000", 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0012000000000002e-05 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501731+0000", 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2971000000000003e-05 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.501908+0000", 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000177534 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502642+0000", 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00073350599999999998 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502692+0000", 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9876999999999997e-05 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503022+0000", 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00033050700000000001 2024-09-21T18:54:22.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.605 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:22 smithi081 podman[73064]: 2024-09-21 18:54:22.518478707 +0000 UTC m=+3.424693742 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T18:54:22.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503072+0000", 2024-09-21T18:54:22.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9666000000000002e-05 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503110+0000", 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7676000000000003e-05 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:439 4.16 4:6849fd06:gc::gc.17:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.502068+0000", 2024-09-21T18:54:22.606 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.013549698999995, 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014609110000000001, 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 439 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502068+0000", 2024-09-21T18:54:22.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502068+0000", 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502069+0000", 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2410000000000001e-06 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502075+0000", 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0170000000000004e-06 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502086+0000", 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.149e-05 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502094+0000", 2024-09-21T18:54:22.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5120000000000002e-06 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502144+0000", 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0645e-05 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502241+0000", 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6947000000000005e-05 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502439+0000", 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019799600000000001 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502884+0000", 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044485299999999999 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.502979+0000", 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5268999999999999e-05 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503421+0000", 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044221000000000001 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503494+0000", 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2300999999999998e-05 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503529+0000", 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5429999999999998e-05 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:443 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.503667+0000", 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.011951241999995, 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021513220000000002, 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 443 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503667+0000", 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503667+0000", 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503668+0000", 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.64e-07 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503673+0000", 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.848e-06 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503680+0000", 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7910000000000005e-06 2024-09-21T18:54:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503685+0000", 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2769999999999999e-06 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503716+0000", 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1250000000000001e-05 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.503763+0000", 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7148999999999999e-05 2024-09-21T18:54:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.504472+0000", 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00070922900000000002 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.504838+0000", 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036587600000000001 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.505560+0000", 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00072165100000000002 2024-09-21T18:54:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.505732+0000", 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017257399999999999 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.505784+0000", 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1895999999999999e-05 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.505818+0000", 2024-09-21T18:54:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3816999999999998e-05 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:451 4.16 4:6849fd06:gc::gc.17:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.506499+0000", 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.009119181000003, 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013453600000000001, 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 451 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506499+0000", 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506499+0000", 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506500+0000", 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.469e-06 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506506+0000", 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2639999999999999e-06 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506520+0000", 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4355000000000001e-05 2024-09-21T18:54:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506527+0000", 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5050000000000001e-06 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506578+0000", 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1100000000000002e-05 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506648+0000", 2024-09-21T18:54:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9757000000000003e-05 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.506804+0000", 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015563799999999999 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.507174+0000", 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037015000000000003 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.507585+0000", 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041098399999999998 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.507748+0000", 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016274799999999999 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.507806+0000", 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8208000000000002e-05 2024-09-21T18:54:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.507844+0000", 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8182000000000001e-05 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:467 4.3 4:c57bc89b:::obj_delete_at_hint.0000000013:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.511950+0000", 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.003667934999996, 2024-09-21T18:54:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001222634, 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 467 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.511950+0000", 2024-09-21T18:54:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.511950+0000", 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.511952+0000", 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1110000000000002e-06 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.511960+0000", 2024-09-21T18:54:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5979999999999999e-06 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.511977+0000", 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7153e-05 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.511985+0000", 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4689999999999997e-06 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.512047+0000", 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1495000000000007e-05 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.512122+0000", 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5289000000000004e-05 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.512233+0000", 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011042 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.512754+0000", 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00052158700000000001 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.513002+0000", 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024780700000000001 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.513085+0000", 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2787000000000005e-05 2024-09-21T18:54:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.513136+0000", 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1239999999999997e-05 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.513173+0000", 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6677999999999997e-05 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:738 4.6 4:6192870f:::obj_delete_at_hint.0000000053:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.611339+0000", 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 67.904279228999997, 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013044249999999999, 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 738 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611339+0000", 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611339+0000", 2024-09-21T18:54:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.634 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:22.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611341+0000", 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7669999999999999e-06 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611354+0000", 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2252999999999999e-05 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611369+0000", 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4919999999999999e-05 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611377+0000", 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3769999999999996e-06 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611436+0000", 2024-09-21T18:54:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8768999999999999e-05 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611572+0000", 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013593899999999999 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.611753+0000", 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018160299999999999 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.612109+0000", 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035538399999999998 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.612364+0000", 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025568999999999999 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.612558+0000", 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000193482 2024-09-21T18:54:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.612606+0000", 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8025000000000002e-05 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.612643+0000", 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7215999999999997e-05 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.14514.0:779 4.1e 4:7db93595:::obj_delete_at_hint.0000000066:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:14.645212+0000", 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 67.870406372000005, 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011578459999999999, 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.14514", 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 779 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645212+0000", 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645212+0000", 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645213+0000", 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2309999999999999e-06 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645218+0000", 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.186e-06 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645229+0000", 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0679000000000001e-05 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645235+0000", 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3450000000000002e-06 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645279+0000", 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3687999999999999e-05 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645346+0000", 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6852000000000003e-05 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645491+0000", 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000145661 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.645784+0000", 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029229000000000001 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.646179+0000", 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039588299999999998 2024-09-21T18:54:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.646273+0000", 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4018999999999995e-05 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.646331+0000", 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7176000000000003e-05 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:14.646369+0000", 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8835999999999999e-05 2024-09-21T18:54:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:22.646 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:54:22.689 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:54:22.689 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:54:22.690 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T18:54:22.690 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T18:54:22.690 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T18:54:22.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T18:54:22.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:22.775 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:22.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:54:22.865 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:22 smithi081 ceph-mon[29384]: pgmap v38: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 12 MiB/s wr, 186 op/s; 123 KiB/s, 13 objects/s recovering 2024-09-21T18:54:22.865 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:22 smithi081 ceph-mon[39192]: pgmap v38: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 12 MiB/s wr, 186 op/s; 123 KiB/s, 13 objects/s recovering 2024-09-21T18:54:22.877 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:22.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:22.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:22 smithi104 ceph-mon[32438]: pgmap v38: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 12 MiB/s wr, 186 op/s; 123 KiB/s, 13 objects/s recovering 2024-09-21T18:54:23.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:23.052 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:23.093 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:54:23.122 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 podman[73064]: 2024-09-21 18:54:23.120076245 +0000 UTC m=+4.026291282 container create dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T18:54:23.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 2 16 7191 7175 224.145 205.5 0.00540786 0.00444865 2024-09-21T18:54:23.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:23.187 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:23.289 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:23.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:23.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.437 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:23.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:23.538 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:23.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:23.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:23.691 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:54:23.691 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T18:54:23.691 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:54:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:23.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:23.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:23.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 podman[73064]: 2024-09-21 18:54:23.970908887 +0000 UTC m=+4.877123928 container init dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 podman[73064]: 2024-09-21 18:54:23.981776723 +0000 UTC m=+4.887991760 container start dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.988Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.988Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.989Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.989Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T18:54:24.067 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.990Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T18:54:24.068 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T18:54:24.069 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T18:54:24.070 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.991Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.992Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T18:54:24.071 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:23 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a[73666]: ts=2024-09-21T18:54:23.992Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T18:54:24.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.137 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:24.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:24.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 3 16 10412 10396 216.529 201.312 0.00361066 0.0046118 2024-09-21T18:54:24.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T18:54:24.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:24.268 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:24.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:24.411 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:24.425 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:24 smithi081 bash[73064]: dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e 2024-09-21T18:54:24.426 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 18:54:24 smithi081 systemd[1]: Started Ceph node-exporter.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:54:24.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:24.560 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:24.604 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:24.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:24.702 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:24.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:24.844 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:24.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:24.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:24 smithi104 ceph-mon[32438]: pgmap v39: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 9.1 MiB/s wr, 145 op/s; 97 KiB/s, 10 objects/s recovering 2024-09-21T18:54:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:24 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:24 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:24 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:24 smithi104 ceph-mon[32438]: osdmap e86: 8 total, 8 up, 7 in 2024-09-21T18:54:25.012 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[39192]: pgmap v39: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 9.1 MiB/s wr, 145 op/s; 97 KiB/s, 10 objects/s recovering 2024-09-21T18:54:25.012 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.012 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.012 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.012 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[39192]: osdmap e86: 8 total, 8 up, 7 in 2024-09-21T18:54:25.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[29384]: pgmap v39: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 9.1 MiB/s wr, 145 op/s; 97 KiB/s, 10 objects/s recovering 2024-09-21T18:54:25.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:24 smithi081 ceph-mon[29384]: osdmap e86: 8 total, 8 up, 7 in 2024-09-21T18:54:25.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.119 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:25.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:25.143 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 4 16 13232 13216 206.457 176.25 0.00696466 0.00483665 2024-09-21T18:54:25.208 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:25.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:25.296 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:25.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:25.432 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:25.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:25.568 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:25.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:25.701 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:25.733 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:54:25.747 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:25 smithi104 systemd[1]: Starting Ceph node-exporter.b for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:54:25.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:25.839 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:25.846 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:54:25.846 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:54:25.886 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:54:25.886 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:54:25.919 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:54:25.920 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:54:25.920 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:25 smithi081 ceph-mon[39192]: Deploying daemon node-exporter.b on smithi104 2024-09-21T18:54:25.920 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2202447073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:54:25.920 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:25 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:25 smithi081 ceph-mon[29384]: Deploying daemon node-exporter.b on smithi104 2024-09-21T18:54:25.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2202447073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:54:25.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:25 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:25.941 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:25.960 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:54:25.960 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:54:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:25.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:25.996 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:54:25.996 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:54:26.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:25 smithi104 ceph-mon[32438]: Deploying daemon node-exporter.b on smithi104 2024-09-21T18:54:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2202447073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:54:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:25 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:26.007 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:25 smithi104 bash[60891]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T18:54:26.031 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:54:26.031 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:54:26.065 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:54:26.065 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:54:26.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:26.098 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:54:26.098 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:54:26.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:26.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 5 16 15803 15787 197.301 160.688 0.00789939 0.00505958 2024-09-21T18:54:26.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:26.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:26.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:26.397 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:26.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:26.507 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:26 smithi104 bash[60891]: Getting image source signatures 2024-09-21T18:54:26.507 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:26 smithi104 bash[60891]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T18:54:26.507 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:26 smithi104 bash[60891]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T18:54:26.507 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:26 smithi104 bash[60891]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T18:54:26.529 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:26.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:26.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:26.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:26.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:26.925 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:26.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:27.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:26 smithi104 ceph-mon[32438]: pgmap v41: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 8.2 MiB/s wr, 130 op/s; 87 KiB/s, 9 objects/s recovering 2024-09-21T18:54:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:26 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2202447073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:54:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:26 smithi104 ceph-mon[32438]: osdmap e87: 8 total, 8 up, 7 in 2024-09-21T18:54:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2895655256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:27.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:27.062 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[29384]: pgmap v41: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 8.2 MiB/s wr, 130 op/s; 87 KiB/s, 9 objects/s recovering 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2202447073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[29384]: osdmap e87: 8 total, 8 up, 7 in 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2895655256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[39192]: pgmap v41: 148 pgs: 148 active+clean; 55 MiB data, 259 MiB used, 626 GiB / 626 GiB avail; 8.2 MiB/s wr, 130 op/s; 87 KiB/s, 9 objects/s recovering 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:27.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2202447073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:54:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[39192]: osdmap e87: 8 total, 8 up, 7 in 2024-09-21T18:54:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2895655256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:27.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 6 15 18599 18584 193.551 174.812 0.00387794 0.00516111 2024-09-21T18:54:27.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:27.202 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:27.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:27.344 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:27.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:27.479 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:27.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:54:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:54:27.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:27.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:27.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.753 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:27.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:27.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:27.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:27.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:27.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:28.001 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:54:28.002 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:54:28.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:54:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:54:28.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:28.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:28.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 7 16 21238 21222 189.452 164.875 0.00356471 0.00527179 2024-09-21T18:54:28.177 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:28.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:28.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:28.443 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:28.507 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 bash[60891]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T18:54:28.507 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 bash[60891]: Writing manifest to image destination 2024-09-21T18:54:28.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:28.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:28.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:28.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:28.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-mon[32438]: pgmap v43: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 400 B/s rd, 134 MiB/s wr, 2.15k op/s; 75 KiB/s, 8 objects/s recovering 2024-09-21T18:54:28.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:28.813 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 podman[60891]: 2024-09-21 18:54:28.470048445 +0000 UTC m=+2.846127969 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T18:54:28.813 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 podman[60891]: 2024-09-21 18:54:28.576463295 +0000 UTC m=+2.952542806 container create a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T18:54:28.813 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 podman[60891]: 2024-09-21 18:54:28.804765729 +0000 UTC m=+3.180845236 container init a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T18:54:28.813 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 podman[60891]: 2024-09-21 18:54:28.811231254 +0000 UTC m=+3.187310765 container start a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T18:54:28.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:28.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:28.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:28.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:28.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:28 smithi081 ceph-mon[39192]: pgmap v43: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 400 B/s rd, 134 MiB/s wr, 2.15k op/s; 75 KiB/s, 8 objects/s recovering 2024-09-21T18:54:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:28 smithi081 ceph-mon[29384]: pgmap v43: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 400 B/s rd, 134 MiB/s wr, 2.15k op/s; 75 KiB/s, 8 objects/s recovering 2024-09-21T18:54:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:29.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.132 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:29.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:29.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 8 15 23666 23651 184.744 151.812 0.0131691 0.00540614 2024-09-21T18:54:29.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T18:54:29.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:29.258 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.817Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T18:54:29.258 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.817Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T18:54:29.258 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.818Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T18:54:29.258 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.818Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T18:54:29.259 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T18:54:29.260 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T18:54:29.261 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T18:54:29.262 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.819Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T18:54:29.263 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.820Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T18:54:29.263 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b[61133]: ts=2024-09-21T18:54:28.820Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T18:54:29.263 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 bash[60891]: a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 2024-09-21T18:54:29.263 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 18:54:28 smithi104 systemd[1]: Started Ceph node-exporter.b for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:54:29.267 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:29.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:29.405 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:29.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:29.524 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:54:29.524 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:54:29.525 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T18:54:29.525 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T18:54:29.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:54:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:29.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:29.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:29.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:54:29.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:29.824 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:29.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:29.957 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:29.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:30.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:30.094 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:30.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 9 16 25388 25372 176.168 107.562 0.0045055 0.0056701 2024-09-21T18:54:30.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:30.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:54:30.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: pgmap v44: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T18:54:30.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:30 smithi104 ceph-mon[32438]: osdmap e88: 8 total, 8 up, 7 in 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: pgmap v44: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T18:54:30.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[29384]: osdmap e88: 8 total, 8 up, 7 in 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: pgmap v44: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T18:54:30.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:30 smithi081 ceph-mon[39192]: osdmap e88: 8 total, 8 up, 7 in 2024-09-21T18:54:30.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.382 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:30.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:30.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:54:30.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:30.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:30.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:30.659 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:30.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.811 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:30.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:30.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:30.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:30.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:31.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:31.073 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:31.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 10 15 27616 27601 172.481 139.312 0.00940943 0.00579266 2024-09-21T18:54:31.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:31.214 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:31.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:31.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:31.480 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:31.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:31.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:31.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:31.742 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:31.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:31 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:31.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:31.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:31.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:31.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:32.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:31 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:31 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:32.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:32.134 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:54:32.134 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:54:32.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 11 16 29449 29433 167.21 114.5 0.00186132 0.00595669 2024-09-21T18:54:32.232 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.264 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:32.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:32.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:32.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:32.414 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:32.417 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:32.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.528 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:32.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:32.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:32.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:32.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:32.783 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:32.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:32 smithi081 ceph-mon[39192]: pgmap v46: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 219 MiB/s wr, 3.51k op/s 2024-09-21T18:54:32.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:32 smithi081 ceph-mon[29384]: pgmap v46: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 219 MiB/s wr, 3.51k op/s 2024-09-21T18:54:32.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:32.909 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:32.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:33.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:32 smithi104 ceph-mon[32438]: pgmap v46: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 219 MiB/s wr, 3.51k op/s 2024-09-21T18:54:33.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:33.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:33.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:33.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 12 16 30973 30957 161.212 95.25 0.00740417 0.00617708 2024-09-21T18:54:33.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:33.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:33.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:33.433 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:33.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:33.555 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:33.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:33.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:33.818 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:33.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:33.946 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:33.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:34.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:34.145 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 13 16 32812 32796 157.652 114.938 0.00370494 0.00633584 2024-09-21T18:54:34.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.209 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:34.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:34.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:34.350 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:34.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:34.476 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:34.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:34.611 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:34.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:34.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:34.839 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:34.873 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:34.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:34.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:34.994 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:34 smithi104 ceph-mon[32438]: pgmap v47: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 212 MiB/s wr, 3.40k op/s 2024-09-21T18:54:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.012 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:35.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:35.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:34 smithi081 ceph-mon[39192]: pgmap v47: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 212 MiB/s wr, 3.40k op/s 2024-09-21T18:54:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:34 smithi081 ceph-mon[29384]: pgmap v47: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 212 MiB/s wr, 3.40k op/s 2024-09-21T18:54:35.112 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:35.144 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 14 16 34529 34513 154.055 107.312 0.00196797 0.00646181 2024-09-21T18:54:35.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:35.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:35.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.408 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:35.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:35.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:35.543 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:35.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:35.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:35.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:35.817 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:35.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:35.939 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:35.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]} : dispatch 2024-09-21T18:54:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T18:54:36.076 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:36.145 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 15 16 36486 36470 151.938 122.312 0.00654184 0.00657712 2024-09-21T18:54:36.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:36.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:36.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:36.460 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.498 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:36.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:36.510 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:36.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T18:54:36.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.645 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:36.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:36.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:36.788 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:36.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:36.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:36.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: pgmap v48: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T18:54:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:36 smithi104 ceph-mon[32438]: osdmap e89: 8 total, 8 up, 7 in 2024-09-21T18:54:37.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: pgmap v48: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[39192]: osdmap e89: 8 total, 8 up, 7 in 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: pgmap v48: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T18:54:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-21T18:54:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-09-21T18:54:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-21T18:54:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T18:54:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:36 smithi081 ceph-mon[29384]: osdmap e89: 8 total, 8 up, 7 in 2024-09-21T18:54:37.085 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:37.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 16 16 38395 38379 149.892 119.312 0.00914302 0.00666652 2024-09-21T18:54:37.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:37.210 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:37.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.345 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:37.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:37.388 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:54:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:54:37.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:37.491 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:37.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:37.630 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:37.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:37.760 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:37.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:54:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:54:37.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:37.891 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:37.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:37.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:38.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:54:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:54:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:37 smithi104 ceph-mon[32438]: osdmap e90: 8 total, 8 up, 7 in 2024-09-21T18:54:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/843160658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:38.027 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[39192]: osdmap e90: 8 total, 8 up, 7 in 2024-09-21T18:54:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/843160658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[29384]: osdmap e90: 8 total, 8 up, 7 in 2024-09-21T18:54:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/843160658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:38.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:38.145 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 17 16 40039 40023 147.119 102.75 0.0249576 0.00679167 2024-09-21T18:54:38.161 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:54:38.222 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:54:38.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:54:38.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:38.297 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:38.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:38.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:38.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:38.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:38.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:38.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:38.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:38.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:38.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:39.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:38 smithi104 ceph-mon[32438]: pgmap v51: 148 pgs: 5 peering, 143 active+clean; 2.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 161 MiB/s wr, 2.58k op/s 2024-09-21T18:54:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:38 smithi104 ceph-mon[32438]: from='client.34757 v1:172.21.15.81:0/3589813082' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:39.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:38 smithi081 ceph-mon[29384]: pgmap v51: 148 pgs: 5 peering, 143 active+clean; 2.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 161 MiB/s wr, 2.58k op/s 2024-09-21T18:54:39.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:39.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:38 smithi081 ceph-mon[29384]: from='client.34757 v1:172.21.15.81:0/3589813082' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:39.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:38 smithi081 ceph-mon[39192]: pgmap v51: 148 pgs: 5 peering, 143 active+clean; 2.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 161 MiB/s wr, 2.58k op/s 2024-09-21T18:54:39.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:39.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:38 smithi081 ceph-mon[39192]: from='client.34757 v1:172.21.15.81:0/3589813082' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:39.076 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:39.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:39.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 18 16 42081 42065 146.035 127.625 0.0079622 0.00684262 2024-09-21T18:54:39.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:39.246 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:39.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:39.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:39.534 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:39.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:39.675 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:39.766 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:54:39.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.814 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:39.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:39.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:39.954 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:39.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/807365982' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:54:40.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/807365982' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:54:40.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/807365982' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:54:40.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:40.092 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:40.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 19 16 43994 43978 144.641 119.562 0.0061628 0.00690792 2024-09-21T18:54:40.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:40.234 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:40.336 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:40.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:40.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:40.492 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-21T18:54:37.384042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2420578843,"num_objects":37125,"num_object_clones":0,"num_object_copies":111375,"num_objects_missing_on_primary":0,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37125,"num_whiteouts":0,"num_read":925,"num_read_kb":782,"num_write":37475,"num_write_kb":2364937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1183576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37481,"ondisk_log_size":37481,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7279824,"kb_used_data":7008772,"kb_used_omap":10,"kb_used_meta":270901,"kb_avail":648936240,"statfs":{"total":671965249536,"available":664510709760,"internally_reserved":0,"allocated":7176982528,"data_stored":7173121544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11153,"internal_metadata":277402735},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1339555840,"num_objects":20440,"num_object_clones":0,"num_object_copies":61320,"num_objects_missing_on_primary":0,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20440,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":20440,"num_write_kb":1308160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.938377"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744367+0000","last_change":"2024-09-21T18:54:17.713866+0000","last_active":"2024-09-21T18:54:35.744367+0000","last_peered":"2024-09-21T18:54:35.744367+0000","last_clean":"2024-09-21T18:54:35.744367+0000","last_became_active":"2024-09-21T18:54:17.713495+0000","last_became_peered":"2024-09-21T18:54:17.713495+0000","last_unstale":"2024-09-21T18:54:35.744367+0000","last_undegraded":"2024-09-21T18:54:35.744367+0000","last_fullsized":"2024-09-21T18:54:35.744367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.249265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":84,"reported_epoch":88,"state":"peering","last_fresh":"2024-09-21T18:54:29.464088+0000","last_change":"2024-09-21T18:54:35.736966+0000","last_active":"2024-09-21T18:54:35.736966+0000","last_peered":"2024-09-21T18:54:29.464088+0000","last_clean":"2024-09-21T18:54:29.464088+0000","last_became_active":"2024-09-21T18:53:09.412193+0000","last_became_peered":"2024-09-21T18:53:09.412193+0000","last_unstale":"2024-09-21T18:54:29.464088+0000","last_undegraded":"2024-09-21T18:54:29.464088+0000","last_fullsized":"2024-09-21T18:54:29.464088+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766269+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:54:35.766269+0000","last_peered":"2024-09-21T18:54:35.766269+0000","last_clean":"2024-09-21T18:54:35.766269+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:54:35.766269+0000","last_undegraded":"2024-09-21T18:54:35.766269+0000","last_fullsized":"2024-09-21T18:54:35.766269+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.278309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736308+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:54:35.736308+0000","last_peered":"2024-09-21T18:54:35.736308+0000","last_clean":"2024-09-21T18:54:35.736308+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:54:35.736308+0000","last_undegraded":"2024-09-21T18:54:35.736308+0000","last_fullsized":"2024-09-21T18:54:35.736308+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.055413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.764589+0000","last_change":"2024-09-21T18:53:13.658931+0000","last_active":"2024-09-21T18:54:35.764589+0000","last_peered":"2024-09-21T18:54:35.764589+0000","last_clean":"2024-09-21T18:54:35.764589+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:54:35.764589+0000","last_undegraded":"2024-09-21T18:54:35.764589+0000","last_fullsized":"2024-09-21T18:54:35.764589+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.771063+0000","last_change":"2024-09-21T18:53:07.390902+0000","last_active":"2024-09-21T18:54:35.771063+0000","last_peered":"2024-09-21T18:54:35.771063+0000","last_clean":"2024-09-21T18:54:35.771063+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:54:35.771063+0000","last_undegraded":"2024-09-21T18:54:35.771063+0000","last_fullsized":"2024-09-21T18:54:35.771063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":118,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.762488+0000","last_change":"2024-09-21T18:54:17.730862+0000","last_active":"2024-09-21T18:54:35.762488+0000","last_peered":"2024-09-21T18:54:35.762488+0000","last_clean":"2024-09-21T18:54:35.762488+0000","last_became_active":"2024-09-21T18:54:17.716359+0000","last_became_peered":"2024-09-21T18:54:17.716359+0000","last_unstale":"2024-09-21T18:54:35.762488+0000","last_undegraded":"2024-09-21T18:54:35.762488+0000","last_fullsized":"2024-09-21T18:54:35.762488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736220+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:54:35.736220+0000","last_peered":"2024-09-21T18:54:35.736220+0000","last_clean":"2024-09-21T18:54:35.736220+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:54:35.736220+0000","last_undegraded":"2024-09-21T18:54:35.736220+0000","last_fullsized":"2024-09-21T18:54:35.736220+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.928369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743432+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:54:35.743432+0000","last_peered":"2024-09-21T18:54:35.743432+0000","last_clean":"2024-09-21T18:54:35.743432+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:54:35.743432+0000","last_undegraded":"2024-09-21T18:54:35.743432+0000","last_fullsized":"2024-09-21T18:54:35.743432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":110,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736573+0000","last_change":"2024-09-21T18:54:17.725564+0000","last_active":"2024-09-21T18:54:35.736573+0000","last_peered":"2024-09-21T18:54:35.736573+0000","last_clean":"2024-09-21T18:54:35.736573+0000","last_became_active":"2024-09-21T18:54:17.704975+0000","last_became_peered":"2024-09-21T18:54:17.704975+0000","last_unstale":"2024-09-21T18:54:35.736573+0000","last_undegraded":"2024-09-21T18:54:35.736573+0000","last_fullsized":"2024-09-21T18:54:35.736573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:14.972606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.972606+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.449994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742695+0000","last_change":"2024-09-21T18:54:17.712927+0000","last_active":"2024-09-21T18:54:35.742695+0000","last_peered":"2024-09-21T18:54:35.742695+0000","last_clean":"2024-09-21T18:54:35.742695+0000","last_became_active":"2024-09-21T18:54:17.712591+0000","last_became_peered":"2024-09-21T18:54:17.712591+0000","last_unstale":"2024-09-21T18:54:35.742695+0000","last_undegraded":"2024-09-21T18:54:35.742695+0000","last_fullsized":"2024-09-21T18:54:35.742695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.765768+0000","last_change":"2024-09-21T18:54:17.704813+0000","last_active":"2024-09-21T18:54:35.765768+0000","last_peered":"2024-09-21T18:54:35.765768+0000","last_clean":"2024-09-21T18:54:35.765768+0000","last_became_active":"2024-09-21T18:54:17.704510+0000","last_became_peered":"2024-09-21T18:54:17.704510+0000","last_unstale":"2024-09-21T18:54:35.765768+0000","last_undegraded":"2024-09-21T18:54:35.765768+0000","last_fullsized":"2024-09-21T18:54:35.765768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.329858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742999+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:54:35.742999+0000","last_peered":"2024-09-21T18:54:35.742999+0000","last_clean":"2024-09-21T18:54:35.742999+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:54:35.742999+0000","last_undegraded":"2024-09-21T18:54:35.742999+0000","last_fullsized":"2024-09-21T18:54:35.742999+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:35.991288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736557+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:54:35.736557+0000","last_peered":"2024-09-21T18:54:35.736557+0000","last_clean":"2024-09-21T18:54:35.736557+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:54:35.736557+0000","last_undegraded":"2024-09-21T18:54:35.736557+0000","last_fullsized":"2024-09-21T18:54:35.736557+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.337954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.756960+0000","last_change":"2024-09-21T18:53:11.412969+0000","last_active":"2024-09-21T18:54:35.756960+0000","last_peered":"2024-09-21T18:54:35.756960+0000","last_clean":"2024-09-21T18:54:35.756960+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:54:35.756960+0000","last_undegraded":"2024-09-21T18:54:35.756960+0000","last_fullsized":"2024-09-21T18:54:35.756960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742569+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:54:35.742569+0000","last_peered":"2024-09-21T18:54:35.742569+0000","last_clean":"2024-09-21T18:54:35.742569+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:54:35.742569+0000","last_undegraded":"2024-09-21T18:54:35.742569+0000","last_fullsized":"2024-09-21T18:54:35.742569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737181+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:54:35.737181+0000","last_peered":"2024-09-21T18:54:35.737181+0000","last_clean":"2024-09-21T18:54:35.737181+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:54:35.737181+0000","last_undegraded":"2024-09-21T18:54:35.737181+0000","last_fullsized":"2024-09-21T18:54:35.737181+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.940396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736680+0000","last_change":"2024-09-21T18:54:17.705466+0000","last_active":"2024-09-21T18:54:35.736680+0000","last_peered":"2024-09-21T18:54:35.736680+0000","last_clean":"2024-09-21T18:54:35.736680+0000","last_became_active":"2024-09-21T18:54:17.705159+0000","last_became_peered":"2024-09-21T18:54:17.705159+0000","last_unstale":"2024-09-21T18:54:35.736680+0000","last_undegraded":"2024-09-21T18:54:35.736680+0000","last_fullsized":"2024-09-21T18:54:35.736680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.944967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761282+0000","last_change":"2024-09-21T18:53:09.407164+0000","last_active":"2024-09-21T18:54:35.761282+0000","last_peered":"2024-09-21T18:54:35.761282+0000","last_clean":"2024-09-21T18:54:35.761282+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:54:35.761282+0000","last_undegraded":"2024-09-21T18:54:35.761282+0000","last_fullsized":"2024-09-21T18:54:35.761282+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766363+0000","last_change":"2024-09-21T18:54:17.714793+0000","last_active":"2024-09-21T18:54:35.766363+0000","last_peered":"2024-09-21T18:54:35.766363+0000","last_clean":"2024-09-21T18:54:35.766363+0000","last_became_active":"2024-09-21T18:54:17.714300+0000","last_became_peered":"2024-09-21T18:54:17.714300+0000","last_unstale":"2024-09-21T18:54:35.766363+0000","last_undegraded":"2024-09-21T18:54:35.766363+0000","last_fullsized":"2024-09-21T18:54:35.766363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.768157+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:54:35.768157+0000","last_peered":"2024-09-21T18:54:35.768157+0000","last_clean":"2024-09-21T18:54:35.768157+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:54:35.768157+0000","last_undegraded":"2024-09-21T18:54:35.768157+0000","last_fullsized":"2024-09-21T18:54:35.768157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736723+0000","last_change":"2024-09-21T18:54:12.987795+0000","last_active":"2024-09-21T18:54:35.736723+0000","last_peered":"2024-09-21T18:54:35.736723+0000","last_clean":"2024-09-21T18:54:35.736723+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:54:35.736723+0000","last_undegraded":"2024-09-21T18:54:35.736723+0000","last_fullsized":"2024-09-21T18:54:35.736723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.987749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.987749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:41.935763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.763342+0000","last_change":"2024-09-21T18:53:09.406892+0000","last_active":"2024-09-21T18:54:35.763342+0000","last_peered":"2024-09-21T18:54:35.763342+0000","last_clean":"2024-09-21T18:54:35.763342+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:54:35.763342+0000","last_undegraded":"2024-09-21T18:54:35.763342+0000","last_fullsized":"2024-09-21T18:54:35.763342+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736818+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:54:35.736818+0000","last_peered":"2024-09-21T18:54:35.736818+0000","last_clean":"2024-09-21T18:54:35.736818+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:54:35.736818+0000","last_undegraded":"2024-09-21T18:54:35.736818+0000","last_fullsized":"2024-09-21T18:54:35.736818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.960588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.758025+0000","last_change":"2024-09-21T18:53:13.426725+0000","last_active":"2024-09-21T18:54:35.758025+0000","last_peered":"2024-09-21T18:54:35.758025+0000","last_clean":"2024-09-21T18:54:35.758025+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:54:35.758025+0000","last_undegraded":"2024-09-21T18:54:35.758025+0000","last_fullsized":"2024-09-21T18:54:35.758025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766386+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:54:35.766386+0000","last_peered":"2024-09-21T18:54:35.766386+0000","last_clean":"2024-09-21T18:54:35.766386+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:54:35.766386+0000","last_undegraded":"2024-09-21T18:54:35.766386+0000","last_fullsized":"2024-09-21T18:54:35.766386+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:43.629105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.758479+0000","last_change":"2024-09-21T18:54:17.738374+0000","last_active":"2024-09-21T18:54:35.758479+0000","last_peered":"2024-09-21T18:54:35.758479+0000","last_clean":"2024-09-21T18:54:35.758479+0000","last_became_active":"2024-09-21T18:54:17.719024+0000","last_became_peered":"2024-09-21T18:54:17.719024+0000","last_unstale":"2024-09-21T18:54:35.758479+0000","last_undegraded":"2024-09-21T18:54:35.758479+0000","last_fullsized":"2024-09-21T18:54:35.758479+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.810583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744688+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:54:35.744688+0000","last_peered":"2024-09-21T18:54:35.744688+0000","last_clean":"2024-09-21T18:54:35.744688+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:54:35.744688+0000","last_undegraded":"2024-09-21T18:54:35.744688+0000","last_fullsized":"2024-09-21T18:54:35.744688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.245724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"89'2282","reported_seq":2313,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.375135+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:54:36.375135+0000","last_peered":"2024-09-21T18:54:36.375135+0000","last_clean":"2024-09-21T18:54:36.375135+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:54:36.375135+0000","last_undegraded":"2024-09-21T18:54:36.375135+0000","last_fullsized":"2024-09-21T18:54:36.375135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:14.400572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149553152,"num_objects":2282,"num_object_clones":0,"num_object_copies":6846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":146048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766979+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:54:35.766979+0000","last_peered":"2024-09-21T18:54:35.766979+0000","last_clean":"2024-09-21T18:54:35.766979+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:54:35.766979+0000","last_undegraded":"2024-09-21T18:54:35.766979+0000","last_fullsized":"2024-09-21T18:54:35.766979+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.762318+0000","last_change":"2024-09-21T18:53:07.398833+0000","last_active":"2024-09-21T18:54:35.762318+0000","last_peered":"2024-09-21T18:54:35.762318+0000","last_clean":"2024-09-21T18:54:35.762318+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:54:35.762318+0000","last_undegraded":"2024-09-21T18:54:35.762318+0000","last_fullsized":"2024-09-21T18:54:35.762318+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"83'9","reported_seq":102,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737303+0000","last_change":"2024-09-21T18:54:17.715008+0000","last_active":"2024-09-21T18:54:35.737303+0000","last_peered":"2024-09-21T18:54:35.737303+0000","last_clean":"2024-09-21T18:54:35.737303+0000","last_became_active":"2024-09-21T18:54:17.706739+0000","last_became_peered":"2024-09-21T18:54:17.706739+0000","last_unstale":"2024-09-21T18:54:35.737303+0000","last_undegraded":"2024-09-21T18:54:35.737303+0000","last_fullsized":"2024-09-21T18:54:35.737303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:37.938529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743263+0000","last_change":"2024-09-21T18:54:17.705694+0000","last_active":"2024-09-21T18:54:35.743263+0000","last_peered":"2024-09-21T18:54:35.743263+0000","last_clean":"2024-09-21T18:54:35.743263+0000","last_became_active":"2024-09-21T18:54:17.705355+0000","last_became_peered":"2024-09-21T18:54:17.705355+0000","last_unstale":"2024-09-21T18:54:35.743263+0000","last_undegraded":"2024-09-21T18:54:35.743263+0000","last_fullsized":"2024-09-21T18:54:35.743263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.222602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"89'2280","reported_seq":2311,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.396285+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:54:36.396285+0000","last_peered":"2024-09-21T18:54:36.396285+0000","last_clean":"2024-09-21T18:54:36.396285+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:54:36.396285+0000","last_undegraded":"2024-09-21T18:54:36.396285+0000","last_fullsized":"2024-09-21T18:54:36.396285+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2280,"log_dups_size":0,"ondisk_log_size":2280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.493653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149422080,"num_objects":2280,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":145920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":127,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.762118+0000","last_change":"2024-09-21T18:54:17.740103+0000","last_active":"2024-09-21T18:54:35.762118+0000","last_peered":"2024-09-21T18:54:35.762118+0000","last_clean":"2024-09-21T18:54:35.762118+0000","last_became_active":"2024-09-21T18:54:17.716713+0000","last_became_peered":"2024-09-21T18:54:17.716713+0000","last_unstale":"2024-09-21T18:54:35.762118+0000","last_undegraded":"2024-09-21T18:54:35.762118+0000","last_fullsized":"2024-09-21T18:54:35.762118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.636848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.769214+0000","last_change":"2024-09-21T18:54:17.719791+0000","last_active":"2024-09-21T18:54:35.769214+0000","last_peered":"2024-09-21T18:54:35.769214+0000","last_clean":"2024-09-21T18:54:35.769214+0000","last_became_active":"2024-09-21T18:54:17.719440+0000","last_became_peered":"2024-09-21T18:54:17.719440+0000","last_unstale":"2024-09-21T18:54:35.769214+0000","last_undegraded":"2024-09-21T18:54:35.769214+0000","last_fullsized":"2024-09-21T18:54:35.769214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.901802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736342+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:54:35.736342+0000","last_peered":"2024-09-21T18:54:35.736342+0000","last_clean":"2024-09-21T18:54:35.736342+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:54:35.736342+0000","last_undegraded":"2024-09-21T18:54:35.736342+0000","last_fullsized":"2024-09-21T18:54:35.736342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736462+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:54:35.736462+0000","last_peered":"2024-09-21T18:54:35.736462+0000","last_clean":"2024-09-21T18:54:35.736462+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:54:35.736462+0000","last_undegraded":"2024-09-21T18:54:35.736462+0000","last_fullsized":"2024-09-21T18:54:35.736462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.926924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"89'2342","reported_seq":2372,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.396743+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:54:36.396743+0000","last_peered":"2024-09-21T18:54:36.396743+0000","last_clean":"2024-09-21T18:54:36.396743+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:54:36.396743+0000","last_undegraded":"2024-09-21T18:54:36.396743+0000","last_fullsized":"2024-09-21T18:54:36.396743+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2342,"log_dups_size":0,"ondisk_log_size":2342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.053908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153485312,"num_objects":2342,"num_object_clones":0,"num_object_copies":7026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2342,"num_write_kb":149888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736477+0000","last_change":"2024-09-21T18:53:09.400587+0000","last_active":"2024-09-21T18:54:35.736477+0000","last_peered":"2024-09-21T18:54:35.736477+0000","last_clean":"2024-09-21T18:54:35.736477+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:54:35.736477+0000","last_undegraded":"2024-09-21T18:54:35.736477+0000","last_fullsized":"2024-09-21T18:54:35.736477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766592+0000","last_change":"2024-09-21T18:54:17.734277+0000","last_active":"2024-09-21T18:54:35.766592+0000","last_peered":"2024-09-21T18:54:35.766592+0000","last_clean":"2024-09-21T18:54:35.766592+0000","last_became_active":"2024-09-21T18:54:17.721086+0000","last_became_peered":"2024-09-21T18:54:17.721086+0000","last_unstale":"2024-09-21T18:54:35.766592+0000","last_undegraded":"2024-09-21T18:54:35.766592+0000","last_fullsized":"2024-09-21T18:54:35.766592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.369107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761949+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:54:35.761949+0000","last_peered":"2024-09-21T18:54:35.761949+0000","last_clean":"2024-09-21T18:54:35.761949+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:54:35.761949+0000","last_undegraded":"2024-09-21T18:54:35.761949+0000","last_fullsized":"2024-09-21T18:54:35.761949+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757368+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:54:35.757368+0000","last_peered":"2024-09-21T18:54:35.757368+0000","last_clean":"2024-09-21T18:54:35.757368+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:54:35.757368+0000","last_undegraded":"2024-09-21T18:54:35.757368+0000","last_fullsized":"2024-09-21T18:54:35.757368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"89'2244","reported_seq":2275,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.372989+0000","last_change":"2024-09-21T18:54:14.452896+0000","last_active":"2024-09-21T18:54:36.372989+0000","last_peered":"2024-09-21T18:54:36.372989+0000","last_clean":"2024-09-21T18:54:36.372989+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:54:36.372989+0000","last_undegraded":"2024-09-21T18:54:36.372989+0000","last_fullsized":"2024-09-21T18:54:36.372989+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.308130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147062784,"num_objects":2244,"num_object_clones":0,"num_object_copies":6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2244,"num_write_kb":143616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.764086+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:54:35.764086+0000","last_peered":"2024-09-21T18:54:35.764086+0000","last_clean":"2024-09-21T18:54:35.764086+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:54:35.764086+0000","last_undegraded":"2024-09-21T18:54:35.764086+0000","last_fullsized":"2024-09-21T18:54:35.764086+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.356779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743493+0000","last_change":"2024-09-21T18:54:15.230912+0000","last_active":"2024-09-21T18:54:35.743493+0000","last_peered":"2024-09-21T18:54:35.743493+0000","last_clean":"2024-09-21T18:54:35.743493+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:54:35.743493+0000","last_undegraded":"2024-09-21T18:54:35.743493+0000","last_fullsized":"2024-09-21T18:54:35.743493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.230861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.230861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.698873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"83'9","reported_seq":59,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744430+0000","last_change":"2024-09-21T18:54:17.722223+0000","last_active":"2024-09-21T18:54:35.744430+0000","last_peered":"2024-09-21T18:54:35.744430+0000","last_clean":"2024-09-21T18:54:35.744430+0000","last_became_active":"2024-09-21T18:54:17.713489+0000","last_became_peered":"2024-09-21T18:54:17.713489+0000","last_unstale":"2024-09-21T18:54:35.744430+0000","last_undegraded":"2024-09-21T18:54:35.744430+0000","last_fullsized":"2024-09-21T18:54:35.744430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:40.298145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761843+0000","last_change":"2024-09-21T18:54:17.714235+0000","last_active":"2024-09-21T18:54:35.761843+0000","last_peered":"2024-09-21T18:54:35.761843+0000","last_clean":"2024-09-21T18:54:35.761843+0000","last_became_active":"2024-09-21T18:54:17.713933+0000","last_became_peered":"2024-09-21T18:54:17.713933+0000","last_unstale":"2024-09-21T18:54:35.761843+0000","last_undegraded":"2024-09-21T18:54:35.761843+0000","last_fullsized":"2024-09-21T18:54:35.761843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.977688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"89'2379","reported_seq":2409,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.369107+0000","last_change":"2024-09-21T18:54:14.451912+0000","last_active":"2024-09-21T18:54:36.369107+0000","last_peered":"2024-09-21T18:54:36.369107+0000","last_clean":"2024-09-21T18:54:36.369107+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:54:36.369107+0000","last_undegraded":"2024-09-21T18:54:36.369107+0000","last_fullsized":"2024-09-21T18:54:36.369107+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2379,"log_dups_size":0,"ondisk_log_size":2379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:29.638453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":7137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":152256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742743+0000","last_change":"2024-09-21T18:54:17.730752+0000","last_active":"2024-09-21T18:54:35.742743+0000","last_peered":"2024-09-21T18:54:35.742743+0000","last_clean":"2024-09-21T18:54:35.742743+0000","last_became_active":"2024-09-21T18:54:17.711274+0000","last_became_peered":"2024-09-21T18:54:17.711274+0000","last_unstale":"2024-09-21T18:54:35.742743+0000","last_undegraded":"2024-09-21T18:54:35.742743+0000","last_fullsized":"2024-09-21T18:54:35.742743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.213832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.767149+0000","last_change":"2024-09-21T18:53:07.400542+0000","last_active":"2024-09-21T18:54:35.767149+0000","last_peered":"2024-09-21T18:54:35.767149+0000","last_clean":"2024-09-21T18:54:35.767149+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:54:35.767149+0000","last_undegraded":"2024-09-21T18:54:35.767149+0000","last_fullsized":"2024-09-21T18:54:35.767149+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742920+0000","last_change":"2024-09-21T18:54:17.710974+0000","last_active":"2024-09-21T18:54:35.742920+0000","last_peered":"2024-09-21T18:54:35.742920+0000","last_clean":"2024-09-21T18:54:35.742920+0000","last_became_active":"2024-09-21T18:54:17.710662+0000","last_became_peered":"2024-09-21T18:54:17.710662+0000","last_unstale":"2024-09-21T18:54:35.742920+0000","last_undegraded":"2024-09-21T18:54:35.742920+0000","last_fullsized":"2024-09-21T18:54:35.742920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.222043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766182+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:54:35.766182+0000","last_peered":"2024-09-21T18:54:35.766182+0000","last_clean":"2024-09-21T18:54:35.766182+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:54:35.766182+0000","last_undegraded":"2024-09-21T18:54:35.766182+0000","last_fullsized":"2024-09-21T18:54:35.766182+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.362211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"89'2273","reported_seq":2311,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.363613+0000","last_change":"2024-09-21T18:54:17.707215+0000","last_active":"2024-09-21T18:54:36.363613+0000","last_peered":"2024-09-21T18:54:36.363613+0000","last_clean":"2024-09-21T18:54:36.363613+0000","last_became_active":"2024-09-21T18:54:17.706866+0000","last_became_peered":"2024-09-21T18:54:17.706866+0000","last_unstale":"2024-09-21T18:54:36.363613+0000","last_undegraded":"2024-09-21T18:54:36.363613+0000","last_fullsized":"2024-09-21T18:54:36.363613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2273,"log_dups_size":0,"ondisk_log_size":2273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.130355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148963328,"num_objects":2273,"num_object_clones":0,"num_object_copies":6819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2273,"num_write_kb":145472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":135,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766890+0000","last_change":"2024-09-21T18:53:09.410172+0000","last_active":"2024-09-21T18:54:35.766890+0000","last_peered":"2024-09-21T18:54:35.766890+0000","last_clean":"2024-09-21T18:54:35.766890+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:54:35.766890+0000","last_undegraded":"2024-09-21T18:54:35.766890+0000","last_fullsized":"2024-09-21T18:54:35.766890+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736869+0000","last_change":"2024-09-21T18:54:17.706922+0000","last_active":"2024-09-21T18:54:35.736869+0000","last_peered":"2024-09-21T18:54:35.736869+0000","last_clean":"2024-09-21T18:54:35.736869+0000","last_became_active":"2024-09-21T18:54:17.706635+0000","last_became_peered":"2024-09-21T18:54:17.706635+0000","last_unstale":"2024-09-21T18:54:35.736869+0000","last_undegraded":"2024-09-21T18:54:35.736869+0000","last_fullsized":"2024-09-21T18:54:35.736869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:09.963263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:09.963263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.094847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743162+0000","last_change":"2024-09-21T18:53:09.398954+0000","last_active":"2024-09-21T18:54:35.743162+0000","last_peered":"2024-09-21T18:54:35.743162+0000","last_clean":"2024-09-21T18:54:35.743162+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:54:35.743162+0000","last_undegraded":"2024-09-21T18:54:35.743162+0000","last_fullsized":"2024-09-21T18:54:35.743162+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766158+0000","last_change":"2024-09-21T18:54:17.713049+0000","last_active":"2024-09-21T18:54:35.766158+0000","last_peered":"2024-09-21T18:54:35.766158+0000","last_clean":"2024-09-21T18:54:35.766158+0000","last_became_active":"2024-09-21T18:54:17.712729+0000","last_became_peered":"2024-09-21T18:54:17.712729+0000","last_unstale":"2024-09-21T18:54:35.766158+0000","last_undegraded":"2024-09-21T18:54:35.766158+0000","last_fullsized":"2024-09-21T18:54:35.766158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.770760+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:54:35.770760+0000","last_peered":"2024-09-21T18:54:35.770760+0000","last_clean":"2024-09-21T18:54:35.770760+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:54:35.770760+0000","last_undegraded":"2024-09-21T18:54:35.770760+0000","last_fullsized":"2024-09-21T18:54:35.770760+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.888758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"89'2439","reported_seq":2465,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.356497+0000","last_change":"2024-09-21T18:54:17.706551+0000","last_active":"2024-09-21T18:54:36.356497+0000","last_peered":"2024-09-21T18:54:36.356497+0000","last_clean":"2024-09-21T18:54:36.356497+0000","last_became_active":"2024-09-21T18:54:17.706169+0000","last_became_peered":"2024-09-21T18:54:17.706169+0000","last_unstale":"2024-09-21T18:54:36.356497+0000","last_undegraded":"2024-09-21T18:54:36.356497+0000","last_fullsized":"2024-09-21T18:54:36.356497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2439,"log_dups_size":0,"ondisk_log_size":2439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.094421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":156096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":2,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-21T18:54:29.453879+0000","last_change":"2024-09-21T18:54:36.378010+0000","last_active":"2024-09-21T18:54:36.378010+0000","last_peered":"2024-09-21T18:54:29.453879+0000","last_clean":"2024-09-21T18:54:29.453879+0000","last_became_active":"2024-09-21T18:54:17.710266+0000","last_became_peered":"2024-09-21T18:54:17.710266+0000","last_unstale":"2024-09-21T18:54:29.453879+0000","last_undegraded":"2024-09-21T18:54:29.453879+0000","last_fullsized":"2024-09-21T18:54:29.453879+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.763819+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:54:35.763819+0000","last_peered":"2024-09-21T18:54:35.763819+0000","last_clean":"2024-09-21T18:54:35.763819+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:54:35.763819+0000","last_undegraded":"2024-09-21T18:54:35.763819+0000","last_fullsized":"2024-09-21T18:54:35.763819+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757372+0000","last_change":"2024-09-21T18:54:17.718825+0000","last_active":"2024-09-21T18:54:35.757372+0000","last_peered":"2024-09-21T18:54:35.757372+0000","last_clean":"2024-09-21T18:54:35.757372+0000","last_became_active":"2024-09-21T18:54:17.710632+0000","last_became_peered":"2024-09-21T18:54:17.710632+0000","last_unstale":"2024-09-21T18:54:35.757372+0000","last_undegraded":"2024-09-21T18:54:35.757372+0000","last_fullsized":"2024-09-21T18:54:35.757372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.946937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761815+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:54:35.761815+0000","last_peered":"2024-09-21T18:54:35.761815+0000","last_clean":"2024-09-21T18:54:35.761815+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:54:35.761815+0000","last_undegraded":"2024-09-21T18:54:35.761815+0000","last_fullsized":"2024-09-21T18:54:35.761815+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.969938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757586+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:54:35.757586+0000","last_peered":"2024-09-21T18:54:35.757586+0000","last_clean":"2024-09-21T18:54:35.757586+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:54:35.757586+0000","last_undegraded":"2024-09-21T18:54:35.757586+0000","last_fullsized":"2024-09-21T18:54:35.757586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:29.965290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"89'2362","reported_seq":2400,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.364405+0000","last_change":"2024-09-21T18:54:17.717837+0000","last_active":"2024-09-21T18:54:36.364405+0000","last_peered":"2024-09-21T18:54:36.364405+0000","last_clean":"2024-09-21T18:54:36.364405+0000","last_became_active":"2024-09-21T18:54:17.717525+0000","last_became_peered":"2024-09-21T18:54:17.717525+0000","last_unstale":"2024-09-21T18:54:36.364405+0000","last_undegraded":"2024-09-21T18:54:36.364405+0000","last_fullsized":"2024-09-21T18:54:36.364405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2362,"log_dups_size":0,"ondisk_log_size":2362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.870954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154796032,"num_objects":2362,"num_object_clones":0,"num_object_copies":7086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":151168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766482+0000","last_change":"2024-09-21T18:54:17.731920+0000","last_active":"2024-09-21T18:54:35.766482+0000","last_peered":"2024-09-21T18:54:35.766482+0000","last_clean":"2024-09-21T18:54:35.766482+0000","last_became_active":"2024-09-21T18:54:17.720574+0000","last_became_peered":"2024-09-21T18:54:17.720574+0000","last_unstale":"2024-09-21T18:54:35.766482+0000","last_undegraded":"2024-09-21T18:54:35.766482+0000","last_fullsized":"2024-09-21T18:54:35.766482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.300639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736342+0000","last_change":"2024-09-21T18:54:17.704631+0000","last_active":"2024-09-21T18:54:35.736342+0000","last_peered":"2024-09-21T18:54:35.736342+0000","last_clean":"2024-09-21T18:54:35.736342+0000","last_became_active":"2024-09-21T18:54:17.704296+0000","last_became_peered":"2024-09-21T18:54:17.704296+0000","last_unstale":"2024-09-21T18:54:35.736342+0000","last_undegraded":"2024-09-21T18:54:35.736342+0000","last_fullsized":"2024-09-21T18:54:35.736342+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:20.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":178,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757207+0000","last_change":"2024-09-21T18:54:17.729001+0000","last_active":"2024-09-21T18:54:35.757207+0000","last_peered":"2024-09-21T18:54:35.757207+0000","last_clean":"2024-09-21T18:54:35.757207+0000","last_became_active":"2024-09-21T18:54:17.713703+0000","last_became_peered":"2024-09-21T18:54:17.713703+0000","last_unstale":"2024-09-21T18:54:35.757207+0000","last_undegraded":"2024-09-21T18:54:35.757207+0000","last_fullsized":"2024-09-21T18:54:35.757207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744287+0000","last_change":"2024-09-21T18:54:17.708152+0000","last_active":"2024-09-21T18:54:35.744287+0000","last_peered":"2024-09-21T18:54:35.744287+0000","last_clean":"2024-09-21T18:54:35.744287+0000","last_became_active":"2024-09-21T18:54:17.707817+0000","last_became_peered":"2024-09-21T18:54:17.707817+0000","last_unstale":"2024-09-21T18:54:35.744287+0000","last_undegraded":"2024-09-21T18:54:35.744287+0000","last_fullsized":"2024-09-21T18:54:35.744287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.765802+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:54:35.765802+0000","last_peered":"2024-09-21T18:54:35.765802+0000","last_clean":"2024-09-21T18:54:35.765802+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:54:35.765802+0000","last_undegraded":"2024-09-21T18:54:35.765802+0000","last_fullsized":"2024-09-21T18:54:35.765802+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"89'2289","reported_seq":2314,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.356266+0000","last_change":"2024-09-21T18:54:17.719358+0000","last_active":"2024-09-21T18:54:36.356266+0000","last_peered":"2024-09-21T18:54:36.356266+0000","last_clean":"2024-09-21T18:54:36.356266+0000","last_became_active":"2024-09-21T18:54:17.718730+0000","last_became_peered":"2024-09-21T18:54:17.718730+0000","last_unstale":"2024-09-21T18:54:36.356266+0000","last_undegraded":"2024-09-21T18:54:36.356266+0000","last_fullsized":"2024-09-21T18:54:36.356266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2289,"log_dups_size":0,"ondisk_log_size":2289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.846251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":6867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":146496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":108,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743838+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:54:35.743838+0000","last_peered":"2024-09-21T18:54:35.743838+0000","last_clean":"2024-09-21T18:54:35.743838+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:54:35.743838+0000","last_undegraded":"2024-09-21T18:54:35.743838+0000","last_fullsized":"2024-09-21T18:54:35.743838+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.249764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.760354+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:54:35.760354+0000","last_peered":"2024-09-21T18:54:35.760354+0000","last_clean":"2024-09-21T18:54:35.760354+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:54:35.760354+0000","last_undegraded":"2024-09-21T18:54:35.760354+0000","last_fullsized":"2024-09-21T18:54:35.760354+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.929847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":130,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757390+0000","last_change":"2024-09-21T18:54:17.736002+0000","last_active":"2024-09-21T18:54:35.757390+0000","last_peered":"2024-09-21T18:54:35.757390+0000","last_clean":"2024-09-21T18:54:35.757390+0000","last_became_active":"2024-09-21T18:54:17.716544+0000","last_became_peered":"2024-09-21T18:54:17.716544+0000","last_unstale":"2024-09-21T18:54:35.757390+0000","last_undegraded":"2024-09-21T18:54:35.757390+0000","last_fullsized":"2024-09-21T18:54:35.757390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:47.704774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.758128+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:54:35.758128+0000","last_peered":"2024-09-21T18:54:35.758128+0000","last_clean":"2024-09-21T18:54:35.758128+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:54:35.758128+0000","last_undegraded":"2024-09-21T18:54:35.758128+0000","last_fullsized":"2024-09-21T18:54:35.758128+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.081383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744148+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:54:35.744148+0000","last_peered":"2024-09-21T18:54:35.744148+0000","last_clean":"2024-09-21T18:54:35.744148+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:54:35.744148+0000","last_undegraded":"2024-09-21T18:54:35.744148+0000","last_fullsized":"2024-09-21T18:54:35.744148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"89'2299","reported_seq":2337,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.350739+0000","last_change":"2024-09-21T18:54:17.710054+0000","last_active":"2024-09-21T18:54:36.350739+0000","last_peered":"2024-09-21T18:54:36.350739+0000","last_clean":"2024-09-21T18:54:36.350739+0000","last_became_active":"2024-09-21T18:54:17.709698+0000","last_became_peered":"2024-09-21T18:54:17.709698+0000","last_unstale":"2024-09-21T18:54:36.350739+0000","last_undegraded":"2024-09-21T18:54:36.350739+0000","last_fullsized":"2024-09-21T18:54:36.350739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2299,"log_dups_size":0,"ondisk_log_size":2299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.567144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150667264,"num_objects":2299,"num_object_clones":0,"num_object_copies":6897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2299,"num_write_kb":147136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737321+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:54:35.737321+0000","last_peered":"2024-09-21T18:54:35.737321+0000","last_clean":"2024-09-21T18:54:35.737321+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:54:35.737321+0000","last_undegraded":"2024-09-21T18:54:35.737321+0000","last_fullsized":"2024-09-21T18:54:35.737321+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.929416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737013+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:54:35.737013+0000","last_peered":"2024-09-21T18:54:35.737013+0000","last_clean":"2024-09-21T18:54:35.737013+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:54:35.737013+0000","last_undegraded":"2024-09-21T18:54:35.737013+0000","last_fullsized":"2024-09-21T18:54:35.737013+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.012525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744066+0000","last_change":"2024-09-21T18:53:11.409004+0000","last_active":"2024-09-21T18:54:35.744066+0000","last_peered":"2024-09-21T18:54:35.744066+0000","last_clean":"2024-09-21T18:54:35.744066+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:54:35.744066+0000","last_undegraded":"2024-09-21T18:54:35.744066+0000","last_fullsized":"2024-09-21T18:54:35.744066+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.759963+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:54:35.759963+0000","last_peered":"2024-09-21T18:54:35.759963+0000","last_clean":"2024-09-21T18:54:35.759963+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:54:35.759963+0000","last_undegraded":"2024-09-21T18:54:35.759963+0000","last_fullsized":"2024-09-21T18:54:35.759963+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.277061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"89'2275","reported_seq":2313,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.361922+0000","last_change":"2024-09-21T18:54:17.708465+0000","last_active":"2024-09-21T18:54:36.361922+0000","last_peered":"2024-09-21T18:54:36.361922+0000","last_clean":"2024-09-21T18:54:36.361922+0000","last_became_active":"2024-09-21T18:54:17.707839+0000","last_became_peered":"2024-09-21T18:54:17.707839+0000","last_unstale":"2024-09-21T18:54:36.361922+0000","last_undegraded":"2024-09-21T18:54:36.361922+0000","last_fullsized":"2024-09-21T18:54:36.361922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2275,"log_dups_size":0,"ondisk_log_size":2275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.899478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149094400,"num_objects":2275,"num_object_clones":0,"num_object_copies":6825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2275,"num_write_kb":145600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736952+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:54:35.736952+0000","last_peered":"2024-09-21T18:54:35.736952+0000","last_clean":"2024-09-21T18:54:35.736952+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:54:35.736952+0000","last_undegraded":"2024-09-21T18:54:35.736952+0000","last_fullsized":"2024-09-21T18:54:35.736952+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743219+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:54:35.743219+0000","last_peered":"2024-09-21T18:54:35.743219+0000","last_clean":"2024-09-21T18:54:35.743219+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:54:35.743219+0000","last_undegraded":"2024-09-21T18:54:35.743219+0000","last_fullsized":"2024-09-21T18:54:35.743219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.289015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.765911+0000","last_change":"2024-09-21T18:54:17.714450+0000","last_active":"2024-09-21T18:54:35.765911+0000","last_peered":"2024-09-21T18:54:35.765911+0000","last_clean":"2024-09-21T18:54:35.765911+0000","last_became_active":"2024-09-21T18:54:17.714167+0000","last_became_peered":"2024-09-21T18:54:17.714167+0000","last_unstale":"2024-09-21T18:54:35.765911+0000","last_undegraded":"2024-09-21T18:54:35.765911+0000","last_fullsized":"2024-09-21T18:54:35.765911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761069+0000","last_change":"2024-09-21T18:53:13.430863+0000","last_active":"2024-09-21T18:54:35.761069+0000","last_peered":"2024-09-21T18:54:35.761069+0000","last_clean":"2024-09-21T18:54:35.761069+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:54:35.761069+0000","last_undegraded":"2024-09-21T18:54:35.761069+0000","last_fullsized":"2024-09-21T18:54:35.761069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:38.403912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"89'2343","reported_seq":2369,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.389896+0000","last_change":"2024-09-21T18:54:17.718909+0000","last_active":"2024-09-21T18:54:36.389896+0000","last_peered":"2024-09-21T18:54:36.389896+0000","last_clean":"2024-09-21T18:54:36.389896+0000","last_became_active":"2024-09-21T18:54:17.718631+0000","last_became_peered":"2024-09-21T18:54:17.718631+0000","last_unstale":"2024-09-21T18:54:36.389896+0000","last_undegraded":"2024-09-21T18:54:36.389896+0000","last_fullsized":"2024-09-21T18:54:36.389896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2343,"log_dups_size":0,"ondisk_log_size":2343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:32.749545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153550848,"num_objects":2343,"num_object_clones":0,"num_object_copies":7029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":149952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.736841+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:54:35.736841+0000","last_peered":"2024-09-21T18:54:35.736841+0000","last_clean":"2024-09-21T18:54:35.736841+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:54:35.736841+0000","last_undegraded":"2024-09-21T18:54:35.736841+0000","last_fullsized":"2024-09-21T18:54:35.736841+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.765003+0000","last_change":"2024-09-21T18:54:17.724937+0000","last_active":"2024-09-21T18:54:35.765003+0000","last_peered":"2024-09-21T18:54:35.765003+0000","last_clean":"2024-09-21T18:54:35.765003+0000","last_became_active":"2024-09-21T18:54:17.710657+0000","last_became_peered":"2024-09-21T18:54:17.710657+0000","last_unstale":"2024-09-21T18:54:35.765003+0000","last_undegraded":"2024-09-21T18:54:35.765003+0000","last_fullsized":"2024-09-21T18:54:35.765003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.355187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761132+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:54:35.761132+0000","last_peered":"2024-09-21T18:54:35.761132+0000","last_clean":"2024-09-21T18:54:35.761132+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:54:35.761132+0000","last_undegraded":"2024-09-21T18:54:35.761132+0000","last_fullsized":"2024-09-21T18:54:35.761132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.995859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737155+0000","last_change":"2024-09-21T18:53:13.428208+0000","last_active":"2024-09-21T18:54:35.737155+0000","last_peered":"2024-09-21T18:54:35.737155+0000","last_clean":"2024-09-21T18:54:35.737155+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:54:35.737155+0000","last_undegraded":"2024-09-21T18:54:35.737155+0000","last_fullsized":"2024-09-21T18:54:35.737155+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"89'2275","reported_seq":2306,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.352546+0000","last_change":"2024-09-21T18:54:14.454300+0000","last_active":"2024-09-21T18:54:36.352546+0000","last_peered":"2024-09-21T18:54:36.352546+0000","last_clean":"2024-09-21T18:54:36.352546+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:54:36.352546+0000","last_undegraded":"2024-09-21T18:54:36.352546+0000","last_fullsized":"2024-09-21T18:54:36.352546+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2275,"log_dups_size":0,"ondisk_log_size":2275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.683811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149094400,"num_objects":2275,"num_object_clones":0,"num_object_copies":6825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2275,"num_write_kb":145600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766017+0000","last_change":"2024-09-21T18:54:17.729395+0000","last_active":"2024-09-21T18:54:35.766017+0000","last_peered":"2024-09-21T18:54:35.766017+0000","last_clean":"2024-09-21T18:54:35.766017+0000","last_became_active":"2024-09-21T18:54:17.714624+0000","last_became_peered":"2024-09-21T18:54:17.714624+0000","last_unstale":"2024-09-21T18:54:35.766017+0000","last_undegraded":"2024-09-21T18:54:35.766017+0000","last_fullsized":"2024-09-21T18:54:35.766017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743121+0000","last_change":"2024-09-21T18:54:13.280500+0000","last_active":"2024-09-21T18:54:35.743121+0000","last_peered":"2024-09-21T18:54:35.743121+0000","last_clean":"2024-09-21T18:54:35.743121+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:54:35.743121+0000","last_undegraded":"2024-09-21T18:54:35.743121+0000","last_fullsized":"2024-09-21T18:54:35.743121+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.280431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.280431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.151455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757768+0000","last_change":"2024-09-21T18:53:11.415649+0000","last_active":"2024-09-21T18:54:35.757768+0000","last_peered":"2024-09-21T18:54:35.757768+0000","last_clean":"2024-09-21T18:54:35.757768+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:54:35.757768+0000","last_undegraded":"2024-09-21T18:54:35.757768+0000","last_fullsized":"2024-09-21T18:54:35.757768+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.756953+0000","last_change":"2024-09-21T18:54:17.717827+0000","last_active":"2024-09-21T18:54:35.756953+0000","last_peered":"2024-09-21T18:54:35.756953+0000","last_clean":"2024-09-21T18:54:35.756953+0000","last_became_active":"2024-09-21T18:54:17.717507+0000","last_became_peered":"2024-09-21T18:54:17.717507+0000","last_unstale":"2024-09-21T18:54:35.756953+0000","last_undegraded":"2024-09-21T18:54:35.756953+0000","last_fullsized":"2024-09-21T18:54:35.756953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.193259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"89'2324","reported_seq":2350,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.378491+0000","last_change":"2024-09-21T18:54:17.712034+0000","last_active":"2024-09-21T18:54:36.378491+0000","last_peered":"2024-09-21T18:54:36.378491+0000","last_clean":"2024-09-21T18:54:36.378491+0000","last_became_active":"2024-09-21T18:54:17.711698+0000","last_became_peered":"2024-09-21T18:54:17.711698+0000","last_unstale":"2024-09-21T18:54:36.378491+0000","last_undegraded":"2024-09-21T18:54:36.378491+0000","last_fullsized":"2024-09-21T18:54:36.378491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.183316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152305664,"num_objects":2324,"num_object_clones":0,"num_object_copies":6972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":148736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743017+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:54:35.743017+0000","last_peered":"2024-09-21T18:54:35.743017+0000","last_clean":"2024-09-21T18:54:35.743017+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:54:35.743017+0000","last_undegraded":"2024-09-21T18:54:35.743017+0000","last_fullsized":"2024-09-21T18:54:35.743017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:36.041097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757783+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:54:35.757783+0000","last_peered":"2024-09-21T18:54:35.757783+0000","last_clean":"2024-09-21T18:54:35.757783+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:54:35.757783+0000","last_undegraded":"2024-09-21T18:54:35.757783+0000","last_fullsized":"2024-09-21T18:54:35.757783+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.128759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.760883+0000","last_change":"2024-09-21T18:54:17.717687+0000","last_active":"2024-09-21T18:54:35.760883+0000","last_peered":"2024-09-21T18:54:35.760883+0000","last_clean":"2024-09-21T18:54:35.760883+0000","last_became_active":"2024-09-21T18:54:17.717380+0000","last_became_peered":"2024-09-21T18:54:17.717380+0000","last_unstale":"2024-09-21T18:54:35.760883+0000","last_undegraded":"2024-09-21T18:54:35.760883+0000","last_fullsized":"2024-09-21T18:54:35.760883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.963253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757301+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:54:35.757301+0000","last_peered":"2024-09-21T18:54:35.757301+0000","last_clean":"2024-09-21T18:54:35.757301+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:54:35.757301+0000","last_undegraded":"2024-09-21T18:54:35.757301+0000","last_fullsized":"2024-09-21T18:54:35.757301+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:18.116478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"89'2280","reported_seq":2311,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.381757+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:54:36.381757+0000","last_peered":"2024-09-21T18:54:36.381757+0000","last_clean":"2024-09-21T18:54:36.381757+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:54:36.381757+0000","last_undegraded":"2024-09-21T18:54:36.381757+0000","last_fullsized":"2024-09-21T18:54:36.381757+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2280,"log_dups_size":0,"ondisk_log_size":2280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.423918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149422080,"num_objects":2280,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":145920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":91,"reported_epoch":88,"state":"peering","last_fresh":"2024-09-21T18:54:29.465902+0000","last_change":"2024-09-21T18:54:35.737100+0000","last_active":"2024-09-21T18:54:35.737100+0000","last_peered":"2024-09-21T18:54:29.465902+0000","last_clean":"2024-09-21T18:54:29.465902+0000","last_became_active":"2024-09-21T18:53:09.406030+0000","last_became_peered":"2024-09-21T18:53:09.406030+0000","last_unstale":"2024-09-21T18:54:29.465902+0000","last_undegraded":"2024-09-21T18:54:29.465902+0000","last_fullsized":"2024-09-21T18:54:29.465902+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.758586+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:54:35.758586+0000","last_peered":"2024-09-21T18:54:35.758586+0000","last_clean":"2024-09-21T18:54:35.758586+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:54:35.758586+0000","last_undegraded":"2024-09-21T18:54:35.758586+0000","last_fullsized":"2024-09-21T18:54:35.758586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:13.545717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757288+0000","last_change":"2024-09-21T18:54:17.708159+0000","last_active":"2024-09-21T18:54:35.757288+0000","last_peered":"2024-09-21T18:54:35.757288+0000","last_clean":"2024-09-21T18:54:35.757288+0000","last_became_active":"2024-09-21T18:54:17.707830+0000","last_became_peered":"2024-09-21T18:54:17.707830+0000","last_unstale":"2024-09-21T18:54:35.757288+0000","last_undegraded":"2024-09-21T18:54:35.757288+0000","last_fullsized":"2024-09-21T18:54:35.757288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:11.994472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.994472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.343603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.764685+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:54:35.764685+0000","last_peered":"2024-09-21T18:54:35.764685+0000","last_clean":"2024-09-21T18:54:35.764685+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:54:35.764685+0000","last_undegraded":"2024-09-21T18:54:35.764685+0000","last_fullsized":"2024-09-21T18:54:35.764685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"89'2240","reported_seq":2271,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:36.370064+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:54:36.370064+0000","last_peered":"2024-09-21T18:54:36.370064+0000","last_clean":"2024-09-21T18:54:36.370064+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:54:36.370064+0000","last_undegraded":"2024-09-21T18:54:36.370064+0000","last_fullsized":"2024-09-21T18:54:36.370064+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.519258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":2240,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2240,"num_write_kb":143360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742296+0000","last_change":"2024-09-21T18:53:09.717860+0000","last_active":"2024-09-21T18:54:35.742296+0000","last_peered":"2024-09-21T18:54:35.742296+0000","last_clean":"2024-09-21T18:54:35.742296+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:54:35.742296+0000","last_undegraded":"2024-09-21T18:54:35.742296+0000","last_fullsized":"2024-09-21T18:54:35.742296+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.759096+0000","last_change":"2024-09-21T18:54:17.732357+0000","last_active":"2024-09-21T18:54:35.759096+0000","last_peered":"2024-09-21T18:54:35.759096+0000","last_clean":"2024-09-21T18:54:35.759096+0000","last_became_active":"2024-09-21T18:54:17.715268+0000","last_became_peered":"2024-09-21T18:54:17.715268+0000","last_unstale":"2024-09-21T18:54:35.759096+0000","last_undegraded":"2024-09-21T18:54:35.759096+0000","last_fullsized":"2024-09-21T18:54:35.759096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.049285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.758019+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:54:35.758019+0000","last_peered":"2024-09-21T18:54:35.758019+0000","last_clean":"2024-09-21T18:54:35.758019+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:54:35.758019+0000","last_undegraded":"2024-09-21T18:54:35.758019+0000","last_fullsized":"2024-09-21T18:54:35.758019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.946870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742846+0000","last_change":"2024-09-21T18:54:17.714328+0000","last_active":"2024-09-21T18:54:35.742846+0000","last_peered":"2024-09-21T18:54:35.742846+0000","last_clean":"2024-09-21T18:54:35.742846+0000","last_became_active":"2024-09-21T18:54:17.713977+0000","last_became_peered":"2024-09-21T18:54:17.713977+0000","last_unstale":"2024-09-21T18:54:35.742846+0000","last_undegraded":"2024-09-21T18:54:35.742846+0000","last_fullsized":"2024-09-21T18:54:35.742846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744404+0000","last_change":"2024-09-21T18:54:17.713398+0000","last_active":"2024-09-21T18:54:35.744404+0000","last_peered":"2024-09-21T18:54:35.744404+0000","last_clean":"2024-09-21T18:54:35.744404+0000","last_became_active":"2024-09-21T18:54:17.713066+0000","last_became_peered":"2024-09-21T18:54:17.713066+0000","last_unstale":"2024-09-21T18:54:35.744404+0000","last_undegraded":"2024-09-21T18:54:35.744404+0000","last_fullsized":"2024-09-21T18:54:35.744404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.224570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.768965+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:54:35.768965+0000","last_peered":"2024-09-21T18:54:35.768965+0000","last_clean":"2024-09-21T18:54:35.768965+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:54:35.768965+0000","last_undegraded":"2024-09-21T18:54:35.768965+0000","last_fullsized":"2024-09-21T18:54:35.768965+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743952+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:54:35.743952+0000","last_peered":"2024-09-21T18:54:35.743952+0000","last_clean":"2024-09-21T18:54:35.743952+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:54:35.743952+0000","last_undegraded":"2024-09-21T18:54:35.743952+0000","last_fullsized":"2024-09-21T18:54:35.743952+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.256441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757512+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:54:35.757512+0000","last_peered":"2024-09-21T18:54:35.757512+0000","last_clean":"2024-09-21T18:54:35.757512+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:54:35.757512+0000","last_undegraded":"2024-09-21T18:54:35.757512+0000","last_fullsized":"2024-09-21T18:54:35.757512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.770918+0000","last_change":"2024-09-21T18:54:17.735434+0000","last_active":"2024-09-21T18:54:35.770918+0000","last_peered":"2024-09-21T18:54:35.770918+0000","last_clean":"2024-09-21T18:54:35.770918+0000","last_became_active":"2024-09-21T18:54:17.719720+0000","last_became_peered":"2024-09-21T18:54:17.719720+0000","last_unstale":"2024-09-21T18:54:35.770918+0000","last_undegraded":"2024-09-21T18:54:35.770918+0000","last_fullsized":"2024-09-21T18:54:35.770918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.758286+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:54:35.758286+0000","last_peered":"2024-09-21T18:54:35.758286+0000","last_clean":"2024-09-21T18:54:35.758286+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:54:35.758286+0000","last_undegraded":"2024-09-21T18:54:35.758286+0000","last_fullsized":"2024-09-21T18:54:35.758286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.194383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.768689+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:54:35.768689+0000","last_peered":"2024-09-21T18:54:35.768689+0000","last_clean":"2024-09-21T18:54:35.768689+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:54:35.768689+0000","last_undegraded":"2024-09-21T18:54:35.768689+0000","last_fullsized":"2024-09-21T18:54:35.768689+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.003241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.760659+0000","last_change":"2024-09-21T18:53:13.423168+0000","last_active":"2024-09-21T18:54:35.760659+0000","last_peered":"2024-09-21T18:54:35.760659+0000","last_clean":"2024-09-21T18:54:35.760659+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:54:35.760659+0000","last_undegraded":"2024-09-21T18:54:35.760659+0000","last_fullsized":"2024-09-21T18:54:35.760659+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:41.401268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743410+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:54:35.743410+0000","last_peered":"2024-09-21T18:54:35.743410+0000","last_clean":"2024-09-21T18:54:35.743410+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:54:35.743410+0000","last_undegraded":"2024-09-21T18:54:35.743410+0000","last_fullsized":"2024-09-21T18:54:35.743410+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.275983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757149+0000","last_change":"2024-09-21T18:54:17.717912+0000","last_active":"2024-09-21T18:54:35.757149+0000","last_peered":"2024-09-21T18:54:35.757149+0000","last_clean":"2024-09-21T18:54:35.757149+0000","last_became_active":"2024-09-21T18:54:17.717617+0000","last_became_peered":"2024-09-21T18:54:17.717617+0000","last_unstale":"2024-09-21T18:54:35.757149+0000","last_undegraded":"2024-09-21T18:54:35.757149+0000","last_fullsized":"2024-09-21T18:54:35.757149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.759977+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:54:35.759977+0000","last_peered":"2024-09-21T18:54:35.759977+0000","last_clean":"2024-09-21T18:54:35.759977+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:54:35.759977+0000","last_undegraded":"2024-09-21T18:54:35.759977+0000","last_fullsized":"2024-09-21T18:54:35.759977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:54:39.168997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737760+0000","last_change":"2024-09-21T18:53:13.423826+0000","last_active":"2024-09-21T18:54:35.737760+0000","last_peered":"2024-09-21T18:54:35.737760+0000","last_clean":"2024-09-21T18:54:35.737760+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:54:35.737760+0000","last_undegraded":"2024-09-21T18:54:35.737760+0000","last_fullsized":"2024-09-21T18:54:35.737760+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761797+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:54:35.761797+0000","last_peered":"2024-09-21T18:54:35.761797+0000","last_clean":"2024-09-21T18:54:35.761797+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:54:35.761797+0000","last_undegraded":"2024-09-21T18:54:35.761797+0000","last_fullsized":"2024-09-21T18:54:35.761797+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757672+0000","last_change":"2024-09-21T18:53:07.403003+0000","last_active":"2024-09-21T18:54:35.757672+0000","last_peered":"2024-09-21T18:54:35.757672+0000","last_clean":"2024-09-21T18:54:35.757672+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:54:35.757672+0000","last_undegraded":"2024-09-21T18:54:35.757672+0000","last_fullsized":"2024-09-21T18:54:35.757672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"83'9","reported_seq":102,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743326+0000","last_change":"2024-09-21T18:54:17.726533+0000","last_active":"2024-09-21T18:54:35.743326+0000","last_peered":"2024-09-21T18:54:35.743326+0000","last_clean":"2024-09-21T18:54:35.743326+0000","last_became_active":"2024-09-21T18:54:17.718821+0000","last_became_peered":"2024-09-21T18:54:17.718821+0000","last_unstale":"2024-09-21T18:54:35.743326+0000","last_undegraded":"2024-09-21T18:54:35.743326+0000","last_fullsized":"2024-09-21T18:54:35.743326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757306+0000","last_change":"2024-09-21T18:54:17.719309+0000","last_active":"2024-09-21T18:54:35.757306+0000","last_peered":"2024-09-21T18:54:35.757306+0000","last_clean":"2024-09-21T18:54:35.757306+0000","last_became_active":"2024-09-21T18:54:17.718560+0000","last_became_peered":"2024-09-21T18:54:17.718560+0000","last_unstale":"2024-09-21T18:54:35.757306+0000","last_undegraded":"2024-09-21T18:54:35.757306+0000","last_fullsized":"2024-09-21T18:54:35.757306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:25.988782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":87,"reported_epoch":88,"state":"peering","last_fresh":"2024-09-21T18:54:29.466243+0000","last_change":"2024-09-21T18:54:35.737628+0000","last_active":"2024-09-21T18:54:35.737628+0000","last_peered":"2024-09-21T18:54:29.466243+0000","last_clean":"2024-09-21T18:54:29.466243+0000","last_became_active":"2024-09-21T18:53:09.407184+0000","last_became_peered":"2024-09-21T18:53:09.407184+0000","last_unstale":"2024-09-21T18:54:29.466243+0000","last_undegraded":"2024-09-21T18:54:29.466243+0000","last_fullsized":"2024-09-21T18:54:29.466243+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.737019+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:54:35.737019+0000","last_peered":"2024-09-21T18:54:35.737019+0000","last_clean":"2024-09-21T18:54:35.737019+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:54:35.737019+0000","last_undegraded":"2024-09-21T18:54:35.737019+0000","last_fullsized":"2024-09-21T18:54:35.737019+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.963073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.763983+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:54:35.763983+0000","last_peered":"2024-09-21T18:54:35.763983+0000","last_clean":"2024-09-21T18:54:35.763983+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:54:35.763983+0000","last_undegraded":"2024-09-21T18:54:35.763983+0000","last_fullsized":"2024-09-21T18:54:35.763983+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.761678+0000","last_change":"2024-09-21T18:54:17.717220+0000","last_active":"2024-09-21T18:54:35.761678+0000","last_peered":"2024-09-21T18:54:35.761678+0000","last_clean":"2024-09-21T18:54:35.761678+0000","last_became_active":"2024-09-21T18:54:17.716004+0000","last_became_peered":"2024-09-21T18:54:17.716004+0000","last_unstale":"2024-09-21T18:54:35.761678+0000","last_undegraded":"2024-09-21T18:54:35.761678+0000","last_fullsized":"2024-09-21T18:54:35.761678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.764250+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:54:35.764250+0000","last_peered":"2024-09-21T18:54:35.764250+0000","last_clean":"2024-09-21T18:54:35.764250+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:54:35.764250+0000","last_undegraded":"2024-09-21T18:54:35.764250+0000","last_fullsized":"2024-09-21T18:54:35.764250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.311285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757701+0000","last_change":"2024-09-21T18:53:07.406445+0000","last_active":"2024-09-21T18:54:35.757701+0000","last_peered":"2024-09-21T18:54:35.757701+0000","last_clean":"2024-09-21T18:54:35.757701+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:54:35.757701+0000","last_undegraded":"2024-09-21T18:54:35.757701+0000","last_fullsized":"2024-09-21T18:54:35.757701+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.764772+0000","last_change":"2024-09-21T18:54:17.715073+0000","last_active":"2024-09-21T18:54:35.764772+0000","last_peered":"2024-09-21T18:54:35.764772+0000","last_clean":"2024-09-21T18:54:35.764772+0000","last_became_active":"2024-09-21T18:54:17.714759+0000","last_became_peered":"2024-09-21T18:54:17.714759+0000","last_unstale":"2024-09-21T18:54:35.764772+0000","last_undegraded":"2024-09-21T18:54:35.764772+0000","last_fullsized":"2024-09-21T18:54:35.764772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.032847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.742221+0000","last_change":"2024-09-21T18:54:17.712675+0000","last_active":"2024-09-21T18:54:35.742221+0000","last_peered":"2024-09-21T18:54:35.742221+0000","last_clean":"2024-09-21T18:54:35.742221+0000","last_became_active":"2024-09-21T18:54:17.712339+0000","last_became_peered":"2024-09-21T18:54:17.712339+0000","last_unstale":"2024-09-21T18:54:35.742221+0000","last_undegraded":"2024-09-21T18:54:35.742221+0000","last_fullsized":"2024-09-21T18:54:35.742221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.762223+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:54:35.762223+0000","last_peered":"2024-09-21T18:54:35.762223+0000","last_clean":"2024-09-21T18:54:35.762223+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:54:35.762223+0000","last_undegraded":"2024-09-21T18:54:35.762223+0000","last_fullsized":"2024-09-21T18:54:35.762223+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744036+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:54:35.744036+0000","last_peered":"2024-09-21T18:54:35.744036+0000","last_clean":"2024-09-21T18:54:35.744036+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:54:35.744036+0000","last_undegraded":"2024-09-21T18:54:35.744036+0000","last_fullsized":"2024-09-21T18:54:35.744036+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.201819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.756820+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:54:35.756820+0000","last_peered":"2024-09-21T18:54:35.756820+0000","last_clean":"2024-09-21T18:54:35.756820+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:54:35.756820+0000","last_undegraded":"2024-09-21T18:54:35.756820+0000","last_fullsized":"2024-09-21T18:54:35.756820+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.082920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.764913+0000","last_change":"2024-09-21T18:54:17.709035+0000","last_active":"2024-09-21T18:54:35.764913+0000","last_peered":"2024-09-21T18:54:35.764913+0000","last_clean":"2024-09-21T18:54:35.764913+0000","last_became_active":"2024-09-21T18:54:17.708626+0000","last_became_peered":"2024-09-21T18:54:17.708626+0000","last_unstale":"2024-09-21T18:54:35.764913+0000","last_undegraded":"2024-09-21T18:54:35.764913+0000","last_fullsized":"2024-09-21T18:54:35.764913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":75,"reported_epoch":88,"state":"peering","last_fresh":"2024-09-21T18:54:29.456163+0000","last_change":"2024-09-21T18:54:35.760120+0000","last_active":"2024-09-21T18:54:35.760120+0000","last_peered":"2024-09-21T18:54:29.456163+0000","last_clean":"2024-09-21T18:54:29.456163+0000","last_became_active":"2024-09-21T18:53:09.409216+0000","last_became_peered":"2024-09-21T18:53:09.409216+0000","last_unstale":"2024-09-21T18:54:29.456163+0000","last_undegraded":"2024-09-21T18:54:29.456163+0000","last_fullsized":"2024-09-21T18:54:29.456163+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744446+0000","last_change":"2024-09-21T18:54:17.719435+0000","last_active":"2024-09-21T18:54:35.744446+0000","last_peered":"2024-09-21T18:54:35.744446+0000","last_clean":"2024-09-21T18:54:35.744446+0000","last_became_active":"2024-09-21T18:54:17.718927+0000","last_became_peered":"2024-09-21T18:54:17.718927+0000","last_unstale":"2024-09-21T18:54:35.744446+0000","last_undegraded":"2024-09-21T18:54:35.744446+0000","last_fullsized":"2024-09-21T18:54:35.744446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.743103+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:54:35.743103+0000","last_peered":"2024-09-21T18:54:35.743103+0000","last_clean":"2024-09-21T18:54:35.743103+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:54:35.743103+0000","last_undegraded":"2024-09-21T18:54:35.743103+0000","last_fullsized":"2024-09-21T18:54:35.743103+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.250351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.756973+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:54:35.756973+0000","last_peered":"2024-09-21T18:54:35.756973+0000","last_clean":"2024-09-21T18:54:35.756973+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:54:35.756973+0000","last_undegraded":"2024-09-21T18:54:35.756973+0000","last_fullsized":"2024-09-21T18:54:35.756973+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.766484+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:54:35.766484+0000","last_peered":"2024-09-21T18:54:35.766484+0000","last_clean":"2024-09-21T18:54:35.766484+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:54:35.766484+0000","last_undegraded":"2024-09-21T18:54:35.766484+0000","last_fullsized":"2024-09-21T18:54:35.766484+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:25.345750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.762248+0000","last_change":"2024-09-21T18:54:17.704421+0000","last_active":"2024-09-21T18:54:35.762248+0000","last_peered":"2024-09-21T18:54:35.762248+0000","last_clean":"2024-09-21T18:54:35.762248+0000","last_became_active":"2024-09-21T18:54:17.704166+0000","last_became_peered":"2024-09-21T18:54:17.704166+0000","last_unstale":"2024-09-21T18:54:35.762248+0000","last_undegraded":"2024-09-21T18:54:35.762248+0000","last_fullsized":"2024-09-21T18:54:35.762248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:13.526296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.744721+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:54:35.744721+0000","last_peered":"2024-09-21T18:54:35.744721+0000","last_clean":"2024-09-21T18:54:35.744721+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:54:35.744721+0000","last_undegraded":"2024-09-21T18:54:35.744721+0000","last_fullsized":"2024-09-21T18:54:35.744721+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.235721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-21T18:54:35.757879+0000","last_change":"2024-09-21T18:54:17.719032+0000","last_active":"2024-09-21T18:54:35.757879+0000","last_peered":"2024-09-21T18:54:35.757879+0000","last_clean":"2024-09-21T18:54:35.757879+0000","last_became_active":"2024-09-21T18:54:17.718245+0000","last_became_peered":"2024-09-21T18:54:17.718245+0000","last_unstale":"2024-09-21T18:54:35.757879+0000","last_undegraded":"2024-09-21T18:54:35.757879+0000","last_fullsized":"2024-09-21T18:54:35.757879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2419982336,"num_objects":36926,"num_object_clones":0,"num_object_copies":110778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36926,"num_write_kb":2363264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7260667904,"data_stored":7260667904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36926,"ondisk_log_size":36926,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266710,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898304,"kb_used_data":871276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92846848,"statfs":{"total":95995035648,"available":95075172352,"internally_reserved":0,"allocated":892186624,"data_stored":891628240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528347,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903328,"kb_used_data":876300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92841824,"statfs":{"total":95995035648,"available":95070027776,"internally_reserved":0,"allocated":897331200,"data_stored":896804559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789982,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762524,"kb_used_data":727176,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92982628,"statfs":{"total":95995035648,"available":95214211072,"internally_reserved":0,"allocated":744628224,"data_stored":744105155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904372,"kb_used_data":873184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92840780,"statfs":{"total":95995035648,"available":95068958720,"internally_reserved":0,"allocated":894140416,"data_stored":893593900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647240,"kb_used_data":1598004,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92097912,"statfs":{"total":95995035648,"available":94308261888,"internally_reserved":0,"allocated":1636356096,"data_stored":1635789898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912476,"kb_used_data":868808,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92832676,"statfs":{"total":95995035648,"available":95060660224,"internally_reserved":0,"allocated":889659392,"data_stored":889136658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251580,"kb_used_data":1194024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92493572,"statfs":{"total":95995035648,"available":94713417728,"internally_reserved":0,"allocated":1222680576,"data_stored":1222063134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1222967296,"data_stored":1222967296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":904790016,"data_stored":904790016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1659043840,"data_stored":1659043840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":910491648,"data_stored":910491648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":751304704,"data_stored":751304704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":902955008,"data_stored":902955008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":909115392,"data_stored":909115392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:54:40.497 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-21T18:54:40.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:40.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:40.639 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:40.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:40.766 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:40.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:40.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:41.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:41.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: pgmap v52: 148 pgs: 5 peering, 143 active+clean; 2.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T18:54:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/807365982' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:54:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T18:54:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: pgmap v52: 148 pgs: 5 peering, 143 active+clean; 2.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/807365982' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T18:54:41.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:40 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T18:54:41.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:54:41.145073+0000 min lat: 0.00134243 max lat: 0.179359 avg lat: 0.00697561 2024-09-21T18:54:41.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:54:41.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 20 16 45869 45853 143.267 117.188 0.0106714 0.00697561 2024-09-21T18:54:41.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:41.191 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: pgmap v52: 148 pgs: 5 peering, 143 active+clean; 2.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/807365982' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T18:54:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:40 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T18:54:41.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:41.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:41.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:41.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:41.457 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:41.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:41.590 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:41.590 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:54:41.590 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:54:41.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13961 7.9 7:908586d8:::benchmark_data_smithi081_73391_object13960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.424826+0000", 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.161115628000001, 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.057775897, 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13961 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.424826+0000", 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.424826+0000", 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.424828+0000", 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.807e-06 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.424853+0000", 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4468e-05 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.424863+0000", 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0519e-05 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.424870+0000", 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425495+0000", 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062445700000000003 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425761+0000", 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026666299999999999 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.426077+0000", 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031512800000000002 2024-09-21T18:54:41.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481400+0000", 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055323107000000003 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482443+0000", 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001043048 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482528+0000", 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5037999999999995e-05 2024-09-21T18:54:41.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482569+0000", 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1146000000000002e-05 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482602+0000", 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3416e-05 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13962 7.9 7:9d4d3909:::benchmark_data_smithi081_73391_object13961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:54:41.596 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.425345+0000", 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.160596863999999, 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.057429877999999997, 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13962 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425345+0000", 2024-09-21T18:54:41.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425345+0000", 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425346+0000", 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425369+0000", 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2889000000000001e-05 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425377+0000", 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5259999999999993e-06 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425384+0000", 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5309999999999998e-06 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427086+0000", 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001701773 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427211+0000", 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012519699999999999 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427346+0000", 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000135506 2024-09-21T18:54:41.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482408+0000", 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055061260000000001 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482492+0000", 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4264000000000005e-05 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482691+0000", 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019943599999999999 2024-09-21T18:54:41.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482731+0000", 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9634e-05 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482775+0000", 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4019e-05 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.602 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13963 7.b 7:df9a7f51:::benchmark_data_smithi081_73391_object13962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.425416+0000", 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.160525917000001, 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055599614999999998, 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.603 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13963 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425416+0000", 2024-09-21T18:54:41.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425416+0000", 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425417+0000", 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1399999999999996e-07 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425440+0000", 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3376000000000001e-05 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425448+0000", 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.875e-06 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425468+0000", 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9995000000000001e-05 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425783+0000", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031491599999999999 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425962+0000", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017868699999999999 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.426077+0000", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114869 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428188+0000", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021116939999999999 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480880+0000", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052691918999999997 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480917+0000", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6977999999999997e-05 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480972+0000", 2024-09-21T18:54:41.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481016+0000", 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4092999999999999e-05 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13965 7.b 7:d5473052:::benchmark_data_smithi081_73391_object13964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.425921+0000", 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.160021482000001, 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.056420126000000001, 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13965 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425921+0000", 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425921+0000", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425933+0000", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2781e-05 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425965+0000", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1359000000000002e-05 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425974+0000", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0489999999999993e-06 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425980+0000", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5010000000000001e-06 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427278+0000", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012979280000000001 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427481+0000", 2024-09-21T18:54:41.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000202981 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427621+0000", 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014011199999999999 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480406+0000", 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052785113000000002 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482003+0000", 2024-09-21T18:54:41.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015968429999999999 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482264+0000", 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000260556 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482306+0000", 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2290999999999999e-05 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482341+0000", 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4612e-05 2024-09-21T18:54:41.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13967 7.9 7:90fba537:::benchmark_data_smithi081_73391_object13966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.425996+0000", 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.159945764, 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.057910428, 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13967 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425996+0000", 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425996+0000", 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.425997+0000", 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1500000000000004e-07 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.426027+0000", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0043999999999999e-05 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.426034+0000", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8229999999999997e-06 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.426039+0000", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4929999999999998e-06 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428089+0000", 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020492290000000001 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428183+0000", 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3985000000000002e-05 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480602+0000", 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052419087000000003 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482453+0000", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001851473 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482968+0000", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00051500099999999996 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.483788+0000", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082038599999999999 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.483851+0000", 2024-09-21T18:54:41.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2658999999999998e-05 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.483907+0000", 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5532999999999998e-05 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13972 7.9 7:9b0b907e:::benchmark_data_smithi081_73391_object13971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.427534+0000", 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.158407589999999, 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.076687018999999995, 2024-09-21T18:54:41.620 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13972 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427534+0000", 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427534+0000", 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427536+0000", 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.218e-06 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427557+0000", 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0951e-05 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427566+0000", 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3070000000000008e-06 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427573+0000", 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.674e-06 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481430+0000", 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053857078000000003 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481593+0000", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000163618 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481730+0000", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013723899999999999 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482879+0000", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001148376 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.483564+0000", 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000685375 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.504042+0000", 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020477795 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.504122+0000", 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0238e-05 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.504221+0000", 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9149999999999998e-05 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:13968 7.0 e86/79 7:07c32104:::benchmark_data_smithi081_73391_object13967:head v 86'896, mlcod=86'893)", 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.427655+0000", 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.158286564000001, 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052740209000000003, 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427655+0000", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427655+0000", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427656+0000", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.16e-07 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427683+0000", 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7072000000000001e-05 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427693+0000", 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4239999999999999e-06 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427699+0000", 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4980000000000003e-06 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:41.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427746+0000", 2024-09-21T18:54:41.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7061000000000002e-05 2024-09-21T18:54:41.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.629 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427763+0000", 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6688e-05 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480361+0000", 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052597478000000003 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480396+0000", 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5071999999999999e-05 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.630 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:13971 7.4 e86/82 7:2f282279:::benchmark_data_smithi081_73391_object13970:head v 86'941, mlcod=86'938)", 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.427696+0000", 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.158245643000001, 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053803489000000003, 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427696+0000", 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427696+0000", 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427697+0000", 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7899999999999997e-07 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427757+0000", 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0244999999999997e-05 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427765+0000", 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9149999999999996e-06 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427771+0000", 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3850000000000001e-06 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427834+0000", 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3204e-05 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427850+0000", 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5855e-05 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481462+0000", 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053612357999999999 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481500+0000", 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7648000000000003e-05 2024-09-21T18:54:41.634 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:13963 7.b e86/79 ondisk, result = 0)", 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.428262+0000", 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.157680031999998, 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052626104999999999, 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428262+0000", 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428262+0000", 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428263+0000", 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428267+0000", 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0520000000000003e-06 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480328+0000", 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052060934000000003 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480337+0000", 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0669999999999998e-06 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480867+0000", 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00052987499999999996 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480874+0000", 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2799999999999998e-06 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480888+0000", 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4064e-05 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:13961 7.9 e86/82 ondisk, result = 0)", 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.428441+0000", 2024-09-21T18:54:41.638 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.157500939999998, 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.054165603999999999, 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428441+0000", 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428441+0000", 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428442+0000", 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0049999999999999e-06 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428446+0000", 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3490000000000002e-06 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480258+0000", 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.051811264000000003 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.480266+0000", 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9610000000000005e-06 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482515+0000", 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022492380000000002 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482524+0000", 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8100000000000004e-06 2024-09-21T18:54:41.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482607+0000", 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2977000000000002e-05 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:13961 7.9 e86/82 ondisk, result = 0)", 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.429138+0000", 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.156804297000001, 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053316277000000002, 2024-09-21T18:54:41.642 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429138+0000", 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429138+0000", 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429139+0000", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0780000000000001e-06 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429143+0000", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2579999999999997e-06 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.479903+0000", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.050759883999999998 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.479914+0000", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1102e-05 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482432+0000", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002517521 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482438+0000", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2199999999999997e-06 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482454+0000", 2024-09-21T18:54:41.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6214000000000001e-05 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:39530 7.9 7:9ef549d3:::benchmark_data_smithi081_73391_object39529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:37.821186+0000", 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.7647563550000003, 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046770700999999998, 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.646 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 39530 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821186+0000", 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821186+0000", 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821187+0000", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8139999999999999e-06 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821210+0000", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2271000000000001e-05 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821220+0000", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0523000000000001e-05 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821228+0000", 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4900000000000003e-06 2024-09-21T18:54:41.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821277+0000", 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9734999999999997e-05 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821360+0000", 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2237999999999994e-05 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.821441+0000", 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1786000000000006e-05 2024-09-21T18:54:41.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.822860+0000", 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014184989999999999 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.867781+0000", 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.044921230999999999 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.867829+0000", 2024-09-21T18:54:41.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7342000000000002e-05 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.867916+0000", 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7929999999999993e-05 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.867956+0000", 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9842000000000002e-05 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.651 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:39958 7.9 7:9d804fcc:::benchmark_data_smithi081_73391_object39957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:38.064307+0000", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.5216348790000001, 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.047623691000000003, 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 39958 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064307+0000", 2024-09-21T18:54:41.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064307+0000", 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064309+0000", 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.08e-06 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064333+0000", 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3892000000000001e-05 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064344+0000", 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1226000000000001e-05 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064351+0000", 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8390000000000001e-06 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064409+0000", 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7982000000000002e-05 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064532+0000", 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012257300000000001 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064664+0000", 2024-09-21T18:54:41.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013210000000000001 2024-09-21T18:54:41.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.657 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.109945+0000", 2024-09-21T18:54:41.657 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045281129000000003 2024-09-21T18:54:41.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.109985+0000", 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0343000000000003e-05 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.111787+0000", 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018012869999999999 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.111884+0000", 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7286000000000003e-05 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.111931+0000", 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6953999999999998e-05 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:39949 7.d e90/82 7:b41f2873:::benchmark_data_smithi081_73391_object39948:head v 90'2481, mlcod=90'2478)", 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:38.064348+0000", 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.5215935979999999, 2024-09-21T18:54:41.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045557092, 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064348+0000", 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064348+0000", 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064349+0000", 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8400000000000005e-07 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064379+0000", 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0303000000000001e-05 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064385+0000", 2024-09-21T18:54:41.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9800000000000003e-06 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064390+0000", 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.702e-06 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064652+0000", 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026229200000000002 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064668+0000", 2024-09-21T18:54:41.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6073999999999999e-05 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.109860+0000", 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045192243 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.109905+0000", 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5014e-05 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:39953 7.d e90/82 7:b34c7278:::benchmark_data_smithi081_73391_object39952:head v 90'2482, mlcod=90'2478)", 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:38.064405+0000", 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.521536861, 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045533557000000002, 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064405+0000", 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064405+0000", 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064406+0000", 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0200000000000003e-07 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064430+0000", 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4332999999999999e-05 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064435+0000", 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3129999999999998e-06 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064439+0000", 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8510000000000001e-06 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.066641+0000", 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002202411 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.066660+0000", 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8669999999999999e-05 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.109916+0000", 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.043255556000000001 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.109939+0000", 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3020999999999999e-05 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:39959 7.9 7:99eb2b8e:::benchmark_data_smithi081_73391_object39958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:38.064466+0000", 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.521475847, 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.047595145999999998, 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 39959 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.668 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064466+0000", 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064466+0000", 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064468+0000", 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.53e-06 2024-09-21T18:54:41.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064490+0000", 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2357e-05 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064501+0000", 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0795000000000001e-05 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.064508+0000", 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0740000000000004e-06 2024-09-21T18:54:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.067692+0000", 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003183651 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.067836+0000", 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014454499999999999 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.067973+0000", 2024-09-21T18:54:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000137119 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.110039+0000", 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.042065911999999997 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.111775+0000", 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001736175 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.111942+0000", 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001671 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.112024+0000", 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1746e-05 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.112061+0000", 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7141999999999998e-05 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:39967 7.9 7:9d83a7c1:::benchmark_data_smithi081_73391_object39966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:38.068502+0000", 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.517439575, 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045945804, 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 39967 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.674 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.068502+0000", 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.068502+0000", 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.068504+0000", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5030000000000001e-06 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.068526+0000", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1866999999999999e-05 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.068534+0000", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9480000000000008e-06 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.068540+0000", 2024-09-21T18:54:41.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.789e-06 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.110094+0000", 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.041554593000000001 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.110245+0000", 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150877 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.110369+0000", 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000124497 2024-09-21T18:54:41.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.112073+0000", 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017039469999999999 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.114239+0000", 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002165756 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.114283+0000", 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4113999999999998e-05 2024-09-21T18:54:41.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.114330+0000", 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6350000000000002e-05 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:38.114448+0000", 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118563 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.679 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:44868 7.b 7:d8c6a79a:::benchmark_data_smithi081_73391_object44867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:40.561536+0000", 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.024405765, 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049845648999999999, 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 44868 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561536+0000", 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561536+0000", 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561538+0000", 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6759999999999999e-06 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561560+0000", 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2160000000000001e-05 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561568+0000", 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9109999999999993e-06 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561574+0000", 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1079999999999998e-06 2024-09-21T18:54:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561808+0000", 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023403900000000001 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561928+0000", 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119838 2024-09-21T18:54:41.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.562187+0000", 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025926299999999997 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.565350+0000", 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031625080000000001 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.565422+0000", 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2744999999999996e-05 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.611212+0000", 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045789942 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.611301+0000", 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8937999999999994e-05 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.611382+0000", 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0520999999999997e-05 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:44869 7.b 7:dc9d9062:::benchmark_data_smithi081_73391_object44868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:40.561625+0000", 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.0243168949999999, 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.054025653, 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.686 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 44869 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561625+0000", 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561625+0000", 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561626+0000", 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8999999999999995e-07 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561645+0000", 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8746e-05 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561651+0000", 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3600000000000001e-06 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.561656+0000", 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9110000000000001e-06 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.563813+0000", 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00215701 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.563929+0000", 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116249 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.564065+0000", 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000135439 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.565567+0000", 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015028629999999999 2024-09-21T18:54:41.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.566355+0000", 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00078774400000000005 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.615500+0000", 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049144673 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.615605+0000", 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010477899999999999 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.615651+0000", 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6088999999999999e-05 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:44883 7.b 7:d202a168:::benchmark_data_smithi081_73391_object44882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:40.570548+0000", 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.0153941660000001, 2024-09-21T18:54:41.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045265519999999997, 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 44883 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570548+0000", 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570548+0000", 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570549+0000", 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.55e-06 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570568+0000", 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8352e-05 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570577+0000", 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3100000000000006e-06 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570582+0000", 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0989999999999998e-06 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570627+0000", 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4839000000000002e-05 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570721+0000", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3653999999999995e-05 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.570817+0000", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6118000000000004e-05 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.571921+0000", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001104474 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.572177+0000", 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025565899999999999 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.615691+0000", 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.043514358000000003 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.615779+0000", 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7996000000000002e-05 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:54:41.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:40.615813+0000", 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4110999999999999e-05 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:54:41.698 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:54:41.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:41.763 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:41.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:54:41.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:41.906 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:41.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:41.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 4.9 scrub starts 2024-09-21T18:54:41.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 4.9 scrub ok 2024-09-21T18:54:41.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 3.13 scrub starts 2024-09-21T18:54:41.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 3.13 scrub ok 2024-09-21T18:54:41.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T18:54:41.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 5.3 scrub starts 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 5.3 scrub ok 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 5.5 scrub starts 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[39192]: 5.5 scrub ok 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 4.9 scrub starts 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 4.9 scrub ok 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 3.13 scrub starts 2024-09-21T18:54:41.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 3.13 scrub ok 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 5.3 scrub starts 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 5.3 scrub ok 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 5.5 scrub starts 2024-09-21T18:54:41.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:41 smithi081 ceph-mon[29384]: 5.5 scrub ok 2024-09-21T18:54:42.007 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:42.037 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:42.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:42.147 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 21 16 47620 47604 141.652 109.438 0.005017 0.00705426 2024-09-21T18:54:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:42.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 4.9 scrub starts 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 4.9 scrub ok 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 3.13 scrub starts 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 3.13 scrub ok 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 5.3 scrub starts 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 5.3 scrub ok 2024-09-21T18:54:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 5.5 scrub starts 2024-09-21T18:54:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:41 smithi104 ceph-mon[32438]: 5.5 scrub ok 2024-09-21T18:54:42.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:42.322 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:42.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:42.462 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:42.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:42.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:42.604 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:42.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:42.737 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:42.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:42.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:42.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:42.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:42.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:54:43.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: 2.0 scrub starts 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: 2.0 scrub ok 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: pgmap v55: 148 pgs: 1 peering, 147 active+clean; 2.5 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 143 MiB/s wr, 2.29k op/s; 136 B/s, 3 objects/s recovering 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:54:43.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: 2.0 scrub starts 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: 2.0 scrub ok 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: pgmap v55: 148 pgs: 1 peering, 147 active+clean; 2.5 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 143 MiB/s wr, 2.29k op/s; 136 B/s, 3 objects/s recovering 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:54:43.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:54:43.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:43.148 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 22 16 49469 49453 140.46 115.562 0.012294 0.00711339 2024-09-21T18:54:43.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: 2.0 scrub starts 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: 2.0 scrub ok 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: pgmap v55: 148 pgs: 1 peering, 147 active+clean; 2.5 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.8 KiB/s rd, 143 MiB/s wr, 2.29k op/s; 136 B/s, 3 objects/s recovering 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:54:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:54:43.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.295 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:43.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:43.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.433 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:43.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:43.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:43.562 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:43.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:43.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.840 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:43.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:43.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T18:54:43.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:43.959 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:43.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:43.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[39192]: osdmap e94: 8 total, 8 up, 7 in 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[39192]: 3.4 scrub starts 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[39192]: 3.4 scrub ok 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[29384]: osdmap e94: 8 total, 8 up, 7 in 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T18:54:44.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[29384]: 3.4 scrub starts 2024-09-21T18:54:44.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:43 smithi081 ceph-mon[29384]: 3.4 scrub ok 2024-09-21T18:54:44.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:44.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:44.149 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 23 15 51122 51107 138.838 103.375 0.00677754 0.00719795 2024-09-21T18:54:44.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:44.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:43 smithi104 ceph-mon[32438]: osdmap e94: 8 total, 8 up, 7 in 2024-09-21T18:54:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:43 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T18:54:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:43 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T18:54:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:43 smithi104 ceph-mon[32438]: 3.4 scrub starts 2024-09-21T18:54:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:43 smithi104 ceph-mon[32438]: 3.4 scrub ok 2024-09-21T18:54:44.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.365 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:44.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:44.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:44.496 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:44.598 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.639 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:44.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:44.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:44.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:44.900 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:44.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:45.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:45.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:45.150 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 24 16 52844 52828 137.534 107.562 0.0155981 0.0072661 2024-09-21T18:54:45.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:45.182 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[29384]: pgmap v58: 148 pgs: 148 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s; 192 B/s, 5 objects/s recovering 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[39192]: pgmap v58: 148 pgs: 148 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s; 192 B/s, 5 objects/s recovering 2024-09-21T18:54:45.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:54:45.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:54:45.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:44 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:44 smithi104 ceph-mon[32438]: pgmap v58: 148 pgs: 148 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s; 192 B/s, 5 objects/s recovering 2024-09-21T18:54:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:44 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:44 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:44 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:54:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:44 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:54:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:44 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:45.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:45.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:45.498 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:54:45.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:54:45.499 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-21T18:54:45.499 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-21T18:54:45.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:54:45.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:45.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:45.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:45.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:45.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:45.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:45.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:45.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:46.016 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:46.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:46.150 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 25 16 54476 54460 136.112 102 0.018453 0.0073415 2024-09-21T18:54:46.156 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:45 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T18:54:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:45 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T18:54:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:45 smithi104 ceph-mon[32438]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T18:54:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:45 smithi104 ceph-mon[32438]: Reconfiguring daemon alertmanager.a on smithi081 2024-09-21T18:54:46.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:46.288 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[29384]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[29384]: Reconfiguring daemon alertmanager.a on smithi081 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[39192]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T18:54:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:45 smithi081 ceph-mon[39192]: Reconfiguring daemon alertmanager.a on smithi081 2024-09-21T18:54:46.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:46.422 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:46.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:46.552 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:46.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:46.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:46.825 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:46.840 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:54:46.892 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:54:46.892 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:54:46.902 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:54:46.902 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:54:46.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: pgmap v59: 148 pgs: 148 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 911 B/s rd, 104 MiB/s wr, 1.66k op/s; 137 B/s, 3 objects/s recovering 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2696131483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: 4.4 scrub starts 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: 4.4 scrub ok 2024-09-21T18:54:46.937 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2298896043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: pgmap v59: 148 pgs: 148 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 911 B/s rd, 104 MiB/s wr, 1.66k op/s; 137 B/s, 3 objects/s recovering 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2696131483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: 4.4 scrub starts 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: 4.4 scrub ok 2024-09-21T18:54:46.938 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2298896043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:46.938 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:54:46.938 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:54:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:46.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:46.981 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:54:46.981 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:54:47.012 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:54:47.012 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:54:47.043 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:54:47.043 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:54:47.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:47.078 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:54:47.078 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:54:47.103 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:47.112 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:54:47.112 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:54:47.132 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:47.150 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 26 16 55948 55932 134.416 92 0.00570992 0.00743414 2024-09-21T18:54:47.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:47.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: pgmap v59: 148 pgs: 148 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 911 B/s rd, 104 MiB/s wr, 1.66k op/s; 137 B/s, 3 objects/s recovering 2024-09-21T18:54:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T18:54:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T18:54:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2696131483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:54:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: 4.4 scrub starts 2024-09-21T18:54:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: 4.4 scrub ok 2024-09-21T18:54:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2298896043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:47.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:47.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:47.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:47.509 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:47.530 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:54:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:54:47.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:47.647 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:47.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:47.780 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:47.864 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:54:47.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:47.923 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:47.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:48.024 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:48.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2696131483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[39192]: osdmap e95: 8 total, 8 up, 7 in 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2696131483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:54:48.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[29384]: osdmap e95: 8 total, 8 up, 7 in 2024-09-21T18:54:48.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T18:54:48.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:47 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T18:54:48.150 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 27 16 57756 57740 133.622 113 0.00957915 0.00747935 2024-09-21T18:54:48.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.203 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:48.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:48.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:54:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:54:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2696131483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:54:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-mon[32438]: osdmap e95: 8 total, 8 up, 7 in 2024-09-21T18:54:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T18:54:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:47 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T18:54:48.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:48.345 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:48.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:48.483 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:48.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:48.626 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:48.698 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:54:48.698 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:54:48.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.761 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:48.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:48.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:48.899 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:48.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:48.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:49.001 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:49.033 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:49.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:49.150 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 28 16 59480 59464 132.696 107.75 0.011514 0.00753111 2024-09-21T18:54:49.169 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:49.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:48 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T18:54:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:48 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T18:54:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:48 smithi104 ceph-mon[32438]: pgmap v61: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 144 MiB/s wr, 2.30k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T18:54:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:48 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T18:54:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:48 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T18:54:49.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.306 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:49.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[39192]: pgmap v61: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 144 MiB/s wr, 2.30k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[29384]: pgmap v61: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 144 MiB/s wr, 2.30k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T18:54:49.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:48 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T18:54:49.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:49.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:49.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:49.577 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:49.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:49.711 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:49.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:49.849 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:49.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T18:54:49.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:49.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:50.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:50.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:50.151 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 29 16 61167 61151 131.756 105.438 0.00971614 0.00758535 2024-09-21T18:54:50.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:50.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:50.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:50.391 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:50.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:50.533 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:50.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:50.673 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:50.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:50 smithi104 ceph-mon[32438]: 4.1 scrub starts 2024-09-21T18:54:50.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:50 smithi104 ceph-mon[32438]: 4.1 scrub ok 2024-09-21T18:54:50.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:50 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T18:54:50.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:50 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T18:54:50.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:50 smithi104 ceph-mon[32438]: pgmap v62: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 673 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T18:54:50.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:50 smithi104 ceph-mon[32438]: osdmap e96: 8 total, 8 up, 7 in 2024-09-21T18:54:50.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:50.809 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[39192]: 4.1 scrub starts 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[39192]: 4.1 scrub ok 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[39192]: pgmap v62: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 673 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[39192]: osdmap e96: 8 total, 8 up, 7 in 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[29384]: 4.1 scrub starts 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[29384]: 4.1 scrub ok 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[29384]: pgmap v62: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 673 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T18:54:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:50 smithi081 ceph-mon[29384]: osdmap e96: 8 total, 8 up, 7 in 2024-09-21T18:54:50.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:50.947 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:50.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:51.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:51.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:54:51.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:51.150 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 30 16 62823 62807 130.813 103.5 0.00738697 0.00763996 2024-09-21T18:54:51.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:51.239 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:51.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:51.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:51.514 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:51.616 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:51.653 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:51.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:51.785 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:51.887 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:51.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:52.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[39192]: 3.1e scrub starts 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[39192]: 3.1e scrub ok 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[29384]: 3.1e scrub starts 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[29384]: 3.1e scrub ok 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T18:54:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T18:54:52.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:52.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:52.151 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 31 16 64342 64326 129.656 94.9375 0.00233572 0.00770484 2024-09-21T18:54:52.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:52.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:51 smithi104 ceph-mon[32438]: 3.1e scrub starts 2024-09-21T18:54:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:51 smithi104 ceph-mon[32438]: 3.1e scrub ok 2024-09-21T18:54:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:51 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T18:54:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:51 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T18:54:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:54:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:54:52.296 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:52.329 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:52.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:52.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:52.473 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:52.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:54:52.575 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:52.611 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:52.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:52.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:52.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:52.879 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:52.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:52.936 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:52.939 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:52.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:53.006 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:53.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: 6.17 scrub starts 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: 6.17 scrub ok 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: 4.16 scrub starts 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: 4.16 scrub ok 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: pgmap v64: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: 4.b scrub starts 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[29384]: 4.b scrub ok 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: 6.17 scrub starts 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: 6.17 scrub ok 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: 4.16 scrub starts 2024-09-21T18:54:53.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: 4.16 scrub ok 2024-09-21T18:54:53.071 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: pgmap v64: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-21T18:54:53.071 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: 4.b scrub starts 2024-09-21T18:54:53.071 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:52 smithi081 ceph-mon[39192]: 4.b scrub ok 2024-09-21T18:54:53.071 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:54:52 smithi081 systemd[1]: Stopping Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:54:53.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:53.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 32 16 65849 65833 128.548 94.1875 0.0141717 0.00777456 2024-09-21T18:54:53.154 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:53.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: 6.17 scrub starts 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: 6.17 scrub ok 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: 4.16 scrub starts 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: 4.16 scrub ok 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: pgmap v64: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: 4.b scrub starts 2024-09-21T18:54:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:52 smithi104 ceph-mon[32438]: 4.b scrub ok 2024-09-21T18:54:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:53.280 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:53.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:53.422 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:53.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:53.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:53.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:53.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:53.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:53.824 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:53.847 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:54:53.847 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:54:53.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:53.962 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:53.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[29384]: 5.1a scrub starts 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[29384]: 5.1a scrub ok 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[39192]: 5.1a scrub starts 2024-09-21T18:54:54.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:53 smithi081 ceph-mon[39192]: 5.1a scrub ok 2024-09-21T18:54:54.064 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:54.094 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:54.151 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 33 16 67383 67367 127.557 95.875 0.00334553 0.00783424 2024-09-21T18:54:54.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:54.225 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:54.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:53 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T18:54:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:53 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T18:54:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:53 smithi104 ceph-mon[32438]: 5.1a scrub starts 2024-09-21T18:54:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:53 smithi104 ceph-mon[32438]: 5.1a scrub ok 2024-09-21T18:54:54.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:54.364 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:54.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:54.514 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:54.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:54.659 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:54.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:54.773 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:54:54.786 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:54.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:54.893 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:54:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T18:54:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T18:54:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[29384]: 3.8 scrub starts 2024-09-21T18:54:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[29384]: 3.8 scrub ok 2024-09-21T18:54:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[29384]: pgmap v65: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-09-21T18:54:54.894 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T18:54:54.895 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T18:54:54.895 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[39192]: 3.8 scrub starts 2024-09-21T18:54:54.895 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[39192]: 3.8 scrub ok 2024-09-21T18:54:54.895 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:54 smithi081 ceph-mon[39192]: pgmap v65: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-09-21T18:54:54.901 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:54:54.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:54.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:55.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:55.069 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:55.151 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 34 16 68820 68804 126.447 89.8125 0.0108564 0.00790432 2024-09-21T18:54:55.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:55.203 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:54 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T18:54:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:54 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T18:54:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:54 smithi104 ceph-mon[32438]: 3.8 scrub starts 2024-09-21T18:54:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:54 smithi104 ceph-mon[32438]: 3.8 scrub ok 2024-09-21T18:54:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:54 smithi104 ceph-mon[32438]: pgmap v65: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-09-21T18:54:55.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:55.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:55.477 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:55.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:55.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:55.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:55.860 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:55.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:56.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:56.039 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:56.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[39192]: 6.1b scrub starts 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[39192]: 6.1b scrub ok 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[39192]: from='client.34787 v1:172.21.15.81:0/2906043307' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3677256851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[29384]: 6.1b scrub starts 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[29384]: 6.1b scrub ok 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[29384]: from='client.34787 v1:172.21.15.81:0/2906043307' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T18:54:56.107 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T18:54:56.108 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3677256851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:56.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:56.151 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 35 16 70342 70326 125.552 95.125 0.00885791 0.0079601 2024-09-21T18:54:56.158 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-21T18:54:53.388128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4140833307,"num_objects":63374,"num_object_clones":0,"num_object_copies":190122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63374,"num_whiteouts":0,"num_read":940,"num_read_kb":797,"num_write":63724,"num_write_kb":4044873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":1184346,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63730,"ondisk_log_size":63730,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12377416,"kb_used_data":11988368,"kb_used_omap":10,"kb_used_meta":388917,"kb_avail":643838648,"statfs":{"total":671965249536,"available":659290775552,"internally_reserved":0,"allocated":12276088832,"data_stored":12271964522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11153,"internal_metadata":398251119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1107361792,"num_objects":16897,"num_object_clones":0,"num_object_copies":50691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16897,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16897,"num_write_kb":1081408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001948"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456565+0000","last_change":"2024-09-21T18:54:17.713866+0000","last_active":"2024-09-21T18:54:49.456565+0000","last_peered":"2024-09-21T18:54:49.456565+0000","last_clean":"2024-09-21T18:54:49.456565+0000","last_became_active":"2024-09-21T18:54:17.713495+0000","last_became_peered":"2024-09-21T18:54:17.713495+0000","last_unstale":"2024-09-21T18:54:49.456565+0000","last_undegraded":"2024-09-21T18:54:49.456565+0000","last_fullsized":"2024-09-21T18:54:49.456565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.249265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":116,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457656+0000","last_change":"2024-09-21T18:54:36.776089+0000","last_active":"2024-09-21T18:54:49.457656+0000","last_peered":"2024-09-21T18:54:49.457656+0000","last_clean":"2024-09-21T18:54:49.457656+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:54:49.457656+0000","last_undegraded":"2024-09-21T18:54:49.457656+0000","last_fullsized":"2024-09-21T18:54:49.457656+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.052772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455683+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:54:49.455683+0000","last_peered":"2024-09-21T18:54:49.455683+0000","last_clean":"2024-09-21T18:54:49.455683+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:54:49.455683+0000","last_undegraded":"2024-09-21T18:54:49.455683+0000","last_fullsized":"2024-09-21T18:54:49.455683+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.278309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456993+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:54:49.456993+0000","last_peered":"2024-09-21T18:54:49.456993+0000","last_clean":"2024-09-21T18:54:49.456993+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:54:49.456993+0000","last_undegraded":"2024-09-21T18:54:49.456993+0000","last_fullsized":"2024-09-21T18:54:49.456993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.055413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":76,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452698+0000","last_change":"2024-09-21T18:53:13.658931+0000","last_active":"2024-09-21T18:54:49.452698+0000","last_peered":"2024-09-21T18:54:49.452698+0000","last_clean":"2024-09-21T18:54:49.452698+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:54:49.452698+0000","last_undegraded":"2024-09-21T18:54:49.452698+0000","last_fullsized":"2024-09-21T18:54:49.452698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":96,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:50.090079+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:54:50.090079+0000","last_peered":"2024-09-21T18:54:50.090079+0000","last_clean":"2024-09-21T18:54:50.090079+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:54:50.090079+0000","last_undegraded":"2024-09-21T18:54:50.090079+0000","last_fullsized":"2024-09-21T18:54:50.090079+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.541042+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":132,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453336+0000","last_change":"2024-09-21T18:54:17.730862+0000","last_active":"2024-09-21T18:54:49.453336+0000","last_peered":"2024-09-21T18:54:49.453336+0000","last_clean":"2024-09-21T18:54:49.453336+0000","last_became_active":"2024-09-21T18:54:17.716359+0000","last_became_peered":"2024-09-21T18:54:17.716359+0000","last_unstale":"2024-09-21T18:54:49.453336+0000","last_undegraded":"2024-09-21T18:54:49.453336+0000","last_fullsized":"2024-09-21T18:54:49.453336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.191168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457499+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:54:49.457499+0000","last_peered":"2024-09-21T18:54:49.457499+0000","last_clean":"2024-09-21T18:54:49.457499+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:54:49.457499+0000","last_undegraded":"2024-09-21T18:54:49.457499+0000","last_fullsized":"2024-09-21T18:54:49.457499+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.928369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456627+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:54:49.456627+0000","last_peered":"2024-09-21T18:54:49.456627+0000","last_clean":"2024-09-21T18:54:49.456627+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:54:49.456627+0000","last_undegraded":"2024-09-21T18:54:49.456627+0000","last_fullsized":"2024-09-21T18:54:49.456627+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":124,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457312+0000","last_change":"2024-09-21T18:54:17.725564+0000","last_active":"2024-09-21T18:54:49.457312+0000","last_peered":"2024-09-21T18:54:49.457312+0000","last_clean":"2024-09-21T18:54:49.457312+0000","last_became_active":"2024-09-21T18:54:17.704975+0000","last_became_peered":"2024-09-21T18:54:17.704975+0000","last_unstale":"2024-09-21T18:54:49.457312+0000","last_undegraded":"2024-09-21T18:54:49.457312+0000","last_fullsized":"2024-09-21T18:54:49.457312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:14.972606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.972606+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.449994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455858+0000","last_change":"2024-09-21T18:54:17.712927+0000","last_active":"2024-09-21T18:54:49.455858+0000","last_peered":"2024-09-21T18:54:49.455858+0000","last_clean":"2024-09-21T18:54:49.455858+0000","last_became_active":"2024-09-21T18:54:17.712591+0000","last_became_peered":"2024-09-21T18:54:17.712591+0000","last_unstale":"2024-09-21T18:54:49.455858+0000","last_undegraded":"2024-09-21T18:54:49.455858+0000","last_fullsized":"2024-09-21T18:54:49.455858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452116+0000","last_change":"2024-09-21T18:54:17.704813+0000","last_active":"2024-09-21T18:54:49.452116+0000","last_peered":"2024-09-21T18:54:49.452116+0000","last_clean":"2024-09-21T18:54:49.452116+0000","last_became_active":"2024-09-21T18:54:17.704510+0000","last_became_peered":"2024-09-21T18:54:17.704510+0000","last_unstale":"2024-09-21T18:54:49.452116+0000","last_undegraded":"2024-09-21T18:54:49.452116+0000","last_fullsized":"2024-09-21T18:54:49.452116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.329858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456111+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:54:49.456111+0000","last_peered":"2024-09-21T18:54:49.456111+0000","last_clean":"2024-09-21T18:54:49.456111+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:54:49.456111+0000","last_undegraded":"2024-09-21T18:54:49.456111+0000","last_fullsized":"2024-09-21T18:54:49.456111+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":101,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457885+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:54:49.457885+0000","last_peered":"2024-09-21T18:54:49.457885+0000","last_clean":"2024-09-21T18:54:49.457885+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:54:49.457885+0000","last_undegraded":"2024-09-21T18:54:49.457885+0000","last_fullsized":"2024-09-21T18:54:49.457885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.337954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":78,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453253+0000","last_change":"2024-09-21T18:53:11.412969+0000","last_active":"2024-09-21T18:54:49.453253+0000","last_peered":"2024-09-21T18:54:49.453253+0000","last_clean":"2024-09-21T18:54:49.453253+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:54:49.453253+0000","last_undegraded":"2024-09-21T18:54:49.453253+0000","last_fullsized":"2024-09-21T18:54:49.453253+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455603+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:54:49.455603+0000","last_peered":"2024-09-21T18:54:49.455603+0000","last_clean":"2024-09-21T18:54:49.455603+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:54:49.455603+0000","last_undegraded":"2024-09-21T18:54:49.455603+0000","last_fullsized":"2024-09-21T18:54:49.455603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.155050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458435+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:54:49.458435+0000","last_peered":"2024-09-21T18:54:49.458435+0000","last_clean":"2024-09-21T18:54:49.458435+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:54:49.458435+0000","last_undegraded":"2024-09-21T18:54:49.458435+0000","last_fullsized":"2024-09-21T18:54:49.458435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.940396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458058+0000","last_change":"2024-09-21T18:54:17.705466+0000","last_active":"2024-09-21T18:54:49.458058+0000","last_peered":"2024-09-21T18:54:49.458058+0000","last_clean":"2024-09-21T18:54:49.458058+0000","last_became_active":"2024-09-21T18:54:17.705159+0000","last_became_peered":"2024-09-21T18:54:17.705159+0000","last_unstale":"2024-09-21T18:54:49.458058+0000","last_undegraded":"2024-09-21T18:54:49.458058+0000","last_fullsized":"2024-09-21T18:54:49.458058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.944967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":116,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458837+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:54:49.458837+0000","last_peered":"2024-09-21T18:54:49.458837+0000","last_clean":"2024-09-21T18:54:49.458837+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:54:49.458837+0000","last_undegraded":"2024-09-21T18:54:49.458837+0000","last_fullsized":"2024-09-21T18:54:49.458837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.670449+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.461302+0000","last_change":"2024-09-21T18:54:17.714793+0000","last_active":"2024-09-21T18:54:49.461302+0000","last_peered":"2024-09-21T18:54:49.461302+0000","last_clean":"2024-09-21T18:54:49.461302+0000","last_became_active":"2024-09-21T18:54:17.714300+0000","last_became_peered":"2024-09-21T18:54:17.714300+0000","last_unstale":"2024-09-21T18:54:49.461302+0000","last_undegraded":"2024-09-21T18:54:49.461302+0000","last_fullsized":"2024-09-21T18:54:49.461302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":76,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453750+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:54:49.453750+0000","last_peered":"2024-09-21T18:54:49.453750+0000","last_clean":"2024-09-21T18:54:49.453750+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:54:49.453750+0000","last_undegraded":"2024-09-21T18:54:49.453750+0000","last_fullsized":"2024-09-21T18:54:49.453750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":95,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457467+0000","last_change":"2024-09-21T18:54:12.987795+0000","last_active":"2024-09-21T18:54:49.457467+0000","last_peered":"2024-09-21T18:54:49.457467+0000","last_clean":"2024-09-21T18:54:49.457467+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:54:49.457467+0000","last_undegraded":"2024-09-21T18:54:49.457467+0000","last_fullsized":"2024-09-21T18:54:49.457467+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.987749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.987749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:41.935763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":115,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456896+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:54:49.456896+0000","last_peered":"2024-09-21T18:54:49.456896+0000","last_clean":"2024-09-21T18:54:49.456896+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:54:49.456896+0000","last_undegraded":"2024-09-21T18:54:49.456896+0000","last_fullsized":"2024-09-21T18:54:49.456896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:06.555236+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457840+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:54:49.457840+0000","last_peered":"2024-09-21T18:54:49.457840+0000","last_clean":"2024-09-21T18:54:49.457840+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:54:49.457840+0000","last_undegraded":"2024-09-21T18:54:49.457840+0000","last_fullsized":"2024-09-21T18:54:49.457840+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.960588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453569+0000","last_change":"2024-09-21T18:53:13.426725+0000","last_active":"2024-09-21T18:54:49.453569+0000","last_peered":"2024-09-21T18:54:49.453569+0000","last_clean":"2024-09-21T18:54:49.453569+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:54:49.453569+0000","last_undegraded":"2024-09-21T18:54:49.453569+0000","last_fullsized":"2024-09-21T18:54:49.453569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":96,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455328+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:54:49.455328+0000","last_peered":"2024-09-21T18:54:49.455328+0000","last_clean":"2024-09-21T18:54:49.455328+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:54:49.455328+0000","last_undegraded":"2024-09-21T18:54:49.455328+0000","last_fullsized":"2024-09-21T18:54:49.455328+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:43.629105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454238+0000","last_change":"2024-09-21T18:54:17.738374+0000","last_active":"2024-09-21T18:54:49.454238+0000","last_peered":"2024-09-21T18:54:49.454238+0000","last_clean":"2024-09-21T18:54:49.454238+0000","last_became_active":"2024-09-21T18:54:17.719024+0000","last_became_peered":"2024-09-21T18:54:17.719024+0000","last_unstale":"2024-09-21T18:54:49.454238+0000","last_undegraded":"2024-09-21T18:54:49.454238+0000","last_fullsized":"2024-09-21T18:54:49.454238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.810583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":110,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459329+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:54:49.459329+0000","last_peered":"2024-09-21T18:54:49.459329+0000","last_clean":"2024-09-21T18:54:49.459329+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:54:49.459329+0000","last_undegraded":"2024-09-21T18:54:49.459329+0000","last_fullsized":"2024-09-21T18:54:49.459329+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.245724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"96'3946","reported_seq":3991,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.342302+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:54:51.342302+0000","last_peered":"2024-09-21T18:54:51.342302+0000","last_clean":"2024-09-21T18:54:51.342302+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:54:51.342302+0000","last_undegraded":"2024-09-21T18:54:51.342302+0000","last_fullsized":"2024-09-21T18:54:51.342302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:14.400572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258605056,"num_objects":3946,"num_object_clones":0,"num_object_copies":11838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":108,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454197+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:54:49.454197+0000","last_peered":"2024-09-21T18:54:49.454197+0000","last_clean":"2024-09-21T18:54:49.454197+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:54:49.454197+0000","last_undegraded":"2024-09-21T18:54:49.454197+0000","last_fullsized":"2024-09-21T18:54:49.454197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.353165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452362+0000","last_change":"2024-09-21T18:53:07.398833+0000","last_active":"2024-09-21T18:54:49.452362+0000","last_peered":"2024-09-21T18:54:49.452362+0000","last_clean":"2024-09-21T18:54:49.452362+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:54:49.452362+0000","last_undegraded":"2024-09-21T18:54:49.452362+0000","last_fullsized":"2024-09-21T18:54:49.452362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"83'9","reported_seq":119,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458518+0000","last_change":"2024-09-21T18:54:17.715008+0000","last_active":"2024-09-21T18:54:49.458518+0000","last_peered":"2024-09-21T18:54:49.458518+0000","last_clean":"2024-09-21T18:54:49.458518+0000","last_became_active":"2024-09-21T18:54:17.706739+0000","last_became_peered":"2024-09-21T18:54:17.706739+0000","last_unstale":"2024-09-21T18:54:49.458518+0000","last_undegraded":"2024-09-21T18:54:49.458518+0000","last_fullsized":"2024-09-21T18:54:49.458518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":82,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456190+0000","last_change":"2024-09-21T18:54:17.705694+0000","last_active":"2024-09-21T18:54:49.456190+0000","last_peered":"2024-09-21T18:54:49.456190+0000","last_clean":"2024-09-21T18:54:49.456190+0000","last_became_active":"2024-09-21T18:54:17.705355+0000","last_became_peered":"2024-09-21T18:54:17.705355+0000","last_unstale":"2024-09-21T18:54:49.456190+0000","last_undegraded":"2024-09-21T18:54:49.456190+0000","last_fullsized":"2024-09-21T18:54:49.456190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.222602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"96'3911","reported_seq":3956,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.397539+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:54:51.397539+0000","last_peered":"2024-09-21T18:54:51.397539+0000","last_clean":"2024-09-21T18:54:51.397539+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:54:51.397539+0000","last_undegraded":"2024-09-21T18:54:51.397539+0000","last_fullsized":"2024-09-21T18:54:51.397539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.493653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":250304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":141,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452144+0000","last_change":"2024-09-21T18:54:17.740103+0000","last_active":"2024-09-21T18:54:49.452144+0000","last_peered":"2024-09-21T18:54:49.452144+0000","last_clean":"2024-09-21T18:54:49.452144+0000","last_became_active":"2024-09-21T18:54:17.716713+0000","last_became_peered":"2024-09-21T18:54:17.716713+0000","last_unstale":"2024-09-21T18:54:49.452144+0000","last_undegraded":"2024-09-21T18:54:49.452144+0000","last_fullsized":"2024-09-21T18:54:49.452144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":95,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454423+0000","last_change":"2024-09-21T18:54:17.719791+0000","last_active":"2024-09-21T18:54:49.454423+0000","last_peered":"2024-09-21T18:54:49.454423+0000","last_clean":"2024-09-21T18:54:49.454423+0000","last_became_active":"2024-09-21T18:54:17.719440+0000","last_became_peered":"2024-09-21T18:54:17.719440+0000","last_unstale":"2024-09-21T18:54:49.454423+0000","last_undegraded":"2024-09-21T18:54:49.454423+0000","last_fullsized":"2024-09-21T18:54:49.454423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.901802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459084+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:54:49.459084+0000","last_peered":"2024-09-21T18:54:49.459084+0000","last_clean":"2024-09-21T18:54:49.459084+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:54:49.459084+0000","last_undegraded":"2024-09-21T18:54:49.459084+0000","last_fullsized":"2024-09-21T18:54:49.459084+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457221+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:54:49.457221+0000","last_peered":"2024-09-21T18:54:49.457221+0000","last_clean":"2024-09-21T18:54:49.457221+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:54:49.457221+0000","last_undegraded":"2024-09-21T18:54:49.457221+0000","last_fullsized":"2024-09-21T18:54:49.457221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"96'4010","reported_seq":4055,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.400729+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:54:51.400729+0000","last_peered":"2024-09-21T18:54:51.400729+0000","last_clean":"2024-09-21T18:54:51.400729+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:54:51.400729+0000","last_undegraded":"2024-09-21T18:54:51.400729+0000","last_fullsized":"2024-09-21T18:54:51.400729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.053908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262799360,"num_objects":4010,"num_object_clones":0,"num_object_copies":12030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4010,"num_write_kb":256640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":112,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459244+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:54:49.459244+0000","last_peered":"2024-09-21T18:54:49.459244+0000","last_clean":"2024-09-21T18:54:49.459244+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:54:49.459244+0000","last_undegraded":"2024-09-21T18:54:49.459244+0000","last_fullsized":"2024-09-21T18:54:49.459244+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.360653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":52,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452093+0000","last_change":"2024-09-21T18:54:17.734277+0000","last_active":"2024-09-21T18:54:49.452093+0000","last_peered":"2024-09-21T18:54:49.452093+0000","last_clean":"2024-09-21T18:54:49.452093+0000","last_became_active":"2024-09-21T18:54:17.721086+0000","last_became_peered":"2024-09-21T18:54:17.721086+0000","last_unstale":"2024-09-21T18:54:49.452093+0000","last_undegraded":"2024-09-21T18:54:49.452093+0000","last_fullsized":"2024-09-21T18:54:49.452093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.369107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":109,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.451903+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:54:49.451903+0000","last_peered":"2024-09-21T18:54:49.451903+0000","last_clean":"2024-09-21T18:54:49.451903+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:54:49.451903+0000","last_undegraded":"2024-09-21T18:54:49.451903+0000","last_fullsized":"2024-09-21T18:54:49.451903+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452919+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:54:49.452919+0000","last_peered":"2024-09-21T18:54:49.452919+0000","last_clean":"2024-09-21T18:54:49.452919+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:54:49.452919+0000","last_undegraded":"2024-09-21T18:54:49.452919+0000","last_fullsized":"2024-09-21T18:54:49.452919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"96'3878","reported_seq":3923,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.365308+0000","last_change":"2024-09-21T18:54:14.452896+0000","last_active":"2024-09-21T18:54:51.365308+0000","last_peered":"2024-09-21T18:54:51.365308+0000","last_clean":"2024-09-21T18:54:51.365308+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:54:51.365308+0000","last_undegraded":"2024-09-21T18:54:51.365308+0000","last_fullsized":"2024-09-21T18:54:51.365308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.308130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254148608,"num_objects":3878,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3878,"num_write_kb":248192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":98,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454550+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:54:49.454550+0000","last_peered":"2024-09-21T18:54:49.454550+0000","last_clean":"2024-09-21T18:54:49.454550+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:54:49.454550+0000","last_undegraded":"2024-09-21T18:54:49.454550+0000","last_fullsized":"2024-09-21T18:54:49.454550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.356779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":95,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456774+0000","last_change":"2024-09-21T18:54:15.230912+0000","last_active":"2024-09-21T18:54:49.456774+0000","last_peered":"2024-09-21T18:54:49.456774+0000","last_clean":"2024-09-21T18:54:49.456774+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:54:49.456774+0000","last_undegraded":"2024-09-21T18:54:49.456774+0000","last_fullsized":"2024-09-21T18:54:49.456774+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.230861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.230861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.698873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"83'9","reported_seq":84,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458808+0000","last_change":"2024-09-21T18:54:41.272637+0000","last_active":"2024-09-21T18:54:49.458808+0000","last_peered":"2024-09-21T18:54:49.458808+0000","last_clean":"2024-09-21T18:54:49.458808+0000","last_became_active":"2024-09-21T18:54:17.713489+0000","last_became_peered":"2024-09-21T18:54:17.713489+0000","last_unstale":"2024-09-21T18:54:49.458808+0000","last_undegraded":"2024-09-21T18:54:49.458808+0000","last_fullsized":"2024-09-21T18:54:49.458808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:47.058473+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.451803+0000","last_change":"2024-09-21T18:54:17.714235+0000","last_active":"2024-09-21T18:54:49.451803+0000","last_peered":"2024-09-21T18:54:49.451803+0000","last_clean":"2024-09-21T18:54:49.451803+0000","last_became_active":"2024-09-21T18:54:17.713933+0000","last_became_peered":"2024-09-21T18:54:17.713933+0000","last_unstale":"2024-09-21T18:54:49.451803+0000","last_undegraded":"2024-09-21T18:54:49.451803+0000","last_fullsized":"2024-09-21T18:54:49.451803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.977688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"96'4030","reported_seq":4075,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.348777+0000","last_change":"2024-09-21T18:54:14.451912+0000","last_active":"2024-09-21T18:54:51.348777+0000","last_peered":"2024-09-21T18:54:51.348777+0000","last_clean":"2024-09-21T18:54:51.348777+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:54:51.348777+0000","last_undegraded":"2024-09-21T18:54:51.348777+0000","last_fullsized":"2024-09-21T18:54:51.348777+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:29.638453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264110080,"num_objects":4030,"num_object_clones":0,"num_object_copies":12090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4030,"num_write_kb":257920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":113,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455946+0000","last_change":"2024-09-21T18:54:17.730752+0000","last_active":"2024-09-21T18:54:49.455946+0000","last_peered":"2024-09-21T18:54:49.455946+0000","last_clean":"2024-09-21T18:54:49.455946+0000","last_became_active":"2024-09-21T18:54:17.711274+0000","last_became_peered":"2024-09-21T18:54:17.711274+0000","last_unstale":"2024-09-21T18:54:49.455946+0000","last_undegraded":"2024-09-21T18:54:49.455946+0000","last_fullsized":"2024-09-21T18:54:49.455946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.213832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454478+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:54:49.454478+0000","last_peered":"2024-09-21T18:54:49.454478+0000","last_clean":"2024-09-21T18:54:49.454478+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:54:49.454478+0000","last_undegraded":"2024-09-21T18:54:49.454478+0000","last_fullsized":"2024-09-21T18:54:49.454478+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.637938+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":85,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459070+0000","last_change":"2024-09-21T18:54:17.710974+0000","last_active":"2024-09-21T18:54:49.459070+0000","last_peered":"2024-09-21T18:54:49.459070+0000","last_clean":"2024-09-21T18:54:49.459070+0000","last_became_active":"2024-09-21T18:54:17.710662+0000","last_became_peered":"2024-09-21T18:54:17.710662+0000","last_unstale":"2024-09-21T18:54:49.459070+0000","last_undegraded":"2024-09-21T18:54:49.459070+0000","last_fullsized":"2024-09-21T18:54:49.459070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.222043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455140+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:54:49.455140+0000","last_peered":"2024-09-21T18:54:49.455140+0000","last_clean":"2024-09-21T18:54:49.455140+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:54:49.455140+0000","last_undegraded":"2024-09-21T18:54:49.455140+0000","last_fullsized":"2024-09-21T18:54:49.455140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.362211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"96'3934","reported_seq":3986,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.365480+0000","last_change":"2024-09-21T18:54:17.707215+0000","last_active":"2024-09-21T18:54:51.365480+0000","last_peered":"2024-09-21T18:54:51.365480+0000","last_clean":"2024-09-21T18:54:51.365480+0000","last_became_active":"2024-09-21T18:54:17.706866+0000","last_became_peered":"2024-09-21T18:54:17.706866+0000","last_unstale":"2024-09-21T18:54:51.365480+0000","last_undegraded":"2024-09-21T18:54:51.365480+0000","last_fullsized":"2024-09-21T18:54:51.365480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.130355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257818624,"num_objects":3934,"num_object_clones":0,"num_object_copies":11802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":251776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":160,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454336+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:54:49.454336+0000","last_peered":"2024-09-21T18:54:49.454336+0000","last_clean":"2024-09-21T18:54:49.454336+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:54:49.454336+0000","last_undegraded":"2024-09-21T18:54:49.454336+0000","last_fullsized":"2024-09-21T18:54:49.454336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:53.227258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":102,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458228+0000","last_change":"2024-09-21T18:54:17.706922+0000","last_active":"2024-09-21T18:54:49.458228+0000","last_peered":"2024-09-21T18:54:49.458228+0000","last_clean":"2024-09-21T18:54:49.458228+0000","last_became_active":"2024-09-21T18:54:17.706635+0000","last_became_peered":"2024-09-21T18:54:17.706635+0000","last_unstale":"2024-09-21T18:54:49.458228+0000","last_undegraded":"2024-09-21T18:54:49.458228+0000","last_fullsized":"2024-09-21T18:54:49.458228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:09.963263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:09.963263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.094847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":105,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456039+0000","last_change":"2024-09-21T18:54:47.179942+0000","last_active":"2024-09-21T18:54:49.456039+0000","last_peered":"2024-09-21T18:54:49.456039+0000","last_clean":"2024-09-21T18:54:49.456039+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:54:49.456039+0000","last_undegraded":"2024-09-21T18:54:49.456039+0000","last_fullsized":"2024-09-21T18:54:49.456039+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:54:47.179869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.179869+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:50.966329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.461115+0000","last_change":"2024-09-21T18:54:41.369741+0000","last_active":"2024-09-21T18:54:49.461115+0000","last_peered":"2024-09-21T18:54:49.461115+0000","last_clean":"2024-09-21T18:54:49.461115+0000","last_became_active":"2024-09-21T18:54:17.712729+0000","last_became_peered":"2024-09-21T18:54:17.712729+0000","last_unstale":"2024-09-21T18:54:49.461115+0000","last_undegraded":"2024-09-21T18:54:49.461115+0000","last_fullsized":"2024-09-21T18:54:49.461115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:55.719422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":76,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452867+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:54:49.452867+0000","last_peered":"2024-09-21T18:54:49.452867+0000","last_clean":"2024-09-21T18:54:49.452867+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:54:49.452867+0000","last_undegraded":"2024-09-21T18:54:49.452867+0000","last_fullsized":"2024-09-21T18:54:49.452867+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"96'4021","reported_seq":4061,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.341601+0000","last_change":"2024-09-21T18:54:17.706551+0000","last_active":"2024-09-21T18:54:51.341601+0000","last_peered":"2024-09-21T18:54:51.341601+0000","last_clean":"2024-09-21T18:54:51.341601+0000","last_became_active":"2024-09-21T18:54:17.706169+0000","last_became_peered":"2024-09-21T18:54:17.706169+0000","last_unstale":"2024-09-21T18:54:51.341601+0000","last_undegraded":"2024-09-21T18:54:51.341601+0000","last_fullsized":"2024-09-21T18:54:51.341601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.094421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":257344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458992+0000","last_change":"2024-09-21T18:54:40.059601+0000","last_active":"2024-09-21T18:54:49.458992+0000","last_peered":"2024-09-21T18:54:49.458992+0000","last_clean":"2024-09-21T18:54:49.458992+0000","last_became_active":"2024-09-21T18:54:36.751672+0000","last_became_peered":"2024-09-21T18:54:36.751672+0000","last_unstale":"2024-09-21T18:54:49.458992+0000","last_undegraded":"2024-09-21T18:54:49.458992+0000","last_fullsized":"2024-09-21T18:54:49.458992+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.628526+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":93,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.451632+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:54:49.451632+0000","last_peered":"2024-09-21T18:54:49.451632+0000","last_clean":"2024-09-21T18:54:49.451632+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:54:49.451632+0000","last_undegraded":"2024-09-21T18:54:49.451632+0000","last_fullsized":"2024-09-21T18:54:49.451632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":107,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458208+0000","last_change":"2024-09-21T18:54:17.718825+0000","last_active":"2024-09-21T18:54:49.458208+0000","last_peered":"2024-09-21T18:54:49.458208+0000","last_clean":"2024-09-21T18:54:49.458208+0000","last_became_active":"2024-09-21T18:54:17.710632+0000","last_became_peered":"2024-09-21T18:54:17.710632+0000","last_unstale":"2024-09-21T18:54:49.458208+0000","last_undegraded":"2024-09-21T18:54:49.458208+0000","last_fullsized":"2024-09-21T18:54:49.458208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.946937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459580+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:54:49.459580+0000","last_peered":"2024-09-21T18:54:49.459580+0000","last_clean":"2024-09-21T18:54:49.459580+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:54:49.459580+0000","last_undegraded":"2024-09-21T18:54:49.459580+0000","last_fullsized":"2024-09-21T18:54:49.459580+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.969938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453240+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:54:49.453240+0000","last_peered":"2024-09-21T18:54:49.453240+0000","last_clean":"2024-09-21T18:54:49.453240+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:54:49.453240+0000","last_undegraded":"2024-09-21T18:54:49.453240+0000","last_fullsized":"2024-09-21T18:54:49.453240+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:29.965290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"96'3998","reported_seq":4050,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.357944+0000","last_change":"2024-09-21T18:54:17.717837+0000","last_active":"2024-09-21T18:54:51.357944+0000","last_peered":"2024-09-21T18:54:51.357944+0000","last_clean":"2024-09-21T18:54:51.357944+0000","last_became_active":"2024-09-21T18:54:17.717525+0000","last_became_peered":"2024-09-21T18:54:17.717525+0000","last_unstale":"2024-09-21T18:54:51.357944+0000","last_undegraded":"2024-09-21T18:54:51.357944+0000","last_fullsized":"2024-09-21T18:54:51.357944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.870954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262012928,"num_objects":3998,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":255872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":91,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.451926+0000","last_change":"2024-09-21T18:54:17.731920+0000","last_active":"2024-09-21T18:54:49.451926+0000","last_peered":"2024-09-21T18:54:49.451926+0000","last_clean":"2024-09-21T18:54:49.451926+0000","last_became_active":"2024-09-21T18:54:17.720574+0000","last_became_peered":"2024-09-21T18:54:17.720574+0000","last_unstale":"2024-09-21T18:54:49.451926+0000","last_undegraded":"2024-09-21T18:54:49.451926+0000","last_fullsized":"2024-09-21T18:54:49.451926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.300639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459165+0000","last_change":"2024-09-21T18:54:17.704631+0000","last_active":"2024-09-21T18:54:49.459165+0000","last_peered":"2024-09-21T18:54:49.459165+0000","last_clean":"2024-09-21T18:54:49.459165+0000","last_became_active":"2024-09-21T18:54:17.704296+0000","last_became_peered":"2024-09-21T18:54:17.704296+0000","last_unstale":"2024-09-21T18:54:49.459165+0000","last_undegraded":"2024-09-21T18:54:49.459165+0000","last_fullsized":"2024-09-21T18:54:49.459165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:20.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":215,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.774034+0000","last_change":"2024-09-21T18:54:41.110834+0000","last_active":"2024-09-21T18:54:49.774034+0000","last_peered":"2024-09-21T18:54:49.774034+0000","last_clean":"2024-09-21T18:54:49.774034+0000","last_became_active":"2024-09-21T18:54:17.713703+0000","last_became_peered":"2024-09-21T18:54:17.713703+0000","last_unstale":"2024-09-21T18:54:49.774034+0000","last_undegraded":"2024-09-21T18:54:49.774034+0000","last_fullsized":"2024-09-21T18:54:49.774034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:41.110776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.110776+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.027705+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":97,"num_read_kb":92,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":94,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456552+0000","last_change":"2024-09-21T18:54:40.112915+0000","last_active":"2024-09-21T18:54:49.456552+0000","last_peered":"2024-09-21T18:54:49.456552+0000","last_clean":"2024-09-21T18:54:49.456552+0000","last_became_active":"2024-09-21T18:54:17.707817+0000","last_became_peered":"2024-09-21T18:54:17.707817+0000","last_unstale":"2024-09-21T18:54:49.456552+0000","last_undegraded":"2024-09-21T18:54:49.456552+0000","last_fullsized":"2024-09-21T18:54:49.456552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.112826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.112826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:08.711823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453172+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:54:49.453172+0000","last_peered":"2024-09-21T18:54:49.453172+0000","last_clean":"2024-09-21T18:54:49.453172+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:54:49.453172+0000","last_undegraded":"2024-09-21T18:54:49.453172+0000","last_fullsized":"2024-09-21T18:54:49.453172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"96'3932","reported_seq":3972,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.365711+0000","last_change":"2024-09-21T18:54:17.719358+0000","last_active":"2024-09-21T18:54:51.365711+0000","last_peered":"2024-09-21T18:54:51.365711+0000","last_clean":"2024-09-21T18:54:51.365711+0000","last_became_active":"2024-09-21T18:54:17.718730+0000","last_became_peered":"2024-09-21T18:54:17.718730+0000","last_unstale":"2024-09-21T18:54:51.365711+0000","last_undegraded":"2024-09-21T18:54:51.365711+0000","last_fullsized":"2024-09-21T18:54:51.365711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.846251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257687552,"num_objects":3932,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":251648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":125,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459902+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:54:49.459902+0000","last_peered":"2024-09-21T18:54:49.459902+0000","last_clean":"2024-09-21T18:54:49.459902+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:54:49.459902+0000","last_undegraded":"2024-09-21T18:54:49.459902+0000","last_fullsized":"2024-09-21T18:54:49.459902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.249764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455421+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:54:49.455421+0000","last_peered":"2024-09-21T18:54:49.455421+0000","last_clean":"2024-09-21T18:54:49.455421+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:54:49.455421+0000","last_undegraded":"2024-09-21T18:54:49.455421+0000","last_fullsized":"2024-09-21T18:54:49.455421+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.929847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":144,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453099+0000","last_change":"2024-09-21T18:54:17.736002+0000","last_active":"2024-09-21T18:54:49.453099+0000","last_peered":"2024-09-21T18:54:49.453099+0000","last_clean":"2024-09-21T18:54:49.453099+0000","last_became_active":"2024-09-21T18:54:17.716544+0000","last_became_peered":"2024-09-21T18:54:17.716544+0000","last_unstale":"2024-09-21T18:54:49.453099+0000","last_undegraded":"2024-09-21T18:54:49.453099+0000","last_fullsized":"2024-09-21T18:54:49.453099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":78,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453914+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:54:49.453914+0000","last_peered":"2024-09-21T18:54:49.453914+0000","last_clean":"2024-09-21T18:54:49.453914+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:54:49.453914+0000","last_undegraded":"2024-09-21T18:54:49.453914+0000","last_fullsized":"2024-09-21T18:54:49.453914+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456458+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:54:49.456458+0000","last_peered":"2024-09-21T18:54:49.456458+0000","last_clean":"2024-09-21T18:54:49.456458+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:54:49.456458+0000","last_undegraded":"2024-09-21T18:54:49.456458+0000","last_fullsized":"2024-09-21T18:54:49.456458+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"96'3963","reported_seq":4015,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.396447+0000","last_change":"2024-09-21T18:54:17.710054+0000","last_active":"2024-09-21T18:54:51.396447+0000","last_peered":"2024-09-21T18:54:51.396447+0000","last_clean":"2024-09-21T18:54:51.396447+0000","last_became_active":"2024-09-21T18:54:17.709698+0000","last_became_peered":"2024-09-21T18:54:17.709698+0000","last_unstale":"2024-09-21T18:54:51.396447+0000","last_undegraded":"2024-09-21T18:54:51.396447+0000","last_fullsized":"2024-09-21T18:54:51.396447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.567144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259719168,"num_objects":3963,"num_object_clones":0,"num_object_copies":11889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3963,"num_write_kb":253632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":102,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458614+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:54:49.458614+0000","last_peered":"2024-09-21T18:54:49.458614+0000","last_clean":"2024-09-21T18:54:49.458614+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:54:49.458614+0000","last_undegraded":"2024-09-21T18:54:49.458614+0000","last_fullsized":"2024-09-21T18:54:49.458614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.929416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459569+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:54:49.459569+0000","last_peered":"2024-09-21T18:54:49.459569+0000","last_clean":"2024-09-21T18:54:49.459569+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:54:49.459569+0000","last_undegraded":"2024-09-21T18:54:49.459569+0000","last_fullsized":"2024-09-21T18:54:49.459569+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.012525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:50.139865+0000","last_change":"2024-09-21T18:54:50.139865+0000","last_active":"2024-09-21T18:54:50.139865+0000","last_peered":"2024-09-21T18:54:50.139865+0000","last_clean":"2024-09-21T18:54:50.139865+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:54:50.139865+0000","last_undegraded":"2024-09-21T18:54:50.139865+0000","last_fullsized":"2024-09-21T18:54:50.139865+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.139807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.139807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.360746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":76,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455406+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:54:49.455406+0000","last_peered":"2024-09-21T18:54:49.455406+0000","last_clean":"2024-09-21T18:54:49.455406+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:54:49.455406+0000","last_undegraded":"2024-09-21T18:54:49.455406+0000","last_fullsized":"2024-09-21T18:54:49.455406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"96'3925","reported_seq":3977,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.382717+0000","last_change":"2024-09-21T18:54:17.708465+0000","last_active":"2024-09-21T18:54:51.382717+0000","last_peered":"2024-09-21T18:54:51.382717+0000","last_clean":"2024-09-21T18:54:51.382717+0000","last_became_active":"2024-09-21T18:54:17.707839+0000","last_became_peered":"2024-09-21T18:54:17.707839+0000","last_unstale":"2024-09-21T18:54:51.382717+0000","last_undegraded":"2024-09-21T18:54:51.382717+0000","last_fullsized":"2024-09-21T18:54:51.382717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.899478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257228800,"num_objects":3925,"num_object_clones":0,"num_object_copies":11775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":251200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":111,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458113+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:54:49.458113+0000","last_peered":"2024-09-21T18:54:49.458113+0000","last_clean":"2024-09-21T18:54:49.458113+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:54:49.458113+0000","last_undegraded":"2024-09-21T18:54:49.458113+0000","last_fullsized":"2024-09-21T18:54:49.458113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459369+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:54:49.459369+0000","last_peered":"2024-09-21T18:54:49.459369+0000","last_clean":"2024-09-21T18:54:49.459369+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:54:49.459369+0000","last_undegraded":"2024-09-21T18:54:49.459369+0000","last_fullsized":"2024-09-21T18:54:49.459369+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.289015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453260+0000","last_change":"2024-09-21T18:54:17.714450+0000","last_active":"2024-09-21T18:54:49.453260+0000","last_peered":"2024-09-21T18:54:49.453260+0000","last_clean":"2024-09-21T18:54:49.453260+0000","last_became_active":"2024-09-21T18:54:17.714167+0000","last_became_peered":"2024-09-21T18:54:17.714167+0000","last_unstale":"2024-09-21T18:54:49.453260+0000","last_undegraded":"2024-09-21T18:54:49.453260+0000","last_fullsized":"2024-09-21T18:54:49.453260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459118+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:54:49.459118+0000","last_peered":"2024-09-21T18:54:49.459118+0000","last_clean":"2024-09-21T18:54:49.459118+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:54:49.459118+0000","last_undegraded":"2024-09-21T18:54:49.459118+0000","last_fullsized":"2024-09-21T18:54:49.459118+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:50.896923+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"96'3940","reported_seq":3979,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.398239+0000","last_change":"2024-09-21T18:54:17.718909+0000","last_active":"2024-09-21T18:54:51.398239+0000","last_peered":"2024-09-21T18:54:51.398239+0000","last_clean":"2024-09-21T18:54:51.398239+0000","last_became_active":"2024-09-21T18:54:17.718631+0000","last_became_peered":"2024-09-21T18:54:17.718631+0000","last_unstale":"2024-09-21T18:54:51.398239+0000","last_undegraded":"2024-09-21T18:54:51.398239+0000","last_fullsized":"2024-09-21T18:54:51.398239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3940,"log_dups_size":0,"ondisk_log_size":3940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:32.749545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258211840,"num_objects":3940,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":252160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":98,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457834+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:54:49.457834+0000","last_peered":"2024-09-21T18:54:49.457834+0000","last_clean":"2024-09-21T18:54:49.457834+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:54:49.457834+0000","last_undegraded":"2024-09-21T18:54:49.457834+0000","last_fullsized":"2024-09-21T18:54:49.457834+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":113,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453608+0000","last_change":"2024-09-21T18:54:17.724937+0000","last_active":"2024-09-21T18:54:49.453608+0000","last_peered":"2024-09-21T18:54:49.453608+0000","last_clean":"2024-09-21T18:54:49.453608+0000","last_became_active":"2024-09-21T18:54:17.710657+0000","last_became_peered":"2024-09-21T18:54:17.710657+0000","last_unstale":"2024-09-21T18:54:49.453608+0000","last_undegraded":"2024-09-21T18:54:49.453608+0000","last_fullsized":"2024-09-21T18:54:49.453608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.355187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":108,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459042+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:54:49.459042+0000","last_peered":"2024-09-21T18:54:49.459042+0000","last_clean":"2024-09-21T18:54:49.459042+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:54:49.459042+0000","last_undegraded":"2024-09-21T18:54:49.459042+0000","last_fullsized":"2024-09-21T18:54:49.459042+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.995859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458353+0000","last_change":"2024-09-21T18:53:13.428208+0000","last_active":"2024-09-21T18:54:49.458353+0000","last_peered":"2024-09-21T18:54:49.458353+0000","last_clean":"2024-09-21T18:54:49.458353+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:54:49.458353+0000","last_undegraded":"2024-09-21T18:54:49.458353+0000","last_fullsized":"2024-09-21T18:54:49.458353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"96'3930","reported_seq":3975,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.353400+0000","last_change":"2024-09-21T18:54:14.454300+0000","last_active":"2024-09-21T18:54:51.353400+0000","last_peered":"2024-09-21T18:54:51.353400+0000","last_clean":"2024-09-21T18:54:51.353400+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:54:51.353400+0000","last_undegraded":"2024-09-21T18:54:51.353400+0000","last_fullsized":"2024-09-21T18:54:51.353400+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.683811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":11790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":251520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":85,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454596+0000","last_change":"2024-09-21T18:54:17.729395+0000","last_active":"2024-09-21T18:54:49.454596+0000","last_peered":"2024-09-21T18:54:49.454596+0000","last_clean":"2024-09-21T18:54:49.454596+0000","last_became_active":"2024-09-21T18:54:17.714624+0000","last_became_peered":"2024-09-21T18:54:17.714624+0000","last_unstale":"2024-09-21T18:54:49.454596+0000","last_undegraded":"2024-09-21T18:54:49.454596+0000","last_fullsized":"2024-09-21T18:54:49.454596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":95,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456241+0000","last_change":"2024-09-21T18:54:13.280500+0000","last_active":"2024-09-21T18:54:49.456241+0000","last_peered":"2024-09-21T18:54:49.456241+0000","last_clean":"2024-09-21T18:54:49.456241+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:54:49.456241+0000","last_undegraded":"2024-09-21T18:54:49.456241+0000","last_fullsized":"2024-09-21T18:54:49.456241+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.280431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.280431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.151455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459659+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:54:49.459659+0000","last_peered":"2024-09-21T18:54:49.459659+0000","last_clean":"2024-09-21T18:54:49.459659+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:54:49.459659+0000","last_undegraded":"2024-09-21T18:54:49.459659+0000","last_fullsized":"2024-09-21T18:54:49.459659+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:49.052101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453667+0000","last_change":"2024-09-21T18:54:17.717827+0000","last_active":"2024-09-21T18:54:49.453667+0000","last_peered":"2024-09-21T18:54:49.453667+0000","last_clean":"2024-09-21T18:54:49.453667+0000","last_became_active":"2024-09-21T18:54:17.717507+0000","last_became_peered":"2024-09-21T18:54:17.717507+0000","last_unstale":"2024-09-21T18:54:49.453667+0000","last_undegraded":"2024-09-21T18:54:49.453667+0000","last_fullsized":"2024-09-21T18:54:49.453667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.193259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"96'4021","reported_seq":4061,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.371388+0000","last_change":"2024-09-21T18:54:17.712034+0000","last_active":"2024-09-21T18:54:51.371388+0000","last_peered":"2024-09-21T18:54:51.371388+0000","last_clean":"2024-09-21T18:54:51.371388+0000","last_became_active":"2024-09-21T18:54:17.711698+0000","last_became_peered":"2024-09-21T18:54:17.711698+0000","last_unstale":"2024-09-21T18:54:51.371388+0000","last_undegraded":"2024-09-21T18:54:51.371388+0000","last_fullsized":"2024-09-21T18:54:51.371388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.183316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":257344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":113,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459172+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:54:49.459172+0000","last_peered":"2024-09-21T18:54:49.459172+0000","last_clean":"2024-09-21T18:54:49.459172+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:54:49.459172+0000","last_undegraded":"2024-09-21T18:54:49.459172+0000","last_fullsized":"2024-09-21T18:54:49.459172+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.210774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453431+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:54:49.453431+0000","last_peered":"2024-09-21T18:54:49.453431+0000","last_clean":"2024-09-21T18:54:49.453431+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:54:49.453431+0000","last_undegraded":"2024-09-21T18:54:49.453431+0000","last_fullsized":"2024-09-21T18:54:49.453431+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.128759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458455+0000","last_change":"2024-09-21T18:54:17.717687+0000","last_active":"2024-09-21T18:54:49.458455+0000","last_peered":"2024-09-21T18:54:49.458455+0000","last_clean":"2024-09-21T18:54:49.458455+0000","last_became_active":"2024-09-21T18:54:17.717380+0000","last_became_peered":"2024-09-21T18:54:17.717380+0000","last_unstale":"2024-09-21T18:54:49.458455+0000","last_undegraded":"2024-09-21T18:54:49.458455+0000","last_fullsized":"2024-09-21T18:54:49.458455+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.963253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453049+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:54:49.453049+0000","last_peered":"2024-09-21T18:54:49.453049+0000","last_clean":"2024-09-21T18:54:49.453049+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:54:49.453049+0000","last_undegraded":"2024-09-21T18:54:49.453049+0000","last_fullsized":"2024-09-21T18:54:49.453049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:18.116478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"96'3883","reported_seq":3928,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.389550+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:54:51.389550+0000","last_peered":"2024-09-21T18:54:51.389550+0000","last_clean":"2024-09-21T18:54:51.389550+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:54:51.389550+0000","last_undegraded":"2024-09-21T18:54:51.389550+0000","last_fullsized":"2024-09-21T18:54:51.389550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.423918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254476288,"num_objects":3883,"num_object_clones":0,"num_object_copies":11649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3883,"num_write_kb":248512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":133,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459393+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:54:49.459393+0000","last_peered":"2024-09-21T18:54:49.459393+0000","last_clean":"2024-09-21T18:54:49.459393+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:54:49.459393+0000","last_undegraded":"2024-09-21T18:54:49.459393+0000","last_fullsized":"2024-09-21T18:54:49.459393+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:51.259913+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":94,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454282+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:54:49.454282+0000","last_peered":"2024-09-21T18:54:49.454282+0000","last_clean":"2024-09-21T18:54:49.454282+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:54:49.454282+0000","last_undegraded":"2024-09-21T18:54:49.454282+0000","last_fullsized":"2024-09-21T18:54:49.454282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:13.545717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":94,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458174+0000","last_change":"2024-09-21T18:54:17.708159+0000","last_active":"2024-09-21T18:54:49.458174+0000","last_peered":"2024-09-21T18:54:49.458174+0000","last_clean":"2024-09-21T18:54:49.458174+0000","last_became_active":"2024-09-21T18:54:17.707830+0000","last_became_peered":"2024-09-21T18:54:17.707830+0000","last_unstale":"2024-09-21T18:54:49.458174+0000","last_undegraded":"2024-09-21T18:54:49.458174+0000","last_fullsized":"2024-09-21T18:54:49.458174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:11.994472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.994472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.343603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":76,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452781+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:54:49.452781+0000","last_peered":"2024-09-21T18:54:49.452781+0000","last_clean":"2024-09-21T18:54:49.452781+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:54:49.452781+0000","last_undegraded":"2024-09-21T18:54:49.452781+0000","last_fullsized":"2024-09-21T18:54:49.452781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"96'3853","reported_seq":3898,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.372257+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:54:51.372257+0000","last_peered":"2024-09-21T18:54:51.372257+0000","last_clean":"2024-09-21T18:54:51.372257+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:54:51.372257+0000","last_undegraded":"2024-09-21T18:54:51.372257+0000","last_fullsized":"2024-09-21T18:54:51.372257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.519258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252510208,"num_objects":3853,"num_object_clones":0,"num_object_copies":11559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":246592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":116,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455614+0000","last_change":"2024-09-21T18:54:44.092986+0000","last_active":"2024-09-21T18:54:49.455614+0000","last_peered":"2024-09-21T18:54:49.455614+0000","last_clean":"2024-09-21T18:54:49.455614+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:54:49.455614+0000","last_undegraded":"2024-09-21T18:54:49.455614+0000","last_fullsized":"2024-09-21T18:54:49.455614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:44.092903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:44.092903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.777104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":111,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453900+0000","last_change":"2024-09-21T18:54:17.732357+0000","last_active":"2024-09-21T18:54:49.453900+0000","last_peered":"2024-09-21T18:54:49.453900+0000","last_clean":"2024-09-21T18:54:49.453900+0000","last_became_active":"2024-09-21T18:54:17.715268+0000","last_became_peered":"2024-09-21T18:54:17.715268+0000","last_unstale":"2024-09-21T18:54:49.453900+0000","last_undegraded":"2024-09-21T18:54:49.453900+0000","last_fullsized":"2024-09-21T18:54:49.453900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.049285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":110,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453904+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:54:49.453904+0000","last_peered":"2024-09-21T18:54:49.453904+0000","last_clean":"2024-09-21T18:54:49.453904+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:54:49.453904+0000","last_undegraded":"2024-09-21T18:54:49.453904+0000","last_fullsized":"2024-09-21T18:54:49.453904+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.946870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":82,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455627+0000","last_change":"2024-09-21T18:54:17.714328+0000","last_active":"2024-09-21T18:54:49.455627+0000","last_peered":"2024-09-21T18:54:49.455627+0000","last_clean":"2024-09-21T18:54:49.455627+0000","last_became_active":"2024-09-21T18:54:17.713977+0000","last_became_peered":"2024-09-21T18:54:17.713977+0000","last_unstale":"2024-09-21T18:54:49.455627+0000","last_undegraded":"2024-09-21T18:54:49.455627+0000","last_fullsized":"2024-09-21T18:54:49.455627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":93,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457406+0000","last_change":"2024-09-21T18:54:17.713398+0000","last_active":"2024-09-21T18:54:49.457406+0000","last_peered":"2024-09-21T18:54:49.457406+0000","last_clean":"2024-09-21T18:54:49.457406+0000","last_became_active":"2024-09-21T18:54:17.713066+0000","last_became_peered":"2024-09-21T18:54:17.713066+0000","last_unstale":"2024-09-21T18:54:49.457406+0000","last_undegraded":"2024-09-21T18:54:49.457406+0000","last_fullsized":"2024-09-21T18:54:49.457406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.224570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":93,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454289+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:54:49.454289+0000","last_peered":"2024-09-21T18:54:49.454289+0000","last_clean":"2024-09-21T18:54:49.454289+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:54:49.454289+0000","last_undegraded":"2024-09-21T18:54:49.454289+0000","last_fullsized":"2024-09-21T18:54:49.454289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456350+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:54:49.456350+0000","last_peered":"2024-09-21T18:54:49.456350+0000","last_clean":"2024-09-21T18:54:49.456350+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:54:49.456350+0000","last_undegraded":"2024-09-21T18:54:49.456350+0000","last_fullsized":"2024-09-21T18:54:49.456350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.256441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453205+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:54:49.453205+0000","last_peered":"2024-09-21T18:54:49.453205+0000","last_clean":"2024-09-21T18:54:49.453205+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:54:49.453205+0000","last_undegraded":"2024-09-21T18:54:49.453205+0000","last_fullsized":"2024-09-21T18:54:49.453205+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":93,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.117250+0000","last_change":"2024-09-21T18:54:51.117250+0000","last_active":"2024-09-21T18:54:51.117250+0000","last_peered":"2024-09-21T18:54:51.117250+0000","last_clean":"2024-09-21T18:54:51.117250+0000","last_became_active":"2024-09-21T18:54:17.719720+0000","last_became_peered":"2024-09-21T18:54:17.719720+0000","last_unstale":"2024-09-21T18:54:51.117250+0000","last_undegraded":"2024-09-21T18:54:51.117250+0000","last_fullsized":"2024-09-21T18:54:51.117250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.430477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":94,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453996+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:54:49.453996+0000","last_peered":"2024-09-21T18:54:49.453996+0000","last_clean":"2024-09-21T18:54:49.453996+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:54:49.453996+0000","last_undegraded":"2024-09-21T18:54:49.453996+0000","last_fullsized":"2024-09-21T18:54:49.453996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.194383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":80,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454154+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:54:49.454154+0000","last_peered":"2024-09-21T18:54:49.454154+0000","last_clean":"2024-09-21T18:54:49.454154+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:54:49.454154+0000","last_undegraded":"2024-09-21T18:54:49.454154+0000","last_fullsized":"2024-09-21T18:54:49.454154+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.003241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458517+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:54:49.458517+0000","last_peered":"2024-09-21T18:54:49.458517+0000","last_clean":"2024-09-21T18:54:49.458517+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:54:49.458517+0000","last_undegraded":"2024-09-21T18:54:49.458517+0000","last_fullsized":"2024-09-21T18:54:49.458517+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:53.334288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":99,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456527+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:54:49.456527+0000","last_peered":"2024-09-21T18:54:49.456527+0000","last_clean":"2024-09-21T18:54:49.456527+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:54:49.456527+0000","last_undegraded":"2024-09-21T18:54:49.456527+0000","last_fullsized":"2024-09-21T18:54:49.456527+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.275983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453396+0000","last_change":"2024-09-21T18:54:17.717912+0000","last_active":"2024-09-21T18:54:49.453396+0000","last_peered":"2024-09-21T18:54:49.453396+0000","last_clean":"2024-09-21T18:54:49.453396+0000","last_became_active":"2024-09-21T18:54:17.717617+0000","last_became_peered":"2024-09-21T18:54:17.717617+0000","last_unstale":"2024-09-21T18:54:49.453396+0000","last_undegraded":"2024-09-21T18:54:49.453396+0000","last_fullsized":"2024-09-21T18:54:49.453396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":109,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455507+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:54:49.455507+0000","last_peered":"2024-09-21T18:54:49.455507+0000","last_clean":"2024-09-21T18:54:49.455507+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:54:49.455507+0000","last_undegraded":"2024-09-21T18:54:49.455507+0000","last_fullsized":"2024-09-21T18:54:49.455507+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:51.062813+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:54:51.062813+0000","last_peered":"2024-09-21T18:54:51.062813+0000","last_clean":"2024-09-21T18:54:51.062813+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:54:51.062813+0000","last_undegraded":"2024-09-21T18:54:51.062813+0000","last_fullsized":"2024-09-21T18:54:51.062813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:14.128893+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":99,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455959+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:54:49.455959+0000","last_peered":"2024-09-21T18:54:49.455959+0000","last_clean":"2024-09-21T18:54:49.455959+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:54:49.455959+0000","last_undegraded":"2024-09-21T18:54:49.455959+0000","last_fullsized":"2024-09-21T18:54:49.455959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":94,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453371+0000","last_change":"2024-09-21T18:54:40.089802+0000","last_active":"2024-09-21T18:54:49.453371+0000","last_peered":"2024-09-21T18:54:49.453371+0000","last_clean":"2024-09-21T18:54:49.453371+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:54:49.453371+0000","last_undegraded":"2024-09-21T18:54:49.453371+0000","last_fullsized":"2024-09-21T18:54:49.453371+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.089723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.089723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.272819+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"83'9","reported_seq":119,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456683+0000","last_change":"2024-09-21T18:54:17.726533+0000","last_active":"2024-09-21T18:54:49.456683+0000","last_peered":"2024-09-21T18:54:49.456683+0000","last_clean":"2024-09-21T18:54:49.456683+0000","last_became_active":"2024-09-21T18:54:17.718821+0000","last_became_peered":"2024-09-21T18:54:17.718821+0000","last_unstale":"2024-09-21T18:54:49.456683+0000","last_undegraded":"2024-09-21T18:54:49.456683+0000","last_fullsized":"2024-09-21T18:54:49.456683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453513+0000","last_change":"2024-09-21T18:54:17.719309+0000","last_active":"2024-09-21T18:54:49.453513+0000","last_peered":"2024-09-21T18:54:49.453513+0000","last_clean":"2024-09-21T18:54:49.453513+0000","last_became_active":"2024-09-21T18:54:17.718560+0000","last_became_peered":"2024-09-21T18:54:17.718560+0000","last_unstale":"2024-09-21T18:54:49.453513+0000","last_undegraded":"2024-09-21T18:54:49.453513+0000","last_fullsized":"2024-09-21T18:54:49.453513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:25.988782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":121,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.457975+0000","last_change":"2024-09-21T18:54:36.773903+0000","last_active":"2024-09-21T18:54:49.457975+0000","last_peered":"2024-09-21T18:54:49.457975+0000","last_clean":"2024-09-21T18:54:49.457975+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:54:49.457975+0000","last_undegraded":"2024-09-21T18:54:49.457975+0000","last_fullsized":"2024-09-21T18:54:49.457975+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.458011+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:54:49.458011+0000","last_peered":"2024-09-21T18:54:49.458011+0000","last_clean":"2024-09-21T18:54:49.458011+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:54:49.458011+0000","last_undegraded":"2024-09-21T18:54:49.458011+0000","last_fullsized":"2024-09-21T18:54:49.458011+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.963073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454409+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:54:49.454409+0000","last_peered":"2024-09-21T18:54:49.454409+0000","last_clean":"2024-09-21T18:54:49.454409+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:54:49.454409+0000","last_undegraded":"2024-09-21T18:54:49.454409+0000","last_fullsized":"2024-09-21T18:54:49.454409+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452858+0000","last_change":"2024-09-21T18:54:17.717220+0000","last_active":"2024-09-21T18:54:49.452858+0000","last_peered":"2024-09-21T18:54:49.452858+0000","last_clean":"2024-09-21T18:54:49.452858+0000","last_became_active":"2024-09-21T18:54:17.716004+0000","last_became_peered":"2024-09-21T18:54:17.716004+0000","last_unstale":"2024-09-21T18:54:49.452858+0000","last_undegraded":"2024-09-21T18:54:49.452858+0000","last_fullsized":"2024-09-21T18:54:49.452858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":99,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.454690+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:54:49.454690+0000","last_peered":"2024-09-21T18:54:49.454690+0000","last_clean":"2024-09-21T18:54:49.454690+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:54:49.454690+0000","last_undegraded":"2024-09-21T18:54:49.454690+0000","last_fullsized":"2024-09-21T18:54:49.454690+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.311285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453355+0000","last_change":"2024-09-21T18:53:07.406445+0000","last_active":"2024-09-21T18:54:49.453355+0000","last_peered":"2024-09-21T18:54:49.453355+0000","last_clean":"2024-09-21T18:54:49.453355+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:54:49.453355+0000","last_undegraded":"2024-09-21T18:54:49.453355+0000","last_fullsized":"2024-09-21T18:54:49.453355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452863+0000","last_change":"2024-09-21T18:54:17.715073+0000","last_active":"2024-09-21T18:54:49.452863+0000","last_peered":"2024-09-21T18:54:49.452863+0000","last_clean":"2024-09-21T18:54:49.452863+0000","last_became_active":"2024-09-21T18:54:17.714759+0000","last_became_peered":"2024-09-21T18:54:17.714759+0000","last_unstale":"2024-09-21T18:54:49.452863+0000","last_undegraded":"2024-09-21T18:54:49.452863+0000","last_fullsized":"2024-09-21T18:54:49.452863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.032847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.455504+0000","last_change":"2024-09-21T18:54:17.712675+0000","last_active":"2024-09-21T18:54:49.455504+0000","last_peered":"2024-09-21T18:54:49.455504+0000","last_clean":"2024-09-21T18:54:49.455504+0000","last_became_active":"2024-09-21T18:54:17.712339+0000","last_became_peered":"2024-09-21T18:54:17.712339+0000","last_unstale":"2024-09-21T18:54:49.455504+0000","last_undegraded":"2024-09-21T18:54:49.455504+0000","last_fullsized":"2024-09-21T18:54:49.455504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":103,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453050+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:54:49.453050+0000","last_peered":"2024-09-21T18:54:49.453050+0000","last_clean":"2024-09-21T18:54:49.453050+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:54:49.453050+0000","last_undegraded":"2024-09-21T18:54:49.453050+0000","last_fullsized":"2024-09-21T18:54:49.453050+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":87,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456425+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:54:49.456425+0000","last_peered":"2024-09-21T18:54:49.456425+0000","last_clean":"2024-09-21T18:54:49.456425+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:54:49.456425+0000","last_undegraded":"2024-09-21T18:54:49.456425+0000","last_fullsized":"2024-09-21T18:54:49.456425+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.201819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":78,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452721+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:54:49.452721+0000","last_peered":"2024-09-21T18:54:49.452721+0000","last_clean":"2024-09-21T18:54:49.452721+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:54:49.452721+0000","last_undegraded":"2024-09-21T18:54:49.452721+0000","last_fullsized":"2024-09-21T18:54:49.452721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.082920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453521+0000","last_change":"2024-09-21T18:54:17.709035+0000","last_active":"2024-09-21T18:54:49.453521+0000","last_peered":"2024-09-21T18:54:49.453521+0000","last_clean":"2024-09-21T18:54:49.453521+0000","last_became_active":"2024-09-21T18:54:17.708626+0000","last_became_peered":"2024-09-21T18:54:17.708626+0000","last_unstale":"2024-09-21T18:54:49.453521+0000","last_undegraded":"2024-09-21T18:54:49.453521+0000","last_fullsized":"2024-09-21T18:54:49.453521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":103,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452052+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:54:49.452052+0000","last_peered":"2024-09-21T18:54:49.452052+0000","last_clean":"2024-09-21T18:54:49.452052+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:54:49.452052+0000","last_undegraded":"2024-09-21T18:54:49.452052+0000","last_fullsized":"2024-09-21T18:54:49.452052+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.456629+0000","last_change":"2024-09-21T18:54:17.719435+0000","last_active":"2024-09-21T18:54:49.456629+0000","last_peered":"2024-09-21T18:54:49.456629+0000","last_clean":"2024-09-21T18:54:49.456629+0000","last_became_active":"2024-09-21T18:54:17.718927+0000","last_became_peered":"2024-09-21T18:54:17.718927+0000","last_unstale":"2024-09-21T18:54:49.456629+0000","last_undegraded":"2024-09-21T18:54:49.456629+0000","last_fullsized":"2024-09-21T18:54:49.456629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":78,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459266+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:54:49.459266+0000","last_peered":"2024-09-21T18:54:49.459266+0000","last_clean":"2024-09-21T18:54:49.459266+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:54:49.459266+0000","last_undegraded":"2024-09-21T18:54:49.459266+0000","last_fullsized":"2024-09-21T18:54:49.459266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.250351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":74,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453156+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:54:49.453156+0000","last_peered":"2024-09-21T18:54:49.453156+0000","last_clean":"2024-09-21T18:54:49.453156+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:54:49.453156+0000","last_undegraded":"2024-09-21T18:54:49.453156+0000","last_fullsized":"2024-09-21T18:54:49.453156+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.461013+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:54:49.461013+0000","last_peered":"2024-09-21T18:54:49.461013+0000","last_clean":"2024-09-21T18:54:49.461013+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:54:49.461013+0000","last_undegraded":"2024-09-21T18:54:49.461013+0000","last_fullsized":"2024-09-21T18:54:49.461013+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:25.345750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":103,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.452285+0000","last_change":"2024-09-21T18:54:17.704421+0000","last_active":"2024-09-21T18:54:49.452285+0000","last_peered":"2024-09-21T18:54:49.452285+0000","last_clean":"2024-09-21T18:54:49.452285+0000","last_became_active":"2024-09-21T18:54:17.704166+0000","last_became_peered":"2024-09-21T18:54:17.704166+0000","last_unstale":"2024-09-21T18:54:49.452285+0000","last_undegraded":"2024-09-21T18:54:49.452285+0000","last_fullsized":"2024-09-21T18:54:49.452285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:13.526296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":101,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.459429+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:54:49.459429+0000","last_peered":"2024-09-21T18:54:49.459429+0000","last_clean":"2024-09-21T18:54:49.459429+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:54:49.459429+0000","last_undegraded":"2024-09-21T18:54:49.459429+0000","last_fullsized":"2024-09-21T18:54:49.459429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.235721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T18:54:49.453833+0000","last_change":"2024-09-21T18:54:17.719032+0000","last_active":"2024-09-21T18:54:49.453833+0000","last_peered":"2024-09-21T18:54:49.453833+0000","last_clean":"2024-09-21T18:54:49.453833+0000","last_became_active":"2024-09-21T18:54:17.718245+0000","last_became_peered":"2024-09-21T18:54:17.718245+0000","last_unstale":"2024-09-21T18:54:49.453833+0000","last_undegraded":"2024-09-21T18:54:49.453833+0000","last_fullsized":"2024-09-21T18:54:49.453833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4140236800,"num_objects":63175,"num_object_clones":0,"num_object_copies":189525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63175,"num_write_kb":4043200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12423528448,"data_stored":12423528448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63175,"ondisk_log_size":63175,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":2090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":92,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266713,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1560976,"kb_used_data":1511744,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92184176,"statfs":{"total":95995035648,"available":94396596224,"internally_reserved":0,"allocated":1548025856,"data_stored":1547429854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528350,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521908,"kb_used_data":1472672,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92223244,"statfs":{"total":95995035648,"available":94436601856,"internally_reserved":0,"allocated":1508016128,"data_stored":1507451869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789985,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301176,"kb_used_data":1247780,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92443976,"statfs":{"total":95995035648,"available":94662631424,"internally_reserved":0,"allocated":1277726720,"data_stored":1277158061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1560452,"kb_used_data":1507060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92184700,"statfs":{"total":95995035648,"available":94397132800,"internally_reserved":0,"allocated":1543229440,"data_stored":1542645306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2827928,"kb_used_data":2756488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":90917224,"statfs":{"total":95995035648,"available":93099237376,"internally_reserved":0,"allocated":2822643712,"data_stored":2822039896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1547840,"kb_used_data":1502768,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92197312,"statfs":{"total":95995035648,"available":94410047488,"internally_reserved":0,"allocated":1538834432,"data_stored":1538254150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2057136,"kb_used_data":1989856,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91688016,"statfs":{"total":95995035648,"available":93888528384,"internally_reserved":0,"allocated":2037612544,"data_stored":2036985386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2087256064,"data_stored":2087256064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1550188544,"data_stored":1550188544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2842951680,"data_stored":2842951680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1552941056,"data_stored":1552941056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1285947392,"data_stored":1285947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1546780672,"data_stored":1546780672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1557463040,"data_stored":1557463040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:54:56.163 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-21T18:54:56.170 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:55 smithi104 ceph-mon[32438]: 6.1b scrub starts 2024-09-21T18:54:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:55 smithi104 ceph-mon[32438]: 6.1b scrub ok 2024-09-21T18:54:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:55 smithi104 ceph-mon[32438]: from='client.34787 v1:172.21.15.81:0/2906043307' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:54:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:55 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T18:54:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:55 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T18:54:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3677256851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:54:56.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:56.302 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:56.404 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:56.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:56.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:56.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:56.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:56.810 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:56.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T18:54:56.840 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:56.856 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:54:56.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:54:56.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:54:56.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:54:56.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:54:56.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:54:56.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:54:56.881 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:54:56.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:56.984 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:56.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[39192]: pgmap v66: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 598 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[39192]: 6.e scrub starts 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[39192]: 6.e scrub ok 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3238807395' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:54:57.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[29384]: pgmap v66: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 598 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-21T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[29384]: 6.e scrub starts 2024-09-21T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[29384]: 6.e scrub ok 2024-09-21T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3238807395' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:54:57.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:57.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:57.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 36 16 71769 71753 124.537 89.1875 0.00829995 0.00802491 2024-09-21T18:54:57.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:56 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T18:54:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:56 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T18:54:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:56 smithi104 ceph-mon[32438]: pgmap v66: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 598 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-21T18:54:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:56 smithi104 ceph-mon[32438]: 6.e scrub starts 2024-09-21T18:54:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:56 smithi104 ceph-mon[32438]: 6.e scrub ok 2024-09-21T18:54:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3238807395' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:54:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.282 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:57.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:57.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:54:57.429 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:54:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:54:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:54:57.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:57.565 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:57.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:57.702 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:57.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:54:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.842 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:57.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:54:57.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:57.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:58.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:57 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3238807395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:54:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:57 smithi104 ceph-mon[32438]: osdmap e97: 8 total, 8 up, 7 in 2024-09-21T18:54:58.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:54:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:54:58.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.120 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:58.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:58.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 37 16 73335 73319 123.816 97.875 0.0132066 0.00807231 2024-09-21T18:54:58.221 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:58.256 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:58.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3238807395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:54:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-mon[29384]: osdmap e97: 8 total, 8 up, 7 in 2024-09-21T18:54:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:54:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3238807395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:54:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:57 smithi081 ceph-mon[39192]: osdmap e97: 8 total, 8 up, 7 in 2024-09-21T18:54:58.357 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:58.392 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:58.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:58.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:58.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:54:58.675 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:54:58.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:58.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:58.947 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:58.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:59.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:54:59.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:54:59.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 38 16 74728 74712 122.849 87.0625 0.0107376 0.00813574 2024-09-21T18:54:59.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:59.211 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:59.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:58 smithi104 ceph-mon[32438]: pgmap v68: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-21T18:54:59.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:54:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:59.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:58 smithi081 ceph-mon[39192]: pgmap v68: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-21T18:54:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:54:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:58 smithi081 ceph-mon[29384]: pgmap v68: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-21T18:54:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:54:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:54:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:59.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:54:59.488 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:54:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:54:59.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:54:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:59.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:54:59.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:54:59.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:54:59.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:54:59.889 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:54:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:54:59.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:54:59.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:00.029 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:00.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:00.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 39 16 76147 76131 121.973 88.6875 0.00989455 0.00819356 2024-09-21T18:55:00.191 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:00.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:00.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:00.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.457 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:00.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:00.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:00.588 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:00.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:00.724 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:00.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:00 smithi104 ceph-mon[32438]: pgmap v69: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T18:55:00.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:00 smithi104 ceph-mon[32438]: osdmap e98: 8 total, 8 up, 7 in 2024-09-21T18:55:00.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:00 smithi081 ceph-mon[29384]: pgmap v69: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T18:55:00.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:00 smithi081 ceph-mon[29384]: osdmap e98: 8 total, 8 up, 7 in 2024-09-21T18:55:00.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:00 smithi081 ceph-mon[39192]: pgmap v69: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T18:55:00.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:00 smithi081 ceph-mon[39192]: osdmap e98: 8 total, 8 up, 7 in 2024-09-21T18:55:00.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:00.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:01.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:01.043 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:01.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:01.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:55:01.152290+0000 min lat: 0.00134243 max lat: 0.179359 avg lat: 0.0082516 2024-09-21T18:55:01.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:55:01.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 40 16 77556 77540 121.124 88.0625 0.0094133 0.0082516 2024-09-21T18:55:01.164 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:55:01.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:55:01.164 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T18:55:01.164 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T18:55:01.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:55:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.187 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:01.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:01.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:01.316 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:01 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[70807]: ts=2024-09-21T18:55:01.057Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T18:55:01.316 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:01 smithi081 podman[77549]: 2024-09-21 18:55:01.099781408 +0000 UTC m=+5.572051877 container died dc002ed31e61611de7f50a18fcf0b9698482e48cd372fe9711acc30231e62ebd (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:01.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:01.463 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:01.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.607 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:01.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:01.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:01.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:01.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:01.868 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:01.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:01.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:01.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:02.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:02.153 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 41 15 78860 78845 120.159 81.5625 0.0106469 0.00831702 2024-09-21T18:55:02.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:55:02.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:02.290 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:02.392 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:02.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.552 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:02.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:02.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:02.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:02.715 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:02 smithi104 ceph-mon[32438]: pgmap v71: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-21T18:55:02.770 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:02 smithi081 ceph-mon[39192]: pgmap v71: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-21T18:55:02.770 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:02 smithi081 ceph-mon[29384]: pgmap v71: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-21T18:55:02.770 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:02 smithi081 podman[77549]: 2024-09-21 18:55:02.490510368 +0000 UTC m=+6.962780839 container cleanup dc002ed31e61611de7f50a18fcf0b9698482e48cd372fe9711acc30231e62ebd (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:02.770 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:02 smithi081 bash[77549]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T18:55:02.771 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:02 smithi081 podman[77927]: 2024-09-21 18:55:02.675703201 +0000 UTC m=+1.563724294 container remove dc002ed31e61611de7f50a18fcf0b9698482e48cd372fe9711acc30231e62ebd (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:02.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:02.823 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:02.925 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:02.973 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:02.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:03.065 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:02 smithi081 podman[77927]: 2024-09-21 18:55:02.76838384 +0000 UTC m=+1.656404939 volume remove e361205deacbdd05c03163b05ff846e20c35e04d36eddb43e55c0d1b53468613 2024-09-21T18:55:03.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:03.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:03.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:03.166 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 42 16 80045 80029 119.024 74 0.00931621 0.00839545 2024-09-21T18:55:03.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:03.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:03.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:03.378 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:03.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:03.435 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:5812 7.e 7:76ddb186:::benchmark_data_smithi081_73391_object5811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:22.687449+0000", 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.78034959, 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.043491952, 2024-09-21T18:55:03.477 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 5812 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.687449+0000", 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.687449+0000", 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.687450+0000", 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1800000000000004e-07 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.687476+0000", 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6546999999999998e-05 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.687484+0000", 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5190000000000003e-06 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.687489+0000", 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4999999999999999e-06 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.689346+0000", 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018566539999999999 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.689825+0000", 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00047927199999999999 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.689972+0000", 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014640500000000001 2024-09-21T18:55:03.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.690633+0000", 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00066142200000000005 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.691534+0000", 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00090115099999999999 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.729841+0000", 2024-09-21T18:55:03.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.038306618000000001 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.729896+0000", 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4919000000000001e-05 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:22.730941+0000", 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001045027 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13973 7.e 7:748b33c0:::benchmark_data_smithi081_73391_object13972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.428862+0000", 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.038936530999997, 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055207547000000003, 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13973 2024-09-21T18:55:03.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428862+0000", 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428862+0000", 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428864+0000", 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8610000000000001e-06 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428885+0000", 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1691e-05 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428895+0000", 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9569999999999997e-06 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428901+0000", 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1720000000000004e-06 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.428962+0000", 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0786999999999999e-05 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429070+0000", 2024-09-21T18:55:03.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107582 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429218+0000", 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014824099999999999 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.429868+0000", 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00064999899999999995 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.431318+0000", 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001449737 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.483397+0000", 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052079602000000003 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.484028+0000", 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063059100000000001 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.484069+0000", 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1326999999999999e-05 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:39517 7.e 7:7031e854:::benchmark_data_smithi081_73391_object39516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:37.814433+0000", 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 25.653365138000002, 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.059190553, 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.490 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 39517 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814433+0000", 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814433+0000", 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814436+0000", 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6620000000000001e-06 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814470+0000", 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3914000000000001e-05 2024-09-21T18:55:03.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814486+0000", 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5879000000000002e-05 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814495+0000", 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1060000000000006e-06 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814619+0000", 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012455499999999999 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814748+0000", 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128238 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.814876+0000", 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128546 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.816766+0000", 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018899489999999999 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817816+0000", 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010498529999999999 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.873023+0000", 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055207587000000002 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.873473+0000", 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044918300000000003 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.873624+0000", 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015108100000000001 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:39524 7.e 7:7b8062a5:::benchmark_data_smithi081_73391_object39523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:37.817908+0000", 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 25.649890301999999, 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.056090520999999997, 2024-09-21T18:55:03.496 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 39524 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817908+0000", 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817908+0000", 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817910+0000", 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1940000000000001e-06 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817933+0000", 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2734e-05 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817944+0000", 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.064e-05 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.817951+0000", 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4529999999999996e-06 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.818213+0000", 2024-09-21T18:55:03.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026150799999999998 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.818326+0000", 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113563 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.818471+0000", 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014463200000000001 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.820154+0000", 2024-09-21T18:55:03.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001683408 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.820520+0000", 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036559500000000003 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.873691+0000", 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053170974000000003 2024-09-21T18:55:03.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.873939+0000", 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024863400000000002 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:37.873999+0000", 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9185999999999999e-05 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:49496 7.e 7:7a624b54:::benchmark_data_smithi081_73391_object49495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:43.161512+0000", 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.306286708000002, 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.044361703000000002, 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.503 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 49496 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161512+0000", 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161512+0000", 2024-09-21T18:55:03.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161514+0000", 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1600000000000001e-06 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161536+0000", 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1871000000000001e-05 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161547+0000", 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1542999999999999e-05 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161554+0000", 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8129999999999998e-06 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161610+0000", 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5695999999999997e-05 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161725+0000", 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115443 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.161850+0000", 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012498500000000001 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.163582+0000", 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001731909 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.164260+0000", 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00067782699999999999 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.205785+0000", 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.041525610999999997 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.205834+0000", 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8238000000000001e-05 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.205873+0000", 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9607000000000002e-05 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:50501 7.e 7:77d4ccf8:::benchmark_data_smithi081_73391_object50500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:43.733488+0000", 2024-09-21T18:55:03.509 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 19.734309880000001, 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.051020784999999999, 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 50501 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733488+0000", 2024-09-21T18:55:03.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733488+0000", 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733490+0000", 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.703e-06 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733510+0000", 2024-09-21T18:55:03.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9542e-05 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733520+0000", 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0376e-05 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733527+0000", 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.951e-06 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733572+0000", 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5179999999999998e-05 2024-09-21T18:55:03.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733653+0000", 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0605999999999994e-05 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.733731+0000", 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8022999999999995e-05 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.734984+0000", 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012534180000000001 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736134+0000", 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011495260000000001 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.784394+0000", 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.048260052999999997 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.784450+0000", 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5726999999999997e-05 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.784509+0000", 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9679999999999998e-05 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:50503 7.e 7:7adac35b:::benchmark_data_smithi081_73391_object50502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:43.736172+0000", 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 19.731626201000001, 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.048530194999999998, 2024-09-21T18:55:03.515 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 50503 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736172+0000", 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736172+0000", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736173+0000", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1800000000000005e-07 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736191+0000", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7620000000000001e-05 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736196+0000", 2024-09-21T18:55:03.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6060000000000002e-06 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736200+0000", 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6959999999999998e-06 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736230+0000", 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0150000000000001e-05 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736304+0000", 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3757999999999994e-05 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.736429+0000", 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000125513 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.737408+0000", 2024-09-21T18:55:03.520 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097874700000000008 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.737756+0000", 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034797099999999998 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.784583+0000", 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046826723000000001 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.784667+0000", 2024-09-21T18:55:03.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4157000000000001e-05 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:43.784702+0000", 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5435999999999998e-05 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:54901 7.e 7:7b80fe6d:::benchmark_data_smithi081_73391_object54900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.448338+0000", 2024-09-21T18:55:03.522 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 17.019460726999998, 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.050457156000000003, 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 54901 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.448338+0000", 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.448338+0000", 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.448338+0000", 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.448356+0000", 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8036000000000001e-05 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.448362+0000", 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6829999999999996e-06 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.448367+0000", 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9570000000000001e-06 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.454170+0000", 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058035939999999996 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.454278+0000", 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107426 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.454402+0000", 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012406 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.526 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456363+0000", 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019608960000000002 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458915+0000", 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002552131 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459319+0000", 2024-09-21T18:55:03.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040379700000000001 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.498749+0000", 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.039430252999999998 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.498795+0000", 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5766000000000002e-05 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:54916 7.e 7:7c91c541:::benchmark_data_smithi081_73391_object54915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.456485+0000", 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 17.011313099999999, 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045906646000000002, 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 54916 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.529 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456485+0000", 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456485+0000", 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456487+0000", 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.689e-06 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456512+0000", 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5069999999999999e-05 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456522+0000", 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0086e-05 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456529+0000", 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7739999999999999e-06 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456588+0000", 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9043999999999999e-05 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456660+0000", 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2433000000000003e-05 2024-09-21T18:55:03.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.456739+0000", 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8953000000000001e-05 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458928+0000", 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021886010000000001 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.500203+0000", 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.041275062000000001 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.502295+0000", 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020916960000000001 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.502348+0000", 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3236000000000003e-05 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.502392+0000", 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4002000000000003e-05 2024-09-21T18:55:03.534 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73488 7.e 7:7c32cab4:::benchmark_data_smithi081_73391_object73487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.250905+0000", 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.2168934159999996, 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049186190999999997, 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.535 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73488 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.250905+0000", 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.250905+0000", 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.250907+0000", 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7099999999999999e-06 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.250977+0000", 2024-09-21T18:55:03.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9927999999999999e-05 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.250987+0000", 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.041e-05 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.250993+0000", 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5060000000000001e-06 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.251032+0000", 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8349999999999997e-05 2024-09-21T18:55:03.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.251136+0000", 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104104 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.251284+0000", 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014794600000000001 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.253657+0000", 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023732689999999999 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.259793+0000", 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061354410000000002 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.300002+0000", 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.040209169000000003 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.300052+0000", 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0416e-05 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.300091+0000", 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8942000000000001e-05 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73513 7.e 7:7ca4825a:::benchmark_data_smithi081_73391_object73512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.303645+0000", 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.164153496, 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045383949999999999, 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.541 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73513 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303645+0000", 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303645+0000", 2024-09-21T18:55:03.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303647+0000", 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7179999999999999e-06 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303669+0000", 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2548000000000002e-05 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303684+0000", 2024-09-21T18:55:03.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4669e-05 2024-09-21T18:55:03.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303691+0000", 2024-09-21T18:55:03.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3799999999999996e-06 2024-09-21T18:55:03.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303781+0000", 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9820000000000004e-05 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.303896+0000", 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115398 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.304023+0000", 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000126747 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.310039+0000", 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060158549999999996 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.311039+0000", 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010002430000000001 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.348766+0000", 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.037726933999999997 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.348975+0000", 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000208524 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.349029+0000", 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4113999999999997e-05 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73821 7.e 7:70c547f6:::benchmark_data_smithi081_73391_object73820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.548 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.537154+0000", 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.930644246, 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046421111000000001, 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73821 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537154+0000", 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537154+0000", 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537156+0000", 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1940000000000001e-06 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537184+0000", 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7208000000000001e-05 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537196+0000", 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2917e-05 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537205+0000", 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0110000000000006e-06 2024-09-21T18:55:03.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537266+0000", 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0946999999999998e-05 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537524+0000", 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025784700000000001 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537737+0000", 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021247 2024-09-21T18:55:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.544846+0000", 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.007109041 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.544853+0000", 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2479999999999999e-06 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583414+0000", 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.038561273 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583525+0000", 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110815 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583575+0000", 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0139999999999998e-05 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73822 7.e 7:7bb7ec41:::benchmark_data_smithi081_73391_object73821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.537229+0000", 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.9305693450000003, 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046520390000000002, 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73822 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.555 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537229+0000", 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537229+0000", 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537230+0000", 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0800000000000003e-07 2024-09-21T18:55:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537251+0000", 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0959e-05 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537259+0000", 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8630000000000008e-06 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.537265+0000", 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2210000000000002e-06 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.539649+0000", 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023838760000000001 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.539773+0000", 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012416599999999999 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.539905+0000", 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013183100000000001 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.544856+0000", 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049512239999999997 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547160+0000", 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023037029999999998 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583624+0000", 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.036464436000000003 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583713+0000", 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8904000000000001e-05 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583749+0000", 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6298999999999999e-05 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73827 7.e 7:74899bb7:::benchmark_data_smithi081_73391_object73826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.538333+0000", 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.929465263, 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045574232999999999, 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73827 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.538333+0000", 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.538333+0000", 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.538335+0000", 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.649e-06 2024-09-21T18:55:03.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.538367+0000", 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2666999999999997e-05 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.538378+0000", 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0648999999999999e-05 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.538385+0000", 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1550000000000001e-06 2024-09-21T18:55:03.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.541949+0000", 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035639759999999999 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542018+0000", 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9047000000000005e-05 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542110+0000", 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1314e-05 2024-09-21T18:55:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547106+0000", 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049969380000000002 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547200+0000", 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3479000000000004e-05 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583784+0000", 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.036584195 2024-09-21T18:55:03.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583867+0000", 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3286000000000002e-05 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583907+0000", 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9878e-05 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73829 7.e 7:71925a73:::benchmark_data_smithi081_73391_object73828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.567 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.540285+0000", 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.927513459, 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.043781173999999999, 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73829 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.540285+0000", 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.540285+0000", 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.540287+0000", 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7659999999999999e-06 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.540308+0000", 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1279000000000001e-05 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.540317+0000", 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.38e-06 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.540323+0000", 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6699999999999999e-06 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.543435+0000", 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031117760000000001 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.543507+0000", 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2663999999999994e-05 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.543583+0000", 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5394000000000004e-05 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547207+0000", 2024-09-21T18:55:03.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003623738 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547245+0000", 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8238000000000002e-05 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.583947+0000", 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.036702452000000003 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.584031+0000", 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3510000000000005e-05 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.584066+0000", 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5306999999999999e-05 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:73834 7.e 7:702c65fe:::benchmark_data_smithi081_73391_object73833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:58.542232+0000", 2024-09-21T18:55:03.573 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.9255658819999999, 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045270592999999998, 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 73834 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542232+0000", 2024-09-21T18:55:03.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542232+0000", 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542233+0000", 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4600000000000003e-07 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542253+0000", 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.982e-05 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542261+0000", 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0870000000000007e-06 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.542267+0000", 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9340000000000003e-06 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.545972+0000", 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037046129999999998 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.546048+0000", 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6341000000000007e-05 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.546129+0000", 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1076999999999996e-05 2024-09-21T18:55:03.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547264+0000", 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011343480000000001 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.547749+0000", 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00048589500000000002 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.587372+0000", 2024-09-21T18:55:03.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0396228 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.587463+0000", 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0818999999999999e-05 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:58.587503+0000", 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0012999999999998e-05 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.579 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:75013 7.e 7:72153c91:::benchmark_data_smithi081_73391_object75012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:59.360942+0000", 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.1068561299999997, 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.062980637000000006, 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 75013 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.360942+0000", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.360942+0000", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.360943+0000", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.186e-06 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.360976+0000", 2024-09-21T18:55:03.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2741999999999999e-05 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.360994+0000", 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7740999999999998e-05 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.361012+0000", 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8366e-05 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.361072+0000", 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9962e-05 2024-09-21T18:55:03.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.361167+0000", 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5284000000000005e-05 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.361355+0000", 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018764000000000001 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.363107+0000", 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017521349999999999 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.365134+0000", 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002026342 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.423831+0000", 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.058696942000000002 2024-09-21T18:55:03.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.423883+0000", 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2061999999999997e-05 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:59.423923+0000", 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0234999999999997e-05 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.585 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:78934 7.e 7:733c2166:::benchmark_data_smithi081_73391_object78933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:02.207095+0000", 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.2607035609999999, 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.062777033999999995, 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 78934 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.586 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207095+0000", 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207095+0000", 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207097+0000", 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3879999999999998e-06 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207123+0000", 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5749e-05 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207136+0000", 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3253e-05 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207145+0000", 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3510000000000002e-06 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207343+0000", 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000198056 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207471+0000", 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012849399999999999 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.207843+0000", 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037205700000000002 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.210624+0000", 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027811400000000001 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.213150+0000", 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025254029999999998 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.269763+0000", 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.056613036999999998 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.269823+0000", 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0634000000000003e-05 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.269872+0000", 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8471999999999999e-05 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:78947 7.e 7:759b33de:::benchmark_data_smithi081_73391_object78946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:02.214677+0000", 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.2531211179999999, 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055325394, 2024-09-21T18:55:03.591 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 78947 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214677+0000", 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214677+0000", 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214680+0000", 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3539999999999998e-06 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214707+0000", 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7637999999999999e-05 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214720+0000", 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3207000000000001e-05 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214729+0000", 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6899999999999998e-06 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214786+0000", 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6761000000000001e-05 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.214911+0000", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000125582 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215047+0000", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013588700000000001 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.220337+0000", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0052892060000000003 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.220397+0000", 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0758999999999999e-05 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.269911+0000", 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049514015000000001 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.269966+0000", 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4737000000000001e-05 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.270003+0000", 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6557999999999998e-05 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:78948 7.e 7:7bd389a1:::benchmark_data_smithi081_73391_object78947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:02.215252+0000", 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.252546068, 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.054869993999999998, 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:55:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 78948 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215252+0000", 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215252+0000", 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215254+0000", 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.243e-06 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215275+0000", 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1761999999999999e-05 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215285+0000", 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9739999999999997e-06 2024-09-21T18:55:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.215293+0000", 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.311e-06 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.217096+0000", 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001803752 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.217244+0000", 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000148128 2024-09-21T18:55:03.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.217395+0000", 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015087400000000001 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.220385+0000", 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029892180000000001 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.220405+0000", 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0460999999999999e-05 2024-09-21T18:55:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.270040+0000", 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049634747 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.270088+0000", 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8489999999999998e-05 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:02.270122+0000", 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4034e-05 2024-09-21T18:55:03.602 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.603 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.603 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.603 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:55:03.603 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:55:03.603 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:55:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:03.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:03.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:03.742 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:03.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:03.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:03.881 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:03.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:03.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:55:03.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:04.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:04.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:04.162 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:04.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:04.166 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 43 16 81316 81300 118.103 79.4375 0.0125937 0.0084619 2024-09-21T18:55:04.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:04.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:04.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:04.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:04 smithi081 ceph-mon[39192]: pgmap v72: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T18:55:04.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:04 smithi081 ceph-mon[29384]: pgmap v72: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T18:55:04.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:04.680 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:04.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:04 smithi104 ceph-mon[32438]: pgmap v72: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T18:55:04.810 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:04.842 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:04.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:04.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:05.018 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:05.018 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:05.061 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:05.061 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:05.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:05.095 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:05.096 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:05.133 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:05.137 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:05.138 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:05.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 44 16 82653 82637 117.318 83.5625 0.00896915 0.00851927 2024-09-21T18:55:05.167 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:05.167 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:05.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:05.233 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:05.233 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:05.234 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:05 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T18:55:05.234 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:05 smithi081 systemd[1]: Stopped Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:55:05.234 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:05 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.197s CPU time. 2024-09-21T18:55:05.234 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:05 smithi081 systemd[1]: Starting Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:55:05.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:05.263 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:05.266 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:05.266 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:05.365 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.402 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:05.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:05.503 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:05.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.672 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:05.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:05.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2624015393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:05.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:05.809 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:05.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2624015393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2624015393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:05.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:05.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:05.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:06.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:06.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 45 16 84114 84098 116.74 91.3125 0.0069083 0.00856163 2024-09-21T18:55:06.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:06.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:06.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:06.487 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:06.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:06.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:06.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:06.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:06 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:06 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:06 smithi104 ceph-mon[32438]: pgmap v73: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 599 B/s rd, 53 MiB/s wr, 842 op/s 2024-09-21T18:55:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:06 smithi104 ceph-mon[32438]: osdmap e99: 8 total, 8 up, 7 in 2024-09-21T18:55:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:06.760 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[29384]: pgmap v73: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 599 B/s rd, 53 MiB/s wr, 842 op/s 2024-09-21T18:55:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[29384]: osdmap e99: 8 total, 8 up, 7 in 2024-09-21T18:55:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:06.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:06.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:06.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[39192]: pgmap v73: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 599 B/s rd, 53 MiB/s wr, 842 op/s 2024-09-21T18:55:06.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[39192]: osdmap e99: 8 total, 8 up, 7 in 2024-09-21T18:55:06.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:06.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:06.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:06.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T18:55:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:06.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:06.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:07.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:07.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:07.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 46 16 85510 85494 116.099 87.25 0.018437 0.00860828 2024-09-21T18:55:07.175 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:07.210 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:55:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:55:07.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.305 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:07.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:07.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.444 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:07.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:07.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:07.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:07.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:07.759 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:55:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:55:07.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:07.855 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:07.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:07.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:07.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:08.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:55:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:55:08.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:08.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:08.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 47 16 86831 86815 115.386 82.5625 0.00671966 0.00866212 2024-09-21T18:55:08.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:08.274 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:08.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.403 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:08.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:08.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:08.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:08.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:08.676 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:08 smithi104 ceph-mon[32438]: pgmap v75: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-21T18:55:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:08.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:08 smithi104 ceph-mon[32438]: from='client.34811 v1:172.21.15.81:0/900097391' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:55:08.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:08.810 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:08 smithi081 ceph-mon[39192]: pgmap v75: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-21T18:55:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:08.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:08 smithi081 ceph-mon[39192]: from='client.34811 v1:172.21.15.81:0/900097391' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:55:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:08 smithi081 ceph-mon[29384]: pgmap v75: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-21T18:55:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:08.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:08 smithi081 ceph-mon[29384]: from='client.34811 v1:172.21.15.81:0/900097391' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:55:08.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:08.946 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:08.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:09.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:09.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 48 15 88193 88178 114.757 85.1875 0.00575613 0.00870966 2024-09-21T18:55:09.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:09.229 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:09.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:09.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:09.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:09.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:09.510 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:09.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:09.721 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-21T18:55:07.392482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5536291355,"num_objects":84667,"num_object_clones":0,"num_object_copies":254001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84667,"num_whiteouts":0,"num_read":955,"num_read_kb":812,"num_write":85017,"num_write_kb":5407625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":1184346,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75723,"ondisk_log_size":75723,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16560508,"kb_used_data":16085376,"kb_used_omap":10,"kb_used_meta":474997,"kb_avail":639655556,"statfs":{"total":671965249536,"available":655007289344,"internally_reserved":0,"allocated":16471425024,"data_stored":16467192894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11153,"internal_metadata":486397039},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":902103040,"num_objects":13765,"num_object_clones":0,"num_object_copies":41295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13765,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13765,"num_write_kb":880960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002911"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492503+0000","last_change":"2024-09-21T18:54:17.713866+0000","last_active":"2024-09-21T18:55:05.492503+0000","last_peered":"2024-09-21T18:55:05.492503+0000","last_clean":"2024-09-21T18:55:05.492503+0000","last_became_active":"2024-09-21T18:54:17.713495+0000","last_became_peered":"2024-09-21T18:54:17.713495+0000","last_unstale":"2024-09-21T18:55:05.492503+0000","last_undegraded":"2024-09-21T18:55:05.492503+0000","last_fullsized":"2024-09-21T18:55:05.492503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.249265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":122,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487679+0000","last_change":"2024-09-21T18:54:36.776089+0000","last_active":"2024-09-21T18:55:05.487679+0000","last_peered":"2024-09-21T18:55:05.487679+0000","last_clean":"2024-09-21T18:55:05.487679+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:55:05.487679+0000","last_undegraded":"2024-09-21T18:55:05.487679+0000","last_fullsized":"2024-09-21T18:55:05.487679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.052772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487351+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:55:05.487351+0000","last_peered":"2024-09-21T18:55:05.487351+0000","last_clean":"2024-09-21T18:55:05.487351+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:55:05.487351+0000","last_undegraded":"2024-09-21T18:55:05.487351+0000","last_fullsized":"2024-09-21T18:55:05.487351+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.278309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487452+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:55:05.487452+0000","last_peered":"2024-09-21T18:55:05.487452+0000","last_clean":"2024-09-21T18:55:05.487452+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:55:05.487452+0000","last_undegraded":"2024-09-21T18:55:05.487452+0000","last_fullsized":"2024-09-21T18:55:05.487452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.055413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":90,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488947+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:55:05.488947+0000","last_peered":"2024-09-21T18:55:05.488947+0000","last_clean":"2024-09-21T18:55:05.488947+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:55:05.488947+0000","last_undegraded":"2024-09-21T18:55:05.488947+0000","last_fullsized":"2024-09-21T18:55:05.488947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.105754+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":102,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489003+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:55:05.489003+0000","last_peered":"2024-09-21T18:55:05.489003+0000","last_clean":"2024-09-21T18:55:05.489003+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:55:05.489003+0000","last_undegraded":"2024-09-21T18:55:05.489003+0000","last_fullsized":"2024-09-21T18:55:05.489003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.541042+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":138,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488860+0000","last_change":"2024-09-21T18:54:17.730862+0000","last_active":"2024-09-21T18:55:05.488860+0000","last_peered":"2024-09-21T18:55:05.488860+0000","last_clean":"2024-09-21T18:55:05.488860+0000","last_became_active":"2024-09-21T18:54:17.716359+0000","last_became_peered":"2024-09-21T18:54:17.716359+0000","last_unstale":"2024-09-21T18:55:05.488860+0000","last_undegraded":"2024-09-21T18:55:05.488860+0000","last_fullsized":"2024-09-21T18:55:05.488860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.191168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487277+0000","last_change":"2024-09-21T18:53:11.410545+0000","last_active":"2024-09-21T18:55:05.487277+0000","last_peered":"2024-09-21T18:55:05.487277+0000","last_clean":"2024-09-21T18:55:05.487277+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:55:05.487277+0000","last_undegraded":"2024-09-21T18:55:05.487277+0000","last_fullsized":"2024-09-21T18:55:05.487277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.928369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492064+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:55:05.492064+0000","last_peered":"2024-09-21T18:55:05.492064+0000","last_clean":"2024-09-21T18:55:05.492064+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:55:05.492064+0000","last_undegraded":"2024-09-21T18:55:05.492064+0000","last_fullsized":"2024-09-21T18:55:05.492064+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":130,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487787+0000","last_change":"2024-09-21T18:54:17.725564+0000","last_active":"2024-09-21T18:55:05.487787+0000","last_peered":"2024-09-21T18:55:05.487787+0000","last_clean":"2024-09-21T18:55:05.487787+0000","last_became_active":"2024-09-21T18:54:17.704975+0000","last_became_peered":"2024-09-21T18:54:17.704975+0000","last_unstale":"2024-09-21T18:55:05.487787+0000","last_undegraded":"2024-09-21T18:55:05.487787+0000","last_fullsized":"2024-09-21T18:55:05.487787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:14.972606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.972606+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.449994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491477+0000","last_change":"2024-09-21T18:54:55.048550+0000","last_active":"2024-09-21T18:55:05.491477+0000","last_peered":"2024-09-21T18:55:05.491477+0000","last_clean":"2024-09-21T18:55:05.491477+0000","last_became_active":"2024-09-21T18:54:17.712591+0000","last_became_peered":"2024-09-21T18:54:17.712591+0000","last_unstale":"2024-09-21T18:55:05.491477+0000","last_undegraded":"2024-09-21T18:55:05.491477+0000","last_fullsized":"2024-09-21T18:55:05.491477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:55.048477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:55.048477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:01.675730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.493783+0000","last_change":"2024-09-21T18:54:17.704813+0000","last_active":"2024-09-21T18:55:05.493783+0000","last_peered":"2024-09-21T18:55:05.493783+0000","last_clean":"2024-09-21T18:55:05.493783+0000","last_became_active":"2024-09-21T18:54:17.704510+0000","last_became_peered":"2024-09-21T18:54:17.704510+0000","last_unstale":"2024-09-21T18:55:05.493783+0000","last_undegraded":"2024-09-21T18:55:05.493783+0000","last_fullsized":"2024-09-21T18:55:05.493783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.329858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491409+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:55:05.491409+0000","last_peered":"2024-09-21T18:55:05.491409+0000","last_clean":"2024-09-21T18:55:05.491409+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:55:05.491409+0000","last_undegraded":"2024-09-21T18:55:05.491409+0000","last_fullsized":"2024-09-21T18:55:05.491409+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":107,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487954+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:55:05.487954+0000","last_peered":"2024-09-21T18:55:05.487954+0000","last_clean":"2024-09-21T18:55:05.487954+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:55:05.487954+0000","last_undegraded":"2024-09-21T18:55:05.487954+0000","last_fullsized":"2024-09-21T18:55:05.487954+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.337954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489730+0000","last_change":"2024-09-21T18:54:53.035653+0000","last_active":"2024-09-21T18:55:05.489730+0000","last_peered":"2024-09-21T18:55:05.489730+0000","last_clean":"2024-09-21T18:55:05.489730+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:55:05.489730+0000","last_undegraded":"2024-09-21T18:55:05.489730+0000","last_fullsized":"2024-09-21T18:55:05.489730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.035514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.035514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.035186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491056+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:55:05.491056+0000","last_peered":"2024-09-21T18:55:05.491056+0000","last_clean":"2024-09-21T18:55:05.491056+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:55:05.491056+0000","last_undegraded":"2024-09-21T18:55:05.491056+0000","last_fullsized":"2024-09-21T18:55:05.491056+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.155050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488123+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:55:05.488123+0000","last_peered":"2024-09-21T18:55:05.488123+0000","last_clean":"2024-09-21T18:55:05.488123+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:55:05.488123+0000","last_undegraded":"2024-09-21T18:55:05.488123+0000","last_fullsized":"2024-09-21T18:55:05.488123+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.940396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488047+0000","last_change":"2024-09-21T18:54:17.705466+0000","last_active":"2024-09-21T18:55:05.488047+0000","last_peered":"2024-09-21T18:55:05.488047+0000","last_clean":"2024-09-21T18:55:05.488047+0000","last_became_active":"2024-09-21T18:54:17.705159+0000","last_became_peered":"2024-09-21T18:54:17.705159+0000","last_unstale":"2024-09-21T18:55:05.488047+0000","last_undegraded":"2024-09-21T18:55:05.488047+0000","last_fullsized":"2024-09-21T18:55:05.488047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.944967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":122,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490672+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:55:05.490672+0000","last_peered":"2024-09-21T18:55:05.490672+0000","last_clean":"2024-09-21T18:55:05.490672+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:55:05.490672+0000","last_undegraded":"2024-09-21T18:55:05.490672+0000","last_fullsized":"2024-09-21T18:55:05.490672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.670449+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486852+0000","last_change":"2024-09-21T18:54:17.714793+0000","last_active":"2024-09-21T18:55:05.486852+0000","last_peered":"2024-09-21T18:55:05.486852+0000","last_clean":"2024-09-21T18:55:05.486852+0000","last_became_active":"2024-09-21T18:54:17.714300+0000","last_became_peered":"2024-09-21T18:54:17.714300+0000","last_unstale":"2024-09-21T18:55:05.486852+0000","last_undegraded":"2024-09-21T18:55:05.486852+0000","last_fullsized":"2024-09-21T18:55:05.486852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488512+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:55:05.488512+0000","last_peered":"2024-09-21T18:55:05.488512+0000","last_clean":"2024-09-21T18:55:05.488512+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:55:05.488512+0000","last_undegraded":"2024-09-21T18:55:05.488512+0000","last_fullsized":"2024-09-21T18:55:05.488512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.092780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487972+0000","last_change":"2024-09-21T18:54:12.987795+0000","last_active":"2024-09-21T18:55:05.487972+0000","last_peered":"2024-09-21T18:55:05.487972+0000","last_clean":"2024-09-21T18:55:05.487972+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:55:05.487972+0000","last_undegraded":"2024-09-21T18:55:05.487972+0000","last_fullsized":"2024-09-21T18:55:05.487972+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.987749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.987749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:41.935763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":121,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488122+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:55:05.488122+0000","last_peered":"2024-09-21T18:55:05.488122+0000","last_clean":"2024-09-21T18:55:05.488122+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:55:05.488122+0000","last_undegraded":"2024-09-21T18:55:05.488122+0000","last_fullsized":"2024-09-21T18:55:05.488122+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:06.555236+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487504+0000","last_change":"2024-09-21T18:53:11.416593+0000","last_active":"2024-09-21T18:55:05.487504+0000","last_peered":"2024-09-21T18:55:05.487504+0000","last_clean":"2024-09-21T18:55:05.487504+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:55:05.487504+0000","last_undegraded":"2024-09-21T18:55:05.487504+0000","last_fullsized":"2024-09-21T18:55:05.487504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.960588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490051+0000","last_change":"2024-09-21T18:54:55.039104+0000","last_active":"2024-09-21T18:55:05.490051+0000","last_peered":"2024-09-21T18:55:05.490051+0000","last_clean":"2024-09-21T18:55:05.490051+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:55:05.490051+0000","last_undegraded":"2024-09-21T18:55:05.490051+0000","last_fullsized":"2024-09-21T18:55:05.490051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:54:55.039055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:55.039055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.098477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":102,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487456+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:55:05.487456+0000","last_peered":"2024-09-21T18:55:05.487456+0000","last_clean":"2024-09-21T18:55:05.487456+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:55:05.487456+0000","last_undegraded":"2024-09-21T18:55:05.487456+0000","last_fullsized":"2024-09-21T18:55:05.487456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:43.629105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490448+0000","last_change":"2024-09-21T18:54:17.738374+0000","last_active":"2024-09-21T18:55:05.490448+0000","last_peered":"2024-09-21T18:55:05.490448+0000","last_clean":"2024-09-21T18:55:05.490448+0000","last_became_active":"2024-09-21T18:54:17.719024+0000","last_became_peered":"2024-09-21T18:54:17.719024+0000","last_unstale":"2024-09-21T18:55:05.490448+0000","last_undegraded":"2024-09-21T18:55:05.490448+0000","last_fullsized":"2024-09-21T18:55:05.490448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.810583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":119,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.495123+0000","last_change":"2024-09-21T18:53:11.408709+0000","last_active":"2024-09-21T18:55:05.495123+0000","last_peered":"2024-09-21T18:55:05.495123+0000","last_clean":"2024-09-21T18:55:05.495123+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:55:05.495123+0000","last_undegraded":"2024-09-21T18:55:05.495123+0000","last_fullsized":"2024-09-21T18:55:05.495123+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.245724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"99'5213","reported_seq":5264,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.358301+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:55:06.358301+0000","last_peered":"2024-09-21T18:55:06.358301+0000","last_clean":"2024-09-21T18:55:06.358301+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:55:06.358301+0000","last_undegraded":"2024-09-21T18:55:06.358301+0000","last_fullsized":"2024-09-21T18:55:06.358301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:14.400572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341639168,"num_objects":5213,"num_object_clones":0,"num_object_copies":15639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5213,"num_write_kb":333632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":114,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487107+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:55:05.487107+0000","last_peered":"2024-09-21T18:55:05.487107+0000","last_clean":"2024-09-21T18:55:05.487107+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:55:05.487107+0000","last_undegraded":"2024-09-21T18:55:05.487107+0000","last_fullsized":"2024-09-21T18:55:05.487107+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.353165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":102,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490505+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:55:05.490505+0000","last_peered":"2024-09-21T18:55:05.490505+0000","last_clean":"2024-09-21T18:55:05.490505+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:55:05.490505+0000","last_undegraded":"2024-09-21T18:55:05.490505+0000","last_fullsized":"2024-09-21T18:55:05.490505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:55.005580+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"83'9","reported_seq":128,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488181+0000","last_change":"2024-09-21T18:54:17.715008+0000","last_active":"2024-09-21T18:55:05.488181+0000","last_peered":"2024-09-21T18:55:05.488181+0000","last_clean":"2024-09-21T18:55:05.488181+0000","last_became_active":"2024-09-21T18:54:17.706739+0000","last_became_peered":"2024-09-21T18:54:17.706739+0000","last_unstale":"2024-09-21T18:55:05.488181+0000","last_undegraded":"2024-09-21T18:55:05.488181+0000","last_fullsized":"2024-09-21T18:55:05.488181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491500+0000","last_change":"2024-09-21T18:54:17.705694+0000","last_active":"2024-09-21T18:55:05.491500+0000","last_peered":"2024-09-21T18:55:05.491500+0000","last_clean":"2024-09-21T18:55:05.491500+0000","last_became_active":"2024-09-21T18:54:17.705355+0000","last_became_peered":"2024-09-21T18:54:17.705355+0000","last_unstale":"2024-09-21T18:55:05.491500+0000","last_undegraded":"2024-09-21T18:55:05.491500+0000","last_fullsized":"2024-09-21T18:55:05.491500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.222602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"99'5239","reported_seq":5290,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.438116+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:55:06.438116+0000","last_peered":"2024-09-21T18:55:06.438116+0000","last_clean":"2024-09-21T18:55:06.438116+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:55:06.438116+0000","last_undegraded":"2024-09-21T18:55:06.438116+0000","last_fullsized":"2024-09-21T18:55:06.438116+0000","mapping_epoch":79,"log_start":"87'1100","ondisk_log_start":"87'1100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.493653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343343104,"num_objects":5239,"num_object_clones":0,"num_object_copies":15717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":335296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489761+0000","last_change":"2024-09-21T18:54:17.740103+0000","last_active":"2024-09-21T18:55:05.489761+0000","last_peered":"2024-09-21T18:55:05.489761+0000","last_clean":"2024-09-21T18:55:05.489761+0000","last_became_active":"2024-09-21T18:54:17.716713+0000","last_became_peered":"2024-09-21T18:54:17.716713+0000","last_unstale":"2024-09-21T18:55:05.489761+0000","last_undegraded":"2024-09-21T18:55:05.489761+0000","last_fullsized":"2024-09-21T18:55:05.489761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489925+0000","last_change":"2024-09-21T18:54:17.719791+0000","last_active":"2024-09-21T18:55:05.489925+0000","last_peered":"2024-09-21T18:55:05.489925+0000","last_clean":"2024-09-21T18:55:05.489925+0000","last_became_active":"2024-09-21T18:54:17.719440+0000","last_became_peered":"2024-09-21T18:54:17.719440+0000","last_unstale":"2024-09-21T18:55:05.489925+0000","last_undegraded":"2024-09-21T18:55:05.489925+0000","last_fullsized":"2024-09-21T18:55:05.489925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.901802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488633+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:55:05.488633+0000","last_peered":"2024-09-21T18:55:05.488633+0000","last_clean":"2024-09-21T18:55:05.488633+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:55:05.488633+0000","last_undegraded":"2024-09-21T18:55:05.488633+0000","last_fullsized":"2024-09-21T18:55:05.488633+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.087833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487703+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:55:05.487703+0000","last_peered":"2024-09-21T18:55:05.487703+0000","last_clean":"2024-09-21T18:55:05.487703+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:55:05.487703+0000","last_undegraded":"2024-09-21T18:55:05.487703+0000","last_fullsized":"2024-09-21T18:55:05.487703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"99'5379","reported_seq":5430,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.432959+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:55:06.432959+0000","last_peered":"2024-09-21T18:55:06.432959+0000","last_clean":"2024-09-21T18:55:06.432959+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:55:06.432959+0000","last_undegraded":"2024-09-21T18:55:06.432959+0000","last_fullsized":"2024-09-21T18:55:06.432959+0000","mapping_epoch":79,"log_start":"87'1200","ondisk_log_start":"87'1200","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.053908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352518144,"num_objects":5379,"num_object_clones":0,"num_object_copies":16137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5379,"num_write_kb":344256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":118,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488663+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:55:05.488663+0000","last_peered":"2024-09-21T18:55:05.488663+0000","last_clean":"2024-09-21T18:55:05.488663+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:55:05.488663+0000","last_undegraded":"2024-09-21T18:55:05.488663+0000","last_fullsized":"2024-09-21T18:55:05.488663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.360653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":58,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486373+0000","last_change":"2024-09-21T18:54:17.734277+0000","last_active":"2024-09-21T18:55:05.486373+0000","last_peered":"2024-09-21T18:55:05.486373+0000","last_clean":"2024-09-21T18:55:05.486373+0000","last_became_active":"2024-09-21T18:54:17.721086+0000","last_became_peered":"2024-09-21T18:54:17.721086+0000","last_unstale":"2024-09-21T18:55:05.486373+0000","last_undegraded":"2024-09-21T18:55:05.486373+0000","last_fullsized":"2024-09-21T18:55:05.486373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.369107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":118,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489198+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:55:05.489198+0000","last_peered":"2024-09-21T18:55:05.489198+0000","last_clean":"2024-09-21T18:55:05.489198+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:55:05.489198+0000","last_undegraded":"2024-09-21T18:55:05.489198+0000","last_fullsized":"2024-09-21T18:55:05.489198+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.132673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490082+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:55:05.490082+0000","last_peered":"2024-09-21T18:55:05.490082+0000","last_clean":"2024-09-21T18:55:05.490082+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:55:05.490082+0000","last_undegraded":"2024-09-21T18:55:05.490082+0000","last_fullsized":"2024-09-21T18:55:05.490082+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"99'5261","reported_seq":5312,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.346687+0000","last_change":"2024-09-21T18:54:14.452896+0000","last_active":"2024-09-21T18:55:06.346687+0000","last_peered":"2024-09-21T18:55:06.346687+0000","last_clean":"2024-09-21T18:55:06.346687+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:55:06.346687+0000","last_undegraded":"2024-09-21T18:55:06.346687+0000","last_fullsized":"2024-09-21T18:55:06.346687+0000","mapping_epoch":79,"log_start":"86'700","ondisk_log_start":"86'700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.308130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344784896,"num_objects":5261,"num_object_clones":0,"num_object_copies":15783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":336704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":104,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486835+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:55:05.486835+0000","last_peered":"2024-09-21T18:55:05.486835+0000","last_clean":"2024-09-21T18:55:05.486835+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:55:05.486835+0000","last_undegraded":"2024-09-21T18:55:05.486835+0000","last_fullsized":"2024-09-21T18:55:05.486835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.356779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492100+0000","last_change":"2024-09-21T18:54:15.230912+0000","last_active":"2024-09-21T18:55:05.492100+0000","last_peered":"2024-09-21T18:55:05.492100+0000","last_clean":"2024-09-21T18:55:05.492100+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:55:05.492100+0000","last_undegraded":"2024-09-21T18:55:05.492100+0000","last_fullsized":"2024-09-21T18:55:05.492100+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.230861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.230861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.698873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"83'9","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.494819+0000","last_change":"2024-09-21T18:54:41.272637+0000","last_active":"2024-09-21T18:55:05.494819+0000","last_peered":"2024-09-21T18:55:05.494819+0000","last_clean":"2024-09-21T18:55:05.494819+0000","last_became_active":"2024-09-21T18:54:17.713489+0000","last_became_peered":"2024-09-21T18:54:17.713489+0000","last_unstale":"2024-09-21T18:55:05.494819+0000","last_undegraded":"2024-09-21T18:55:05.494819+0000","last_fullsized":"2024-09-21T18:55:05.494819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:47.058473+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487510+0000","last_change":"2024-09-21T18:54:17.714235+0000","last_active":"2024-09-21T18:55:05.487510+0000","last_peered":"2024-09-21T18:55:05.487510+0000","last_clean":"2024-09-21T18:55:05.487510+0000","last_became_active":"2024-09-21T18:54:17.713933+0000","last_became_peered":"2024-09-21T18:54:17.713933+0000","last_unstale":"2024-09-21T18:55:05.487510+0000","last_undegraded":"2024-09-21T18:55:05.487510+0000","last_fullsized":"2024-09-21T18:55:05.487510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.977688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"99'5364","reported_seq":5413,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.387106+0000","last_change":"2024-09-21T18:54:14.451912+0000","last_active":"2024-09-21T18:55:06.387106+0000","last_peered":"2024-09-21T18:55:06.387106+0000","last_clean":"2024-09-21T18:55:06.387106+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:55:06.387106+0000","last_undegraded":"2024-09-21T18:55:06.387106+0000","last_fullsized":"2024-09-21T18:55:06.387106+0000","mapping_epoch":79,"log_start":"85'400","ondisk_log_start":"85'400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:29.638453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351535104,"num_objects":5364,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":343296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":119,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491703+0000","last_change":"2024-09-21T18:54:17.730752+0000","last_active":"2024-09-21T18:55:05.491703+0000","last_peered":"2024-09-21T18:55:05.491703+0000","last_clean":"2024-09-21T18:55:05.491703+0000","last_became_active":"2024-09-21T18:54:17.711274+0000","last_became_peered":"2024-09-21T18:54:17.711274+0000","last_unstale":"2024-09-21T18:55:05.491703+0000","last_undegraded":"2024-09-21T18:55:05.491703+0000","last_fullsized":"2024-09-21T18:55:05.491703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.213832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489044+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:55:05.489044+0000","last_peered":"2024-09-21T18:55:05.489044+0000","last_clean":"2024-09-21T18:55:05.489044+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:55:05.489044+0000","last_undegraded":"2024-09-21T18:55:05.489044+0000","last_fullsized":"2024-09-21T18:55:05.489044+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.637938+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":91,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.493856+0000","last_change":"2024-09-21T18:54:17.710974+0000","last_active":"2024-09-21T18:55:05.493856+0000","last_peered":"2024-09-21T18:55:05.493856+0000","last_clean":"2024-09-21T18:55:05.493856+0000","last_became_active":"2024-09-21T18:54:17.710662+0000","last_became_peered":"2024-09-21T18:54:17.710662+0000","last_unstale":"2024-09-21T18:55:05.493856+0000","last_undegraded":"2024-09-21T18:55:05.493856+0000","last_fullsized":"2024-09-21T18:55:05.493856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.222043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487264+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:55:05.487264+0000","last_peered":"2024-09-21T18:55:05.487264+0000","last_clean":"2024-09-21T18:55:05.487264+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:55:05.487264+0000","last_undegraded":"2024-09-21T18:55:05.487264+0000","last_fullsized":"2024-09-21T18:55:05.487264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.362211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"99'5296","reported_seq":5354,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.368248+0000","last_change":"2024-09-21T18:54:17.707215+0000","last_active":"2024-09-21T18:55:06.368248+0000","last_peered":"2024-09-21T18:55:06.368248+0000","last_clean":"2024-09-21T18:55:06.368248+0000","last_became_active":"2024-09-21T18:54:17.706866+0000","last_became_peered":"2024-09-21T18:54:17.706866+0000","last_unstale":"2024-09-21T18:55:06.368248+0000","last_undegraded":"2024-09-21T18:55:06.368248+0000","last_fullsized":"2024-09-21T18:55:06.368248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.130355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347078656,"num_objects":5296,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5296,"num_write_kb":338944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":169,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488869+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:55:05.488869+0000","last_peered":"2024-09-21T18:55:05.488869+0000","last_clean":"2024-09-21T18:55:05.488869+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:55:05.488869+0000","last_undegraded":"2024-09-21T18:55:05.488869+0000","last_fullsized":"2024-09-21T18:55:05.488869+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:53.227258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":108,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488468+0000","last_change":"2024-09-21T18:54:17.706922+0000","last_active":"2024-09-21T18:55:05.488468+0000","last_peered":"2024-09-21T18:55:05.488468+0000","last_clean":"2024-09-21T18:55:05.488468+0000","last_became_active":"2024-09-21T18:54:17.706635+0000","last_became_peered":"2024-09-21T18:54:17.706635+0000","last_unstale":"2024-09-21T18:55:05.488468+0000","last_undegraded":"2024-09-21T18:55:05.488468+0000","last_fullsized":"2024-09-21T18:55:05.488468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:09.963263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:09.963263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.094847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":111,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491198+0000","last_change":"2024-09-21T18:54:47.179942+0000","last_active":"2024-09-21T18:55:05.491198+0000","last_peered":"2024-09-21T18:55:05.491198+0000","last_clean":"2024-09-21T18:55:05.491198+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:55:05.491198+0000","last_undegraded":"2024-09-21T18:55:05.491198+0000","last_fullsized":"2024-09-21T18:55:05.491198+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:54:47.179869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.179869+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:50.966329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":53,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486468+0000","last_change":"2024-09-21T18:54:41.369741+0000","last_active":"2024-09-21T18:55:05.486468+0000","last_peered":"2024-09-21T18:55:05.486468+0000","last_clean":"2024-09-21T18:55:05.486468+0000","last_became_active":"2024-09-21T18:54:17.712729+0000","last_became_peered":"2024-09-21T18:54:17.712729+0000","last_unstale":"2024-09-21T18:55:05.486468+0000","last_undegraded":"2024-09-21T18:55:05.486468+0000","last_fullsized":"2024-09-21T18:55:05.486468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:55.719422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488280+0000","last_change":"2024-09-21T18:53:13.428232+0000","last_active":"2024-09-21T18:55:05.488280+0000","last_peered":"2024-09-21T18:55:05.488280+0000","last_clean":"2024-09-21T18:55:05.488280+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:55:05.488280+0000","last_undegraded":"2024-09-21T18:55:05.488280+0000","last_fullsized":"2024-09-21T18:55:05.488280+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"99'5354","reported_seq":5400,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.331699+0000","last_change":"2024-09-21T18:54:17.706551+0000","last_active":"2024-09-21T18:55:06.331699+0000","last_peered":"2024-09-21T18:55:06.331699+0000","last_clean":"2024-09-21T18:55:06.331699+0000","last_became_active":"2024-09-21T18:54:17.706169+0000","last_became_peered":"2024-09-21T18:54:17.706169+0000","last_unstale":"2024-09-21T18:55:06.331699+0000","last_undegraded":"2024-09-21T18:55:06.331699+0000","last_fullsized":"2024-09-21T18:55:06.331699+0000","mapping_epoch":82,"log_start":"86'800","ondisk_log_start":"86'800","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.094421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350879744,"num_objects":5354,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":342656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":50,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490556+0000","last_change":"2024-09-21T18:54:40.059601+0000","last_active":"2024-09-21T18:55:05.490556+0000","last_peered":"2024-09-21T18:55:05.490556+0000","last_clean":"2024-09-21T18:55:05.490556+0000","last_became_active":"2024-09-21T18:54:36.751672+0000","last_became_peered":"2024-09-21T18:54:36.751672+0000","last_unstale":"2024-09-21T18:55:05.490556+0000","last_undegraded":"2024-09-21T18:55:05.490556+0000","last_fullsized":"2024-09-21T18:55:05.490556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.628526+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486832+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:55:05.486832+0000","last_peered":"2024-09-21T18:55:05.486832+0000","last_clean":"2024-09-21T18:55:05.486832+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:55:05.486832+0000","last_undegraded":"2024-09-21T18:55:05.486832+0000","last_fullsized":"2024-09-21T18:55:05.486832+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":113,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490425+0000","last_change":"2024-09-21T18:54:17.718825+0000","last_active":"2024-09-21T18:55:05.490425+0000","last_peered":"2024-09-21T18:55:05.490425+0000","last_clean":"2024-09-21T18:55:05.490425+0000","last_became_active":"2024-09-21T18:54:17.710632+0000","last_became_peered":"2024-09-21T18:54:17.710632+0000","last_unstale":"2024-09-21T18:55:05.490425+0000","last_undegraded":"2024-09-21T18:55:05.490425+0000","last_fullsized":"2024-09-21T18:55:05.490425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:53:05.370660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.946937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491252+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:55:05.491252+0000","last_peered":"2024-09-21T18:55:05.491252+0000","last_clean":"2024-09-21T18:55:05.491252+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:55:05.491252+0000","last_undegraded":"2024-09-21T18:55:05.491252+0000","last_fullsized":"2024-09-21T18:55:05.491252+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.969938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491668+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:55:05.491668+0000","last_peered":"2024-09-21T18:55:05.491668+0000","last_clean":"2024-09-21T18:55:05.491668+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:55:05.491668+0000","last_undegraded":"2024-09-21T18:55:05.491668+0000","last_fullsized":"2024-09-21T18:55:05.491668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:29.965290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"99'5382","reported_seq":5440,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.324123+0000","last_change":"2024-09-21T18:54:17.717837+0000","last_active":"2024-09-21T18:55:06.324123+0000","last_peered":"2024-09-21T18:55:06.324123+0000","last_clean":"2024-09-21T18:55:06.324123+0000","last_became_active":"2024-09-21T18:54:17.717525+0000","last_became_peered":"2024-09-21T18:54:17.717525+0000","last_unstale":"2024-09-21T18:55:06.324123+0000","last_undegraded":"2024-09-21T18:55:06.324123+0000","last_fullsized":"2024-09-21T18:55:06.324123+0000","mapping_epoch":82,"log_start":"85'600","ondisk_log_start":"85'600","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.870954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352714752,"num_objects":5382,"num_object_clones":0,"num_object_copies":16146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5382,"num_write_kb":344448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":97,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486251+0000","last_change":"2024-09-21T18:54:17.731920+0000","last_active":"2024-09-21T18:55:05.486251+0000","last_peered":"2024-09-21T18:55:05.486251+0000","last_clean":"2024-09-21T18:55:05.486251+0000","last_became_active":"2024-09-21T18:54:17.720574+0000","last_became_peered":"2024-09-21T18:54:17.720574+0000","last_unstale":"2024-09-21T18:55:05.486251+0000","last_undegraded":"2024-09-21T18:55:05.486251+0000","last_fullsized":"2024-09-21T18:55:05.486251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.300639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488593+0000","last_change":"2024-09-21T18:54:17.704631+0000","last_active":"2024-09-21T18:55:05.488593+0000","last_peered":"2024-09-21T18:55:05.488593+0000","last_clean":"2024-09-21T18:55:05.488593+0000","last_became_active":"2024-09-21T18:54:17.704296+0000","last_became_peered":"2024-09-21T18:54:17.704296+0000","last_unstale":"2024-09-21T18:55:05.488593+0000","last_undegraded":"2024-09-21T18:55:05.488593+0000","last_fullsized":"2024-09-21T18:55:05.488593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:20.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":236,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489811+0000","last_change":"2024-09-21T18:54:41.110834+0000","last_active":"2024-09-21T18:55:05.489811+0000","last_peered":"2024-09-21T18:55:05.489811+0000","last_clean":"2024-09-21T18:55:05.489811+0000","last_became_active":"2024-09-21T18:54:17.713703+0000","last_became_peered":"2024-09-21T18:54:17.713703+0000","last_unstale":"2024-09-21T18:55:05.489811+0000","last_undegraded":"2024-09-21T18:55:05.489811+0000","last_fullsized":"2024-09-21T18:55:05.489811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:41.110776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.110776+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.027705+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492447+0000","last_change":"2024-09-21T18:54:40.112915+0000","last_active":"2024-09-21T18:55:05.492447+0000","last_peered":"2024-09-21T18:55:05.492447+0000","last_clean":"2024-09-21T18:55:05.492447+0000","last_became_active":"2024-09-21T18:54:17.707817+0000","last_became_peered":"2024-09-21T18:54:17.707817+0000","last_unstale":"2024-09-21T18:55:05.492447+0000","last_undegraded":"2024-09-21T18:55:05.492447+0000","last_fullsized":"2024-09-21T18:55:05.492447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.112826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.112826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:08.711823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486200+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:55:05.486200+0000","last_peered":"2024-09-21T18:55:05.486200+0000","last_clean":"2024-09-21T18:55:05.486200+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:55:05.486200+0000","last_undegraded":"2024-09-21T18:55:05.486200+0000","last_fullsized":"2024-09-21T18:55:05.486200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:01.474112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"99'5188","reported_seq":5234,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.380054+0000","last_change":"2024-09-21T18:54:17.719358+0000","last_active":"2024-09-21T18:55:06.380054+0000","last_peered":"2024-09-21T18:55:06.380054+0000","last_clean":"2024-09-21T18:55:06.380054+0000","last_became_active":"2024-09-21T18:54:17.718730+0000","last_became_peered":"2024-09-21T18:54:17.718730+0000","last_unstale":"2024-09-21T18:55:06.380054+0000","last_undegraded":"2024-09-21T18:55:06.380054+0000","last_fullsized":"2024-09-21T18:55:06.380054+0000","mapping_epoch":82,"log_start":"85'400","ondisk_log_start":"85'400","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4788,"log_dups_size":0,"ondisk_log_size":4788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.846251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":332032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":134,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.495075+0000","last_change":"2024-09-21T18:53:09.719368+0000","last_active":"2024-09-21T18:55:05.495075+0000","last_peered":"2024-09-21T18:55:05.495075+0000","last_clean":"2024-09-21T18:55:05.495075+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:55:05.495075+0000","last_undegraded":"2024-09-21T18:55:05.495075+0000","last_fullsized":"2024-09-21T18:55:05.495075+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.249764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486160+0000","last_change":"2024-09-21T18:53:07.398998+0000","last_active":"2024-09-21T18:55:05.486160+0000","last_peered":"2024-09-21T18:55:05.486160+0000","last_clean":"2024-09-21T18:55:05.486160+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:55:05.486160+0000","last_undegraded":"2024-09-21T18:55:05.486160+0000","last_fullsized":"2024-09-21T18:55:05.486160+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.929847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":151,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490002+0000","last_change":"2024-09-21T18:54:17.736002+0000","last_active":"2024-09-21T18:55:05.490002+0000","last_peered":"2024-09-21T18:55:05.490002+0000","last_clean":"2024-09-21T18:55:05.490002+0000","last_became_active":"2024-09-21T18:54:17.716544+0000","last_became_peered":"2024-09-21T18:54:17.716544+0000","last_unstale":"2024-09-21T18:55:05.490002+0000","last_undegraded":"2024-09-21T18:55:05.490002+0000","last_fullsized":"2024-09-21T18:55:05.490002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":84,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490177+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:55:05.490177+0000","last_peered":"2024-09-21T18:55:05.490177+0000","last_clean":"2024-09-21T18:55:05.490177+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:55:05.490177+0000","last_undegraded":"2024-09-21T18:55:05.490177+0000","last_fullsized":"2024-09-21T18:55:05.490177+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492353+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:55:05.492353+0000","last_peered":"2024-09-21T18:55:05.492353+0000","last_clean":"2024-09-21T18:55:05.492353+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:55:05.492353+0000","last_undegraded":"2024-09-21T18:55:05.492353+0000","last_fullsized":"2024-09-21T18:55:05.492353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"99'5311","reported_seq":5368,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.397148+0000","last_change":"2024-09-21T18:54:17.710054+0000","last_active":"2024-09-21T18:55:06.397148+0000","last_peered":"2024-09-21T18:55:06.397148+0000","last_clean":"2024-09-21T18:55:06.397148+0000","last_became_active":"2024-09-21T18:54:17.709698+0000","last_became_peered":"2024-09-21T18:54:17.709698+0000","last_unstale":"2024-09-21T18:55:06.397148+0000","last_undegraded":"2024-09-21T18:55:06.397148+0000","last_fullsized":"2024-09-21T18:55:06.397148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.567144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348061696,"num_objects":5311,"num_object_clones":0,"num_object_copies":15933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5311,"num_write_kb":339904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":108,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488217+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:55:05.488217+0000","last_peered":"2024-09-21T18:55:05.488217+0000","last_clean":"2024-09-21T18:55:05.488217+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:55:05.488217+0000","last_undegraded":"2024-09-21T18:55:05.488217+0000","last_fullsized":"2024-09-21T18:55:05.488217+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.929416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488818+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:55:05.488818+0000","last_peered":"2024-09-21T18:55:05.488818+0000","last_clean":"2024-09-21T18:55:05.488818+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:55:05.488818+0000","last_undegraded":"2024-09-21T18:55:05.488818+0000","last_fullsized":"2024-09-21T18:55:05.488818+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.012525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492288+0000","last_change":"2024-09-21T18:54:50.139865+0000","last_active":"2024-09-21T18:55:05.492288+0000","last_peered":"2024-09-21T18:55:05.492288+0000","last_clean":"2024-09-21T18:55:05.492288+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:55:05.492288+0000","last_undegraded":"2024-09-21T18:55:05.492288+0000","last_fullsized":"2024-09-21T18:55:05.492288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.139807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.139807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.360746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.485544+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:55:05.485544+0000","last_peered":"2024-09-21T18:55:05.485544+0000","last_clean":"2024-09-21T18:55:05.485544+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:55:05.485544+0000","last_undegraded":"2024-09-21T18:55:05.485544+0000","last_fullsized":"2024-09-21T18:55:05.485544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"99'5224","reported_seq":5282,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.362500+0000","last_change":"2024-09-21T18:54:17.708465+0000","last_active":"2024-09-21T18:55:06.362500+0000","last_peered":"2024-09-21T18:55:06.362500+0000","last_clean":"2024-09-21T18:55:06.362500+0000","last_became_active":"2024-09-21T18:54:17.707839+0000","last_became_peered":"2024-09-21T18:54:17.707839+0000","last_unstale":"2024-09-21T18:55:06.362500+0000","last_undegraded":"2024-09-21T18:55:06.362500+0000","last_fullsized":"2024-09-21T18:55:06.362500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.899478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342360064,"num_objects":5224,"num_object_clones":0,"num_object_copies":15672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5224,"num_write_kb":334336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":117,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488204+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:55:05.488204+0000","last_peered":"2024-09-21T18:55:05.488204+0000","last_clean":"2024-09-21T18:55:05.488204+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:55:05.488204+0000","last_undegraded":"2024-09-21T18:55:05.488204+0000","last_fullsized":"2024-09-21T18:55:05.488204+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.077501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.494323+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:55:05.494323+0000","last_peered":"2024-09-21T18:55:05.494323+0000","last_clean":"2024-09-21T18:55:05.494323+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:55:05.494323+0000","last_undegraded":"2024-09-21T18:55:05.494323+0000","last_fullsized":"2024-09-21T18:55:05.494323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:24.289015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486295+0000","last_change":"2024-09-21T18:54:17.714450+0000","last_active":"2024-09-21T18:55:05.486295+0000","last_peered":"2024-09-21T18:55:05.486295+0000","last_clean":"2024-09-21T18:55:05.486295+0000","last_became_active":"2024-09-21T18:54:17.714167+0000","last_became_peered":"2024-09-21T18:54:17.714167+0000","last_unstale":"2024-09-21T18:55:05.486295+0000","last_undegraded":"2024-09-21T18:55:05.486295+0000","last_fullsized":"2024-09-21T18:55:05.486295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490691+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:55:05.490691+0000","last_peered":"2024-09-21T18:55:05.490691+0000","last_clean":"2024-09-21T18:55:05.490691+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:55:05.490691+0000","last_undegraded":"2024-09-21T18:55:05.490691+0000","last_fullsized":"2024-09-21T18:55:05.490691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:50.896923+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"99'5280","reported_seq":5325,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.431231+0000","last_change":"2024-09-21T18:54:17.718909+0000","last_active":"2024-09-21T18:55:06.431231+0000","last_peered":"2024-09-21T18:55:06.431231+0000","last_clean":"2024-09-21T18:55:06.431231+0000","last_became_active":"2024-09-21T18:54:17.718631+0000","last_became_peered":"2024-09-21T18:54:17.718631+0000","last_unstale":"2024-09-21T18:55:06.431231+0000","last_undegraded":"2024-09-21T18:55:06.431231+0000","last_fullsized":"2024-09-21T18:55:06.431231+0000","mapping_epoch":82,"log_start":"87'1100","ondisk_log_start":"87'1100","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:32.749545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346030080,"num_objects":5280,"num_object_clones":0,"num_object_copies":15840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5280,"num_write_kb":337920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":104,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487556+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:55:05.487556+0000","last_peered":"2024-09-21T18:55:05.487556+0000","last_clean":"2024-09-21T18:55:05.487556+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:55:05.487556+0000","last_undegraded":"2024-09-21T18:55:05.487556+0000","last_fullsized":"2024-09-21T18:55:05.487556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.102897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":119,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490732+0000","last_change":"2024-09-21T18:54:17.724937+0000","last_active":"2024-09-21T18:55:05.490732+0000","last_peered":"2024-09-21T18:55:05.490732+0000","last_clean":"2024-09-21T18:55:05.490732+0000","last_became_active":"2024-09-21T18:54:17.710657+0000","last_became_peered":"2024-09-21T18:54:17.710657+0000","last_unstale":"2024-09-21T18:55:05.490732+0000","last_undegraded":"2024-09-21T18:55:05.490732+0000","last_fullsized":"2024-09-21T18:55:05.490732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.355187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":117,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492489+0000","last_change":"2024-09-21T18:53:11.413473+0000","last_active":"2024-09-21T18:55:05.492489+0000","last_peered":"2024-09-21T18:55:05.492489+0000","last_clean":"2024-09-21T18:55:05.492489+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:55:05.492489+0000","last_undegraded":"2024-09-21T18:55:05.492489+0000","last_fullsized":"2024-09-21T18:55:05.492489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.995859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488093+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:55:05.488093+0000","last_peered":"2024-09-21T18:55:05.488093+0000","last_clean":"2024-09-21T18:55:05.488093+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:55:05.488093+0000","last_undegraded":"2024-09-21T18:55:05.488093+0000","last_fullsized":"2024-09-21T18:55:05.488093+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:04.802370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"99'5257","reported_seq":5308,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.363779+0000","last_change":"2024-09-21T18:54:14.454300+0000","last_active":"2024-09-21T18:55:06.363779+0000","last_peered":"2024-09-21T18:55:06.363779+0000","last_clean":"2024-09-21T18:55:06.363779+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:55:06.363779+0000","last_undegraded":"2024-09-21T18:55:06.363779+0000","last_fullsized":"2024-09-21T18:55:06.363779+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.683811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344522752,"num_objects":5257,"num_object_clones":0,"num_object_copies":15771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":336448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487401+0000","last_change":"2024-09-21T18:54:52.386165+0000","last_active":"2024-09-21T18:55:05.487401+0000","last_peered":"2024-09-21T18:55:05.487401+0000","last_clean":"2024-09-21T18:55:05.487401+0000","last_became_active":"2024-09-21T18:54:17.714624+0000","last_became_peered":"2024-09-21T18:54:17.714624+0000","last_unstale":"2024-09-21T18:55:05.487401+0000","last_undegraded":"2024-09-21T18:55:05.487401+0000","last_fullsized":"2024-09-21T18:55:05.487401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.346786+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":101,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491616+0000","last_change":"2024-09-21T18:54:13.280500+0000","last_active":"2024-09-21T18:55:05.491616+0000","last_peered":"2024-09-21T18:55:05.491616+0000","last_clean":"2024-09-21T18:55:05.491616+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:55:05.491616+0000","last_undegraded":"2024-09-21T18:55:05.491616+0000","last_fullsized":"2024-09-21T18:55:05.491616+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.280431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.280431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.151455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491216+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:55:05.491216+0000","last_peered":"2024-09-21T18:55:05.491216+0000","last_clean":"2024-09-21T18:55:05.491216+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:55:05.491216+0000","last_undegraded":"2024-09-21T18:55:05.491216+0000","last_fullsized":"2024-09-21T18:55:05.491216+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:49.052101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489673+0000","last_change":"2024-09-21T18:54:17.717827+0000","last_active":"2024-09-21T18:55:05.489673+0000","last_peered":"2024-09-21T18:55:05.489673+0000","last_clean":"2024-09-21T18:55:05.489673+0000","last_became_active":"2024-09-21T18:54:17.717507+0000","last_became_peered":"2024-09-21T18:54:17.717507+0000","last_unstale":"2024-09-21T18:55:05.489673+0000","last_undegraded":"2024-09-21T18:55:05.489673+0000","last_fullsized":"2024-09-21T18:55:05.489673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.193259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"99'5362","reported_seq":5407,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.414370+0000","last_change":"2024-09-21T18:54:17.712034+0000","last_active":"2024-09-21T18:55:06.414370+0000","last_peered":"2024-09-21T18:55:06.414370+0000","last_clean":"2024-09-21T18:55:06.414370+0000","last_became_active":"2024-09-21T18:54:17.711698+0000","last_became_peered":"2024-09-21T18:54:17.711698+0000","last_unstale":"2024-09-21T18:55:06.414370+0000","last_undegraded":"2024-09-21T18:55:06.414370+0000","last_fullsized":"2024-09-21T18:55:06.414370+0000","mapping_epoch":82,"log_start":"87'1100","ondisk_log_start":"87'1100","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.183316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351404032,"num_objects":5362,"num_object_clones":0,"num_object_copies":16086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":343168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":119,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.494051+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:55:05.494051+0000","last_peered":"2024-09-21T18:55:05.494051+0000","last_clean":"2024-09-21T18:55:05.494051+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:55:05.494051+0000","last_undegraded":"2024-09-21T18:55:05.494051+0000","last_fullsized":"2024-09-21T18:55:05.494051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.210774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491968+0000","last_change":"2024-09-21T18:53:07.406676+0000","last_active":"2024-09-21T18:55:05.491968+0000","last_peered":"2024-09-21T18:55:05.491968+0000","last_clean":"2024-09-21T18:55:05.491968+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:55:05.491968+0000","last_undegraded":"2024-09-21T18:55:05.491968+0000","last_fullsized":"2024-09-21T18:55:05.491968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.128759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":92,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490349+0000","last_change":"2024-09-21T18:54:17.717687+0000","last_active":"2024-09-21T18:55:05.490349+0000","last_peered":"2024-09-21T18:55:05.490349+0000","last_clean":"2024-09-21T18:55:05.490349+0000","last_became_active":"2024-09-21T18:54:17.717380+0000","last_became_peered":"2024-09-21T18:54:17.717380+0000","last_unstale":"2024-09-21T18:55:05.490349+0000","last_undegraded":"2024-09-21T18:55:05.490349+0000","last_fullsized":"2024-09-21T18:55:05.490349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.963253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489837+0000","last_change":"2024-09-21T18:53:13.428783+0000","last_active":"2024-09-21T18:55:05.489837+0000","last_peered":"2024-09-21T18:55:05.489837+0000","last_clean":"2024-09-21T18:55:05.489837+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:55:05.489837+0000","last_undegraded":"2024-09-21T18:55:05.489837+0000","last_fullsized":"2024-09-21T18:55:05.489837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:18.116478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"99'5152","reported_seq":5203,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.446401+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:55:06.446401+0000","last_peered":"2024-09-21T18:55:06.446401+0000","last_clean":"2024-09-21T18:55:06.446401+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:55:06.446401+0000","last_undegraded":"2024-09-21T18:55:06.446401+0000","last_fullsized":"2024-09-21T18:55:06.446401+0000","mapping_epoch":79,"log_start":"87'1000","ondisk_log_start":"87'1000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.423918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337641472,"num_objects":5152,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5152,"num_write_kb":329728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":139,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488739+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:55:05.488739+0000","last_peered":"2024-09-21T18:55:05.488739+0000","last_clean":"2024-09-21T18:55:05.488739+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:55:05.488739+0000","last_undegraded":"2024-09-21T18:55:05.488739+0000","last_fullsized":"2024-09-21T18:55:05.488739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:51.259913+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490464+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:55:05.490464+0000","last_peered":"2024-09-21T18:55:05.490464+0000","last_clean":"2024-09-21T18:55:05.490464+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:55:05.490464+0000","last_undegraded":"2024-09-21T18:55:05.490464+0000","last_fullsized":"2024-09-21T18:55:05.490464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:13.545717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490484+0000","last_change":"2024-09-21T18:54:17.708159+0000","last_active":"2024-09-21T18:55:05.490484+0000","last_peered":"2024-09-21T18:55:05.490484+0000","last_clean":"2024-09-21T18:55:05.490484+0000","last_became_active":"2024-09-21T18:54:17.707830+0000","last_became_peered":"2024-09-21T18:54:17.707830+0000","last_unstale":"2024-09-21T18:55:05.490484+0000","last_undegraded":"2024-09-21T18:55:05.490484+0000","last_fullsized":"2024-09-21T18:55:05.490484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:11.994472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.994472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:28.343603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489433+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:55:05.489433+0000","last_peered":"2024-09-21T18:55:05.489433+0000","last_clean":"2024-09-21T18:55:05.489433+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:55:05.489433+0000","last_undegraded":"2024-09-21T18:55:05.489433+0000","last_fullsized":"2024-09-21T18:55:05.489433+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"99'5206","reported_seq":5257,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:06.408068+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:55:06.408068+0000","last_peered":"2024-09-21T18:55:06.408068+0000","last_clean":"2024-09-21T18:55:06.408068+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:55:06.408068+0000","last_undegraded":"2024-09-21T18:55:06.408068+0000","last_fullsized":"2024-09-21T18:55:06.408068+0000","mapping_epoch":79,"log_start":"86'900","ondisk_log_start":"86'900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:21.519258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341180416,"num_objects":5206,"num_object_clones":0,"num_object_copies":15618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":333184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":122,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490910+0000","last_change":"2024-09-21T18:54:44.092986+0000","last_active":"2024-09-21T18:55:05.490910+0000","last_peered":"2024-09-21T18:55:05.490910+0000","last_clean":"2024-09-21T18:55:05.490910+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:55:05.490910+0000","last_undegraded":"2024-09-21T18:55:05.490910+0000","last_fullsized":"2024-09-21T18:55:05.490910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:44.092903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:44.092903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.777104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":117,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490763+0000","last_change":"2024-09-21T18:54:17.732357+0000","last_active":"2024-09-21T18:55:05.490763+0000","last_peered":"2024-09-21T18:55:05.490763+0000","last_clean":"2024-09-21T18:55:05.490763+0000","last_became_active":"2024-09-21T18:54:17.715268+0000","last_became_peered":"2024-09-21T18:54:17.715268+0000","last_unstale":"2024-09-21T18:55:05.490763+0000","last_undegraded":"2024-09-21T18:55:05.490763+0000","last_fullsized":"2024-09-21T18:55:05.490763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.049285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":119,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489781+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:55:05.489781+0000","last_peered":"2024-09-21T18:55:05.489781+0000","last_clean":"2024-09-21T18:55:05.489781+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:55:05.489781+0000","last_undegraded":"2024-09-21T18:55:05.489781+0000","last_fullsized":"2024-09-21T18:55:05.489781+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:26.946870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":96,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490934+0000","last_change":"2024-09-21T18:54:53.051485+0000","last_active":"2024-09-21T18:55:05.490934+0000","last_peered":"2024-09-21T18:55:05.490934+0000","last_clean":"2024-09-21T18:55:05.490934+0000","last_became_active":"2024-09-21T18:54:17.713977+0000","last_became_peered":"2024-09-21T18:54:17.713977+0000","last_unstale":"2024-09-21T18:55:05.490934+0000","last_undegraded":"2024-09-21T18:55:05.490934+0000","last_fullsized":"2024-09-21T18:55:05.490934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.375794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.494194+0000","last_change":"2024-09-21T18:54:17.713398+0000","last_active":"2024-09-21T18:55:05.494194+0000","last_peered":"2024-09-21T18:55:05.494194+0000","last_clean":"2024-09-21T18:55:05.494194+0000","last_became_active":"2024-09-21T18:54:17.713066+0000","last_became_peered":"2024-09-21T18:54:17.713066+0000","last_unstale":"2024-09-21T18:55:05.494194+0000","last_undegraded":"2024-09-21T18:55:05.494194+0000","last_fullsized":"2024-09-21T18:55:05.494194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:22.224570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489780+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:55:05.489780+0000","last_peered":"2024-09-21T18:55:05.489780+0000","last_clean":"2024-09-21T18:55:05.489780+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:55:05.489780+0000","last_undegraded":"2024-09-21T18:55:05.489780+0000","last_fullsized":"2024-09-21T18:55:05.489780+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:58.061399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491653+0000","last_change":"2024-09-21T18:53:11.408926+0000","last_active":"2024-09-21T18:55:05.491653+0000","last_peered":"2024-09-21T18:55:05.491653+0000","last_clean":"2024-09-21T18:55:05.491653+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:55:05.491653+0000","last_undegraded":"2024-09-21T18:55:05.491653+0000","last_fullsized":"2024-09-21T18:55:05.491653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:27.256441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491494+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:55:05.491494+0000","last_peered":"2024-09-21T18:55:05.491494+0000","last_clean":"2024-09-21T18:55:05.491494+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:55:05.491494+0000","last_undegraded":"2024-09-21T18:55:05.491494+0000","last_fullsized":"2024-09-21T18:55:05.491494+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":99,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488841+0000","last_change":"2024-09-21T18:54:51.117250+0000","last_active":"2024-09-21T18:55:05.488841+0000","last_peered":"2024-09-21T18:55:05.488841+0000","last_clean":"2024-09-21T18:55:05.488841+0000","last_became_active":"2024-09-21T18:54:17.719720+0000","last_became_peered":"2024-09-21T18:54:17.719720+0000","last_unstale":"2024-09-21T18:55:05.488841+0000","last_undegraded":"2024-09-21T18:55:05.488841+0000","last_fullsized":"2024-09-21T18:55:05.488841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.430477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490090+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:55:05.490090+0000","last_peered":"2024-09-21T18:55:05.490090+0000","last_clean":"2024-09-21T18:55:05.490090+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:55:05.490090+0000","last_undegraded":"2024-09-21T18:55:05.490090+0000","last_fullsized":"2024-09-21T18:55:05.490090+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.194383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":86,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489636+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:55:05.489636+0000","last_peered":"2024-09-21T18:55:05.489636+0000","last_clean":"2024-09-21T18:55:05.489636+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:55:05.489636+0000","last_undegraded":"2024-09-21T18:55:05.489636+0000","last_fullsized":"2024-09-21T18:55:05.489636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.003241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490328+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:55:05.490328+0000","last_peered":"2024-09-21T18:55:05.490328+0000","last_clean":"2024-09-21T18:55:05.490328+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:55:05.490328+0000","last_undegraded":"2024-09-21T18:55:05.490328+0000","last_fullsized":"2024-09-21T18:55:05.490328+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:53.334288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":105,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491974+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:55:05.491974+0000","last_peered":"2024-09-21T18:55:05.491974+0000","last_clean":"2024-09-21T18:55:05.491974+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:55:05.491974+0000","last_undegraded":"2024-09-21T18:55:05.491974+0000","last_fullsized":"2024-09-21T18:55:05.491974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.275983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489873+0000","last_change":"2024-09-21T18:54:17.717912+0000","last_active":"2024-09-21T18:55:05.489873+0000","last_peered":"2024-09-21T18:55:05.489873+0000","last_clean":"2024-09-21T18:55:05.489873+0000","last_became_active":"2024-09-21T18:54:17.717617+0000","last_became_peered":"2024-09-21T18:54:17.717617+0000","last_unstale":"2024-09-21T18:55:05.489873+0000","last_undegraded":"2024-09-21T18:55:05.489873+0000","last_fullsized":"2024-09-21T18:55:05.489873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":118,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.485665+0000","last_change":"2024-09-21T18:53:11.414447+0000","last_active":"2024-09-21T18:55:05.485665+0000","last_peered":"2024-09-21T18:55:05.485665+0000","last_clean":"2024-09-21T18:55:05.485665+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:55:05.485665+0000","last_undegraded":"2024-09-21T18:55:05.485665+0000","last_fullsized":"2024-09-21T18:55:05.485665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487803+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:55:05.487803+0000","last_peered":"2024-09-21T18:55:05.487803+0000","last_clean":"2024-09-21T18:55:05.487803+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:55:05.487803+0000","last_undegraded":"2024-09-21T18:55:05.487803+0000","last_fullsized":"2024-09-21T18:55:05.487803+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:14.128893+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":105,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.485960+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:55:05.485960+0000","last_peered":"2024-09-21T18:55:05.485960+0000","last_clean":"2024-09-21T18:55:05.485960+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:55:05.485960+0000","last_undegraded":"2024-09-21T18:55:05.485960+0000","last_fullsized":"2024-09-21T18:55:05.485960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:01.112400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490391+0000","last_change":"2024-09-21T18:54:40.089802+0000","last_active":"2024-09-21T18:55:05.490391+0000","last_peered":"2024-09-21T18:55:05.490391+0000","last_clean":"2024-09-21T18:55:05.490391+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:55:05.490391+0000","last_undegraded":"2024-09-21T18:55:05.490391+0000","last_fullsized":"2024-09-21T18:55:05.490391+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.089723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.089723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.272819+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"83'9","reported_seq":128,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491734+0000","last_change":"2024-09-21T18:54:17.726533+0000","last_active":"2024-09-21T18:55:05.491734+0000","last_peered":"2024-09-21T18:55:05.491734+0000","last_clean":"2024-09-21T18:55:05.491734+0000","last_became_active":"2024-09-21T18:54:17.718821+0000","last_became_peered":"2024-09-21T18:54:17.718821+0000","last_unstale":"2024-09-21T18:55:05.491734+0000","last_undegraded":"2024-09-21T18:55:05.491734+0000","last_fullsized":"2024-09-21T18:55:05.491734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:02.213185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490115+0000","last_change":"2024-09-21T18:54:17.719309+0000","last_active":"2024-09-21T18:55:05.490115+0000","last_peered":"2024-09-21T18:55:05.490115+0000","last_clean":"2024-09-21T18:55:05.490115+0000","last_became_active":"2024-09-21T18:54:17.718560+0000","last_became_peered":"2024-09-21T18:54:17.718560+0000","last_unstale":"2024-09-21T18:55:05.490115+0000","last_undegraded":"2024-09-21T18:55:05.490115+0000","last_fullsized":"2024-09-21T18:55:05.490115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:25.988782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":127,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487976+0000","last_change":"2024-09-21T18:54:36.773903+0000","last_active":"2024-09-21T18:55:05.487976+0000","last_peered":"2024-09-21T18:55:05.487976+0000","last_clean":"2024-09-21T18:55:05.487976+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:55:05.487976+0000","last_undegraded":"2024-09-21T18:55:05.487976+0000","last_fullsized":"2024-09-21T18:55:05.487976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.487662+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:55:05.487662+0000","last_peered":"2024-09-21T18:55:05.487662+0000","last_clean":"2024-09-21T18:55:05.487662+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:55:05.487662+0000","last_undegraded":"2024-09-21T18:55:05.487662+0000","last_fullsized":"2024-09-21T18:55:05.487662+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.963073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486712+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:55:05.486712+0000","last_peered":"2024-09-21T18:55:05.486712+0000","last_clean":"2024-09-21T18:55:05.486712+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:55:05.486712+0000","last_undegraded":"2024-09-21T18:55:05.486712+0000","last_fullsized":"2024-09-21T18:55:05.486712+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488343+0000","last_change":"2024-09-21T18:54:17.717220+0000","last_active":"2024-09-21T18:55:05.488343+0000","last_peered":"2024-09-21T18:55:05.488343+0000","last_clean":"2024-09-21T18:55:05.488343+0000","last_became_active":"2024-09-21T18:54:17.716004+0000","last_became_peered":"2024-09-21T18:54:17.716004+0000","last_unstale":"2024-09-21T18:55:05.488343+0000","last_undegraded":"2024-09-21T18:55:05.488343+0000","last_fullsized":"2024-09-21T18:55:05.488343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":105,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486912+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:55:05.486912+0000","last_peered":"2024-09-21T18:55:05.486912+0000","last_clean":"2024-09-21T18:55:05.486912+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:55:05.486912+0000","last_undegraded":"2024-09-21T18:55:05.486912+0000","last_fullsized":"2024-09-21T18:55:05.486912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.311285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":100,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491884+0000","last_change":"2024-09-21T18:54:52.054723+0000","last_active":"2024-09-21T18:55:05.491884+0000","last_peered":"2024-09-21T18:55:05.491884+0000","last_clean":"2024-09-21T18:55:05.491884+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:55:05.491884+0000","last_undegraded":"2024-09-21T18:55:05.491884+0000","last_fullsized":"2024-09-21T18:55:05.491884+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:52.054655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.054655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:17.905775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.495059+0000","last_change":"2024-09-21T18:54:17.715073+0000","last_active":"2024-09-21T18:55:05.495059+0000","last_peered":"2024-09-21T18:55:05.495059+0000","last_clean":"2024-09-21T18:55:05.495059+0000","last_became_active":"2024-09-21T18:54:17.714759+0000","last_became_peered":"2024-09-21T18:54:17.714759+0000","last_unstale":"2024-09-21T18:55:05.495059+0000","last_undegraded":"2024-09-21T18:55:05.495059+0000","last_fullsized":"2024-09-21T18:55:05.495059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.032847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490794+0000","last_change":"2024-09-21T18:54:17.712675+0000","last_active":"2024-09-21T18:55:05.490794+0000","last_peered":"2024-09-21T18:55:05.490794+0000","last_clean":"2024-09-21T18:55:05.490794+0000","last_became_active":"2024-09-21T18:54:17.712339+0000","last_became_peered":"2024-09-21T18:54:17.712339+0000","last_unstale":"2024-09-21T18:55:05.490794+0000","last_undegraded":"2024-09-21T18:55:05.490794+0000","last_fullsized":"2024-09-21T18:55:05.490794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":109,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.488493+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:55:05.488493+0000","last_peered":"2024-09-21T18:55:05.488493+0000","last_clean":"2024-09-21T18:55:05.488493+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:55:05.488493+0000","last_undegraded":"2024-09-21T18:55:05.488493+0000","last_fullsized":"2024-09-21T18:55:05.488493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.065143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.491767+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:55:05.491767+0000","last_peered":"2024-09-21T18:55:05.491767+0000","last_clean":"2024-09-21T18:55:05.491767+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:55:05.491767+0000","last_undegraded":"2024-09-21T18:55:05.491767+0000","last_fullsized":"2024-09-21T18:55:05.491767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.201819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":84,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489516+0000","last_change":"2024-09-21T18:53:11.412945+0000","last_active":"2024-09-21T18:55:05.489516+0000","last_peered":"2024-09-21T18:55:05.489516+0000","last_clean":"2024-09-21T18:55:05.489516+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:55:05.489516+0000","last_undegraded":"2024-09-21T18:55:05.489516+0000","last_fullsized":"2024-09-21T18:55:05.489516+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:23.082920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":89,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490631+0000","last_change":"2024-09-21T18:54:17.709035+0000","last_active":"2024-09-21T18:55:05.490631+0000","last_peered":"2024-09-21T18:55:05.490631+0000","last_clean":"2024-09-21T18:55:05.490631+0000","last_became_active":"2024-09-21T18:54:17.708626+0000","last_became_peered":"2024-09-21T18:54:17.708626+0000","last_unstale":"2024-09-21T18:55:05.490631+0000","last_undegraded":"2024-09-21T18:55:05.490631+0000","last_fullsized":"2024-09-21T18:55:05.490631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":109,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489651+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:55:05.489651+0000","last_peered":"2024-09-21T18:55:05.489651+0000","last_clean":"2024-09-21T18:55:05.489651+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:55:05.489651+0000","last_undegraded":"2024-09-21T18:55:05.489651+0000","last_fullsized":"2024-09-21T18:55:05.489651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.492538+0000","last_change":"2024-09-21T18:54:17.719435+0000","last_active":"2024-09-21T18:55:05.492538+0000","last_peered":"2024-09-21T18:55:05.492538+0000","last_clean":"2024-09-21T18:55:05.492538+0000","last_became_active":"2024-09-21T18:54:17.718927+0000","last_became_peered":"2024-09-21T18:54:17.718927+0000","last_unstale":"2024-09-21T18:55:05.492538+0000","last_undegraded":"2024-09-21T18:55:05.492538+0000","last_fullsized":"2024-09-21T18:55:05.492538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.142303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":84,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.495978+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:55:05.495978+0000","last_peered":"2024-09-21T18:55:05.495978+0000","last_clean":"2024-09-21T18:55:05.495978+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:55:05.495978+0000","last_undegraded":"2024-09-21T18:55:05.495978+0000","last_fullsized":"2024-09-21T18:55:05.495978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:19.250351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":80,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489665+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:55:05.489665+0000","last_peered":"2024-09-21T18:55:05.489665+0000","last_clean":"2024-09-21T18:55:05.489665+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:55:05.489665+0000","last_undegraded":"2024-09-21T18:55:05.489665+0000","last_fullsized":"2024-09-21T18:55:05.489665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.129027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":81,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.486364+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:55:05.486364+0000","last_peered":"2024-09-21T18:55:05.486364+0000","last_clean":"2024-09-21T18:55:05.486364+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:55:05.486364+0000","last_undegraded":"2024-09-21T18:55:05.486364+0000","last_fullsized":"2024-09-21T18:55:05.486364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:25.345750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":109,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.489935+0000","last_change":"2024-09-21T18:54:17.704421+0000","last_active":"2024-09-21T18:55:05.489935+0000","last_peered":"2024-09-21T18:55:05.489935+0000","last_clean":"2024-09-21T18:55:05.489935+0000","last_became_active":"2024-09-21T18:54:17.704166+0000","last_became_peered":"2024-09-21T18:54:17.704166+0000","last_unstale":"2024-09-21T18:55:05.489935+0000","last_undegraded":"2024-09-21T18:55:05.489935+0000","last_fullsized":"2024-09-21T18:55:05.489935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:13.526296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":107,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.495122+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:55:05.495122+0000","last_peered":"2024-09-21T18:55:05.495122+0000","last_clean":"2024-09-21T18:55:05.495122+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:55:05.495122+0000","last_undegraded":"2024-09-21T18:55:05.495122+0000","last_fullsized":"2024-09-21T18:55:05.495122+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:16.235721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T18:55:05.490276+0000","last_change":"2024-09-21T18:54:17.719032+0000","last_active":"2024-09-21T18:55:05.490276+0000","last_peered":"2024-09-21T18:55:05.490276+0000","last_clean":"2024-09-21T18:55:05.490276+0000","last_became_active":"2024-09-21T18:54:17.718245+0000","last_became_peered":"2024-09-21T18:54:17.718245+0000","last_unstale":"2024-09-21T18:55:05.490276+0000","last_undegraded":"2024-09-21T18:55:05.490276+0000","last_fullsized":"2024-09-21T18:55:05.490276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.074471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5535694848,"num_objects":84468,"num_object_clones":0,"num_object_copies":253404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84468,"num_write_kb":5405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16609181696,"data_stored":16609181696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75168,"ondisk_log_size":75168,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":2090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266716,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2081016,"kb_used_data":2017892,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91664136,"statfs":{"total":95995035648,"available":93864075264,"internally_reserved":0,"allocated":2066321408,"data_stored":2065709238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2052360,"kb_used_data":1989240,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91692792,"statfs":{"total":95995035648,"available":93893419008,"internally_reserved":0,"allocated":2036981760,"data_stored":2036406689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789988,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713436,"kb_used_data":1668360,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92031716,"statfs":{"total":95995035648,"available":94240477184,"internally_reserved":0,"allocated":1708400640,"data_stored":1707815813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018922,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088872,"kb_used_data":2021592,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91656280,"statfs":{"total":95995035648,"available":93856030720,"internally_reserved":0,"allocated":2070110208,"data_stored":2069509906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3768060,"kb_used_data":3682732,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":89977092,"statfs":{"total":95995035648,"available":92136542208,"internally_reserved":0,"allocated":3771117568,"data_stored":3770497584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":87357898},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092772,"kb_used_data":2017172,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91652380,"statfs":{"total":95995035648,"available":93852037120,"internally_reserved":0,"allocated":2065584128,"data_stored":2064987678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2763992,"kb_used_data":2688388,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":90981160,"statfs":{"total":95995035648,"available":93164707840,"internally_reserved":0,"allocated":2752909312,"data_stored":2752265986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2796224512,"data_stored":2796224512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2071986176,"data_stored":2071986176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3799187456,"data_stored":3799187456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2079850496,"data_stored":2079850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1712193536,"data_stored":1712193536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2074017792,"data_stored":2074017792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2075721728,"data_stored":2075721728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:55:09.727 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-21T18:55:09.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:09.786 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:09.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:09.920 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:09.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:09.999 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:10.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:10.068 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:10.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 49 16 89614 89598 114.226 88.75 0.0118999 0.00874975 2024-09-21T18:55:10.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:10.203 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:10.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:10.316 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:09 smithi081 podman[78889]: 2024-09-21 18:55:09.873727023 +0000 UTC m=+0.035892351 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T18:55:10.316 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:09 smithi081 podman[78889]: 2024-09-21 18:55:09.983414806 +0000 UTC m=+0.145580119 volume create 2005a17249674d689c7c7cc70f7f5490fcc2afee88e7625862dd1d440fc6505b 2024-09-21T18:55:10.316 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 podman[78889]: 2024-09-21 18:55:10.092528649 +0000 UTC m=+0.254693966 container create 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:10.340 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:10.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:10.473 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:10.575 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:10.593 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-mon[39192]: pgmap v76: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-21T18:55:10.593 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-mon[39192]: osdmap e100: 8 total, 8 up, 7 in 2024-09-21T18:55:10.593 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-mon[29384]: pgmap v76: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-21T18:55:10.593 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-mon[29384]: osdmap e100: 8 total, 8 up, 7 in 2024-09-21T18:55:10.594 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 podman[78889]: 2024-09-21 18:55:10.479605847 +0000 UTC m=+0.641771170 container init 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:10.594 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 podman[78889]: 2024-09-21 18:55:10.48433856 +0000 UTC m=+0.646503875 container start 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:10.594 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.505Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T18:55:10.594 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.505Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T18:55:10.594 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.507Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.81 port=9094 2024-09-21T18:55:10.594 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.511Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.549Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.549Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.551Z caller=main.go:394 level=info component=configuration msg="skipping creation of receiver not referenced by any route" receiver=custom-receiver 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.552Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:10.552Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 bash[78889]: 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 2024-09-21T18:55:10.595 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:10 smithi081 systemd[1]: Started Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:55:10.606 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:10.701 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:10.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:10.736 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:10.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:10 smithi104 ceph-mon[32438]: pgmap v76: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-21T18:55:10.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:10 smithi104 ceph-mon[32438]: osdmap e100: 8 total, 8 up, 7 in 2024-09-21T18:55:10.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:10.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:10.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:11.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:11.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 50 16 90933 90917 113.59 82.4375 0.015845 0.00879889 2024-09-21T18:55:11.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.292 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:11.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:11.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:11.436 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:11.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:11.564 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:11.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:11.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:11 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:11 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:11 smithi104 ceph-mon[32438]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T18:55:11.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:11.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:11 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:11.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:11 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:11.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:11 smithi081 ceph-mon[39192]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T18:55:11.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:11 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:11.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:11 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:11.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:11 smithi081 ceph-mon[29384]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T18:55:11.821 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:11.822 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:11.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:11.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:55:11.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:11.983 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:11.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:12.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:12.133 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:12.167 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 51 16 92184 92168 112.896 78.1875 0.0121492 0.00885311 2024-09-21T18:55:12.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:12.274 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:12.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:12.414 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:12.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:12.549 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:12.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T18:55:12.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:12.754 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:55:12.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[39192]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T18:55:12.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[39192]: pgmap v78: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T18:55:12.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3485042454' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:12.778 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[39192]: from='client.34817 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:12.778 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:12.512Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000875631s 2024-09-21T18:55:12.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[29384]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T18:55:12.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[29384]: pgmap v78: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T18:55:12.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3485042454' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:12.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:12 smithi081 ceph-mon[29384]: from='client.34817 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:12.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:12.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:12 smithi104 ceph-mon[32438]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T18:55:12.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:12 smithi104 ceph-mon[32438]: pgmap v78: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T18:55:12.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3485042454' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:12.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:12 smithi104 ceph-mon[32438]: from='client.34817 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:12.840 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:12.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:12.990 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:12.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:13.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:13.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:13.170 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 52 16 93505 93489 112.313 82.5625 0.0122436 0.00889875 2024-09-21T18:55:13.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:13.266 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:13.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:13.398 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:13.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:13.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:13.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:13.672 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:13.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.810 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:13.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:13.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:13.952 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:13.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:13 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:13 smithi104 ceph-mon[32438]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:13 smithi104 ceph-mon[32438]: osdmap e101: 8 total, 8 up, 7 in 2024-09-21T18:55:14.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:14.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:13 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:13 smithi081 ceph-mon[29384]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:13 smithi081 ceph-mon[29384]: osdmap e101: 8 total, 8 up, 7 in 2024-09-21T18:55:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:13 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:13 smithi081 ceph-mon[39192]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:13 smithi081 ceph-mon[39192]: osdmap e101: 8 total, 8 up, 7 in 2024-09-21T18:55:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:14.145 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:14.145 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:14.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 53 16 94847 94831 111.776 83.875 0.0109234 0.00894151 2024-09-21T18:55:14.182 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:14.182 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:14.184 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:14.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:14.228 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:14.228 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.241 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:14.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:14.261 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:14.262 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:14.278 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:14.278 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:14.295 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:14.295 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:14.327 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:14.327 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:14.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:14.360 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:14.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:14.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:14.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:14.621 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:14.657 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:14.729 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:55:14.729 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:55:14.729 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T18:55:14.729 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-09-21T18:55:14.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T18:55:14.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:14.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:14.861 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:14 smithi104 ceph-mon[32438]: pgmap v80: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-21T18:55:14.862 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3510061842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:14.862 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:14 smithi104 ceph-mon[32438]: osdmap e102: 8 total, 8 up, 7 in 2024-09-21T18:55:14.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:14.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:15.016 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:14 smithi081 ceph-mon[39192]: pgmap v80: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-21T18:55:15.016 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3510061842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:15.016 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:14 smithi081 ceph-mon[39192]: osdmap e102: 8 total, 8 up, 7 in 2024-09-21T18:55:15.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:14 smithi081 ceph-mon[29384]: pgmap v80: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-21T18:55:15.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3510061842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:15.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:14 smithi081 ceph-mon[29384]: osdmap e102: 8 total, 8 up, 7 in 2024-09-21T18:55:15.042 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:55:15.042 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:55:15.043 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:55:15.043 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:55:15.043 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:55:15.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.106 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:15.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:15.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 54 16 96087 96071 111.142 77.5 0.00659336 0.00899393 2024-09-21T18:55:15.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:15.257 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 systemd[1]: Stopping Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:55:15.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:15.381 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:15.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:15.519 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:15.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:15.662 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:15.711 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T18:55:15.711 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T18:55:15.711 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.416Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.418Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.418Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[58195]: ts=2024-09-21T18:55:15.418Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T18:55:15.712 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 podman[64871]: 2024-09-21 18:55:15.426607716 +0000 UTC m=+0.145893349 container died 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:15.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:15.794 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:15.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:15.901 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:55:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:15.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:16.006 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 podman[64871]: 2024-09-21 18:55:15.815760703 +0000 UTC m=+0.535046327 container cleanup 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:16.007 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:15 smithi104 bash[64871]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T18:55:16.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:16.067 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:16.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 55 16 97472 97456 110.695 86.5625 0.00683787 0.00902911 2024-09-21T18:55:16.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:16.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:16.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:16.342 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:16.345 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:16 smithi104 podman[64881]: 2024-09-21 18:55:16.092366148 +0000 UTC m=+0.656208432 container remove 5ac65de841da5b42b7b179cdd3dc8408be948ac7222d960248514250f8a4df50 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:16.345 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:16 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T18:55:16.345 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:16 smithi104 systemd[1]: Stopped Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:55:16.345 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:16 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.288s CPU time. 2024-09-21T18:55:16.345 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:16 smithi104 systemd[1]: Starting Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T18:55:16.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:16.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T18:55:16.513 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:16.596 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:16.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:16.641 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:16.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:16.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:16.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:16.952 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:16.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:17.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:16 smithi104 ceph-mon[32438]: pgmap v82: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-21T18:55:17.007 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:16 smithi104 podman[65048]: 2024-09-21 18:55:16.518814145 +0000 UTC m=+0.025125790 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T18:55:17.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:17.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:16 smithi081 ceph-mon[29384]: pgmap v82: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-21T18:55:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:16 smithi081 ceph-mon[39192]: pgmap v82: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-21T18:55:17.073 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:17.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 56 16 98654 98638 110.037 73.875 0.00647588 0.00908374 2024-09-21T18:55:17.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:17.205 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:17.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:17.342 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:17.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:17.477 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:17.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:17.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.741 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:17.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:17.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:17.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:55:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:17.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:17.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:17.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:18.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:55:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:55:18.007 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:17 smithi104 podman[65048]: 2024-09-21 18:55:17.5527999 +0000 UTC m=+1.059111537 container create 7747803a501c1cd4b2e20b982793f87eb5dee4ba75f5a20a9b2872e4977ba154 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:18.045 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:18.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:18.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 57 15 99864 99849 109.435 75.6875 0.0182701 0.00913305 2024-09-21T18:55:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.199 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:18.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:18.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.341 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:18.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:18.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.488 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:18.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:18.589 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:18.628 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:18.650 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.653 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:18.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:18.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T18:55:18.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:18.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:18.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:18.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[29384]: pgmap v83: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T18:55:18.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:18.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[29384]: osdmap e103: 8 total, 8 up, 7 in 2024-09-21T18:55:18.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/113887006' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:55:18.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:18.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:18.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[39192]: pgmap v83: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T18:55:18.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:18.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[39192]: osdmap e103: 8 total, 8 up, 7 in 2024-09-21T18:55:18.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/113887006' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:55:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:18.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:18.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:18.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:19.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:19.085 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:19.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:18 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:19.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:18 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:19.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:18 smithi104 ceph-mon[32438]: pgmap v83: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T18:55:19.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:19.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:18 smithi104 ceph-mon[32438]: osdmap e103: 8 total, 8 up, 7 in 2024-09-21T18:55:19.153 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/113887006' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:55:19.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 58 16 101006 100990 108.778 71.3125 0.00994031 0.00918796 2024-09-21T18:55:19.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:19.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:19.363 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:19.411 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 podman[65048]: 2024-09-21 18:55:19.315121949 +0000 UTC m=+2.821433594 container init 7747803a501c1cd4b2e20b982793f87eb5dee4ba75f5a20a9b2872e4977ba154 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:19.411 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 podman[65048]: 2024-09-21 18:55:19.324886102 +0000 UTC m=+2.831197735 container start 7747803a501c1cd4b2e20b982793f87eb5dee4ba75f5a20a9b2872e4977ba154 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T18:55:19.411 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.358Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.358Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.358Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi104 (none))" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.358Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.358Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.367Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.370Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.371Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.371Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.374Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.374Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.933µs 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.375Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.375Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T18:55:19.412 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.377Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.377Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=51.482µs wal_replay_duration=2.578992ms wbl_replay_duration=132ns total_replay_duration=2.731578ms 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.381Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.381Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.381Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.409Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.653293ms db_storage=1.058µs remote_storage=1.641µs web_handler=522ns query_engine=929ns scrape=730.315µs scrape_sd=991.762µs notify=28.502µs notify_sd=24.61µs rules=25.120219ms tracing=9.2µs 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.409Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T18:55:19.413 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T18:55:19.409Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T18:55:19.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:19.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.641 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:19.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:19.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:19.797 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:19.798 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:55:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:19.899 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:19.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:19.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/113887006' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T18:55:19.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:19 smithi081 ceph-mon[29384]: osdmap e104: 8 total, 8 up, 7 in 2024-09-21T18:55:19.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/425898598' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:55:19.996 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/113887006' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T18:55:19.996 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:19 smithi081 ceph-mon[39192]: osdmap e104: 8 total, 8 up, 7 in 2024-09-21T18:55:19.996 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/425898598' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:55:20.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:20.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:20.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 59 16 102129 102113 108.124 70.1875 0.0113312 0.00924442 2024-09-21T18:55:20.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:20.204 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/113887006' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T18:55:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-mon[32438]: osdmap e104: 8 total, 8 up, 7 in 2024-09-21T18:55:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/425898598' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:55:20.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:20.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:20.471 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:20.479 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:20.479 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:20.492 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:20.492 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:20.521 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:20.521 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:20.555 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:20.556 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:20.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:20.590 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:20.590 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:20.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:20.620 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:20.657 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:20.657 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:20.657 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T18:55:20.519Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.007986262s 2024-09-21T18:55:20.692 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:20.693 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:20.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:20.757 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:20.804 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:20 smithi104 bash[65048]: 7747803a501c1cd4b2e20b982793f87eb5dee4ba75f5a20a9b2872e4977ba154 2024-09-21T18:55:20.804 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 18:55:20 smithi104 systemd[1]: Started Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T18:55:20.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:20.869 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:20.869 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:20.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:20.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:20.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: pgmap v86: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/425898598' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: osdmap e105: 8 total, 8 up, 7 in 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4052148437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi081.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.948 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: pgmap v86: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/425898598' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: osdmap e105: 8 total, 8 up, 7 in 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4052148437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.949 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi081.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T18:55:20.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T18:55:20.951 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:20] ENGINE Bus STOPPING 2024-09-21T18:55:20.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:21.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:21.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:21.167 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:21.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:55:21.167361+0000 min lat: 0.00134243 max lat: 0.179359 avg lat: 0.00930265 2024-09-21T18:55:21.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:55:21.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 60 16 103201 103185 107.438 67 0.00776098 0.00930265 2024-09-21T18:55:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: pgmap v86: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T18:55:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/425898598' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:55:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: osdmap e105: 8 total, 8 up, 7 in 2024-09-21T18:55:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4052148437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi081.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T18:55:21.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T18:55:21.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:21.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:21.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T18:55:21.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:20] ENGINE Bus STOPPED 2024-09-21T18:55:21.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:20 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:20] ENGINE Bus STARTING 2024-09-21T18:55:21.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Serving on http://:::9283 2024-09-21T18:55:21.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Bus STARTED 2024-09-21T18:55:21.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Bus STOPPING 2024-09-21T18:55:21.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:21.447 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:21.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:21.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.586 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:21.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:21.672 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T18:55:21.672 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Bus STOPPED 2024-09-21T18:55:21.672 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Bus STARTING 2024-09-21T18:55:21.672 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Serving on http://:::9283 2024-09-21T18:55:21.672 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Bus STARTED 2024-09-21T18:55:21.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:21.723 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:21.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:21.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:21.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:21.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T18:55:21.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.010 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:22.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:22.035 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:21] ENGINE Bus STOPPING 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi081.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: 4.0 scrub starts 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: 4.0 scrub ok 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:55:22.036 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi081.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: 4.0 scrub starts 2024-09-21T18:55:22.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: 4.0 scrub ok 2024-09-21T18:55:22.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:55:22.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T18:55:22.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T18:55:22.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:22.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:22.149 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:22.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 61 16 104501 104485 107.009 81.25 0.00913344 0.00934025 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi081.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: 4.0 scrub starts 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: 4.0 scrub ok 2024-09-21T18:55:22.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:55:22.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T18:55:22.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T18:55:22.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:22.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:22.287 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T18:55:22.287 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:22] ENGINE Bus STOPPED 2024-09-21T18:55:22.287 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:22] ENGINE Bus STARTING 2024-09-21T18:55:22.287 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:22] ENGINE Serving on http://:::9283 2024-09-21T18:55:22.295 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:22.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:22.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:22.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:18:55:22] ENGINE Bus STARTED 2024-09-21T18:55:22.570 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:22.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:22.758 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:22.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:22.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:22.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:22.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:23.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:23.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: pgmap v88: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T18:55:23.114 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:22 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T18:55:23.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: pgmap v88: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T18:55:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T18:55:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: pgmap v88: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-21T18:55:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T18:55:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T18:55:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:22 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T18:55:23.168 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 62 16 105776 105760 106.568 79.6875 0.00873161 0.00937929 2024-09-21T18:55:23.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:23.290 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:23.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:23.423 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:23.423 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:23.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:23.567 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:23.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:23.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:23.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:23.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:23.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:23.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:24.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:23 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T18:55:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:23 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T18:55:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:23 smithi104 ceph-mon[32438]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:24.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:24.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T18:55:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:24.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:24.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 63 16 106919 106903 106.01 71.4375 0.0038654 0.00942717 2024-09-21T18:55:24.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:24.242 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:24.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:23 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T18:55:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:23 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T18:55:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:23 smithi081 ceph-mon[29384]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:23 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T18:55:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:23 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T18:55:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:23 smithi081 ceph-mon[39192]: from='mon.? v1:172.21.15.104:0/3803381667' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T18:55:24.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:24.374 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:24.476 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:24.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:24.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.647 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:24.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:24.748 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:24.782 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:24.802 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:55:24.803 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:55:24.803 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T18:55:24.803 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T18:55:24.803 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T18:55:24.803 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-21T18:55:24.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:24.922 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:24.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:24.936 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:25.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: pgmap v89: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 955 op/s 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: osdmap e106: 8 total, 8 up, 7 in 2024-09-21T18:55:25.105 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: 3.7 scrub starts 2024-09-21T18:55:25.106 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: 3.7 scrub ok 2024-09-21T18:55:25.106 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: 6.8 scrub starts 2024-09-21T18:55:25.106 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:24 smithi104 ceph-mon[32438]: 6.8 scrub ok 2024-09-21T18:55:25.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:25.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 64 16 108029 108013 105.438 69.375 0.0133019 0.00947988 2024-09-21T18:55:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:25.293 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: pgmap v89: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 955 op/s 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: osdmap e106: 8 total, 8 up, 7 in 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: 3.7 scrub starts 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: 3.7 scrub ok 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: 6.8 scrub starts 2024-09-21T18:55:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[39192]: 6.8 scrub ok 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: pgmap v89: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 955 op/s 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: osdmap e106: 8 total, 8 up, 7 in 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: 3.7 scrub starts 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: 3.7 scrub ok 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: 6.8 scrub starts 2024-09-21T18:55:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:24 smithi081 ceph-mon[29384]: 6.8 scrub ok 2024-09-21T18:55:25.330 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:25.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:25.466 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:25.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:25.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:25.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:25.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:25.837 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:25.872 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:25.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:25.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:26.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[29384]: 5.1c scrub starts 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[29384]: 5.1c scrub ok 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[29384]: 3.10 scrub starts 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[29384]: 3.10 scrub ok 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T18:55:26.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[39192]: 5.1c scrub starts 2024-09-21T18:55:26.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[39192]: 5.1c scrub ok 2024-09-21T18:55:26.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[39192]: 3.10 scrub starts 2024-09-21T18:55:26.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[39192]: 3.10 scrub ok 2024-09-21T18:55:26.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T18:55:26.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:25 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T18:55:26.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:26.145 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:26.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 65 16 109367 109351 105.102 83.625 0.0138015 0.00951017 2024-09-21T18:55:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:25 smithi104 ceph-mon[32438]: 5.1c scrub starts 2024-09-21T18:55:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:25 smithi104 ceph-mon[32438]: 5.1c scrub ok 2024-09-21T18:55:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:25 smithi104 ceph-mon[32438]: 3.10 scrub starts 2024-09-21T18:55:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:25 smithi104 ceph-mon[32438]: 3.10 scrub ok 2024-09-21T18:55:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:25 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T18:55:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:25 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T18:55:26.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:26.295 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:26.321 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:26.321 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:26.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:26.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.571 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:26.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:26.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.711 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:26.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:26.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:26.844 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:26.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:26.982 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:26.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:27.058 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:55:27.058 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T18:55:27.058 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[39192]: 3.1f scrub starts 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[39192]: 3.1f scrub ok 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[39192]: pgmap v91: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 773 B/s rd, 54 MiB/s wr, 865 op/s 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[39192]: 5.14 scrub starts 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[39192]: 5.14 scrub ok 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[29384]: 3.1f scrub starts 2024-09-21T18:55:27.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[29384]: 3.1f scrub ok 2024-09-21T18:55:27.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[29384]: pgmap v91: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 773 B/s rd, 54 MiB/s wr, 865 op/s 2024-09-21T18:55:27.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[29384]: 5.14 scrub starts 2024-09-21T18:55:27.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:26 smithi081 ceph-mon[29384]: 5.14 scrub ok 2024-09-21T18:55:27.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:27.125 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:27.169 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 66 16 110651 110635 104.726 80.25 0.0127666 0.00954403 2024-09-21T18:55:27.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:26 smithi104 ceph-mon[32438]: 3.1f scrub starts 2024-09-21T18:55:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:26 smithi104 ceph-mon[32438]: 3.1f scrub ok 2024-09-21T18:55:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:26 smithi104 ceph-mon[32438]: pgmap v91: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 773 B/s rd, 54 MiB/s wr, 865 op/s 2024-09-21T18:55:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:26 smithi104 ceph-mon[32438]: 5.14 scrub starts 2024-09-21T18:55:27.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:26 smithi104 ceph-mon[32438]: 5.14 scrub ok 2024-09-21T18:55:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:27.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:55:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:55:27.360 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:27.400 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:27.436 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T18:55:27.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:27.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:27.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:27.674 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:27.728 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:55:27.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:27.816 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:27.918 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:27.930 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:27.950 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:27 smithi104 ceph-mon[32438]: 6.6 scrub starts 2024-09-21T18:55:27.950 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:27 smithi104 ceph-mon[32438]: 6.6 scrub ok 2024-09-21T18:55:27.951 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:27 smithi104 ceph-mon[32438]: 3.d scrub starts 2024-09-21T18:55:27.951 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:27 smithi104 ceph-mon[32438]: 3.d scrub ok 2024-09-21T18:55:27.951 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:27 smithi104 ceph-mon[32438]: 7.e scrub starts 2024-09-21T18:55:27.951 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:55:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:55:27.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:27.959 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:27.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:28.060 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[29384]: 6.6 scrub starts 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[29384]: 6.6 scrub ok 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[29384]: 3.d scrub starts 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[29384]: 3.d scrub ok 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[29384]: 7.e scrub starts 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[39192]: 6.6 scrub starts 2024-09-21T18:55:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[39192]: 6.6 scrub ok 2024-09-21T18:55:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[39192]: 3.d scrub starts 2024-09-21T18:55:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[39192]: 3.d scrub ok 2024-09-21T18:55:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:27 smithi081 ceph-mon[39192]: 7.e scrub starts 2024-09-21T18:55:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:28.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:28.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:28.170 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 67 15 111855 111840 104.287 75.3125 0.00580003 0.00958416 2024-09-21T18:55:28.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:28.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:55:28.241 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:28.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:28.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:28.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:28.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:28.466 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:28.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:28.557 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:28.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:28.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:28.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:28.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:28.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:28.819 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:28.819 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:28.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:28.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:28.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: pgmap v92: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 3.6 scrub starts 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 3.6 scrub ok 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T18:55:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:28 smithi104 ceph-mon[32438]: 7.8 scrub starts 2024-09-21T18:55:29.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:29.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:29.170 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 68 16 112875 112859 103.69 63.6875 0.00672556 0.00963817 2024-09-21T18:55:29.182 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: pgmap v92: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 3.6 scrub starts 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 3.6 scrub ok 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[29384]: 7.8 scrub starts 2024-09-21T18:55:29.184 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: pgmap v92: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 3.6 scrub starts 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 3.6 scrub ok 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T18:55:29.185 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:28 smithi081 ceph-mon[39192]: 7.8 scrub starts 2024-09-21T18:55:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:29.227 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:29.227 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:29.263 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:29.303 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:29.304 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:29.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:29.335 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:29.335 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:29.336 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:29.367 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:29.368 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:29.404 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:29.404 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:29.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:29.440 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:29.440 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:29.472 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:29.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:29.601 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:29.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:29.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:29.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:29.879 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:29.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:29.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:30.016 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:30.099 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: osdmap e107: 8 total, 8 up, 7 in 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1606116200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: osdmap e108: 8 total, 8 up, 7 in 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: 7.e scrub ok 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:30.100 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:29 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:30.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:30.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T18:55:30.159 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: osdmap e107: 8 total, 8 up, 7 in 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1606116200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: osdmap e108: 8 total, 8 up, 7 in 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: 7.e scrub ok 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:30.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:30.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: osdmap e107: 8 total, 8 up, 7 in 2024-09-21T18:55:30.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1606116200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:30.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: osdmap e108: 8 total, 8 up, 7 in 2024-09-21T18:55:30.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: 7.e scrub ok 2024-09-21T18:55:30.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:30.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:29 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:30.170 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 69 16 113973 113957 103.182 68.625 0.00942664 0.00968632 2024-09-21T18:55:30.173 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:30.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:30.305 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:30.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:30.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:30.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:30.588 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:30.622 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:30.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:30.722 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:30.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:30.854 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:30.900 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:55:30.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:30.992 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:30.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:31.039 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:30 smithi104 ceph-mon[32438]: pgmap v94: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T18:55:31.039 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:30 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T18:55:31.039 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:30 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T18:55:31.039 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/466142214' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:31.039 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:30 smithi104 ceph-mon[32438]: from='client.34844 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:31.039 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:30 smithi104 ceph-mon[32438]: 7.8 scrub ok 2024-09-21T18:55:31.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:31.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:31.170 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 70 16 114935 114919 102.567 60.125 0.010227 0.00974485 2024-09-21T18:55:31.211 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:55:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:55:31.212 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T18:55:31.212 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:55:31.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:31.257 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[29384]: pgmap v94: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/466142214' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[29384]: from='client.34844 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[29384]: 7.8 scrub ok 2024-09-21T18:55:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[39192]: pgmap v94: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T18:55:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T18:55:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T18:55:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/466142214' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[39192]: from='client.34844 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:30 smithi081 ceph-mon[39192]: 7.8 scrub ok 2024-09-21T18:55:31.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:31.393 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:31.494 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:31.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:31.677 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:31.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:31.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:31.918 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:31.951 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:31.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:32.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:32.089 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:32.170 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 71 16 116162 116146 102.202 76.6875 0.00594049 0.00977898 2024-09-21T18:55:32.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:32.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: 7.0 scrub starts 2024-09-21T18:55:32.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T18:55:32.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T18:55:32.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:55:32.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: from='client.34844 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: osdmap e109: 8 total, 8 up, 7 in 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: 7.0 scrub starts 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: from='client.34844 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: osdmap e109: 8 total, 8 up, 7 in 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T18:55:32.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:31 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T18:55:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: 7.0 scrub starts 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: from='client.34844 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: osdmap e109: 8 total, 8 up, 7 in 2024-09-21T18:55:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T18:55:32.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:31 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T18:55:32.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:32.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:32.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.492 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:32.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:32.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.632 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:32.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:32.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:32.766 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:32.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:32.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:32.976 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:55:32.976 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:55:32.976 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-21T18:55:32.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:55:33.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:33.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:33.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T18:55:33.153 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:32 smithi104 ceph-mon[32438]: 3.2 scrub starts 2024-09-21T18:55:33.153 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:32 smithi104 ceph-mon[32438]: 3.2 scrub ok 2024-09-21T18:55:33.153 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:32 smithi104 ceph-mon[32438]: pgmap v97: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T18:55:33.153 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:32 smithi104 ceph-mon[32438]: 6.18 scrub starts 2024-09-21T18:55:33.153 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:32 smithi104 ceph-mon[32438]: 6.18 scrub ok 2024-09-21T18:55:33.154 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:32 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T18:55:33.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:33.172 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 72 16 117285 117269 101.755 70.1875 0.0136164 0.00982263 2024-09-21T18:55:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.217 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:33.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[39192]: 3.2 scrub starts 2024-09-21T18:55:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[39192]: 3.2 scrub ok 2024-09-21T18:55:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[39192]: pgmap v97: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T18:55:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[39192]: 6.18 scrub starts 2024-09-21T18:55:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[39192]: 6.18 scrub ok 2024-09-21T18:55:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T18:55:33.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[29384]: 3.2 scrub starts 2024-09-21T18:55:33.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[29384]: 3.2 scrub ok 2024-09-21T18:55:33.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[29384]: pgmap v97: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T18:55:33.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[29384]: 6.18 scrub starts 2024-09-21T18:55:33.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[29384]: 6.18 scrub ok 2024-09-21T18:55:33.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:32 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T18:55:33.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:33.453 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:33.487 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:33.493 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:33.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:33.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:33.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:33.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:33.921 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:33.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:34.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:34.066 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.078 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:34.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:34.082 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:55:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:55:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T18:55:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T18:55:34.083 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:55:34.172 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 73 16 118416 118400 101.329 70.6875 0.00715142 0.00986468 2024-09-21T18:55:34.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:34.209 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:34.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[29384]: 6.0 scrub starts 2024-09-21T18:55:34.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[29384]: 6.0 scrub ok 2024-09-21T18:55:34.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T18:55:34.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[29384]: 7.0 scrub ok 2024-09-21T18:55:34.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T18:55:34.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T18:55:34.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[39192]: 6.0 scrub starts 2024-09-21T18:55:34.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[39192]: 6.0 scrub ok 2024-09-21T18:55:34.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T18:55:34.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[39192]: 7.0 scrub ok 2024-09-21T18:55:34.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T18:55:34.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:33 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T18:55:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:33 smithi104 ceph-mon[32438]: 6.0 scrub starts 2024-09-21T18:55:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:33 smithi104 ceph-mon[32438]: 6.0 scrub ok 2024-09-21T18:55:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:33 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T18:55:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:33 smithi104 ceph-mon[32438]: 7.0 scrub ok 2024-09-21T18:55:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:33 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T18:55:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:33 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T18:55:34.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:34.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:55:34.348 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:34.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:34.513 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:34.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:34.649 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:34.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:34.799 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:34.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T18:55:34.900 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:34.937 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:34.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:35.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:35.080 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:35.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:35.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:55:35.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 74 16 119707 119691 101.049 80.6875 0.0127432 0.00989162 2024-09-21T18:55:35.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[29384]: 5.13 scrub starts 2024-09-21T18:55:35.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[29384]: 5.13 scrub ok 2024-09-21T18:55:35.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[29384]: pgmap v98: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-21T18:55:35.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[29384]: osdmap e110: 8 total, 8 up, 7 in 2024-09-21T18:55:35.176 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[39192]: 5.13 scrub starts 2024-09-21T18:55:35.176 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[39192]: 5.13 scrub ok 2024-09-21T18:55:35.176 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[39192]: pgmap v98: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-21T18:55:35.176 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:34 smithi081 ceph-mon[39192]: osdmap e110: 8 total, 8 up, 7 in 2024-09-21T18:55:35.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:35.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:35.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:34 smithi104 ceph-mon[32438]: 5.13 scrub starts 2024-09-21T18:55:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:34 smithi104 ceph-mon[32438]: 5.13 scrub ok 2024-09-21T18:55:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:34 smithi104 ceph-mon[32438]: pgmap v98: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-21T18:55:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:34 smithi104 ceph-mon[32438]: osdmap e110: 8 total, 8 up, 7 in 2024-09-21T18:55:35.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:35.422 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:35.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:35.483 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:35.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:35.557 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:35.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:35.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:35.709 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:35.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:35.795 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:13968 7.0 7:07c32104:::benchmark_data_smithi081_73391_object13967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:25.426751+0000", 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.361611436999993, 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.054114677, 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.796 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 13968 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426751+0000", 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426751+0000", 2024-09-21T18:55:35.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426752+0000", 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.446e-06 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426780+0000", 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7593000000000002e-05 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426791+0000", 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0801999999999999e-05 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426798+0000", 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.396e-06 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426856+0000", 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8319999999999997e-05 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.426976+0000", 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119628 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.427102+0000", 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012564999999999999 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.428154+0000", 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010523189999999999 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.428895+0000", 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074118899999999995 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.480702+0000", 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051806456000000001 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.480810+0000", 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107846 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.480866+0000", 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6032000000000002e-05 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:13976 7.0 7:02bba0bd:::benchmark_data_smithi081_73391_object13975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:25.429412+0000", 2024-09-21T18:55:35.802 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.358950308000004, 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.054160280999999998, 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 13976 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429412+0000", 2024-09-21T18:55:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429412+0000", 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429425+0000", 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3004e-05 2024-09-21T18:55:35.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429462+0000", 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6634000000000002e-05 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429482+0000", 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0695000000000001e-05 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429498+0000", 2024-09-21T18:55:35.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5925000000000001e-05 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429649+0000", 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015033000000000001 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429774+0000", 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012532099999999999 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.429911+0000", 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137047 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.430876+0000", 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00096484999999999995 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.482769+0000", 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051893373999999999 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.483447+0000", 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067779199999999998 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.483508+0000", 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0619999999999999e-05 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:25.483572+0000", 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4689000000000004e-05 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.808 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54661 7.0 7:07cacdd3:::benchmark_data_smithi081_73391_object54660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.264263+0000", 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.524098983000002, 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069566144999999996, 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.809 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54661 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264263+0000", 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264263+0000", 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264265+0000", 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8989999999999999e-06 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264329+0000", 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3916000000000003e-05 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264354+0000", 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5173000000000001e-05 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264361+0000", 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7630000000000003e-06 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264406+0000", 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4493999999999998e-05 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264494+0000", 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8230000000000001e-05 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264583+0000", 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9458999999999998e-05 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.268277+0000", 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036939899999999999 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.268288+0000", 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0331e-05 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333729+0000", 2024-09-21T18:55:35.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.065441705000000003 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333784+0000", 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5055000000000001e-05 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333830+0000", 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5130000000000003e-05 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54662 7.0 7:0fe82cc8:::benchmark_data_smithi081_73391_object54661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.264375+0000", 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.523987765000001, 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069597117999999999, 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54662 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264375+0000", 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264375+0000", 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264376+0000", 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5799999999999998e-07 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264404+0000", 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8065999999999999e-05 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264409+0000", 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.665e-06 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264413+0000", 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0319999999999997e-06 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.266408+0000", 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019946170000000002 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.266516+0000", 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108414 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.266658+0000", 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141827 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.268354+0000", 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016960689999999999 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.270100+0000", 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017457460000000001 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333874+0000", 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.063773683999999997 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333931+0000", 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6997999999999999e-05 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333972+0000", 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1041999999999998e-05 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.821 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54682 7.0 7:0114f529:::benchmark_data_smithi081_73391_object54681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.275862+0000", 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.512499914000003, 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060413308999999998, 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54682 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.275862+0000", 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.275862+0000", 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.275865+0000", 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3410000000000001e-06 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.275921+0000", 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6329999999999998e-05 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.275938+0000", 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6727000000000001e-05 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.275946+0000", 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0479999999999999e-06 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.276010+0000", 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.389e-05 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.276142+0000", 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013179399999999999 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.276259+0000", 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117749 2024-09-21T18:55:35.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.277637+0000", 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001377421 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.278301+0000", 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00066463600000000002 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.336146+0000", 2024-09-21T18:55:35.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.057844128000000002 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.336232+0000", 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6976000000000002e-05 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.336276+0000", 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3269000000000002e-05 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:78940 7.0 7:0864e58f:::benchmark_data_smithi081_73391_object78939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:02.211546+0000", 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.576816778999998, 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064420430000000001, 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.828 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78940 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.211546+0000", 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.211546+0000", 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.211548+0000", 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3659999999999999e-06 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212602+0000", 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001054372 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212618+0000", 2024-09-21T18:55:35.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.524e-05 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212625+0000", 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6550000000000004e-06 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212745+0000", 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011971 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212849+0000", 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103978 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.213271+0000", 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000422178 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.217957+0000", 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004685982 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.217998+0000", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0633000000000002e-05 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.275799+0000", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.057800909999999997 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.275918+0000", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119132 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.275966+0000", 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8273999999999999e-05 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:78943 7.0 7:06134009:::benchmark_data_smithi081_73391_object78942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:02.212645+0000", 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.57571729, 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066485445000000004, 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.834 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 78943 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212645+0000", 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212645+0000", 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212646+0000", 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.161e-06 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212674+0000", 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7631999999999999e-05 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212682+0000", 2024-09-21T18:55:35.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7039999999999994e-06 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.212687+0000", 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0860000000000001e-06 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.215763+0000", 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030764619999999999 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.215873+0000", 2024-09-21T18:55:35.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109868 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.216015+0000", 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141731 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.218091+0000", 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020767649999999999 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.218741+0000", 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00064925599999999997 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.279048+0000", 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060307552 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.279087+0000", 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8578000000000002e-05 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:02.279131+0000", 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3649999999999997e-05 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:98540 7.0 7:0fcbc423:::benchmark_data_smithi081_73391_object98539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.043170+0000", 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.745192346, 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.061531638999999999, 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.840 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 98540 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043170+0000", 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043170+0000", 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043172+0000", 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3439999999999999e-06 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043216+0000", 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3383999999999997e-05 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043234+0000", 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8085e-05 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043243+0000", 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5459999999999997e-06 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043305+0000", 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2071999999999998e-05 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043460+0000", 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154069 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.043625+0000", 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016505299999999999 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.046311+0000", 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026859480000000001 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.046486+0000", 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000175816 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.104561+0000", 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058074369000000001 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.104655+0000", 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4077000000000001e-05 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.104702+0000", 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6876000000000003e-05 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99372 7.0 7:018c5cb9:::benchmark_data_smithi081_73391_object99371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.08662043, 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100414906, 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99372 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:55:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701744+0000", 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3889999999999999e-06 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701794+0000", 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0031000000000003e-05 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701809+0000", 2024-09-21T18:55:35.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4639e-05 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701817+0000", 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0090000000000007e-06 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701879+0000", 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1820000000000001e-05 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.702011+0000", 2024-09-21T18:55:35.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013177800000000001 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.702160+0000", 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014952100000000001 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.704287+0000", 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021267270000000001 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.704646+0000", 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035917799999999998 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802063+0000", 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097417227999999995 2024-09-21T18:55:35.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802117+0000", 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3488e-05 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802157+0000", 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0098000000000001e-05 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99378 7.0 7:0717396a:::benchmark_data_smithi081_73391_object99377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.080608431999998, 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094525928999999995, 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99378 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.854 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707756+0000", 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1110000000000002e-06 2024-09-21T18:55:35.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707812+0000", 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6208000000000001e-05 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707824+0000", 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1838000000000001e-05 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707832+0000", 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5560000000000002e-06 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707876+0000", 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4354000000000003e-05 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707962+0000", 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5986e-05 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.708079+0000", 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117478 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710054+0000", 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019749530000000002 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710919+0000", 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00086474499999999999 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802199+0000", 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.091279896999999999 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802247+0000", 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7550999999999999e-05 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802280+0000", 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3251999999999999e-05 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99386 7.0 7:06617387:::benchmark_data_smithi081_73391_object99385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:55:35.859 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.074524625999999, 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088596121999999999, 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99386 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713840+0000", 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7990000000000001e-06 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713885+0000", 2024-09-21T18:55:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5139999999999998e-05 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713895+0000", 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0193999999999999e-05 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713901+0000", 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7049999999999996e-06 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713952+0000", 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1641999999999998e-05 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.714080+0000", 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012742999999999999 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.714184+0000", 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010401899999999999 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.718889+0000", 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047058029999999997 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.718919+0000", 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9291000000000001e-05 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802318+0000", 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083398876999999996 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802398+0000", 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0391999999999999e-05 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802434+0000", 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5830000000000001e-05 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99389 7.0 7:02d2e699:::benchmark_data_smithi081_73391_object99388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:55:35.865 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.073207986, 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087440860999999995, 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99389 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715156+0000", 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6980000000000001e-06 2024-09-21T18:55:35.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715227+0000", 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0919000000000003e-05 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715243+0000", 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5559000000000001e-05 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715250+0000", 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6340000000000001e-06 2024-09-21T18:55:35.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716563+0000", 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013130399999999999 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716688+0000", 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012472500000000001 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716876+0000", 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000187586 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.719345+0000", 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024699539999999999 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720446+0000", 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011003219999999999 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802480+0000", 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082033990000000001 2024-09-21T18:55:35.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802566+0000", 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6119999999999995e-05 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802595+0000", 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9314000000000001e-05 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114116 7.0 7:06cfcf3e:::benchmark_data_smithi081_73391_object114115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:35.871 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.335474+0000", 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.4528880619999995, 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.059105802999999998, 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114116 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.335474+0000", 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.335474+0000", 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.335477+0000", 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2249999999999999e-06 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.335525+0000", 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8041000000000003e-05 2024-09-21T18:55:35.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.335543+0000", 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8498000000000001e-05 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.335552+0000", 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3049999999999998e-06 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.345225+0000", 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0096729910000000006 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.345360+0000", 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134313 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.345520+0000", 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016049400000000001 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.353704+0000", 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.008183559 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.353715+0000", 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1759999999999999e-05 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.394464+0000", 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.040749002999999999 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.394519+0000", 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4416999999999998e-05 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.394580+0000", 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1197000000000004e-05 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114537 7.0 7:04929e7d:::benchmark_data_smithi081_73391_object114536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.0649644519999999, 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105603195, 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114537 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723400+0000", 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.119e-06 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723433+0000", 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3302000000000001e-05 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723444+0000", 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0523000000000001e-05 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723452+0000", 2024-09-21T18:55:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8199999999999997e-06 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727483+0000", 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040313839999999998 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727571+0000", 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7770999999999997e-05 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727655+0000", 2024-09-21T18:55:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4322999999999999e-05 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730033+0000", 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002377948 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730045+0000", 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2091e-05 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828840+0000", 2024-09-21T18:55:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098794333999999998 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828919+0000", 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9752000000000005e-05 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.829001+0000", 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1828000000000003e-05 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114544 7.0 7:016dd1d4:::benchmark_data_smithi081_73391_object114543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.0574406559999998, 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101948043, 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114544 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730924+0000", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1280000000000002e-06 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730973+0000", 2024-09-21T18:55:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8687000000000003e-05 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730987+0000", 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.448e-05 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730996+0000", 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9719999999999998e-06 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731081+0000", 2024-09-21T18:55:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5292999999999998e-05 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731162+0000", 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0603000000000001e-05 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731260+0000", 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8259999999999998e-05 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.733069+0000", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018090840000000001 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.780503+0000", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.047434142999999998 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832639+0000", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052135212 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832711+0000", 2024-09-21T18:55:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2327000000000001e-05 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832870+0000", 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015885399999999999 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114602 7.0 7:0e092d8d:::benchmark_data_smithi081_73391_object114601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.868863+0000", 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.9194997039999997, 2024-09-21T18:55:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052542667000000001, 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114602 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.868863+0000", 2024-09-21T18:55:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.868863+0000", 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.868864+0000", 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0869999999999999e-06 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.869023+0000", 2024-09-21T18:55:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015889499999999999 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.869033+0000", 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0668e-05 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.869040+0000", 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.793e-06 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.893250+0000", 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024210008000000002 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.893395+0000", 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014503399999999999 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.893576+0000", 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018084199999999999 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.906713+0000", 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013136841 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.907432+0000", 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000719081 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.921296+0000", 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013864489000000001 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.921351+0000", 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4818999999999998e-05 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.921405+0000", 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4110000000000002e-05 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:115640 7.0 7:07e50b4e:::benchmark_data_smithi081_73391_object115639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:31.711141+0000", 2024-09-21T18:55:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.0772212320000003, 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070270899999999997, 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 115640 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711141+0000", 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711141+0000", 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711144+0000", 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4980000000000001e-06 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711178+0000", 2024-09-21T18:55:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4239999999999997e-05 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711190+0000", 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2078e-05 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711198+0000", 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8010000000000004e-06 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.718891+0000", 2024-09-21T18:55:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076928630000000003 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.718976+0000", 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5426000000000005e-05 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.719071+0000", 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4860999999999999e-05 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.721212+0000", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021414110000000002 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.722474+0000", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012613839999999999 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.781300+0000", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058826330000000003 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.781356+0000", 2024-09-21T18:55:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5823999999999999e-05 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.781412+0000", 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6184000000000003e-05 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:116129 7.0 7:08f433c2:::benchmark_data_smithi081_73391_object116128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:32.139455+0000", 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.648907726, 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076617837999999994, 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 116129 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139455+0000", 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139455+0000", 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139457+0000", 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9130000000000001e-06 2024-09-21T18:55:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139488+0000", 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1940000000000003e-05 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139497+0000", 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1000000000000004e-06 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139503+0000", 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.981e-06 2024-09-21T18:55:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.195546+0000", 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056043530000000001 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.195659+0000", 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011290600000000001 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.195796+0000", 2024-09-21T18:55:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000136747 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.214061+0000", 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018265019 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.214100+0000", 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8948e-05 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.215953+0000", 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001852921 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216012+0000", 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8866000000000002e-05 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216072+0000", 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0967000000000001e-05 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:116144 7.a e109/79 7:500af0ec:::benchmark_data_smithi081_73391_object116143:head v 109'7133, mlcod=109'7131)", 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:32.151495+0000", 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.636866935, 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.059983533999999998, 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151495+0000", 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151495+0000", 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151497+0000", 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.889e-06 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151581+0000", 2024-09-21T18:55:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3727999999999995e-05 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.191324+0000", 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.039742863000000003 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.191334+0000", 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0263e-05 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.191383+0000", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8539999999999999e-05 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.191405+0000", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1803000000000001e-05 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.211440+0000", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020035659000000001 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.211479+0000", 2024-09-21T18:55:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8788999999999997e-05 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:116148 7.0 7:047926a3:::benchmark_data_smithi081_73391_object116147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:32.151668+0000", 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6366947619999999, 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064488434999999997, 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 116148 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151668+0000", 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151668+0000", 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151670+0000", 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1919999999999999e-06 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.151721+0000", 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1205999999999997e-05 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.191332+0000", 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.039611039000000001 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.191342+0000", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0079e-05 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.197904+0000", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006561814 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.198015+0000", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111329 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.211600+0000", 2024-09-21T18:55:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013584568999999999 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.215899+0000", 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042988360000000003 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.215943+0000", 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4233999999999997e-05 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216082+0000", 2024-09-21T18:55:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013938399999999999 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216117+0000", 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5131e-05 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216156+0000", 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8621999999999997e-05 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:35.919 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:55:35.921 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:55:35.926 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:55:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:35.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:36.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:36.071 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:36.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:36.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 75 16 120722 120706 100.548 63.4375 0.0125762 0.00994105 2024-09-21T18:55:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: 5.17 scrub starts 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: 5.17 scrub ok 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T18:55:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:35 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:36.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: 5.17 scrub starts 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: 5.17 scrub ok 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:36.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: 5.17 scrub starts 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: 5.17 scrub ok 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:36.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:35 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:36.341 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:36.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:36.464 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:36.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:36.603 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:36.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.747 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:36.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:36.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T18:55:36.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:36.890 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:36.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:36.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: 3.16 scrub starts 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: 3.16 scrub ok 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: pgmap v100: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: from='client.34850 v1:172.21.15.81:0/1455030259' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T18:55:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:36 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T18:55:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:37.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:37.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 76 16 121945 121929 100.231 76.4375 0.00773149 0.00997255 2024-09-21T18:55:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:37.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:37.199 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:37.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: 3.16 scrub starts 2024-09-21T18:55:37.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: 3.16 scrub ok 2024-09-21T18:55:37.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: pgmap v100: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: from='client.34850 v1:172.21.15.81:0/1455030259' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: 3.16 scrub starts 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: 3.16 scrub ok 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: pgmap v100: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: from='client.34850 v1:172.21.15.81:0/1455030259' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:55:37.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:55:37.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:37.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T18:55:37.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:36 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T18:55:37.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:37.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:37.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.471 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:37.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:37.540 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:55:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:55:37.568 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:37.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:37.603 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:37.603 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.617 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:37.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:37.642 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:37.642 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:37.670 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:37.670 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:37.701 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:37.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:37.717 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:37.739 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:37.757 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:37.774 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:37.774 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:37.796 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:55:37.796 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:122738 7.b e110/79 7:d6f6cc69:::benchmark_data_smithi081_73391_object122737:head v 110'7540, mlcod=110'7537)", 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:37.787695+0000", 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0051858859999999998, 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052125369999999997, 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787695+0000", 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787695+0000", 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787696+0000", 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0999999999999997e-07 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787760+0000", 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3577999999999993e-05 2024-09-21T18:55:37.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787767+0000", 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2779999999999996e-06 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787772+0000", 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0340000000000004e-06 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787822+0000", 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9549000000000003e-05 2024-09-21T18:55:37.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.787839+0000", 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7036000000000001e-05 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:122724 7.c 7:320a9e6b:::benchmark_data_smithi081_73391_object122723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:37.776311+0000", 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0165702, 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016656838, 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:37.800 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122724 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776311+0000", 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:37.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776311+0000", 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776312+0000", 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.424e-06 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776357+0000", 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4719000000000003e-05 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776368+0000", 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0764999999999999e-05 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776387+0000", 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9139000000000001e-05 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776493+0000", 2024-09-21T18:55:37.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010568 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776615+0000", 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012213 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.776755+0000", 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000139682 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780146+0000", 2024-09-21T18:55:37.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003391294 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:122728 7.c 7:3277f842:::benchmark_data_smithi081_73391_object122727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:37.780457+0000", 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.012424220999999999, 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012578064, 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:37.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122728 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780457+0000", 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780457+0000", 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780459+0000", 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7659999999999999e-06 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780614+0000", 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015487399999999999 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780624+0000", 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0344e-05 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780631+0000", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3849999999999996e-06 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780684+0000", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2629e-05 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780819+0000", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013519500000000001 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:55:37.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.780973+0000", 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154289 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.789480+0000", 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0085065009999999996 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:122732 7.c 7:3cb99d09:::benchmark_data_smithi081_73391_object122731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:55:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:37.781020+0000", 2024-09-21T18:55:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.011861226000000001, 2024-09-21T18:55:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012100184999999999, 2024-09-21T18:55:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:37.810 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122732 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781020+0000", 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:37.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781020+0000", 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781021+0000", 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1170000000000001e-06 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781052+0000", 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0939000000000003e-05 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781060+0000", 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2830000000000001e-06 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781067+0000", 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1760000000000004e-06 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.786445+0000", 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053779489999999999 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.786583+0000", 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137723 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.786780+0000", 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000197238 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:122725 7.4 e110/82 7:294b6e5c:::benchmark_data_smithi081_73391_object122724:head v 110'7724, mlcod=110'7721)", 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:37.781240+0000", 2024-09-21T18:55:37.814 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.011640714, 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011936921, 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781240+0000", 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781240+0000", 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781242+0000", 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0550000000000001e-06 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781264+0000", 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2028999999999999e-05 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781274+0000", 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0258000000000001e-05 2024-09-21T18:55:37.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.781280+0000", 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8019999999999997e-06 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.786596+0000", 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053167099999999997 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.786614+0000", 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8144000000000001e-05 2024-09-21T18:55:37.817 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:122735 7.c 7:3dea3af1:::benchmark_data_smithi081_73391_object122734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:37.784295+0000", 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0085866510000000007, 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089356090000000006, 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:37.818 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122735 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.784295+0000", 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.784295+0000", 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.784296+0000", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7370000000000001e-06 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.784334+0000", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7481000000000002e-05 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.784343+0000", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0480000000000004e-06 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.784349+0000", 2024-09-21T18:55:37.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.579e-06 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.789644+0000", 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052947239999999998 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.789753+0000", 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010898100000000001 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:37.789895+0000", 2024-09-21T18:55:37.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014175200000000001 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 6 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:37.822 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:55:37.822 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:37.835 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:37.835 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:37.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:37.898 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:37.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:37.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:38.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 3.5 scrub starts 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 3.5 scrub ok 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 5.2 scrub starts 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 5.2 scrub ok 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T18:55:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3662780564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 3.5 scrub starts 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 3.5 scrub ok 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 5.2 scrub starts 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 5.2 scrub ok 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T18:55:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3662780564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:38.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:38.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 77 16 123253 123237 99.9906 81.75 0.0163276 0.00999683 2024-09-21T18:55:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 3.5 scrub starts 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 3.5 scrub ok 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 5.2 scrub starts 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 5.2 scrub ok 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T18:55:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3662780564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:38.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:38.290 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-21T18:55:35.400514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7555455515,"num_objects":115477,"num_object_clones":0,"num_object_copies":346431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115477,"num_whiteouts":0,"num_read":980,"num_read_kb":837,"num_write":115827,"num_write_kb":7379465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":1184346,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77133,"ondisk_log_size":77133,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22602960,"kb_used_data":22034812,"kb_used_omap":10,"kb_used_meta":567989,"kb_avail":633613104,"statfs":{"total":671965249536,"available":648819818496,"internally_reserved":0,"allocated":22563647488,"data_stored":22559044780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11153,"internal_metadata":581620847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":365363200,"num_objects":5575,"num_object_clones":0,"num_object_copies":16725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5575,"num_write_kb":356800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001604"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911500+0000","last_change":"2024-09-21T18:55:25.183356+0000","last_active":"2024-09-21T18:55:30.911500+0000","last_peered":"2024-09-21T18:55:30.911500+0000","last_clean":"2024-09-21T18:55:30.911500+0000","last_became_active":"2024-09-21T18:54:17.713495+0000","last_became_peered":"2024-09-21T18:54:17.713495+0000","last_unstale":"2024-09-21T18:55:30.911500+0000","last_undegraded":"2024-09-21T18:55:30.911500+0000","last_fullsized":"2024-09-21T18:55:30.911500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:37.231364+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":142,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895910+0000","last_change":"2024-09-21T18:54:36.776089+0000","last_active":"2024-09-21T18:55:30.895910+0000","last_peered":"2024-09-21T18:55:30.895910+0000","last_clean":"2024-09-21T18:55:30.895910+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:55:30.895910+0000","last_undegraded":"2024-09-21T18:55:30.895910+0000","last_fullsized":"2024-09-21T18:55:30.895910+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.052772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905310+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:55:30.905310+0000","last_peered":"2024-09-21T18:55:30.905310+0000","last_clean":"2024-09-21T18:55:30.905310+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:55:30.905310+0000","last_undegraded":"2024-09-21T18:55:30.905310+0000","last_fullsized":"2024-09-21T18:55:30.905310+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.278309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":101,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896130+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:55:30.896130+0000","last_peered":"2024-09-21T18:55:30.896130+0000","last_clean":"2024-09-21T18:55:30.896130+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:55:30.896130+0000","last_undegraded":"2024-09-21T18:55:30.896130+0000","last_fullsized":"2024-09-21T18:55:30.896130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.055413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895667+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:55:30.895667+0000","last_peered":"2024-09-21T18:55:30.895667+0000","last_clean":"2024-09-21T18:55:30.895667+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:55:30.895667+0000","last_undegraded":"2024-09-21T18:55:30.895667+0000","last_fullsized":"2024-09-21T18:55:30.895667+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.105754+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":123,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896749+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:55:30.896749+0000","last_peered":"2024-09-21T18:55:30.896749+0000","last_clean":"2024-09-21T18:55:30.896749+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:55:30.896749+0000","last_undegraded":"2024-09-21T18:55:30.896749+0000","last_fullsized":"2024-09-21T18:55:30.896749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.541042+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":158,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896127+0000","last_change":"2024-09-21T18:54:17.730862+0000","last_active":"2024-09-21T18:55:30.896127+0000","last_peered":"2024-09-21T18:55:30.896127+0000","last_clean":"2024-09-21T18:55:30.896127+0000","last_became_active":"2024-09-21T18:54:17.716359+0000","last_became_peered":"2024-09-21T18:54:17.716359+0000","last_unstale":"2024-09-21T18:55:30.896127+0000","last_undegraded":"2024-09-21T18:55:30.896127+0000","last_fullsized":"2024-09-21T18:55:30.896127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.191168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895759+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:55:30.895759+0000","last_peered":"2024-09-21T18:55:30.895759+0000","last_clean":"2024-09-21T18:55:30.895759+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:55:30.895759+0000","last_undegraded":"2024-09-21T18:55:30.895759+0000","last_fullsized":"2024-09-21T18:55:30.895759+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:51.178034+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911197+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:55:30.911197+0000","last_peered":"2024-09-21T18:55:30.911197+0000","last_clean":"2024-09-21T18:55:30.911197+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:55:30.911197+0000","last_undegraded":"2024-09-21T18:55:30.911197+0000","last_fullsized":"2024-09-21T18:55:30.911197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.277892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":150,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896424+0000","last_change":"2024-09-21T18:54:17.725564+0000","last_active":"2024-09-21T18:55:30.896424+0000","last_peered":"2024-09-21T18:55:30.896424+0000","last_clean":"2024-09-21T18:55:30.896424+0000","last_became_active":"2024-09-21T18:54:17.704975+0000","last_became_peered":"2024-09-21T18:54:17.704975+0000","last_unstale":"2024-09-21T18:55:30.896424+0000","last_undegraded":"2024-09-21T18:55:30.896424+0000","last_fullsized":"2024-09-21T18:55:30.896424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:14.972606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.972606+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":120,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902903+0000","last_change":"2024-09-21T18:54:55.048550+0000","last_active":"2024-09-21T18:55:30.902903+0000","last_peered":"2024-09-21T18:55:30.902903+0000","last_clean":"2024-09-21T18:55:30.902903+0000","last_became_active":"2024-09-21T18:54:17.712591+0000","last_became_peered":"2024-09-21T18:54:17.712591+0000","last_unstale":"2024-09-21T18:55:30.902903+0000","last_undegraded":"2024-09-21T18:55:30.902903+0000","last_fullsized":"2024-09-21T18:55:30.902903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:55.048477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:55.048477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:01.675730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.906352+0000","last_change":"2024-09-21T18:54:17.704813+0000","last_active":"2024-09-21T18:55:30.906352+0000","last_peered":"2024-09-21T18:55:30.906352+0000","last_clean":"2024-09-21T18:55:30.906352+0000","last_became_active":"2024-09-21T18:54:17.704510+0000","last_became_peered":"2024-09-21T18:54:17.704510+0000","last_unstale":"2024-09-21T18:55:30.906352+0000","last_undegraded":"2024-09-21T18:55:30.906352+0000","last_fullsized":"2024-09-21T18:55:30.906352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902454+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:55:30.902454+0000","last_peered":"2024-09-21T18:55:30.902454+0000","last_clean":"2024-09-21T18:55:30.902454+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:55:30.902454+0000","last_undegraded":"2024-09-21T18:55:30.902454+0000","last_fullsized":"2024-09-21T18:55:30.902454+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:14.247778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":127,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896216+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:55:30.896216+0000","last_peered":"2024-09-21T18:55:30.896216+0000","last_clean":"2024-09-21T18:55:30.896216+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:55:30.896216+0000","last_undegraded":"2024-09-21T18:55:30.896216+0000","last_fullsized":"2024-09-21T18:55:30.896216+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899000+0000","last_change":"2024-09-21T18:54:53.035653+0000","last_active":"2024-09-21T18:55:30.899000+0000","last_peered":"2024-09-21T18:55:30.899000+0000","last_clean":"2024-09-21T18:55:30.899000+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:55:30.899000+0000","last_undegraded":"2024-09-21T18:55:30.899000+0000","last_fullsized":"2024-09-21T18:55:30.899000+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.035514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.035514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.035186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":101,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902102+0000","last_change":"2024-09-21T18:53:13.427489+0000","last_active":"2024-09-21T18:55:30.902102+0000","last_peered":"2024-09-21T18:55:30.902102+0000","last_clean":"2024-09-21T18:55:30.902102+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:55:30.902102+0000","last_undegraded":"2024-09-21T18:55:30.902102+0000","last_fullsized":"2024-09-21T18:55:30.902102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.155050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.901879+0000","last_change":"2024-09-21T18:53:13.657625+0000","last_active":"2024-09-21T18:55:30.901879+0000","last_peered":"2024-09-21T18:55:30.901879+0000","last_clean":"2024-09-21T18:55:30.901879+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:55:30.901879+0000","last_undegraded":"2024-09-21T18:55:30.901879+0000","last_fullsized":"2024-09-21T18:55:30.901879+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896401+0000","last_change":"2024-09-21T18:55:29.982578+0000","last_active":"2024-09-21T18:55:30.896401+0000","last_peered":"2024-09-21T18:55:30.896401+0000","last_clean":"2024-09-21T18:55:30.896401+0000","last_became_active":"2024-09-21T18:54:17.705159+0000","last_became_peered":"2024-09-21T18:54:17.705159+0000","last_unstale":"2024-09-21T18:55:30.896401+0000","last_undegraded":"2024-09-21T18:55:30.896401+0000","last_fullsized":"2024-09-21T18:55:30.896401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:46.625994+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":142,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899006+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:55:30.899006+0000","last_peered":"2024-09-21T18:55:30.899006+0000","last_clean":"2024-09-21T18:55:30.899006+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:55:30.899006+0000","last_undegraded":"2024-09-21T18:55:30.899006+0000","last_fullsized":"2024-09-21T18:55:30.899006+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.670449+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904604+0000","last_change":"2024-09-21T18:54:17.714793+0000","last_active":"2024-09-21T18:55:30.904604+0000","last_peered":"2024-09-21T18:55:30.904604+0000","last_clean":"2024-09-21T18:55:30.904604+0000","last_became_active":"2024-09-21T18:54:17.714300+0000","last_became_peered":"2024-09-21T18:54:17.714300+0000","last_unstale":"2024-09-21T18:55:30.904604+0000","last_undegraded":"2024-09-21T18:55:30.904604+0000","last_fullsized":"2024-09-21T18:55:30.904604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:08.542177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897920+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:55:30.897920+0000","last_peered":"2024-09-21T18:55:30.897920+0000","last_clean":"2024-09-21T18:55:30.897920+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:55:30.897920+0000","last_undegraded":"2024-09-21T18:55:30.897920+0000","last_fullsized":"2024-09-21T18:55:30.897920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.092780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896582+0000","last_change":"2024-09-21T18:54:12.987795+0000","last_active":"2024-09-21T18:55:30.896582+0000","last_peered":"2024-09-21T18:55:30.896582+0000","last_clean":"2024-09-21T18:55:30.896582+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:55:30.896582+0000","last_undegraded":"2024-09-21T18:55:30.896582+0000","last_fullsized":"2024-09-21T18:55:30.896582+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.987749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.987749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:41.935763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":141,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897089+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:55:30.897089+0000","last_peered":"2024-09-21T18:55:30.897089+0000","last_clean":"2024-09-21T18:55:30.897089+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:55:30.897089+0000","last_undegraded":"2024-09-21T18:55:30.897089+0000","last_fullsized":"2024-09-21T18:55:30.897089+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:06.555236+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896550+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:55:30.896550+0000","last_peered":"2024-09-21T18:55:30.896550+0000","last_clean":"2024-09-21T18:55:30.896550+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:55:30.896550+0000","last_undegraded":"2024-09-21T18:55:30.896550+0000","last_fullsized":"2024-09-21T18:55:30.896550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:28.697127+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900065+0000","last_change":"2024-09-21T18:54:55.039104+0000","last_active":"2024-09-21T18:55:30.900065+0000","last_peered":"2024-09-21T18:55:30.900065+0000","last_clean":"2024-09-21T18:55:30.900065+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:55:30.900065+0000","last_undegraded":"2024-09-21T18:55:30.900065+0000","last_fullsized":"2024-09-21T18:55:30.900065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:54:55.039055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:55.039055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.098477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":122,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905387+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:55:30.905387+0000","last_peered":"2024-09-21T18:55:30.905387+0000","last_clean":"2024-09-21T18:55:30.905387+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:55:30.905387+0000","last_undegraded":"2024-09-21T18:55:30.905387+0000","last_fullsized":"2024-09-21T18:55:30.905387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:43.629105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899705+0000","last_change":"2024-09-21T18:54:17.738374+0000","last_active":"2024-09-21T18:55:30.899705+0000","last_peered":"2024-09-21T18:55:30.899705+0000","last_clean":"2024-09-21T18:55:30.899705+0000","last_became_active":"2024-09-21T18:54:17.719024+0000","last_became_peered":"2024-09-21T18:54:17.719024+0000","last_unstale":"2024-09-21T18:55:30.899705+0000","last_undegraded":"2024-09-21T18:55:30.899705+0000","last_fullsized":"2024-09-21T18:55:30.899705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":153,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905925+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:55:30.905925+0000","last_peered":"2024-09-21T18:55:30.905925+0000","last_clean":"2024-09-21T18:55:30.905925+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:55:30.905925+0000","last_undegraded":"2024-09-21T18:55:30.905925+0000","last_fullsized":"2024-09-21T18:55:30.905925+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:41.932240+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"109'7121","reported_seq":7191,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.416893+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:55:31.416893+0000","last_peered":"2024-09-21T18:55:31.416893+0000","last_clean":"2024-09-21T18:55:31.416893+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:55:31.416893+0000","last_undegraded":"2024-09-21T18:55:31.416893+0000","last_fullsized":"2024-09-21T18:55:31.416893+0000","mapping_epoch":79,"log_start":"88'1500","ondisk_log_start":"88'1500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:14.413165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466681856,"num_objects":7121,"num_object_clones":0,"num_object_copies":21363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7121,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":135,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.906442+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:55:30.906442+0000","last_peered":"2024-09-21T18:55:30.906442+0000","last_clean":"2024-09-21T18:55:30.906442+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:55:30.906442+0000","last_undegraded":"2024-09-21T18:55:30.906442+0000","last_fullsized":"2024-09-21T18:55:30.906442+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.353165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":123,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.898215+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:55:30.898215+0000","last_peered":"2024-09-21T18:55:30.898215+0000","last_clean":"2024-09-21T18:55:30.898215+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:55:30.898215+0000","last_undegraded":"2024-09-21T18:55:30.898215+0000","last_fullsized":"2024-09-21T18:55:30.898215+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:55.005580+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"83'9","reported_seq":154,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.901999+0000","last_change":"2024-09-21T18:54:17.715008+0000","last_active":"2024-09-21T18:55:30.901999+0000","last_peered":"2024-09-21T18:55:30.901999+0000","last_clean":"2024-09-21T18:55:30.901999+0000","last_became_active":"2024-09-21T18:54:17.706739+0000","last_became_peered":"2024-09-21T18:54:17.706739+0000","last_unstale":"2024-09-21T18:55:30.901999+0000","last_undegraded":"2024-09-21T18:55:30.901999+0000","last_fullsized":"2024-09-21T18:55:30.901999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:13.070639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":117,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.910906+0000","last_change":"2024-09-21T18:55:27.156403+0000","last_active":"2024-09-21T18:55:30.910906+0000","last_peered":"2024-09-21T18:55:30.910906+0000","last_clean":"2024-09-21T18:55:30.910906+0000","last_became_active":"2024-09-21T18:54:17.705355+0000","last_became_peered":"2024-09-21T18:54:17.705355+0000","last_unstale":"2024-09-21T18:55:30.910906+0000","last_undegraded":"2024-09-21T18:55:30.910906+0000","last_fullsized":"2024-09-21T18:55:30.910906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:27.156343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.156343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:38.809711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"109'7121","reported_seq":7192,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.502157+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:55:31.502157+0000","last_peered":"2024-09-21T18:55:31.502157+0000","last_clean":"2024-09-21T18:55:31.502157+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:55:31.502157+0000","last_undegraded":"2024-09-21T18:55:31.502157+0000","last_fullsized":"2024-09-21T18:55:31.502157+0000","mapping_epoch":79,"log_start":"93'3000","ondisk_log_start":"93'3000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.493653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466681856,"num_objects":7121,"num_object_clones":0,"num_object_copies":21363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7121,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":176,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897864+0000","last_change":"2024-09-21T18:55:20.959813+0000","last_active":"2024-09-21T18:55:30.897864+0000","last_peered":"2024-09-21T18:55:30.897864+0000","last_clean":"2024-09-21T18:55:30.897864+0000","last_became_active":"2024-09-21T18:54:17.716713+0000","last_became_peered":"2024-09-21T18:54:17.716713+0000","last_unstale":"2024-09-21T18:55:30.897864+0000","last_undegraded":"2024-09-21T18:55:30.897864+0000","last_fullsized":"2024-09-21T18:55:30.897864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:38.798346+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":130,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.898196+0000","last_change":"2024-09-21T18:55:24.017676+0000","last_active":"2024-09-21T18:55:30.898196+0000","last_peered":"2024-09-21T18:55:30.898196+0000","last_clean":"2024-09-21T18:55:30.898196+0000","last_became_active":"2024-09-21T18:54:17.719440+0000","last_became_peered":"2024-09-21T18:54:17.719440+0000","last_unstale":"2024-09-21T18:55:30.898196+0000","last_undegraded":"2024-09-21T18:55:30.898196+0000","last_fullsized":"2024-09-21T18:55:30.898196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:32.825552+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905033+0000","last_change":"2024-09-21T18:53:11.411576+0000","last_active":"2024-09-21T18:55:30.905033+0000","last_peered":"2024-09-21T18:55:30.905033+0000","last_clean":"2024-09-21T18:55:30.905033+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:55:30.905033+0000","last_undegraded":"2024-09-21T18:55:30.905033+0000","last_fullsized":"2024-09-21T18:55:30.905033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.087833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896376+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:55:30.896376+0000","last_peered":"2024-09-21T18:55:30.896376+0000","last_clean":"2024-09-21T18:55:30.896376+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:55:30.896376+0000","last_undegraded":"2024-09-21T18:55:30.896376+0000","last_fullsized":"2024-09-21T18:55:30.896376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:09.095433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"109'7245","reported_seq":7316,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.506747+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:55:31.506747+0000","last_peered":"2024-09-21T18:55:31.506747+0000","last_clean":"2024-09-21T18:55:31.506747+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:55:31.506747+0000","last_undegraded":"2024-09-21T18:55:31.506747+0000","last_fullsized":"2024-09-21T18:55:31.506747+0000","mapping_epoch":79,"log_start":"93'3100","ondisk_log_start":"93'3100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474808320,"num_objects":7245,"num_object_clones":0,"num_object_copies":21735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":139,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905175+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:55:30.905175+0000","last_peered":"2024-09-21T18:55:30.905175+0000","last_clean":"2024-09-21T18:55:30.905175+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:55:30.905175+0000","last_undegraded":"2024-09-21T18:55:30.905175+0000","last_fullsized":"2024-09-21T18:55:30.905175+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.360653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904444+0000","last_change":"2024-09-21T18:55:27.735922+0000","last_active":"2024-09-21T18:55:30.904444+0000","last_peered":"2024-09-21T18:55:30.904444+0000","last_clean":"2024-09-21T18:55:30.904444+0000","last_became_active":"2024-09-21T18:54:17.721086+0000","last_became_peered":"2024-09-21T18:54:17.721086+0000","last_unstale":"2024-09-21T18:55:30.904444+0000","last_undegraded":"2024-09-21T18:55:30.904444+0000","last_fullsized":"2024-09-21T18:55:30.904444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:33.097047+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":144,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897599+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:55:30.897599+0000","last_peered":"2024-09-21T18:55:30.897599+0000","last_clean":"2024-09-21T18:55:30.897599+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:55:30.897599+0000","last_undegraded":"2024-09-21T18:55:30.897599+0000","last_fullsized":"2024-09-21T18:55:30.897599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.132673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":101,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899478+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:55:30.899478+0000","last_peered":"2024-09-21T18:55:30.899478+0000","last_clean":"2024-09-21T18:55:30.899478+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:55:30.899478+0000","last_undegraded":"2024-09-21T18:55:30.899478+0000","last_fullsized":"2024-09-21T18:55:30.899478+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:09.052091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"109'7190","reported_seq":7261,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.366626+0000","last_change":"2024-09-21T18:54:14.452896+0000","last_active":"2024-09-21T18:55:31.366626+0000","last_peered":"2024-09-21T18:55:31.366626+0000","last_clean":"2024-09-21T18:55:31.366626+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:55:31.366626+0000","last_undegraded":"2024-09-21T18:55:31.366626+0000","last_fullsized":"2024-09-21T18:55:31.366626+0000","mapping_epoch":79,"log_start":"91'2700","ondisk_log_start":"91'2700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:37.308130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471203840,"num_objects":7190,"num_object_clones":0,"num_object_copies":21570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7190,"num_write_kb":460160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":125,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904181+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:55:30.904181+0000","last_peered":"2024-09-21T18:55:30.904181+0000","last_clean":"2024-09-21T18:55:30.904181+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:55:30.904181+0000","last_undegraded":"2024-09-21T18:55:30.904181+0000","last_fullsized":"2024-09-21T18:55:30.904181+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.356779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911269+0000","last_change":"2024-09-21T18:54:15.230912+0000","last_active":"2024-09-21T18:55:30.911269+0000","last_peered":"2024-09-21T18:55:30.911269+0000","last_clean":"2024-09-21T18:55:30.911269+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:55:30.911269+0000","last_undegraded":"2024-09-21T18:55:30.911269+0000","last_fullsized":"2024-09-21T18:55:30.911269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.230861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.230861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.698873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"83'9","reported_seq":118,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902096+0000","last_change":"2024-09-21T18:54:41.272637+0000","last_active":"2024-09-21T18:55:30.902096+0000","last_peered":"2024-09-21T18:55:30.902096+0000","last_clean":"2024-09-21T18:55:30.902096+0000","last_became_active":"2024-09-21T18:54:17.713489+0000","last_became_peered":"2024-09-21T18:54:17.713489+0000","last_unstale":"2024-09-21T18:55:30.902096+0000","last_undegraded":"2024-09-21T18:55:30.902096+0000","last_fullsized":"2024-09-21T18:55:30.902096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:47.058473+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897504+0000","last_change":"2024-09-21T18:54:17.714235+0000","last_active":"2024-09-21T18:55:30.897504+0000","last_peered":"2024-09-21T18:55:30.897504+0000","last_clean":"2024-09-21T18:55:30.897504+0000","last_became_active":"2024-09-21T18:54:17.713933+0000","last_became_peered":"2024-09-21T18:54:17.713933+0000","last_unstale":"2024-09-21T18:55:30.897504+0000","last_undegraded":"2024-09-21T18:55:30.897504+0000","last_fullsized":"2024-09-21T18:55:30.897504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"109'7323","reported_seq":7428,"reported_epoch":109,"state":"active+clean+scrubbing","last_fresh":"2024-09-21T18:55:31.421187+0000","last_change":"2024-09-21T18:55:30.108541+0000","last_active":"2024-09-21T18:55:31.421187+0000","last_peered":"2024-09-21T18:55:31.421187+0000","last_clean":"2024-09-21T18:55:31.421187+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:55:31.421187+0000","last_undegraded":"2024-09-21T18:55:31.421187+0000","last_fullsized":"2024-09-21T18:55:31.421187+0000","mapping_epoch":79,"log_start":"89'2400","ondisk_log_start":"89'2400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":3299,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479920128,"num_objects":7323,"num_object_clones":0,"num_object_copies":21969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7323,"num_write_kb":468672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":147,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904063+0000","last_change":"2024-09-21T18:55:21.240927+0000","last_active":"2024-09-21T18:55:30.904063+0000","last_peered":"2024-09-21T18:55:30.904063+0000","last_clean":"2024-09-21T18:55:30.904063+0000","last_became_active":"2024-09-21T18:54:17.711274+0000","last_became_peered":"2024-09-21T18:54:17.711274+0000","last_unstale":"2024-09-21T18:55:30.904063+0000","last_undegraded":"2024-09-21T18:55:30.904063+0000","last_fullsized":"2024-09-21T18:55:30.904063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:44.588012+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":122,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904472+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:55:30.904472+0000","last_peered":"2024-09-21T18:55:30.904472+0000","last_clean":"2024-09-21T18:55:30.904472+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:55:30.904472+0000","last_undegraded":"2024-09-21T18:55:30.904472+0000","last_fullsized":"2024-09-21T18:55:30.904472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.637938+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.901179+0000","last_change":"2024-09-21T18:54:17.710974+0000","last_active":"2024-09-21T18:55:30.901179+0000","last_peered":"2024-09-21T18:55:30.901179+0000","last_clean":"2024-09-21T18:55:30.901179+0000","last_became_active":"2024-09-21T18:54:17.710662+0000","last_became_peered":"2024-09-21T18:54:17.710662+0000","last_unstale":"2024-09-21T18:55:30.901179+0000","last_undegraded":"2024-09-21T18:55:30.901179+0000","last_fullsized":"2024-09-21T18:55:30.901179+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.222043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905220+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:55:30.905220+0000","last_peered":"2024-09-21T18:55:30.905220+0000","last_clean":"2024-09-21T18:55:30.905220+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:55:30.905220+0000","last_undegraded":"2024-09-21T18:55:30.905220+0000","last_fullsized":"2024-09-21T18:55:30.905220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.362211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"109'7223","reported_seq":7301,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.378045+0000","last_change":"2024-09-21T18:54:17.707215+0000","last_active":"2024-09-21T18:55:31.378045+0000","last_peered":"2024-09-21T18:55:31.378045+0000","last_clean":"2024-09-21T18:55:31.378045+0000","last_became_active":"2024-09-21T18:54:17.706866+0000","last_became_peered":"2024-09-21T18:54:17.706866+0000","last_unstale":"2024-09-21T18:55:31.378045+0000","last_undegraded":"2024-09-21T18:55:31.378045+0000","last_fullsized":"2024-09-21T18:55:31.378045+0000","mapping_epoch":82,"log_start":"88'1600","ondisk_log_start":"88'1600","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.130355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473366528,"num_objects":7223,"num_object_clones":0,"num_object_copies":21669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":195,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904217+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:55:30.904217+0000","last_peered":"2024-09-21T18:55:30.904217+0000","last_clean":"2024-09-21T18:55:30.904217+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:55:30.904217+0000","last_undegraded":"2024-09-21T18:55:30.904217+0000","last_fullsized":"2024-09-21T18:55:30.904217+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:53.227258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":129,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896586+0000","last_change":"2024-09-21T18:54:17.706922+0000","last_active":"2024-09-21T18:55:30.896586+0000","last_peered":"2024-09-21T18:55:30.896586+0000","last_clean":"2024-09-21T18:55:30.896586+0000","last_became_active":"2024-09-21T18:54:17.706635+0000","last_became_peered":"2024-09-21T18:54:17.706635+0000","last_unstale":"2024-09-21T18:55:30.896586+0000","last_undegraded":"2024-09-21T18:55:30.896586+0000","last_fullsized":"2024-09-21T18:55:30.896586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:09.963263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:09.963263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":132,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.903795+0000","last_change":"2024-09-21T18:54:47.179942+0000","last_active":"2024-09-21T18:55:30.903795+0000","last_peered":"2024-09-21T18:55:30.903795+0000","last_clean":"2024-09-21T18:55:30.903795+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:55:30.903795+0000","last_undegraded":"2024-09-21T18:55:30.903795+0000","last_fullsized":"2024-09-21T18:55:30.903795+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:54:47.179869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.179869+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:50.966329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904223+0000","last_change":"2024-09-21T18:54:41.369741+0000","last_active":"2024-09-21T18:55:30.904223+0000","last_peered":"2024-09-21T18:55:30.904223+0000","last_clean":"2024-09-21T18:55:30.904223+0000","last_became_active":"2024-09-21T18:54:17.712729+0000","last_became_peered":"2024-09-21T18:54:17.712729+0000","last_unstale":"2024-09-21T18:55:30.904223+0000","last_undegraded":"2024-09-21T18:55:30.904223+0000","last_fullsized":"2024-09-21T18:55:30.904223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:55.719422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896435+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:55:30.896435+0000","last_peered":"2024-09-21T18:55:30.896435+0000","last_clean":"2024-09-21T18:55:30.896435+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:55:30.896435+0000","last_undegraded":"2024-09-21T18:55:30.896435+0000","last_fullsized":"2024-09-21T18:55:30.896435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:41.866717+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"109'7240","reported_seq":7306,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.395736+0000","last_change":"2024-09-21T18:54:17.706551+0000","last_active":"2024-09-21T18:55:31.395736+0000","last_peered":"2024-09-21T18:55:31.395736+0000","last_clean":"2024-09-21T18:55:31.395736+0000","last_became_active":"2024-09-21T18:54:17.706169+0000","last_became_peered":"2024-09-21T18:54:17.706169+0000","last_unstale":"2024-09-21T18:55:31.395736+0000","last_undegraded":"2024-09-21T18:55:31.395736+0000","last_fullsized":"2024-09-21T18:55:31.395736+0000","mapping_epoch":82,"log_start":"90'2700","ondisk_log_start":"90'2700","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.094421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895273+0000","last_change":"2024-09-21T18:54:40.059601+0000","last_active":"2024-09-21T18:55:30.895273+0000","last_peered":"2024-09-21T18:55:30.895273+0000","last_clean":"2024-09-21T18:55:30.895273+0000","last_became_active":"2024-09-21T18:54:36.751672+0000","last_became_peered":"2024-09-21T18:54:36.751672+0000","last_unstale":"2024-09-21T18:55:30.895273+0000","last_undegraded":"2024-09-21T18:55:30.895273+0000","last_fullsized":"2024-09-21T18:55:30.895273+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:40.628526+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":120,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904702+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:55:30.904702+0000","last_peered":"2024-09-21T18:55:30.904702+0000","last_clean":"2024-09-21T18:55:30.904702+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:55:30.904702+0000","last_undegraded":"2024-09-21T18:55:30.904702+0000","last_fullsized":"2024-09-21T18:55:30.904702+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.531180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":142,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.894910+0000","last_change":"2024-09-21T18:55:25.237254+0000","last_active":"2024-09-21T18:55:30.894910+0000","last_peered":"2024-09-21T18:55:30.894910+0000","last_clean":"2024-09-21T18:55:30.894910+0000","last_became_active":"2024-09-21T18:54:17.710632+0000","last_became_peered":"2024-09-21T18:54:17.710632+0000","last_unstale":"2024-09-21T18:55:30.894910+0000","last_undegraded":"2024-09-21T18:55:30.894910+0000","last_fullsized":"2024-09-21T18:55:30.894910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:25.237173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.237173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:40.388556+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899440+0000","last_change":"2024-09-21T18:53:11.413811+0000","last_active":"2024-09-21T18:55:30.899440+0000","last_peered":"2024-09-21T18:55:30.899440+0000","last_clean":"2024-09-21T18:55:30.899440+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:55:30.899440+0000","last_undegraded":"2024-09-21T18:55:30.899440+0000","last_fullsized":"2024-09-21T18:55:30.899440+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.969938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900346+0000","last_change":"2024-09-21T18:53:13.657616+0000","last_active":"2024-09-21T18:55:30.900346+0000","last_peered":"2024-09-21T18:55:30.900346+0000","last_clean":"2024-09-21T18:55:30.900346+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:55:30.900346+0000","last_undegraded":"2024-09-21T18:55:30.900346+0000","last_fullsized":"2024-09-21T18:55:30.900346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"109'7392","reported_seq":7470,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.384255+0000","last_change":"2024-09-21T18:54:17.717837+0000","last_active":"2024-09-21T18:55:31.384255+0000","last_peered":"2024-09-21T18:55:31.384255+0000","last_clean":"2024-09-21T18:55:31.384255+0000","last_became_active":"2024-09-21T18:54:17.717525+0000","last_became_peered":"2024-09-21T18:54:17.717525+0000","last_unstale":"2024-09-21T18:55:31.384255+0000","last_undegraded":"2024-09-21T18:55:31.384255+0000","last_fullsized":"2024-09-21T18:55:31.384255+0000","mapping_epoch":82,"log_start":"90'2600","ondisk_log_start":"90'2600","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484442112,"num_objects":7392,"num_object_clones":0,"num_object_copies":22176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":473088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":125,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904200+0000","last_change":"2024-09-21T18:55:23.675202+0000","last_active":"2024-09-21T18:55:30.904200+0000","last_peered":"2024-09-21T18:55:30.904200+0000","last_clean":"2024-09-21T18:55:30.904200+0000","last_became_active":"2024-09-21T18:54:17.720574+0000","last_became_peered":"2024-09-21T18:54:17.720574+0000","last_unstale":"2024-09-21T18:55:30.904200+0000","last_undegraded":"2024-09-21T18:55:30.904200+0000","last_fullsized":"2024-09-21T18:55:30.904200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:50.745330+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905106+0000","last_change":"2024-09-21T18:55:22.082609+0000","last_active":"2024-09-21T18:55:30.905106+0000","last_peered":"2024-09-21T18:55:30.905106+0000","last_clean":"2024-09-21T18:55:30.905106+0000","last_became_active":"2024-09-21T18:54:17.704296+0000","last_became_peered":"2024-09-21T18:54:17.704296+0000","last_unstale":"2024-09-21T18:55:30.905106+0000","last_undegraded":"2024-09-21T18:55:30.905106+0000","last_fullsized":"2024-09-21T18:55:30.905106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:40.041103+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":282,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899139+0000","last_change":"2024-09-21T18:54:41.110834+0000","last_active":"2024-09-21T18:55:30.899139+0000","last_peered":"2024-09-21T18:55:30.899139+0000","last_clean":"2024-09-21T18:55:30.899139+0000","last_became_active":"2024-09-21T18:54:17.713703+0000","last_became_peered":"2024-09-21T18:54:17.713703+0000","last_unstale":"2024-09-21T18:55:30.899139+0000","last_undegraded":"2024-09-21T18:55:30.899139+0000","last_fullsized":"2024-09-21T18:55:30.899139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:41.110776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.110776+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.027705+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":137,"num_read_kb":132,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911448+0000","last_change":"2024-09-21T18:54:40.112915+0000","last_active":"2024-09-21T18:55:30.911448+0000","last_peered":"2024-09-21T18:55:30.911448+0000","last_clean":"2024-09-21T18:55:30.911448+0000","last_became_active":"2024-09-21T18:54:17.707817+0000","last_became_peered":"2024-09-21T18:54:17.707817+0000","last_unstale":"2024-09-21T18:55:30.911448+0000","last_undegraded":"2024-09-21T18:55:30.911448+0000","last_fullsized":"2024-09-21T18:55:30.911448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.112826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.112826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:08.711823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.903877+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:55:30.903877+0000","last_peered":"2024-09-21T18:55:30.903877+0000","last_clean":"2024-09-21T18:55:30.903877+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:55:30.903877+0000","last_undegraded":"2024-09-21T18:55:30.903877+0000","last_fullsized":"2024-09-21T18:55:30.903877+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:01.474112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"109'7177","reported_seq":7243,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.390847+0000","last_change":"2024-09-21T18:54:17.719358+0000","last_active":"2024-09-21T18:55:31.390847+0000","last_peered":"2024-09-21T18:55:31.390847+0000","last_clean":"2024-09-21T18:55:31.390847+0000","last_became_active":"2024-09-21T18:54:17.718730+0000","last_became_peered":"2024-09-21T18:54:17.718730+0000","last_unstale":"2024-09-21T18:55:31.390847+0000","last_undegraded":"2024-09-21T18:55:31.390847+0000","last_fullsized":"2024-09-21T18:55:31.390847+0000","mapping_epoch":82,"log_start":"90'2400","ondisk_log_start":"90'2400","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.846251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470351872,"num_objects":7177,"num_object_clones":0,"num_object_copies":21531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7177,"num_write_kb":459328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":168,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902956+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:55:30.902956+0000","last_peered":"2024-09-21T18:55:30.902956+0000","last_clean":"2024-09-21T18:55:30.902956+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:55:30.902956+0000","last_undegraded":"2024-09-21T18:55:30.902956+0000","last_fullsized":"2024-09-21T18:55:30.902956+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:40.490018+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":123,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.069114+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:55:31.069114+0000","last_peered":"2024-09-21T18:55:31.069114+0000","last_clean":"2024-09-21T18:55:31.069114+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:55:31.069114+0000","last_undegraded":"2024-09-21T18:55:31.069114+0000","last_fullsized":"2024-09-21T18:55:31.069114+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:36.735348+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":171,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899896+0000","last_change":"2024-09-21T18:54:17.736002+0000","last_active":"2024-09-21T18:55:30.899896+0000","last_peered":"2024-09-21T18:55:30.899896+0000","last_clean":"2024-09-21T18:55:30.899896+0000","last_became_active":"2024-09-21T18:54:17.716544+0000","last_became_peered":"2024-09-21T18:54:17.716544+0000","last_unstale":"2024-09-21T18:55:30.899896+0000","last_undegraded":"2024-09-21T18:55:30.899896+0000","last_fullsized":"2024-09-21T18:55:30.899896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:16.070251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":105,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900273+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:55:30.900273+0000","last_peered":"2024-09-21T18:55:30.900273+0000","last_clean":"2024-09-21T18:55:30.900273+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:55:30.900273+0000","last_undegraded":"2024-09-21T18:55:30.900273+0000","last_fullsized":"2024-09-21T18:55:30.900273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:14.064293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":102,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911364+0000","last_change":"2024-09-21T18:53:13.430342+0000","last_active":"2024-09-21T18:55:30.911364+0000","last_peered":"2024-09-21T18:55:30.911364+0000","last_clean":"2024-09-21T18:55:30.911364+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:55:30.911364+0000","last_undegraded":"2024-09-21T18:55:30.911364+0000","last_fullsized":"2024-09-21T18:55:30.911364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:09.291303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"109'7226","reported_seq":7304,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.406749+0000","last_change":"2024-09-21T18:54:17.710054+0000","last_active":"2024-09-21T18:55:31.406749+0000","last_peered":"2024-09-21T18:55:31.406749+0000","last_clean":"2024-09-21T18:55:31.406749+0000","last_became_active":"2024-09-21T18:54:17.709698+0000","last_became_peered":"2024-09-21T18:54:17.709698+0000","last_unstale":"2024-09-21T18:55:31.406749+0000","last_undegraded":"2024-09-21T18:55:31.406749+0000","last_fullsized":"2024-09-21T18:55:31.406749+0000","mapping_epoch":82,"log_start":"88'1600","ondisk_log_start":"88'1600","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.567144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473563136,"num_objects":7226,"num_object_clones":0,"num_object_copies":21678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7226,"num_write_kb":462464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":129,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902047+0000","last_change":"2024-09-21T18:53:09.407269+0000","last_active":"2024-09-21T18:55:30.902047+0000","last_peered":"2024-09-21T18:55:30.902047+0000","last_clean":"2024-09-21T18:55:30.902047+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:55:30.902047+0000","last_undegraded":"2024-09-21T18:55:30.902047+0000","last_fullsized":"2024-09-21T18:55:30.902047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:31.929416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905421+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:55:30.905421+0000","last_peered":"2024-09-21T18:55:30.905421+0000","last_clean":"2024-09-21T18:55:30.905421+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:55:30.905421+0000","last_undegraded":"2024-09-21T18:55:30.905421+0000","last_fullsized":"2024-09-21T18:55:30.905421+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.012525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911297+0000","last_change":"2024-09-21T18:54:50.139865+0000","last_active":"2024-09-21T18:55:30.911297+0000","last_peered":"2024-09-21T18:55:30.911297+0000","last_clean":"2024-09-21T18:55:30.911297+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:55:30.911297+0000","last_undegraded":"2024-09-21T18:55:30.911297+0000","last_fullsized":"2024-09-21T18:55:30.911297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.139807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.139807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.360746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895531+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:55:30.895531+0000","last_peered":"2024-09-21T18:55:30.895531+0000","last_clean":"2024-09-21T18:55:30.895531+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:55:30.895531+0000","last_undegraded":"2024-09-21T18:55:30.895531+0000","last_fullsized":"2024-09-21T18:55:30.895531+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:18.885373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"109'7175","reported_seq":7333,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.402230+0000","last_change":"2024-09-21T18:55:29.650392+0000","last_active":"2024-09-21T18:55:31.402230+0000","last_peered":"2024-09-21T18:55:31.402230+0000","last_clean":"2024-09-21T18:55:31.402230+0000","last_became_active":"2024-09-21T18:54:17.707839+0000","last_became_peered":"2024-09-21T18:54:17.707839+0000","last_unstale":"2024-09-21T18:55:31.402230+0000","last_undegraded":"2024-09-21T18:55:31.402230+0000","last_fullsized":"2024-09-21T18:55:31.402230+0000","mapping_epoch":82,"log_start":"88'1700","ondisk_log_start":"88'1700","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:48.209775+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470220800,"num_objects":7175,"num_object_clones":0,"num_object_copies":21525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7175,"num_write_kb":459200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":138,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897055+0000","last_change":"2024-09-21T18:53:09.406752+0000","last_active":"2024-09-21T18:55:30.897055+0000","last_peered":"2024-09-21T18:55:30.897055+0000","last_clean":"2024-09-21T18:55:30.897055+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:55:30.897055+0000","last_undegraded":"2024-09-21T18:55:30.897055+0000","last_fullsized":"2024-09-21T18:55:30.897055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.077501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.901459+0000","last_change":"2024-09-21T18:53:07.400601+0000","last_active":"2024-09-21T18:55:30.901459+0000","last_peered":"2024-09-21T18:55:30.901459+0000","last_clean":"2024-09-21T18:55:30.901459+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:55:30.901459+0000","last_undegraded":"2024-09-21T18:55:30.901459+0000","last_fullsized":"2024-09-21T18:55:30.901459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.903972+0000","last_change":"2024-09-21T18:54:17.714450+0000","last_active":"2024-09-21T18:55:30.903972+0000","last_peered":"2024-09-21T18:55:30.903972+0000","last_clean":"2024-09-21T18:55:30.903972+0000","last_became_active":"2024-09-21T18:54:17.714167+0000","last_became_peered":"2024-09-21T18:54:17.714167+0000","last_unstale":"2024-09-21T18:55:30.903972+0000","last_undegraded":"2024-09-21T18:55:30.903972+0000","last_fullsized":"2024-09-21T18:55:30.903972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:06.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895413+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:55:30.895413+0000","last_peered":"2024-09-21T18:55:30.895413+0000","last_clean":"2024-09-21T18:55:30.895413+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:55:30.895413+0000","last_undegraded":"2024-09-21T18:55:30.895413+0000","last_fullsized":"2024-09-21T18:55:30.895413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:50.896923+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"109'7173","reported_seq":7239,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.509251+0000","last_change":"2024-09-21T18:54:17.718909+0000","last_active":"2024-09-21T18:55:31.509251+0000","last_peered":"2024-09-21T18:55:31.509251+0000","last_clean":"2024-09-21T18:55:31.509251+0000","last_became_active":"2024-09-21T18:54:17.718631+0000","last_became_peered":"2024-09-21T18:54:17.718631+0000","last_unstale":"2024-09-21T18:55:31.509251+0000","last_undegraded":"2024-09-21T18:55:31.509251+0000","last_fullsized":"2024-09-21T18:55:31.509251+0000","mapping_epoch":82,"log_start":"93'3000","ondisk_log_start":"93'3000","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:32.749545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470089728,"num_objects":7173,"num_object_clones":0,"num_object_copies":21519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":459072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":125,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896639+0000","last_change":"2024-09-21T18:53:09.401652+0000","last_active":"2024-09-21T18:55:30.896639+0000","last_peered":"2024-09-21T18:55:30.896639+0000","last_clean":"2024-09-21T18:55:30.896639+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:55:30.896639+0000","last_undegraded":"2024-09-21T18:55:30.896639+0000","last_fullsized":"2024-09-21T18:55:30.896639+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.102897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":140,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896060+0000","last_change":"2024-09-21T18:54:17.724937+0000","last_active":"2024-09-21T18:55:30.896060+0000","last_peered":"2024-09-21T18:55:30.896060+0000","last_clean":"2024-09-21T18:55:30.896060+0000","last_became_active":"2024-09-21T18:54:17.710657+0000","last_became_peered":"2024-09-21T18:54:17.710657+0000","last_unstale":"2024-09-21T18:55:30.896060+0000","last_undegraded":"2024-09-21T18:55:30.896060+0000","last_fullsized":"2024-09-21T18:55:30.896060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":151,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896132+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:55:30.896132+0000","last_peered":"2024-09-21T18:55:30.896132+0000","last_clean":"2024-09-21T18:55:30.896132+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:55:30.896132+0000","last_undegraded":"2024-09-21T18:55:30.896132+0000","last_fullsized":"2024-09-21T18:55:30.896132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:36.583654+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.901829+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:55:30.901829+0000","last_peered":"2024-09-21T18:55:30.901829+0000","last_clean":"2024-09-21T18:55:30.901829+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:55:30.901829+0000","last_undegraded":"2024-09-21T18:55:30.901829+0000","last_fullsized":"2024-09-21T18:55:30.901829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:04.802370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"109'7252","reported_seq":7403,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.410944+0000","last_change":"2024-09-21T18:55:30.532749+0000","last_active":"2024-09-21T18:55:31.410944+0000","last_peered":"2024-09-21T18:55:31.410944+0000","last_clean":"2024-09-21T18:55:31.410944+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:55:31.410944+0000","last_undegraded":"2024-09-21T18:55:31.410944+0000","last_fullsized":"2024-09-21T18:55:31.410944+0000","mapping_epoch":79,"log_start":"88'1700","ondisk_log_start":"88'1700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7201","last_scrub_stamp":"2024-09-21T18:55:30.532526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.532526+0000","objects_scrubbed":7133,"log_size":5552,"log_dups_size":0,"ondisk_log_size":5552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:46.105611+0000","scrub_duration":2233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475267072,"num_objects":7252,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7252,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":119,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905519+0000","last_change":"2024-09-21T18:54:52.386165+0000","last_active":"2024-09-21T18:55:30.905519+0000","last_peered":"2024-09-21T18:55:30.905519+0000","last_clean":"2024-09-21T18:55:30.905519+0000","last_became_active":"2024-09-21T18:54:17.714624+0000","last_became_peered":"2024-09-21T18:54:17.714624+0000","last_unstale":"2024-09-21T18:55:30.905519+0000","last_undegraded":"2024-09-21T18:55:30.905519+0000","last_fullsized":"2024-09-21T18:55:30.905519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:00.346786+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902595+0000","last_change":"2024-09-21T18:54:13.280500+0000","last_active":"2024-09-21T18:55:30.902595+0000","last_peered":"2024-09-21T18:55:30.902595+0000","last_clean":"2024-09-21T18:55:30.902595+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:55:30.902595+0000","last_undegraded":"2024-09-21T18:55:30.902595+0000","last_fullsized":"2024-09-21T18:55:30.902595+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.280431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.280431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.151455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895905+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:55:30.895905+0000","last_peered":"2024-09-21T18:55:30.895905+0000","last_clean":"2024-09-21T18:55:30.895905+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:55:30.895905+0000","last_undegraded":"2024-09-21T18:55:30.895905+0000","last_fullsized":"2024-09-21T18:55:30.895905+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:49.052101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902844+0000","last_change":"2024-09-21T18:54:17.717827+0000","last_active":"2024-09-21T18:55:30.902844+0000","last_peered":"2024-09-21T18:55:30.902844+0000","last_clean":"2024-09-21T18:55:30.902844+0000","last_became_active":"2024-09-21T18:54:17.717507+0000","last_became_peered":"2024-09-21T18:54:17.717507+0000","last_unstale":"2024-09-21T18:55:30.902844+0000","last_undegraded":"2024-09-21T18:55:30.902844+0000","last_fullsized":"2024-09-21T18:55:30.902844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"109'7241","reported_seq":7307,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.440514+0000","last_change":"2024-09-21T18:54:17.712034+0000","last_active":"2024-09-21T18:55:31.440514+0000","last_peered":"2024-09-21T18:55:31.440514+0000","last_clean":"2024-09-21T18:55:31.440514+0000","last_became_active":"2024-09-21T18:54:17.711698+0000","last_became_peered":"2024-09-21T18:54:17.711698+0000","last_unstale":"2024-09-21T18:55:31.440514+0000","last_undegraded":"2024-09-21T18:55:31.440514+0000","last_fullsized":"2024-09-21T18:55:31.440514+0000","mapping_epoch":82,"log_start":"93'3000","ondisk_log_start":"93'3000","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:39.183316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474546176,"num_objects":7241,"num_object_clones":0,"num_object_copies":21723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":463424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":140,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.901262+0000","last_change":"2024-09-21T18:53:09.717996+0000","last_active":"2024-09-21T18:55:30.901262+0000","last_peered":"2024-09-21T18:55:30.901262+0000","last_clean":"2024-09-21T18:55:30.901262+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:55:30.901262+0000","last_undegraded":"2024-09-21T18:55:30.901262+0000","last_fullsized":"2024-09-21T18:55:30.901262+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:38.210774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900701+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:55:30.900701+0000","last_peered":"2024-09-21T18:55:30.900701+0000","last_clean":"2024-09-21T18:55:30.900701+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:55:30.900701+0000","last_undegraded":"2024-09-21T18:55:30.900701+0000","last_fullsized":"2024-09-21T18:55:30.900701+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:50.542633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895366+0000","last_change":"2024-09-21T18:54:17.717687+0000","last_active":"2024-09-21T18:55:30.895366+0000","last_peered":"2024-09-21T18:55:30.895366+0000","last_clean":"2024-09-21T18:55:30.895366+0000","last_became_active":"2024-09-21T18:54:17.717380+0000","last_became_peered":"2024-09-21T18:54:17.717380+0000","last_unstale":"2024-09-21T18:55:30.895366+0000","last_undegraded":"2024-09-21T18:55:30.895366+0000","last_fullsized":"2024-09-21T18:55:30.895366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:30.963253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899229+0000","last_change":"2024-09-21T18:55:24.049546+0000","last_active":"2024-09-21T18:55:30.899229+0000","last_peered":"2024-09-21T18:55:30.899229+0000","last_clean":"2024-09-21T18:55:30.899229+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:55:30.899229+0000","last_undegraded":"2024-09-21T18:55:30.899229+0000","last_fullsized":"2024-09-21T18:55:30.899229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.049407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.049407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:40.272741+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"109'7081","reported_seq":7152,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.481528+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:55:31.481528+0000","last_peered":"2024-09-21T18:55:31.481528+0000","last_clean":"2024-09-21T18:55:31.481528+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:55:31.481528+0000","last_undegraded":"2024-09-21T18:55:31.481528+0000","last_fullsized":"2024-09-21T18:55:31.481528+0000","mapping_epoch":79,"log_start":"92'2900","ondisk_log_start":"92'2900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:34.423918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464060416,"num_objects":7081,"num_object_clones":0,"num_object_copies":21243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":453184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":160,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.905292+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:55:30.905292+0000","last_peered":"2024-09-21T18:55:30.905292+0000","last_clean":"2024-09-21T18:55:30.905292+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:55:30.905292+0000","last_undegraded":"2024-09-21T18:55:30.905292+0000","last_fullsized":"2024-09-21T18:55:30.905292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:51.259913+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899996+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:55:30.899996+0000","last_peered":"2024-09-21T18:55:30.899996+0000","last_clean":"2024-09-21T18:55:30.899996+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:55:30.899996+0000","last_undegraded":"2024-09-21T18:55:30.899996+0000","last_fullsized":"2024-09-21T18:55:30.899996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:18.077219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.894794+0000","last_change":"2024-09-21T18:54:17.708159+0000","last_active":"2024-09-21T18:55:30.894794+0000","last_peered":"2024-09-21T18:55:30.894794+0000","last_clean":"2024-09-21T18:55:30.894794+0000","last_became_active":"2024-09-21T18:54:17.707830+0000","last_became_peered":"2024-09-21T18:54:17.707830+0000","last_unstale":"2024-09-21T18:55:30.894794+0000","last_undegraded":"2024-09-21T18:55:30.894794+0000","last_fullsized":"2024-09-21T18:55:30.894794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:11.994472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.994472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895789+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:55:30.895789+0000","last_peered":"2024-09-21T18:55:30.895789+0000","last_clean":"2024-09-21T18:55:30.895789+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:55:30.895789+0000","last_undegraded":"2024-09-21T18:55:30.895789+0000","last_fullsized":"2024-09-21T18:55:30.895789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.087297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"109'7098","reported_seq":7168,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.458183+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:55:31.458183+0000","last_peered":"2024-09-21T18:55:31.458183+0000","last_clean":"2024-09-21T18:55:31.458183+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:55:31.458183+0000","last_undegraded":"2024-09-21T18:55:31.458183+0000","last_fullsized":"2024-09-21T18:55:31.458183+0000","mapping_epoch":79,"log_start":"92'2800","ondisk_log_start":"92'2800","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465174528,"num_objects":7098,"num_object_clones":0,"num_object_copies":21294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7098,"num_write_kb":454272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":143,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.906748+0000","last_change":"2024-09-21T18:54:44.092986+0000","last_active":"2024-09-21T18:55:30.906748+0000","last_peered":"2024-09-21T18:55:30.906748+0000","last_clean":"2024-09-21T18:55:30.906748+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:55:30.906748+0000","last_undegraded":"2024-09-21T18:55:30.906748+0000","last_fullsized":"2024-09-21T18:55:30.906748+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:44.092903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:44.092903+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:10.777104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":138,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900810+0000","last_change":"2024-09-21T18:54:17.732357+0000","last_active":"2024-09-21T18:55:30.900810+0000","last_peered":"2024-09-21T18:55:30.900810+0000","last_clean":"2024-09-21T18:55:30.900810+0000","last_became_active":"2024-09-21T18:54:17.715268+0000","last_became_peered":"2024-09-21T18:54:17.715268+0000","last_unstale":"2024-09-21T18:55:30.900810+0000","last_undegraded":"2024-09-21T18:55:30.900810+0000","last_fullsized":"2024-09-21T18:55:30.900810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:35.049285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":145,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.903076+0000","last_change":"2024-09-21T18:53:11.413849+0000","last_active":"2024-09-21T18:55:30.903076+0000","last_peered":"2024-09-21T18:55:30.903076+0000","last_clean":"2024-09-21T18:55:30.903076+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:55:30.903076+0000","last_undegraded":"2024-09-21T18:55:30.903076+0000","last_fullsized":"2024-09-21T18:55:30.903076+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":117,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902640+0000","last_change":"2024-09-21T18:54:53.051485+0000","last_active":"2024-09-21T18:55:30.902640+0000","last_peered":"2024-09-21T18:55:30.902640+0000","last_clean":"2024-09-21T18:55:30.902640+0000","last_became_active":"2024-09-21T18:54:17.713977+0000","last_became_peered":"2024-09-21T18:54:17.713977+0000","last_unstale":"2024-09-21T18:55:30.902640+0000","last_undegraded":"2024-09-21T18:55:30.902640+0000","last_fullsized":"2024-09-21T18:55:30.902640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:07.375794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900969+0000","last_change":"2024-09-21T18:55:24.758362+0000","last_active":"2024-09-21T18:55:30.900969+0000","last_peered":"2024-09-21T18:55:30.900969+0000","last_clean":"2024-09-21T18:55:30.900969+0000","last_became_active":"2024-09-21T18:54:17.713066+0000","last_became_peered":"2024-09-21T18:54:17.713066+0000","last_unstale":"2024-09-21T18:55:30.900969+0000","last_undegraded":"2024-09-21T18:55:30.900969+0000","last_fullsized":"2024-09-21T18:55:30.900969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:51.736347+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":119,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.898084+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:55:30.898084+0000","last_peered":"2024-09-21T18:55:30.898084+0000","last_clean":"2024-09-21T18:55:30.898084+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:55:30.898084+0000","last_undegraded":"2024-09-21T18:55:30.898084+0000","last_fullsized":"2024-09-21T18:55:30.898084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:58.061399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.910292+0000","last_change":"2024-09-21T18:55:28.120107+0000","last_active":"2024-09-21T18:55:30.910292+0000","last_peered":"2024-09-21T18:55:30.910292+0000","last_clean":"2024-09-21T18:55:30.910292+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:55:30.910292+0000","last_undegraded":"2024-09-21T18:55:30.910292+0000","last_fullsized":"2024-09-21T18:55:30.910292+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.119928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.119928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:38.372293+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":100,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900262+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:55:30.900262+0000","last_peered":"2024-09-21T18:55:30.900262+0000","last_clean":"2024-09-21T18:55:30.900262+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:55:30.900262+0000","last_undegraded":"2024-09-21T18:55:30.900262+0000","last_fullsized":"2024-09-21T18:55:30.900262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:11.051623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":119,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896570+0000","last_change":"2024-09-21T18:54:51.117250+0000","last_active":"2024-09-21T18:55:30.896570+0000","last_peered":"2024-09-21T18:55:30.896570+0000","last_clean":"2024-09-21T18:55:30.896570+0000","last_became_active":"2024-09-21T18:54:17.719720+0000","last_became_peered":"2024-09-21T18:54:17.719720+0000","last_unstale":"2024-09-21T18:55:30.896570+0000","last_undegraded":"2024-09-21T18:55:30.896570+0000","last_fullsized":"2024-09-21T18:55:30.896570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.430477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.903265+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:55:30.903265+0000","last_peered":"2024-09-21T18:55:30.903265+0000","last_clean":"2024-09-21T18:55:30.903265+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:55:30.903265+0000","last_undegraded":"2024-09-21T18:55:30.903265+0000","last_fullsized":"2024-09-21T18:55:30.903265+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:36.194383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897991+0000","last_change":"2024-09-21T18:53:11.410408+0000","last_active":"2024-09-21T18:55:30.897991+0000","last_peered":"2024-09-21T18:55:30.897991+0000","last_clean":"2024-09-21T18:55:30.897991+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:55:30.897991+0000","last_undegraded":"2024-09-21T18:55:30.897991+0000","last_fullsized":"2024-09-21T18:55:30.897991+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.003241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.898734+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:55:30.898734+0000","last_peered":"2024-09-21T18:55:30.898734+0000","last_clean":"2024-09-21T18:55:30.898734+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:55:30.898734+0000","last_undegraded":"2024-09-21T18:55:30.898734+0000","last_fullsized":"2024-09-21T18:55:30.898734+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:53.334288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":126,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911151+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:55:30.911151+0000","last_peered":"2024-09-21T18:55:30.911151+0000","last_clean":"2024-09-21T18:55:30.911151+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:55:30.911151+0000","last_undegraded":"2024-09-21T18:55:30.911151+0000","last_fullsized":"2024-09-21T18:55:30.911151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:17.222845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899165+0000","last_change":"2024-09-21T18:54:17.717912+0000","last_active":"2024-09-21T18:55:30.899165+0000","last_peered":"2024-09-21T18:55:30.899165+0000","last_clean":"2024-09-21T18:55:30.899165+0000","last_became_active":"2024-09-21T18:54:17.717617+0000","last_became_peered":"2024-09-21T18:54:17.717617+0000","last_unstale":"2024-09-21T18:55:30.899165+0000","last_undegraded":"2024-09-21T18:55:30.899165+0000","last_fullsized":"2024-09-21T18:55:30.899165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:12.058747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":152,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895629+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:55:30.895629+0000","last_peered":"2024-09-21T18:55:30.895629+0000","last_clean":"2024-09-21T18:55:30.895629+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:55:30.895629+0000","last_undegraded":"2024-09-21T18:55:30.895629+0000","last_fullsized":"2024-09-21T18:55:30.895629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:27.681314+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897107+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:55:30.897107+0000","last_peered":"2024-09-21T18:55:30.897107+0000","last_clean":"2024-09-21T18:55:30.897107+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:55:30.897107+0000","last_undegraded":"2024-09-21T18:55:30.897107+0000","last_fullsized":"2024-09-21T18:55:30.897107+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:14.128893+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":126,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897585+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:55:30.897585+0000","last_peered":"2024-09-21T18:55:30.897585+0000","last_clean":"2024-09-21T18:55:30.897585+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:55:30.897585+0000","last_undegraded":"2024-09-21T18:55:30.897585+0000","last_fullsized":"2024-09-21T18:55:30.897585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:01.112400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899850+0000","last_change":"2024-09-21T18:54:40.089802+0000","last_active":"2024-09-21T18:55:30.899850+0000","last_peered":"2024-09-21T18:55:30.899850+0000","last_clean":"2024-09-21T18:55:30.899850+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:55:30.899850+0000","last_undegraded":"2024-09-21T18:55:30.899850+0000","last_fullsized":"2024-09-21T18:55:30.899850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.089723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.089723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:05.272819+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"83'9","reported_seq":153,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911013+0000","last_change":"2024-09-21T18:54:17.726533+0000","last_active":"2024-09-21T18:55:30.911013+0000","last_peered":"2024-09-21T18:55:30.911013+0000","last_clean":"2024-09-21T18:55:30.911013+0000","last_became_active":"2024-09-21T18:54:17.718821+0000","last_became_peered":"2024-09-21T18:54:17.718821+0000","last_unstale":"2024-09-21T18:55:30.911013+0000","last_undegraded":"2024-09-21T18:55:30.911013+0000","last_fullsized":"2024-09-21T18:55:30.911013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:02.213185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899171+0000","last_change":"2024-09-21T18:54:17.719309+0000","last_active":"2024-09-21T18:55:30.899171+0000","last_peered":"2024-09-21T18:55:30.899171+0000","last_clean":"2024-09-21T18:55:30.899171+0000","last_became_active":"2024-09-21T18:54:17.718560+0000","last_became_peered":"2024-09-21T18:54:17.718560+0000","last_unstale":"2024-09-21T18:55:30.899171+0000","last_undegraded":"2024-09-21T18:55:30.899171+0000","last_fullsized":"2024-09-21T18:55:30.899171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":148,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.896354+0000","last_change":"2024-09-21T18:54:36.773903+0000","last_active":"2024-09-21T18:55:30.896354+0000","last_peered":"2024-09-21T18:55:30.896354+0000","last_clean":"2024-09-21T18:55:30.896354+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:55:30.896354+0000","last_undegraded":"2024-09-21T18:55:30.896354+0000","last_fullsized":"2024-09-21T18:55:30.896354+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:08.081305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902493+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:55:30.902493+0000","last_peered":"2024-09-21T18:55:30.902493+0000","last_clean":"2024-09-21T18:55:30.902493+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:55:30.902493+0000","last_undegraded":"2024-09-21T18:55:30.902493+0000","last_fullsized":"2024-09-21T18:55:30.902493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:18.054974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904063+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:55:30.904063+0000","last_peered":"2024-09-21T18:55:30.904063+0000","last_clean":"2024-09-21T18:55:30.904063+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:55:30.904063+0000","last_undegraded":"2024-09-21T18:55:30.904063+0000","last_fullsized":"2024-09-21T18:55:30.904063+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:09.539693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895717+0000","last_change":"2024-09-21T18:54:17.717220+0000","last_active":"2024-09-21T18:55:30.895717+0000","last_peered":"2024-09-21T18:55:30.895717+0000","last_clean":"2024-09-21T18:55:30.895717+0000","last_became_active":"2024-09-21T18:54:17.716004+0000","last_became_peered":"2024-09-21T18:54:17.716004+0000","last_unstale":"2024-09-21T18:55:30.895717+0000","last_undegraded":"2024-09-21T18:55:30.895717+0000","last_fullsized":"2024-09-21T18:55:30.895717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:15.950621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":126,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904550+0000","last_change":"2024-09-21T18:53:09.406111+0000","last_active":"2024-09-21T18:55:30.904550+0000","last_peered":"2024-09-21T18:55:30.904550+0000","last_clean":"2024-09-21T18:55:30.904550+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:55:30.904550+0000","last_undegraded":"2024-09-21T18:55:30.904550+0000","last_fullsized":"2024-09-21T18:55:30.904550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.900604+0000","last_change":"2024-09-21T18:54:52.054723+0000","last_active":"2024-09-21T18:55:30.900604+0000","last_peered":"2024-09-21T18:55:30.900604+0000","last_clean":"2024-09-21T18:55:30.900604+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:55:30.900604+0000","last_undegraded":"2024-09-21T18:55:30.900604+0000","last_fullsized":"2024-09-21T18:55:30.900604+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:52.054655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.054655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:17.905775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895865+0000","last_change":"2024-09-21T18:54:17.715073+0000","last_active":"2024-09-21T18:55:30.895865+0000","last_peered":"2024-09-21T18:55:30.895865+0000","last_clean":"2024-09-21T18:55:30.895865+0000","last_became_active":"2024-09-21T18:54:17.714759+0000","last_became_peered":"2024-09-21T18:54:17.714759+0000","last_unstale":"2024-09-21T18:55:30.895865+0000","last_undegraded":"2024-09-21T18:55:30.895865+0000","last_fullsized":"2024-09-21T18:55:30.895865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.906484+0000","last_change":"2024-09-21T18:54:17.712675+0000","last_active":"2024-09-21T18:55:30.906484+0000","last_peered":"2024-09-21T18:55:30.906484+0000","last_clean":"2024-09-21T18:55:30.906484+0000","last_became_active":"2024-09-21T18:54:17.712339+0000","last_became_peered":"2024-09-21T18:54:17.712339+0000","last_unstale":"2024-09-21T18:55:30.906484+0000","last_undegraded":"2024-09-21T18:55:30.906484+0000","last_fullsized":"2024-09-21T18:55:30.906484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:11.251204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":130,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895883+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:55:30.895883+0000","last_peered":"2024-09-21T18:55:30.895883+0000","last_clean":"2024-09-21T18:55:30.895883+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:55:30.895883+0000","last_undegraded":"2024-09-21T18:55:30.895883+0000","last_fullsized":"2024-09-21T18:55:30.895883+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.065143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.910373+0000","last_change":"2024-09-21T18:53:07.402429+0000","last_active":"2024-09-21T18:55:30.910373+0000","last_peered":"2024-09-21T18:55:30.910373+0000","last_clean":"2024-09-21T18:55:30.910373+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:55:30.910373+0000","last_undegraded":"2024-09-21T18:55:30.910373+0000","last_fullsized":"2024-09-21T18:55:30.910373+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:33.201819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:31.026916+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:55:31.026916+0000","last_peered":"2024-09-21T18:55:31.026916+0000","last_clean":"2024-09-21T18:55:31.026916+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:55:31.026916+0000","last_undegraded":"2024-09-21T18:55:31.026916+0000","last_fullsized":"2024-09-21T18:55:31.026916+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:47.356589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":109,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.895970+0000","last_change":"2024-09-21T18:54:17.709035+0000","last_active":"2024-09-21T18:55:30.895970+0000","last_peered":"2024-09-21T18:55:30.895970+0000","last_clean":"2024-09-21T18:55:30.895970+0000","last_became_active":"2024-09-21T18:54:17.708626+0000","last_became_peered":"2024-09-21T18:54:17.708626+0000","last_unstale":"2024-09-21T18:55:30.895970+0000","last_undegraded":"2024-09-21T18:55:30.895970+0000","last_fullsized":"2024-09-21T18:55:30.895970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:17.870602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":130,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.897808+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:55:30.897808+0000","last_peered":"2024-09-21T18:55:30.897808+0000","last_clean":"2024-09-21T18:55:30.897808+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:55:30.897808+0000","last_undegraded":"2024-09-21T18:55:30.897808+0000","last_fullsized":"2024-09-21T18:55:30.897808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:13.996232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.911525+0000","last_change":"2024-09-21T18:54:17.719435+0000","last_active":"2024-09-21T18:55:30.911525+0000","last_peered":"2024-09-21T18:55:30.911525+0000","last_clean":"2024-09-21T18:55:30.911525+0000","last_became_active":"2024-09-21T18:54:17.718927+0000","last_became_peered":"2024-09-21T18:54:17.718927+0000","last_unstale":"2024-09-21T18:55:30.911525+0000","last_undegraded":"2024-09-21T18:55:30.911525+0000","last_fullsized":"2024-09-21T18:55:30.911525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.142303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":105,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.902207+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:55:30.902207+0000","last_peered":"2024-09-21T18:55:30.902207+0000","last_clean":"2024-09-21T18:55:30.902207+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:55:30.902207+0000","last_undegraded":"2024-09-21T18:55:30.902207+0000","last_fullsized":"2024-09-21T18:55:30.902207+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:19.521468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":100,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.898900+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:55:30.898900+0000","last_peered":"2024-09-21T18:55:30.898900+0000","last_clean":"2024-09-21T18:55:30.898900+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:55:30.898900+0000","last_undegraded":"2024-09-21T18:55:30.898900+0000","last_fullsized":"2024-09-21T18:55:30.898900+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:59.129027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":101,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.904134+0000","last_change":"2024-09-21T18:53:13.427548+0000","last_active":"2024-09-21T18:55:30.904134+0000","last_peered":"2024-09-21T18:55:30.904134+0000","last_clean":"2024-09-21T18:55:30.904134+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:55:30.904134+0000","last_undegraded":"2024-09-21T18:55:30.904134+0000","last_fullsized":"2024-09-21T18:55:30.904134+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":129,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.898113+0000","last_change":"2024-09-21T18:54:17.704421+0000","last_active":"2024-09-21T18:55:30.898113+0000","last_peered":"2024-09-21T18:55:30.898113+0000","last_clean":"2024-09-21T18:55:30.898113+0000","last_became_active":"2024-09-21T18:54:17.704166+0000","last_became_peered":"2024-09-21T18:54:17.704166+0000","last_unstale":"2024-09-21T18:55:30.898113+0000","last_undegraded":"2024-09-21T18:55:30.898113+0000","last_fullsized":"2024-09-21T18:55:30.898113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":128,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.906042+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:55:30.906042+0000","last_peered":"2024-09-21T18:55:30.906042+0000","last_clean":"2024-09-21T18:55:30.906042+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:55:30.906042+0000","last_undegraded":"2024-09-21T18:55:30.906042+0000","last_fullsized":"2024-09-21T18:55:30.906042+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:56:16.478974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-21T18:55:30.899393+0000","last_change":"2024-09-21T18:54:17.719032+0000","last_active":"2024-09-21T18:55:30.899393+0000","last_peered":"2024-09-21T18:55:30.899393+0000","last_clean":"2024-09-21T18:55:30.899393+0000","last_became_active":"2024-09-21T18:54:17.718245+0000","last_became_peered":"2024-09-21T18:54:17.718245+0000","last_unstale":"2024-09-21T18:55:30.899393+0000","last_undegraded":"2024-09-21T18:55:30.899393+0000","last_fullsized":"2024-09-21T18:55:30.899393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:55:57.074471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7554859008,"num_objects":115278,"num_object_clones":0,"num_object_copies":345834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115278,"num_write_kb":7377792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22665822208,"data_stored":22665822208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76578,"ondisk_log_size":76578,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":2090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":137,"num_read_kb":132,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266721,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2840688,"kb_used_data":2765084,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":90904464,"statfs":{"total":95995035648,"available":93086171136,"internally_reserved":0,"allocated":2831446016,"data_stored":2830780136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528358,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2813256,"kb_used_data":2737648,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":90931896,"statfs":{"total":95995035648,"available":93114261504,"internally_reserved":0,"allocated":2803351552,"data_stored":2802722789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2353052,"kb_used_data":2285760,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91392100,"statfs":{"total":95995035648,"available":93585510400,"internally_reserved":0,"allocated":2340618240,"data_stored":2339979703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018927,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2830372,"kb_used_data":2754768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":90914780,"statfs":{"total":95995035648,"available":93096734720,"internally_reserved":0,"allocated":2820882432,"data_stored":2820228420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5167032,"kb_used_data":5041444,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":88578120,"statfs":{"total":95995035648,"available":90703994880,"internally_reserved":0,"allocated":5162438656,"data_stored":5161764962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2832096,"kb_used_data":2764812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":90913056,"statfs":{"total":95995035648,"available":93094969344,"internally_reserved":0,"allocated":2831167488,"data_stored":2830517328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3766464,"kb_used_data":3685296,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":89978688,"statfs":{"total":95995035648,"available":92138176512,"internally_reserved":0,"allocated":3773743104,"data_stored":3773051442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3801939968,"data_stored":3801939968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2834038784,"data_stored":2834038784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5185142784,"data_stored":5185142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2831024128,"data_stored":2831024128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2348875776,"data_stored":2348875776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2824404992,"data_stored":2824404992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2840395776,"data_stored":2840395776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:55:38.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-21T18:55:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:38.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:38.450 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:38.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:38.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T18:55:38.701 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:38.753 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:38.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:38.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:38.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:38.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:38 smithi104 ceph-mon[32438]: 3.c scrub starts 2024-09-21T18:55:38.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:38 smithi104 ceph-mon[32438]: 3.c scrub ok 2024-09-21T18:55:38.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:38 smithi104 ceph-mon[32438]: pgmap v101: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T18:55:38.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:38 smithi104 ceph-mon[32438]: 7.d scrub starts 2024-09-21T18:55:38.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:38.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:39.033 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:39.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.173 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:39.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:39.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 78 15 124515 124500 99.7207 78.9375 0.0119009 0.0100229 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[29384]: 3.c scrub starts 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[29384]: 3.c scrub ok 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[29384]: pgmap v101: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[29384]: 7.d scrub starts 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[39192]: 3.c scrub starts 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[39192]: 3.c scrub ok 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[39192]: pgmap v101: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T18:55:39.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[39192]: 7.d scrub starts 2024-09-21T18:55:39.219 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:39.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:39.308 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:39.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:39.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:39.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:39.575 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:39.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:39.717 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:39.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:39.872 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:39.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:39.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:40.031 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:39 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T18:55:40.031 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:39 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T18:55:40.031 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:39 smithi104 ceph-mon[32438]: 6.16 scrub starts 2024-09-21T18:55:40.031 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:39 smithi104 ceph-mon[32438]: 6.16 scrub ok 2024-09-21T18:55:40.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:40.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:40.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:40.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 79 16 125681 125665 99.3802 72.8125 0.00694077 0.0100581 2024-09-21T18:55:40.190 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T18:55:40.190 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T18:55:40.190 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[39192]: 6.16 scrub starts 2024-09-21T18:55:40.190 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[39192]: 6.16 scrub ok 2024-09-21T18:55:40.191 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T18:55:40.191 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T18:55:40.191 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[29384]: 6.16 scrub starts 2024-09-21T18:55:40.191 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:39 smithi081 ceph-mon[29384]: 6.16 scrub ok 2024-09-21T18:55:40.199 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:40.199 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:40.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.266 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:40.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:40.367 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:40.404 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:40.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:40.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:40.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:40.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.692 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:40.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:40.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:40.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.834 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:40.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:40.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T18:55:40.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:40.952 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 7 pgp_num to 16 2024-09-21T18:55:40.964 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:55:40.968 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:40.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:40.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:55:41.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:41.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:41.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:55:41.172295+0000 min lat: 0.00134243 max lat: 0.179359 avg lat: 0.0100822 2024-09-21T18:55:41.173 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:55:41.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 80 16 126965 126949 99.141 80.25 0.0101549 0.0100822 2024-09-21T18:55:41.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:41.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: pgmap v102: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T18:55:41.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 4.a scrub starts 2024-09-21T18:55:41.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 4.a scrub ok 2024-09-21T18:55:41.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4189702762' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:55:41.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: from='client.34862 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 5.9 scrub starts 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 5.9 scrub ok 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: 7.d scrub ok 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/777938696' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: pgmap v102: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 4.a scrub starts 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 4.a scrub ok 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4189702762' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: from='client.34862 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T18:55:41.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T18:55:41.230 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 5.9 scrub starts 2024-09-21T18:55:41.230 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 5.9 scrub ok 2024-09-21T18:55:41.230 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: 7.d scrub ok 2024-09-21T18:55:41.230 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/777938696' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T18:55:41.235 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.254 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:41.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:41.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: pgmap v102: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T18:55:41.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 4.a scrub starts 2024-09-21T18:55:41.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 4.a scrub ok 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4189702762' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: from='client.34862 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 5.9 scrub starts 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 5.9 scrub ok 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: 7.d scrub ok 2024-09-21T18:55:41.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/777938696' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T18:55:41.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:41.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:41.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:41.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:41.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:41.765 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:41.802 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:41.843 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:41.843 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:41.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:41.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:41.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:42.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:42.071 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:42.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:42.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 81 16 128345 128329 98.9816 86.25 0.0189601 0.0100989 2024-09-21T18:55:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.209 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:42.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/777938696' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: osdmap e111: 8 total, 8 up, 7 in 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T18:55:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:41 smithi104 ceph-mon[32438]: pgmap v104: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 721 op/s 2024-09-21T18:55:42.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/777938696' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: osdmap e111: 8 total, 8 up, 7 in 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[39192]: pgmap v104: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 721 op/s 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:55:42.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/777938696' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T18:55:42.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: osdmap e111: 8 total, 8 up, 7 in 2024-09-21T18:55:42.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T18:55:42.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:41 smithi081 ceph-mon[29384]: pgmap v104: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 721 op/s 2024-09-21T18:55:42.348 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:42.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:42.482 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:42.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:42.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:42.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:42.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:42.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T18:55:42.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:42.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:42.989 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:42.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.046 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:43.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:43.171 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:43.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 82 16 129729 129713 98.8288 86.5 0.00996658 0.010114 2024-09-21T18:55:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:43.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:43.327 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:43.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:43.468 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:43.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:43.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:43.732 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:43.732 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:43.742 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:43.742 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:43.767 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:43.767 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:43.801 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:43.801 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:43.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:43.836 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:43.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:43.871 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:43.871 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:43.887 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:43.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:43.904 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:43.904 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:43.935 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:43.936 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:43.971 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:43.971 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:43.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:44.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:55:44.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:44.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:44.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 83 16 131011 130995 98.6035 80.125 0.00619255 0.0101373 2024-09-21T18:55:44.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:44.310 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:44.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T18:55:44.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:44.474 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:44.576 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:44.642 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:44 smithi081 ceph-mon[39192]: pgmap v105: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T18:55:44.642 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2677943011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:44.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:44 smithi081 ceph-mon[29384]: pgmap v105: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T18:55:44.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2677943011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:44.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:44.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:44 smithi104 ceph-mon[32438]: pgmap v105: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T18:55:44.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2677943011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:44.758 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:44.815 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:44.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:44.894 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:44.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:44.987 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:44.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.046 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:45.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:45.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:45.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 84 16 132307 132291 98.3939 81 0.00561558 0.0101593 2024-09-21T18:55:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.178 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:45.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:45.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:45.318 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:45.411 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:55:45.411 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:55:45.411 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T18:55:45.411 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T18:55:45.411 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:55:45.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:45.459 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:45.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:45.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:45.730 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:45.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:45.867 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:45.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:45.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:46.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T18:55:46.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:46.139 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:46.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 85 16 133643 133627 98.2186 83.5 0.00821009 0.0101773 2024-09-21T18:55:46.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:46.279 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:46.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:46.414 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:55:46.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:46.566 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:46.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:46.702 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:46.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:46 smithi104 ceph-mon[32438]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-21T18:55:46.779 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:46 smithi081 ceph-mon[39192]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-21T18:55:46.780 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:46 smithi081 ceph-mon[29384]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-21T18:55:46.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:46.836 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:46.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:46.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:55:46.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:47.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:47.047 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54919 7.5 7:a6ed1b73:::benchmark_data_smithi081_73391_object54918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.457046+0000", 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 60.586063123000002, 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.072552502000000005, 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54919 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457046+0000", 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457046+0000", 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457048+0000", 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7039999999999999e-06 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457088+0000", 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0391e-05 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457104+0000", 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5909999999999998e-05 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457112+0000", 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2239999999999998e-06 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457195+0000", 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3146000000000007e-05 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457341+0000", 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014639199999999999 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457459+0000", 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011788799999999999 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461185+0000", 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037262430000000002 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.501979+0000", 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.040793665 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.529482+0000", 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027503393000000001 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.529552+0000", 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9307000000000006e-05 2024-09-21T18:55:47.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.529599+0000", 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7239e-05 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54921 7.5 7:a314aa6c:::benchmark_data_smithi081_73391_object54920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 60.585979645000002, 2024-09-21T18:55:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.074546457999999996, 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54921 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6599999999999996e-07 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457166+0000", 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5052000000000003e-05 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457172+0000", 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457178+0000", 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9259999999999997e-06 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459247+0000", 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020685429999999999 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459371+0000", 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012465100000000001 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459488+0000", 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011696000000000001 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461191+0000", 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001702924 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.502032+0000", 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.040840333999999999 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531576+0000", 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029544002999999999 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531634+0000", 2024-09-21T18:55:47.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8527999999999999e-05 2024-09-21T18:55:47.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531676+0000", 2024-09-21T18:55:47.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2054000000000001e-05 2024-09-21T18:55:47.061 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.061 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.061 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54929 7.5 7:af33521c:::benchmark_data_smithi081_73391_object54928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.461656+0000", 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 60.581453160000002, 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076295636, 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.062 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.063 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.063 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54929 2024-09-21T18:55:47.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461656+0000", 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461656+0000", 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461657+0000", 2024-09-21T18:55:47.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.094e-06 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461720+0000", 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2421000000000004e-05 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461729+0000", 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9320000000000002e-06 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461735+0000", 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4479999999999999e-06 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461793+0000", 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7732000000000002e-05 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461920+0000", 2024-09-21T18:55:47.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000126957 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.462069+0000", 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014942500000000001 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.463573+0000", 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015035960000000001 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.502383+0000", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.038809896000000003 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537843+0000", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.035460301 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537910+0000", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537952+0000", 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1569000000000001e-05 2024-09-21T18:55:47.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:75008 7.6 7:6c860821:::benchmark_data_smithi081_73391_object75007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:59.356808+0000", 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 47.686301161999999, 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069982419000000004, 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.069 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 75008 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.356808+0000", 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.356808+0000", 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.356810+0000", 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9910000000000001e-06 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.356842+0000", 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.133e-05 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.356854+0000", 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2490999999999999e-05 2024-09-21T18:55:47.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.356862+0000", 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7209999999999994e-06 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.357321+0000", 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045868100000000001 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.357431+0000", 2024-09-21T18:55:47.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011053900000000001 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.357560+0000", 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128489 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.360178+0000", 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002618623 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.361447+0000", 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001268421 2024-09-21T18:55:47.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.426677+0000", 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.065230609999999994 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.426728+0000", 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1233000000000002e-05 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:59.426791+0000", 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2290000000000002e-05 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.074 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99380 7.5 7:a5c9fcf3:::benchmark_data_smithi081_73391_object99379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.710751+0000", 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.332358722999999, 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079490342000000005, 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.075 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99380 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710751+0000", 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710751+0000", 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710752+0000", 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7129999999999999e-06 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710804+0000", 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1554999999999997e-05 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710813+0000", 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3589999999999997e-06 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710820+0000", 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7950000000000002e-06 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710870+0000", 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0315000000000002e-05 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710979+0000", 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010852 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.711073+0000", 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4493000000000005e-05 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.712997+0000", 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019239000000000001 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713798+0000", 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080073700000000004 2024-09-21T18:55:47.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790157+0000", 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076358910000000002 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790215+0000", 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8378999999999998e-05 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790241+0000", 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5666000000000001e-05 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.080 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99392 7.5 7:a86b0216:::benchmark_data_smithi081_73391_object99391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.720196+0000", 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.322913150000002, 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070161993000000006, 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99392 2024-09-21T18:55:47.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720196+0000", 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720196+0000", 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720199+0000", 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3120000000000001e-06 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720255+0000", 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5890999999999997e-05 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720269+0000", 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4453999999999999e-05 2024-09-21T18:55:47.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720277+0000", 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5099999999999998e-06 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720337+0000", 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0001999999999999e-05 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720484+0000", 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014647600000000001 2024-09-21T18:55:47.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720671+0000", 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018671900000000001 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.723494+0000", 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028234520000000002 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.723561+0000", 2024-09-21T18:55:47.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7002000000000007e-05 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790277+0000", 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066716083999999995 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790325+0000", 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7567e-05 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790358+0000", 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3524e-05 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101889 7.f e105/82 7:f2f30f0c:::benchmark_data_smithi081_73391_object101888:head v 105'6311, mlcod=105'6308)", 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.908994+0000", 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.134115272999999, 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087156066000000004, 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908995+0000", 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9800000000000003e-07 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909027+0000", 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2353e-05 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909033+0000", 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1859999999999997e-06 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909038+0000", 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7249999999999997e-06 2024-09-21T18:55:47.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909131+0000", 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2363999999999999e-05 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909149+0000", 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8450000000000001e-05 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996090+0000", 2024-09-21T18:55:47.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086940559000000001 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996150+0000", 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0730999999999998e-05 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101894 7.8 e105/79 7:1138a4d6:::benchmark_data_smithi081_73391_object101893:head v 105'6424, mlcod=105'6422)", 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910208+0000", 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.132901553, 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086428049000000007, 2024-09-21T18:55:47.091 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910209+0000", 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.24e-06 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910259+0000", 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9756000000000003e-05 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910269+0000", 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0035e-05 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910276+0000", 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8040000000000004e-06 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914457+0000", 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041814679999999998 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914478+0000", 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0528999999999999e-05 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996573+0000", 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082095020000000005 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996636+0000", 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3196999999999998e-05 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101892 7.f e105/82 7:f5e7532a:::benchmark_data_smithi081_73391_object101891:head v 105'6312, mlcod=105'6308)", 2024-09-21T18:55:47.095 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910906+0000", 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.132203692000001, 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085280135000000007, 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T18:55:47.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910907+0000", 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910962+0000", 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5427000000000003e-05 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910972+0000", 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2520000000000005e-06 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910978+0000", 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.117e-06 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914620+0000", 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036426869999999999 2024-09-21T18:55:47.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914639+0000", 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8848999999999999e-05 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996155+0000", 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.081515729999999995 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996186+0000", 2024-09-21T18:55:47.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0874000000000003e-05 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101891 7.c e105/79 7:3b3bc22b:::benchmark_data_smithi081_73391_object101890:head v 105'6296, mlcod=105'6294)", 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.912575+0000", 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.130534102999999, 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084158549999999999, 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912575+0000", 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912575+0000", 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912577+0000", 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.316e-06 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912603+0000", 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5993999999999999e-05 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912614+0000", 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1082e-05 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912621+0000", 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.063e-06 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.913518+0000", 2024-09-21T18:55:47.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089733900000000004 2024-09-21T18:55:47.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.913546+0000", 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7746000000000002e-05 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916419+0000", 2024-09-21T18:55:47.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028730499999999998 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996734+0000", 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080314960000000005 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101901 7.5 7:a8e9101c:::benchmark_data_smithi081_73391_object101900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916321+0000", 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.126788547, 2024-09-21T18:55:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087318653999999996, 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101901 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T18:55:47.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916329+0000", 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3990000000000004e-06 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916364+0000", 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4808999999999998e-05 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916374+0000", 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0298e-05 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916382+0000", 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996259+0000", 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079877760000000006 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996412+0000", 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015201099999999999 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996586+0000", 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017437 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003419+0000", 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068332979999999998 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003455+0000", 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5438000000000002e-05 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003478+0000", 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3048e-05 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003584+0000", 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106331 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003640+0000", 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5583e-05 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.111 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101902 7.5 7:a3b6d405:::benchmark_data_smithi081_73391_object101901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916400+0000", 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.126709538, 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087823038000000006, 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101902 2024-09-21T18:55:47.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916401+0000", 2024-09-21T18:55:47.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5899999999999996e-07 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916430+0000", 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9076000000000001e-05 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996083+0000", 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079653529000000001 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996103+0000", 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0015999999999999e-05 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998560+0000", 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002457126 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998676+0000", 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115328 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998809+0000", 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133767 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003647+0000", 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048375040000000003 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003682+0000", 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:55:47.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004144+0000", 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046231000000000001 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004190+0000", 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5958999999999998e-05 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004223+0000", 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2840999999999999e-05 2024-09-21T18:55:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114519 7.6 7:6b5e3486:::benchmark_data_smithi081_73391_object114518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709106+0000", 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.334003720999998, 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12712907900000001, 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.118 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114519 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709107+0000", 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.66e-06 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709157+0000", 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9911999999999999e-05 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709171+0000", 2024-09-21T18:55:47.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3947e-05 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709179+0000", 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3529999999999997e-06 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716464+0000", 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072858719999999997 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716590+0000", 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554999999999999 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716718+0000", 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012810499999999999 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723181+0000", 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006462944 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723217+0000", 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5593000000000003e-05 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836081+0000", 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11286423700000001 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836188+0000", 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107315 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836235+0000", 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6591000000000002e-05 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114521 7.6 7:6fe397a7:::benchmark_data_smithi081_73391_object114520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:47.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709300+0000", 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.333809171999999, 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12709793799999999, 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114521 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709301+0000", 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6199999999999996e-07 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709354+0000", 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2936999999999998e-05 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709361+0000", 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.097e-06 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709367+0000", 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6550000000000001e-06 2024-09-21T18:55:47.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719193+0000", 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098261459999999991 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719453+0000", 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026041699999999998 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.720410+0000", 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095699200000000004 2024-09-21T18:55:47.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723227+0000", 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002816888 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723832+0000", 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060487799999999999 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836280+0000", 2024-09-21T18:55:47.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11244789099999999 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836362+0000", 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1967999999999999e-05 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836398+0000", 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6207000000000001e-05 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114538 7.6 7:657d97a1:::benchmark_data_smithi081_73391_object114537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723638+0000", 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.319471499999999, 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11594671600000001, 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114538 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723640+0000", 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.801e-06 2024-09-21T18:55:47.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723678+0000", 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7873000000000001e-05 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723688+0000", 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0071000000000001e-05 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723695+0000", 2024-09-21T18:55:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3429999999999998e-06 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723868+0000", 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017280900000000001 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724003+0000", 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013505 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724166+0000", 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162764 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729028+0000", 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048623590000000001 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729096+0000", 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7625000000000004e-05 2024-09-21T18:55:47.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836438+0000", 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107342302 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839527+0000", 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030888589999999998 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839585+0000", 2024-09-21T18:55:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7859999999999998e-05 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114539 7.6 7:6de988ff:::benchmark_data_smithi081_73391_object114538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723706+0000", 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.319403049000002, 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11604534699999999, 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114539 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723707+0000", 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4399999999999999e-07 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723743+0000", 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5395999999999998e-05 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723749+0000", 2024-09-21T18:55:47.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1850000000000001e-06 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723754+0000", 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6219999999999997e-06 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726769+0000", 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030147289999999998 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726889+0000", 2024-09-21T18:55:47.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012032999999999999 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727041+0000", 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015136699999999999 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729293+0000", 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002252085 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730090+0000", 2024-09-21T18:55:47.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00079701999999999996 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839629+0000", 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10953908599999999 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839710+0000", 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0630000000000006e-05 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839752+0000", 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2153000000000001e-05 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121044 7.6 7:62f149b0:::benchmark_data_smithi081_73391_object121043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.421001+0000", 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.62210887, 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084735675999999996, 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121044 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421002+0000", 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.336e-06 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421062+0000", 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0475e-05 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421072+0000", 2024-09-21T18:55:47.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9389999999999992e-06 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421078+0000", 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2870000000000004e-06 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421128+0000", 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9938000000000002e-05 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421252+0000", 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012434499999999999 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421447+0000", 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019541299999999999 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425265+0000", 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038176640000000001 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428690+0000", 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034252710000000001 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505655+0000", 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076964640000000001 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505691+0000", 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5887999999999999e-05 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505736+0000", 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5479999999999998e-05 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121051 7.6 7:619bd2cd:::benchmark_data_smithi081_73391_object121050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.425560+0000", 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.617549649000001, 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088495110000000002, 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121051 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425562+0000", 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.154e-06 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425607+0000", 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5229999999999999e-05 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425620+0000", 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2346e-05 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425627+0000", 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425678+0000", 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1053000000000001e-05 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425785+0000", 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107099 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425916+0000", 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013072999999999999 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428156+0000", 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022408649999999999 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428832+0000", 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067540999999999999 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.513961+0000", 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085129596000000002 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514015+0000", 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3313000000000003e-05 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514055+0000", 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0156e-05 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:133485 7.5 7:aae62e2e:::benchmark_data_smithi081_73391_object133484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:46.013699+0000", 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.029410253, 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071373945999999994, 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133485 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.013699+0000", 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.013699+0000", 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.013701+0000", 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.373e-06 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.013730+0000", 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9356000000000002e-05 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.013737+0000", 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9500000000000004e-06 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.013743+0000", 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8089999999999998e-06 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.014358+0000", 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00061548500000000003 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.014466+0000", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107896 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.014607+0000", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014066 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.017605+0000", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029986800000000001 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.022411+0000", 2024-09-21T18:55:47.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004805597 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.085007+0000", 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.062596385000000004 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.085040+0000", 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2935999999999998e-05 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.085073+0000", 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2818999999999999e-05 2024-09-21T18:55:47.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:133496 7.5 7:a3e205bf:::benchmark_data_smithi081_73391_object133495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:46.018117+0000", 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.0249926899999999, 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071355328999999995, 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133496 2024-09-21T18:55:47.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018117+0000", 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018117+0000", 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018118+0000", 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.344e-06 2024-09-21T18:55:47.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018173+0000", 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4493999999999997e-05 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018185+0000", 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2644e-05 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018193+0000", 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3719999999999999e-06 2024-09-21T18:55:47.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018243+0000", 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0025999999999998e-05 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018358+0000", 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115278 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.018487+0000", 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128627 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.022696+0000", 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042095689999999998 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.024423+0000", 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017267140000000001 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089345+0000", 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064921956000000003 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089427+0000", 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2248000000000003e-05 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089472+0000", 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5056999999999999e-05 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:55:47.166 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:55:47.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:47.174 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 86 16 134858 134842 97.9595 75.9375 0.00995529 0.0102036 2024-09-21T18:55:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.196 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:47.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:47.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:47.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:47.354 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:47.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:47.468 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:47.484 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:55:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:55:47.498 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:55:47.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:47.611 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:47.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3350205420' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:47.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:47 smithi104 ceph-mon[32438]: from='client.34877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:47.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:47.745 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3350205420' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:47.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:47 smithi081 ceph-mon[39192]: from='client.34877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3350205420' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:47.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:47 smithi081 ceph-mon[29384]: from='client.34877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:55:47.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:47.889 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:47.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:47.934 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:55:47.934 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:55:47.934 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-21T18:55:47.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd in 0 2024-09-21T18:55:47.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:48.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:55:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:55:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.039 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:48.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:48.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:48.175 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 87 16 136142 136126 97.7553 80.25 0.00992272 0.0102251 2024-09-21T18:55:48.184 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:48.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:48.314 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:48.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:48.446 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:48.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:48.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:48.686 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:48.719 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:48.736 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[39192]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:55:48.736 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:48.736 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[39192]: from='client.34877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[39192]: osdmap e112: 8 total, 8 up, 7 in 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[29384]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[29384]: from='client.34877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[29384]: osdmap e112: 8 total, 8 up, 7 in 2024-09-21T18:55:48.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:48.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:48 smithi104 ceph-mon[32438]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:55:48.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:48 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:48.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:48 smithi104 ceph-mon[32438]: from='client.34877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:55:48.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:48 smithi104 ceph-mon[32438]: osdmap e112: 8 total, 8 up, 7 in 2024-09-21T18:55:48.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:48.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:48.860 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:48.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:48.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.002 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:49.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:49.040 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:49.040 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:49.074 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:49.075 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:49.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:49.108 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:49.109 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:49.120 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:49.135 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:49.143 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:49.143 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:49.175 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:49.175 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:49.175 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 88 15 137288 137273 97.459 71.6875 0.0124977 0.0102562 2024-09-21T18:55:49.211 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:49.211 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:49.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:49.244 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:49.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:49.274 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:49.275 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:49.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.397 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:49.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:49.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:49.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:49.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2931980518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2931980518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:49.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.665 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:49.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:49.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T18:55:49.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2931980518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:49.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:49.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:49.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:49.973 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:49.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:50.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.119 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:50.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:50.175 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 89 16 138533 138517 97.2375 77.75 0.00999483 0.0102799 2024-09-21T18:55:50.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:50.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:50.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:50.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:50.498 INFO:teuthology.orchestra.run.smithi081.stderr:marked in osd.0. 2024-09-21T18:55:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:50.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-mon[39192]: pgmap v109: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1654979136' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-mon[39192]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T18:55:50.299+0000 7ff1af1de640 -1 mon.a@0(leader).osd e112 definitely_dead 0 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-mon[29384]: pgmap v109: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1654979136' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T18:55:50.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:50 smithi081 ceph-mon[29384]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T18:55:50.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:50.655 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:50.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:50.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:50.786 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:55:50.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:50.901 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:50.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:51.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:51.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:50 smithi104 ceph-mon[32438]: pgmap v109: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T18:55:51.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1654979136' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T18:55:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:50 smithi104 ceph-mon[32438]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-21T18:55:51.023 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:51.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:51.148 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:51.154 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-21T18:55:51.175 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 90 16 138996 138980 96.4788 28.9375 0.00547027 0.0102853 2024-09-21T18:55:51.203 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:51.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:51.272 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:51.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:51.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:51.546 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:51.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:51.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[29384]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[29384]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[29384]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[39192]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[39192]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:51.706 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:51.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:51 smithi081 ceph-mon[39192]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T18:55:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:51 smithi104 ceph-mon[32438]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-21T18:55:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:51 smithi104 ceph-mon[32438]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T18:55:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:55:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:55:51.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:51 smithi104 ceph-mon[32438]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T18:55:51.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 18:55:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=infra.usagestats t=2024-09-21T18:55:51.391099565Z level=info msg="Usage stats are ready to report" 2024-09-21T18:55:51.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:51.793 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:51.876 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:51.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:51.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:52.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:52.043 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:52.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:52.229 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 91 16 138996 138980 95.4189 0 - 0.0102853 2024-09-21T18:55:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:52.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:52.344 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:55:52.345 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:55:52.356 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:52.380 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:52.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:52.496 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:55:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.512 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:52.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:52.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:55:52.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:52.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:52.757 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[29384]: pgmap v111: 148 pgs: 4 unknown, 19 peering, 1 active+clean+scrubbing, 124 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 14 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2679400094' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2679400094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[29384]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[39192]: pgmap v111: 148 pgs: 4 unknown, 19 peering, 1 active+clean+scrubbing, 124 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T18:55:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 14 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-21T18:55:52.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2679400094' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:55:52.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:52.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2679400094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:55:52.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:52 smithi081 ceph-mon[39192]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T18:55:52.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:52.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:52.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:52.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:53.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:52 smithi104 ceph-mon[32438]: pgmap v111: 148 pgs: 4 unknown, 19 peering, 1 active+clean+scrubbing, 124 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T18:55:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:52 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 14 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-09-21T18:55:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2679400094' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:55:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:52 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:55:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2679400094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:55:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:52 smithi104 ceph-mon[32438]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T18:55:53.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:53.122 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:53.144 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total time run: 91.7564 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total writes made: 138996 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Write size: 65536 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object size: 65536 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Bandwidth (MB/sec): 94.6773 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Bandwidth: 34.8693 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max bandwidth (MB/sec): 242.938 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min bandwidth (MB/sec): 0 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average IOPS: 1514 2024-09-21T18:55:53.176 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev IOPS: 557.908 2024-09-21T18:55:53.177 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max IOPS: 3887 2024-09-21T18:55:53.177 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min IOPS: 0 2024-09-21T18:55:53.177 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average Latency(s): 0.0105574 2024-09-21T18:55:53.177 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Latency(s): 0.0265788 2024-09-21T18:55:53.177 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max latency(s): 2.39831 2024-09-21T18:55:53.177 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min latency(s): 0.00134243 2024-09-21T18:55:53.181 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T18:55:53.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:53.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:53.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.419 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:53.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:53.521 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:53.549 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:53.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:53.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:53.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:53.808 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.f starting backfill to osd.0 from (0'0,0'0] MAX to 112'8668 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.6 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8679 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.d starting backfill to osd.0 from (0'0,0'0] MAX to 112'8808 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.7 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8661 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.e starting backfill to osd.0 from (0'0,0'0] MAX to 112'8567 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.9 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8715 2024-09-21T18:55:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.5 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8911 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 7.4 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8754 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.f starting backfill to osd.0 from (0'0,0'0] MAX to 112'8668 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.6 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8679 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.d starting backfill to osd.0 from (0'0,0'0] MAX to 112'8808 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.7 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8661 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.e starting backfill to osd.0 from (0'0,0'0] MAX to 112'8567 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.9 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8715 2024-09-21T18:55:53.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.5 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8911 2024-09-21T18:55:53.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 7.4 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8754 2024-09-21T18:55:53.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:53.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:53.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T18:55:53.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:53 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T18:55:53.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:53.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.f starting backfill to osd.0 from (0'0,0'0] MAX to 112'8668 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.6 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8679 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.d starting backfill to osd.0 from (0'0,0'0] MAX to 112'8808 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.7 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8661 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.e starting backfill to osd.0 from (0'0,0'0] MAX to 112'8567 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.9 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8715 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.5 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8911 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 7.4 starting backfill to osd.0 from (0'0,0'0] MAX to 112'8754 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:54.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T18:55:54.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:53 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T18:55:54.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:54.073 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:54.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:54.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:54.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:54.334 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:54.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:54.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:54.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T18:55:54.581 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.607 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:54.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:54.615 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:54.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:54.760 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T18:55:54.760 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T18:55:54.760 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:54.760 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:54.760 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[39192]: pgmap v114: 148 pgs: 2 remapped+peering, 2 unknown, 42 peering, 102 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 173 B/s rd, 0 op/s; 149 B/s, 0 objects/s recovering 2024-09-21T18:55:54.760 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[39192]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T18:55:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.761 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:54.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:54.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T18:55:54.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T18:55:54.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:54.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:54.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[29384]: pgmap v114: 148 pgs: 2 remapped+peering, 2 unknown, 42 peering, 102 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 173 B/s rd, 0 op/s; 149 B/s, 0 objects/s recovering 2024-09-21T18:55:54.764 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:54 smithi081 ceph-mon[29384]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T18:55:54.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:54.887 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:54.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:54.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:54 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T18:55:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:54 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T18:55:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:54 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:54 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:54 smithi104 ceph-mon[32438]: pgmap v114: 148 pgs: 2 remapped+peering, 2 unknown, 42 peering, 102 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 173 B/s rd, 0 op/s; 149 B/s, 0 objects/s recovering 2024-09-21T18:55:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:54 smithi104 ceph-mon[32438]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T18:55:55.011 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:55.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:55.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:55.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:55.270 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:55.329 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:55.336 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:55:55.336 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:55:55.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:55.381 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:55:55.381 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:55:55.411 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:55.423 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:55:55.423 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:55:55.457 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:55:55.457 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:55:55.489 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:55:55.489 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:55:55.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:55.520 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:55:55.520 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:55:55.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:55.554 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:55:55.555 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:55:55.594 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:55:55.594 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:55:55.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:55.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:55.681 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1294721995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:55.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1294721995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:55.782 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:55.818 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:55.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:55:55.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:55.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:55.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:56.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1294721995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:55:56.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:56.155 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:55:56.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:55:56.156 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T18:55:56.156 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T18:55:56.156 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T18:55:56.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T18:55:56.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:56.207 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:56.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:56.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:56.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:55:56.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:55:56.548 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T18:55:56.583 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:56.611 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:56.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T18:55:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T18:55:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:55:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[39192]: pgmap v116: 148 pgs: 2 remapped+peering, 2 unknown, 42 peering, 102 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 208 B/s rd, 0 op/s; 179 B/s, 0 objects/s recovering 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[39192]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[29384]: pgmap v116: 148 pgs: 2 remapped+peering, 2 unknown, 42 peering, 102 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 208 B/s rd, 0 op/s; 179 B/s, 0 objects/s recovering 2024-09-21T18:55:56.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:56 smithi081 ceph-mon[29384]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T18:55:56.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:56.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:56.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:56.997 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:57.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:56 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T18:55:57.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:56 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T18:55:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:56 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:55:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:56 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:55:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:56 smithi104 ceph-mon[32438]: pgmap v116: 148 pgs: 2 remapped+peering, 2 unknown, 42 peering, 102 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 208 B/s rd, 0 op/s; 179 B/s, 0 objects/s recovering 2024-09-21T18:55:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:56 smithi104 ceph-mon[32438]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T18:55:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:55:57.037 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:55:57.037 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:55:57.037 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:55:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:55:57.064 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:57.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:57.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:55:57.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:57.313 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:57.332 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:55:57.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:57.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:57.462 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:57.547 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:55:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:55:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:55:57.563 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:57.592 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:57.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:57.742 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:57.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:57 smithi081 ceph-mon[29384]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T18:55:57.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:57 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 14 pgs peering) 2024-09-21T18:55:57.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:57 smithi081 ceph-mon[39192]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T18:55:57.808 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:57 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 14 pgs peering) 2024-09-21T18:55:57.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T18:55:57.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:57.870 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:57.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:57.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:58.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:55:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:55:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:57 smithi104 ceph-mon[32438]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T18:55:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:57 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 14 pgs peering) 2024-09-21T18:55:58.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:58.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:58.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.294 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:58.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:58.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:55:58.408 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:55:58.409 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T18:55:58.409 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:55:58.418 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:55:58.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:58.562 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:58.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:58.694 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:58.695 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:55:58.734 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[29384]: pgmap v118: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.0 KiB/s rd, 378 MiB/s wr, 6.63k op/s; 66649/407244 objects misplaced (16.366%); 14 MiB/s, 240 objects/s recovering 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[39192]: pgmap v118: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.0 KiB/s rd, 378 MiB/s wr, 6.63k op/s; 66649/407244 objects misplaced (16.366%); 14 MiB/s, 240 objects/s recovering 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T18:55:58.735 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:58 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T18:55:58.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:58.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:58.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:55:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:58.971 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:58.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:55:59.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:58 smithi104 ceph-mon[32438]: pgmap v118: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.0 KiB/s rd, 378 MiB/s wr, 6.63k op/s; 66649/407244 objects misplaced (16.366%); 14 MiB/s, 240 objects/s recovering 2024-09-21T18:55:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:55:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:58 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T18:55:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:58 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T18:55:59.072 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:59.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:59.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:55:59.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:55:59.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:59.390 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:59.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:55:59.524 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:55:59.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:55:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:55:59.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:55:59.787 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:55:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:55:59.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:55:59.926 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:55:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:55:59.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:00.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:55:59 smithi104 ceph-mon[32438]: 7.1 scrub starts 2024-09-21T18:56:00.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:00.054 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:55:59 smithi081 ceph-mon[29384]: 7.1 scrub starts 2024-09-21T18:56:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:55:59 smithi081 ceph-mon[39192]: 7.1 scrub starts 2024-09-21T18:56:00.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:00.190 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:00.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.331 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:00.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:00.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:00.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:00.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:00.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:00.717 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:00.730 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:00.745 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:00.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:00.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:00.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:56:00.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[29384]: pgmap v119: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 888 B/s rd, 158 MiB/s wr, 3.03k op/s; 66649/407244 objects misplaced (16.366%); 12 MiB/s, 203 objects/s recovering 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[29384]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[29384]: 4.c scrub starts 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[29384]: 4.c scrub ok 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:56:00.902 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[39192]: pgmap v119: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 888 B/s rd, 158 MiB/s wr, 3.03k op/s; 66649/407244 objects misplaced (16.366%); 12 MiB/s, 203 objects/s recovering 2024-09-21T18:56:00.903 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[39192]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T18:56:00.903 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[39192]: 4.c scrub starts 2024-09-21T18:56:00.903 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:00 smithi081 ceph-mon[39192]: 4.c scrub ok 2024-09-21T18:56:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:00 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:56:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:00 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:56:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:00 smithi104 ceph-mon[32438]: pgmap v119: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 888 B/s rd, 158 MiB/s wr, 3.03k op/s; 66649/407244 objects misplaced (16.366%); 12 MiB/s, 203 objects/s recovering 2024-09-21T18:56:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:00 smithi104 ceph-mon[32438]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T18:56:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:00 smithi104 ceph-mon[32438]: 4.c scrub starts 2024-09-21T18:56:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:00 smithi104 ceph-mon[32438]: 4.c scrub ok 2024-09-21T18:56:00.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:01.018 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:01.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:01.158 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:01.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:01.298 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:01.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:01.432 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:01.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:01.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:01.686 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:01.714 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:01.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:01.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:01.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:01 smithi081 ceph-mon[29384]: 3.3 scrub starts 2024-09-21T18:56:01.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:01 smithi081 ceph-mon[29384]: 3.3 scrub ok 2024-09-21T18:56:01.936 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:01 smithi081 ceph-mon[39192]: 3.3 scrub starts 2024-09-21T18:56:01.936 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:01 smithi081 ceph-mon[39192]: 3.3 scrub ok 2024-09-21T18:56:01.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:01.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:01.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:02.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:01 smithi104 ceph-mon[32438]: 3.3 scrub starts 2024-09-21T18:56:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:01 smithi104 ceph-mon[32438]: 3.3 scrub ok 2024-09-21T18:56:02.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:02.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:02.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:02.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:02.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:02.382 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:02.484 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:02.512 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:02.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:02.640 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:02.649 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:56:02.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:02 smithi081 ceph-mon[39192]: 7.1 scrub ok 2024-09-21T18:56:02.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:02 smithi081 ceph-mon[39192]: pgmap v121: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 779 B/s rd, 139 MiB/s wr, 2.65k op/s; 66649/407244 objects misplaced (16.366%); 11 MiB/s, 178 objects/s recovering 2024-09-21T18:56:02.682 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1431582560' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:02.682 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:02 smithi081 ceph-mon[29384]: 7.1 scrub ok 2024-09-21T18:56:02.682 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:02 smithi081 ceph-mon[29384]: pgmap v121: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 779 B/s rd, 139 MiB/s wr, 2.65k op/s; 66649/407244 objects misplaced (16.366%); 11 MiB/s, 178 objects/s recovering 2024-09-21T18:56:02.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1431582560' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:02.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.788 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:02.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:02.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:02.915 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:02.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:03.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:02 smithi104 ceph-mon[32438]: 7.1 scrub ok 2024-09-21T18:56:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:02 smithi104 ceph-mon[32438]: pgmap v121: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 779 B/s rd, 139 MiB/s wr, 2.65k op/s; 66649/407244 objects misplaced (16.366%); 11 MiB/s, 178 objects/s recovering 2024-09-21T18:56:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1431582560' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:03.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:03.042 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:03.144 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34376.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:53:06.958110+0000", 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 175.96955401299999, 2024-09-21T18:56:03.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.44773363700000002, 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34376", 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/449352202", 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 1 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958110+0000", 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958110+0000", 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958119+0000", 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1530000000000003e-06 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958127+0000", 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3969999999999997e-06 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958138+0000", 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1051e-05 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958148+0000", 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9539999999999999e-06 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958204+0000", 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6134999999999997e-05 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:06.958212+0000", 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8120000000000008e-06 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.169 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.405799+0000", 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.44758725500000002 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:53:07.405844+0000", 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4879999999999997e-05 2024-09-21T18:56:03.170 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:144261 7.e e117/114 7:7e80c5a1:::benchmark_data_smithi081_73391_object5262:head v 117'8920, mlcod=117'8916)", 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:58.224809+0000", 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7028550110000005, 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.035222403999999999, 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.171 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.224809+0000", 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.224809+0000", 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.224810+0000", 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3000000000000001e-07 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.172 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.224814+0000", 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1649999999999999e-06 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.224826+0000", 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2568999999999999e-05 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.224831+0000", 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4179999999999999e-06 2024-09-21T18:56:03.173 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.239122+0000", 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014290730999999999 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.239176+0000", 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4135000000000003e-05 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.260009+0000", 2024-09-21T18:56:03.174 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020833221999999998 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:58.260032+0000", 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2634e-05 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(7.5 117/114 [PushOp(7:a1bcf2b5:::benchmark_data_smithi081_73391_object112545:head, version: 106'7221, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a1bcf2b5:::benchmark_data_smithi081_73391_object112545:head@106'7221, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.357406+0000", 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.5702583949999998, 2024-09-21T18:56:03.175 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069891840999999996, 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357406+0000", 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357406+0000", 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.176 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357407+0000", 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.33e-06 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357489+0000", 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2019000000000002e-05 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357517+0000", 2024-09-21T18:56:03.177 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8149000000000001e-05 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357549+0000", 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1736999999999998e-05 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357599+0000", 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0228000000000001e-05 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357606+0000", 2024-09-21T18:56:03.178 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9240000000000001e-06 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.427298+0000", 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069691454 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_backfill(progress 7.5 e 117/117 lb 7:a1bc2f0b:::benchmark_data_smithi081_73391_object49173:head)", 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.357562+0000", 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.57010264, 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.071851529999999997, 2024-09-21T18:56:03.179 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357562+0000", 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357562+0000", 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357562+0000", 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.46e-07 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357566+0000", 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9230000000000004e-06 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357569+0000", 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5669999999999999e-06 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.357572+0000", 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9950000000000001e-06 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.429355+0000", 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.071782538000000007 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.429371+0000", 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5610000000000001e-05 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.182 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.429413+0000", 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2551e-05 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(7.5 118/114 [PushOp(7:a1e67716:::benchmark_data_smithi081_73391_object127972:head, version: 111'8219, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a1e67716:::benchmark_data_smithi081_73391_object127972:head@111'8219, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.848978+0000", 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.0786861220000001, 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032157401000000002, 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.183 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.848978+0000", 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.184 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.185 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.185 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.848978+0000", 2024-09-21T18:56:03.185 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.848979+0000", 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.263e-06 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849030+0000", 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0676000000000001e-05 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.186 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849057+0000", 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6792999999999999e-05 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849081+0000", 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4102999999999999e-05 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849210+0000", 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012876200000000001 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.187 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849216+0000", 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4500000000000001e-06 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.881135+0000", 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031919353999999997 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_backfill(progress 7.5 e 118/118 lb 7:a1e6499a:::benchmark_data_smithi081_73391_object56236:head)", 2024-09-21T18:56:03.188 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.849113+0000", 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.0785509040000001, 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032123277999999998, 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849113+0000", 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.189 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849113+0000", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849114+0000", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9999999999999995e-07 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849128+0000", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3776e-05 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849143+0000", 2024-09-21T18:56:03.190 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4776e-05 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.849158+0000", 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4983999999999999e-05 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.881191+0000", 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032033264999999998 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.881203+0000", 2024-09-21T18:56:03.191 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2238999999999999e-05 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.881237+0000", 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3238000000000003e-05 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(7.5 118/114 [PushOp(7:a1f6cdd6:::benchmark_data_smithi081_73391_object91252:head, version: 100'5815, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a1f6cdd6:::benchmark_data_smithi081_73391_object91252:head@100'5815, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:00.231421+0000", 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 2.6962427579999999, 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049217258999999999, 2024-09-21T18:56:03.192 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231421+0000", 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231421+0000", 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.193 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231422+0000", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9400000000000005e-07 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231503+0000", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0673000000000005e-05 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231514+0000", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1240000000000001e-05 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231523+0000", 2024-09-21T18:56:03.194 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7199999999999995e-06 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231569+0000", 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6242999999999998e-05 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.280401+0000", 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.048832078000000001 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.195 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.280639+0000", 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023771099999999999 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_backfill(progress 7.5 e 118/118 lb 7:a1f6b2eb:::benchmark_data_smithi081_73391_object136303:head)", 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:00.231535+0000", 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 2.6961293450000001, 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049201887999999999, 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231535+0000", 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231535+0000", 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231535+0000", 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0699999999999998e-07 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231539+0000", 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1230000000000002e-06 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231543+0000", 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6650000000000001e-06 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.231546+0000", 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9040000000000001e-06 2024-09-21T18:56:03.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.280693+0000", 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049146859000000001 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.280708+0000", 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4848000000000001e-05 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:00.280737+0000", 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9181999999999999e-05 2024-09-21T18:56:03.199 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:147572 7.e e118/114 7:7819e614:::benchmark_data_smithi081_73391_object8573:head v 118'9138, mlcod=118'9136)", 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:01.766042+0000", 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.161622551, 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032063636999999999, 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.200 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.766042+0000", 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.766042+0000", 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.766043+0000", 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.314e-06 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.201 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.766050+0000", 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.766061+0000", 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0472000000000001e-05 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.766068+0000", 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0060000000000003e-06 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.775111+0000", 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0090438159999999997 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.775127+0000", 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5814000000000001e-05 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.797909+0000", 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022781793000000002 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.203 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.798105+0000", 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000196214 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:147582 7.e e118/114 7:7f2f2fc6:::benchmark_data_smithi081_73391_object8583:head v 118'9139, mlcod=118'9136)", 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:01.771729+0000", 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.15593513, 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030988803999999998, 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.204 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.771729+0000", 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.771729+0000", 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.771731+0000", 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8729999999999999e-06 2024-09-21T18:56:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.771740+0000", 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6270000000000006e-06 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.771755+0000", 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5753e-05 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.771764+0000", 2024-09-21T18:56:03.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3280000000000006e-06 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.781947+0000", 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010183329 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.781989+0000", 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2341e-05 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.802680+0000", 2024-09-21T18:56:03.207 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020691030999999999 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:01.802718+0000", 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7521999999999998e-05 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:147793 7.e e118/114 7:731f6c1d:::benchmark_data_smithi081_73391_object8794:head v 118'9147, mlcod=118'9145)", 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.009200+0000", 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.91846419999999995, 2024-09-21T18:56:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.036277917999999999, 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.009200+0000", 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.009200+0000", 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.009202+0000", 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.009209+0000", 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6690000000000006e-06 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.009223+0000", 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3811e-05 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.009234+0000", 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0839000000000001e-05 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.018199+0000", 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0089648019999999991 2024-09-21T18:56:03.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.020134+0000", 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00193501 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.045395+0000", 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025261349999999998 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.045478+0000", 2024-09-21T18:56:03.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2696999999999998e-05 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(7.6 118/114 [PushOp(7:61663830:::benchmark_data_smithi081_73391_object112636:head, version: 107'7010, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:61663830:::benchmark_data_smithi081_73391_object112636:head@107'7010, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.011091+0000", 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.91657331399999997, 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.034574365000000003, 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.011091+0000", 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.011091+0000", 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.011091+0000", 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7899999999999999e-07 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.011119+0000", 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7242000000000001e-05 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.011133+0000", 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4218e-05 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.011148+0000", 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4769999999999999e-05 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.215 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.037436+0000", 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.026288249 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.037450+0000", 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4552000000000001e-05 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.045665+0000", 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082149549999999995 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.216 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(7.5 118/114 [PushOp(7:a26f3c8d:::benchmark_data_smithi081_73391_object133040:head, version: 111'8544, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a26f3c8d:::benchmark_data_smithi081_73391_object133040:head@111'8544, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.086971+0000", 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.84069341200000003, 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046106331, 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.086971+0000", 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.086971+0000", 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.086972+0000", 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1820000000000001e-06 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087020+0000", 2024-09-21T18:56:03.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8124000000000002e-05 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087034+0000", 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4105e-05 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087045+0000", 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0307e-05 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.132332+0000", 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045287544999999998 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.132735+0000", 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040287899999999998 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.133077+0000", 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034218900000000001 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_backfill(progress 7.5 e 118/118 lb 7:a26eb4b5:::benchmark_data_smithi081_73391_object87755:head)", 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.087059+0000", 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.84060506700000004, 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046202403000000003, 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087059+0000", 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087059+0000", 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087060+0000", 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0499999999999999e-07 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087064+0000", 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5650000000000003e-06 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087068+0000", 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2590000000000002e-06 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.087072+0000", 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3100000000000001e-06 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.133169+0000", 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046097069999999997 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.133208+0000", 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8886e-05 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.133262+0000", 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3908e-05 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:147908 7.7 e118/114 7:e6381b19:::benchmark_data_smithi081_73391_object8909:head v 118'9222, mlcod=118'9220)", 2024-09-21T18:56:03.224 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.204809+0000", 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.72285566099999998, 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031793421000000002, 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204809+0000", 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204809+0000", 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204810+0000", 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2550000000000001e-06 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204817+0000", 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1300000000000003e-06 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204827+0000", 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0247e-05 2024-09-21T18:56:03.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204834+0000", 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3640000000000004e-06 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204965+0000", 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013184700000000001 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.204983+0000", 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7876999999999998e-05 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.236505+0000", 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031521501 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.236602+0000", 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7200000000000004e-05 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:148016 7.e e118/114 7:7b5728d8:::benchmark_data_smithi081_73391_object9017:head v 118'9161, mlcod=118'9158)", 2024-09-21T18:56:03.228 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.360500+0000", 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.56716422099999997, 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.038419030999999999, 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.360500+0000", 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.360500+0000", 2024-09-21T18:56:03.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.360501+0000", 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2580000000000001e-06 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.360508+0000", 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9500000000000004e-06 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.360518+0000", 2024-09-21T18:56:03.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9380000000000004e-06 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.360525+0000", 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8900000000000001e-06 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366335+0000", 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058100529999999999 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366403+0000", 2024-09-21T18:56:03.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7873999999999994e-05 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.390041+0000", 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.023638211999999999 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.398919+0000", 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0088778559999999999 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:148022 7.e e118/114 7:707b6ea8:::benchmark_data_smithi081_73391_object9023:head v 118'9162, mlcod=118'9159)", 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.366632+0000", 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.56103239699999996, 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032377947999999997, 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366632+0000", 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366632+0000", 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366633+0000", 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4410000000000001e-06 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366640+0000", 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0770000000000002e-06 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366658+0000", 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8046e-05 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.366665+0000", 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4860000000000002e-06 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.374059+0000", 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00739407 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.374079+0000", 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0553000000000001e-05 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.398923+0000", 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.024843649999999998 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.399010+0000", 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6625000000000005e-05 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:148031 7.6 e118/114 7:69dece02:::benchmark_data_smithi081_73391_object9032:head v 118'9227, mlcod=118'9223)", 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.368580+0000", 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.55908459200000005, 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032650712999999998, 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.368580+0000", 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.368580+0000", 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.368581+0000", 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.302e-06 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.368588+0000", 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.579e-06 2024-09-21T18:56:03.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.368600+0000", 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2629000000000001e-05 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.368607+0000", 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3659999999999997e-06 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.378165+0000", 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0095587750000000003 2024-09-21T18:56:03.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.378189+0000", 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.404e-05 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.401033+0000", 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022844162000000001 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.401230+0000", 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019686000000000001 2024-09-21T18:56:03.240 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:148028 7.e e118/114 7:762836cf:::benchmark_data_smithi081_73391_object9029:head v 118'9163, mlcod=118'9159)", 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.370697+0000", 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.55696680499999995, 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.033774945000000001, 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.370697+0000", 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.370697+0000", 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.370699+0000", 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.124e-06 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.370706+0000", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1400000000000002e-06 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.370715+0000", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2229999999999996e-06 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.370721+0000", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2960000000000004e-06 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.383772+0000", 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013050581 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.383799+0000", 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7345e-05 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.403687+0000", 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019888303 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.404472+0000", 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00078493300000000003 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:148375 7.4 e119/114 7:21d4f642:::benchmark_data_smithi081_73391_object9376:head v 119'9392, mlcod=119'9389)", 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.802000+0000", 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.125664517, 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030850239000000002, 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802000+0000", 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802000+0000", 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802001+0000", 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4190000000000001e-06 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802008+0000", 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.545e-06 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802022+0000", 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.415e-05 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802029+0000", 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0210000000000002e-06 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802119+0000", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0643999999999995e-05 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.802139+0000", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9368000000000002e-05 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.832810+0000", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030671057000000002 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.832850+0000", 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0034999999999999e-05 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:03.249 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:56:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:03.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:03.375 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:03.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:03.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:03.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:03.784 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:03.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:03.927 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:03.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:04.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:56:04.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1431582560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[39192]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[39192]: 4.1a scrub starts 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[39192]: 4.1a scrub ok 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1431582560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[29384]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[29384]: 4.1a scrub starts 2024-09-21T18:56:04.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:03 smithi081 ceph-mon[29384]: 4.1a scrub ok 2024-09-21T18:56:04.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:03 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:56:04.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1431582560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:03 smithi104 ceph-mon[32438]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T18:56:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:03 smithi104 ceph-mon[32438]: 4.1a scrub starts 2024-09-21T18:56:04.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:03 smithi104 ceph-mon[32438]: 4.1a scrub ok 2024-09-21T18:56:04.028 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:04.056 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:04.158 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:04.193 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:04.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:04.333 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:04.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:04.473 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:04.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:04.607 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:04.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.745 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:04.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:04.846 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:04.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:04.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:04.976 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:04.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:05.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:04 smithi104 ceph-mon[32438]: pgmap v123: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 139 MiB/s wr, 3.28k op/s; 62836/392655 objects misplaced (16.003%); 22 MiB/s, 363 objects/s recovering 2024-09-21T18:56:05.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:04 smithi081 ceph-mon[39192]: pgmap v123: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 139 MiB/s wr, 3.28k op/s; 62836/392655 objects misplaced (16.003%); 22 MiB/s, 363 objects/s recovering 2024-09-21T18:56:05.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:04 smithi081 ceph-mon[29384]: pgmap v123: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 139 MiB/s wr, 3.28k op/s; 62836/392655 objects misplaced (16.003%); 22 MiB/s, 363 objects/s recovering 2024-09-21T18:56:05.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:05.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:05.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:05.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:05.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:05.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:05.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:05.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T18:56:05.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:05.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:05.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:05.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:05.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:05.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:05.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:06.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:06.226 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:06.229 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:56:06.229 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:56:06.252 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:06.277 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:56:06.278 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:56:06.300 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:06.308 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:56:06.309 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:56:06.350 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:56:06.350 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:56:06.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:06.381 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:06.389 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:56:06.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T18:56:06.390 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:56:06.391 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:56:06.425 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:56:06.425 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:56:06.458 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:56:06.458 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:56:06.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:06.495 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:56:06.495 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:56:06.511 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:06.613 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.678 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:06.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:06.779 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.806 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:06.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:06.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:56:06.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:06.932 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:06.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:07.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:07.059 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:07.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:07.172 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[39192]: pgmap v124: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 608 op/s; 62836/392655 objects misplaced (16.003%); 11 MiB/s, 181 objects/s recovering 2024-09-21T18:56:07.172 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:07.172 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:07.172 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[39192]: 5.1 scrub starts 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[39192]: 5.1 scrub ok 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/967661809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[29384]: pgmap v124: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 608 op/s; 62836/392655 objects misplaced (16.003%); 11 MiB/s, 181 objects/s recovering 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[29384]: 5.1 scrub starts 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[29384]: 5.1 scrub ok 2024-09-21T18:56:07.173 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/967661809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:07.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:06 smithi104 ceph-mon[32438]: pgmap v124: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 608 op/s; 62836/392655 objects misplaced (16.003%); 11 MiB/s, 181 objects/s recovering 2024-09-21T18:56:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:06 smithi104 ceph-mon[32438]: 5.1 scrub starts 2024-09-21T18:56:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:06 smithi104 ceph-mon[32438]: 5.1 scrub ok 2024-09-21T18:56:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/967661809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:07.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:07.363 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:07.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:07.486 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:56:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:56:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:56:07.492 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:07.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:07.625 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:07.727 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:07.864 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:07.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:07.975 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:56:07.975 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:56:07.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:08.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:56:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:56:08.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:08.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:08.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:08.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:08.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:08.419 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:08.521 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:08.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:08.683 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:08.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:08.819 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:08.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:08.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:08.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:09.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:09.078 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:09.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:09.207 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:08 smithi104 ceph-mon[32438]: pgmap v125: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 59290/378942 objects misplaced (15.646%); 22 MiB/s, 353 objects/s recovering 2024-09-21T18:56:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:08 smithi104 ceph-mon[32438]: 3.1a scrub starts 2024-09-21T18:56:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:08 smithi104 ceph-mon[32438]: 3.1a scrub ok 2024-09-21T18:56:09.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:09.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[29384]: pgmap v125: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 59290/378942 objects misplaced (15.646%); 22 MiB/s, 353 objects/s recovering 2024-09-21T18:56:09.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:09.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[29384]: 3.1a scrub starts 2024-09-21T18:56:09.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[29384]: 3.1a scrub ok 2024-09-21T18:56:09.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[39192]: pgmap v125: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 59290/378942 objects misplaced (15.646%); 22 MiB/s, 353 objects/s recovering 2024-09-21T18:56:09.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:09.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[39192]: 3.1a scrub starts 2024-09-21T18:56:09.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:08 smithi081 ceph-mon[39192]: 3.1a scrub ok 2024-09-21T18:56:09.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:09.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.493 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:09.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:09.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:56:09.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:09.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:09.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:09.822 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:09.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:09.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:09 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T18:56:09.940 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:09 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T18:56:09.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:09.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:10.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:10.127 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:10.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:10.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:09 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T18:56:10.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:09 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T18:56:10.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:09 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T18:56:10.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:09 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T18:56:10.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:10.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:10.509 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:10.611 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:10.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:10.796 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:10.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:10.936 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:10.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T18:56:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:10.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:10.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:11.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:11.140 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:11.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:11.161 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:11.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:10 smithi104 ceph-mon[32438]: pgmap v126: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 963 op/s; 59290/378942 objects misplaced (15.646%); 18 MiB/s, 288 objects/s recovering 2024-09-21T18:56:11.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:10 smithi081 ceph-mon[29384]: pgmap v126: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 963 op/s; 59290/378942 objects misplaced (15.646%); 18 MiB/s, 288 objects/s recovering 2024-09-21T18:56:11.283 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:10 smithi081 ceph-mon[39192]: pgmap v126: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 963 op/s; 59290/378942 objects misplaced (15.646%); 18 MiB/s, 288 objects/s recovering 2024-09-21T18:56:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:11.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:11.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:11.441 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:11.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:11.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:11.591 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:11.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:11.692 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:11.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:11.822 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:11.847 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:11.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:11.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.005 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:12.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:12.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:12.131 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:12.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:12.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:12.305 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:12.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.445 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:12.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:12.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:12.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:12.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:12.703 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:12.802 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:12.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:12.864 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:56:12.864 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:56:12.903 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:56:12.903 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:56:12.934 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:12.938 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:56:12.938 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:56:12.960 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:12.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:12.968 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T18:56:12.980 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:56:12.986 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:56:12.986 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:56:13.031 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:56:13.031 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:56:13.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:13.074 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:56:13.074 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:56:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:13.113 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:56:13.113 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:56:13.146 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:56:13.146 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:56:13.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:13.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[39192]: pgmap v127: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s; 59290/378942 objects misplaced (15.646%); 18 MiB/s, 282 objects/s recovering 2024-09-21T18:56:13.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4228368663' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:56:13.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2133323664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:13.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[39192]: from='client.34916 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:13.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1589592138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:13.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[29384]: pgmap v127: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s; 59290/378942 objects misplaced (15.646%); 18 MiB/s, 282 objects/s recovering 2024-09-21T18:56:13.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4228368663' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:56:13.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2133323664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:13.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[29384]: from='client.34916 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:13.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1589592138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:12 smithi104 ceph-mon[32438]: pgmap v127: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s; 59290/378942 objects misplaced (15.646%); 18 MiB/s, 282 objects/s recovering 2024-09-21T18:56:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4228368663' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:56:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2133323664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:12 smithi104 ceph-mon[32438]: from='client.34916 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1589592138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:13.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:13.379 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:13.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:13.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:13.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:13.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.777 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:13.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:13.878 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:13.918 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:13.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:14.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:14.041 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:56:14.041 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:56:14.041 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:56:14.041 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:56:14.041 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:56:14.047 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:14.149 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:14.181 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:13 smithi104 ceph-mon[32438]: 6.7 scrub starts 2024-09-21T18:56:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:13 smithi104 ceph-mon[32438]: 6.7 scrub ok 2024-09-21T18:56:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:13 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4228368663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T18:56:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:13 smithi104 ceph-mon[32438]: from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:56:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:13 smithi104 ceph-mon[32438]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T18:56:14.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[39192]: 6.7 scrub starts 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[39192]: 6.7 scrub ok 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4228368663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[39192]: from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[39192]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[29384]: 6.7 scrub starts 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[29384]: 6.7 scrub ok 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4228368663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T18:56:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[29384]: from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:56:14.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:13 smithi081 ceph-mon[29384]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T18:56:14.414 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:14.443 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:14.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:14.569 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:14.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:14.705 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:14.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:14.841 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:14.942 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:14.969 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:14.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:15.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:15.108 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:15.210 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:15.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:14 smithi104 ceph-mon[32438]: pgmap v129: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 929 op/s; 55648/364779 objects misplaced (15.255%); 18 MiB/s, 285 objects/s recovering 2024-09-21T18:56:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:14 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:56:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:14 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:56:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:14 smithi081 ceph-mon[29384]: pgmap v129: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 929 op/s; 55648/364779 objects misplaced (15.255%); 18 MiB/s, 285 objects/s recovering 2024-09-21T18:56:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:14 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:56:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:14 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:56:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:14 smithi081 ceph-mon[39192]: pgmap v129: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 929 op/s; 55648/364779 objects misplaced (15.255%); 18 MiB/s, 285 objects/s recovering 2024-09-21T18:56:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:14 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:56:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:14 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:56:15.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:15.363 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:15.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:15.494 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:15.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:15.634 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:15.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:15.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T18:56:15.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:15.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:15.904 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:15.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:16.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:16.033 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:16.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:16.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:16.164 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:16.214 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:15 smithi081 ceph-mon[39192]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T18:56:16.214 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:15 smithi081 ceph-mon[39192]: pgmap v131: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 590 op/s; 55648/364779 objects misplaced (15.255%); 12 MiB/s, 185 objects/s recovering 2024-09-21T18:56:16.214 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:15 smithi081 ceph-mon[29384]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T18:56:16.215 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:15 smithi081 ceph-mon[29384]: pgmap v131: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 590 op/s; 55648/364779 objects misplaced (15.255%); 12 MiB/s, 185 objects/s recovering 2024-09-21T18:56:16.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:15 smithi104 ceph-mon[32438]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T18:56:16.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:15 smithi104 ceph-mon[32438]: pgmap v131: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 590 op/s; 55648/364779 objects misplaced (15.255%); 12 MiB/s, 185 objects/s recovering 2024-09-21T18:56:16.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:16.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T18:56:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:16.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:16.358 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:16.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:16.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:16.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:16.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:56:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.604 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:16.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:16.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.764 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:16.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:16.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:16.891 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:16.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:16.993 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:17.020 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:17.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:56:17.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:17.175 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:17.249 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:17.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.302 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:17.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:17.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:17.430 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:56:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:56:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:56:17.469 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:17.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:17.586 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:17.650 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:17.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:17.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:17.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:17.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:17.937 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:56:17.937 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:56:17.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:56:17.937 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:13972 7.9 7:9b0b907e:::benchmark_data_smithi081_73391_object13971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:25.427534+0000", 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 112.50525905800001, 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.076687018999999995, 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 13972 2024-09-21T18:56:17.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427534+0000", 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427534+0000", 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427536+0000", 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.218e-06 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427557+0000", 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0951e-05 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427566+0000", 2024-09-21T18:56:17.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3070000000000008e-06 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.427573+0000", 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.674e-06 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481430+0000", 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053857078000000003 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481593+0000", 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000163618 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.481730+0000", 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013723899999999999 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.482879+0000", 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001148376 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.483564+0000", 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000685375 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.504042+0000", 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020477795 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.504122+0000", 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0238e-05 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:25.504221+0000", 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9149999999999998e-05 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:54919 7.5 e94/82 7:a6ed1b73:::benchmark_data_smithi081_73391_object54918:head v 94'3481, mlcod=94'3478)", 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.457562+0000", 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 91.475231266999998, 2024-09-21T18:56:17.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.071749502000000007, 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457562+0000", 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457562+0000", 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457563+0000", 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.175e-06 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457674+0000", 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111103 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457683+0000", 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8140000000000007e-06 2024-09-21T18:56:17.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457689+0000", 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7679999999999997e-06 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457840+0000", 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015101000000000001 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.457858+0000", 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8315999999999999e-05 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.461996+0000", 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041372780000000003 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529312+0000", 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.067316037999999995 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:54922 7.d e94/82 7:b40a5803:::benchmark_data_smithi081_73391_object54921:head v 94'3447, mlcod=94'3444)", 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.458048+0000", 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 91.474745127000006, 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.071294892999999998, 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458048+0000", 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458048+0000", 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458049+0000", 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8999999999999996e-07 2024-09-21T18:56:17.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458164+0000", 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011514399999999999 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458173+0000", 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.458178+0000", 2024-09-21T18:56:17.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3600000000000004e-06 2024-09-21T18:56:17.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460236+0000", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002057924 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460253+0000", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7051e-05 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529317+0000", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069064023000000002 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529343+0000", 2024-09-21T18:56:17.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6197999999999999e-05 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:54923 7.b 7:dace79c5:::benchmark_data_smithi081_73391_object54922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.459001+0000", 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 91.473792201999998, 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.070825013000000006, 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:17.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 54923 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459001+0000", 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459001+0000", 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459003+0000", 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459027+0000", 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3842e-05 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459038+0000", 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0817e-05 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459045+0000", 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1400000000000002e-06 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460270+0000", 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001225119 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460573+0000", 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030314200000000001 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460697+0000", 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012391400000000001 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529430+0000", 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.068733048000000005 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529580+0000", 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149806 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529617+0000", 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7166000000000003e-05 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529785+0000", 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000167267 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529826+0000", 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1662000000000002e-05 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.959 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:54921 7.5 e94/82 7:a314aa6c:::benchmark_data_smithi081_73391_object54920:head v 94'3482, mlcod=94'3478)", 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.459617+0000", 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 91.473176464000005, 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.071807165000000006, 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459617+0000", 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459617+0000", 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459618+0000", 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.159e-06 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459756+0000", 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013828199999999999 2024-09-21T18:56:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459766+0000", 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3959999999999995e-06 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.459772+0000", 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9150000000000001e-06 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460409+0000", 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063733400000000003 2024-09-21T18:56:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.460425+0000", 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6053999999999999e-05 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.531374+0000", 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.070948836000000001 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.531424+0000", 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0188999999999997e-05 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:54929 7.5 e94/82 7:af33521c:::benchmark_data_smithi081_73391_object54928:head v 94'3483, mlcod=94'3479)", 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:54:46.462155+0000", 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 91.470638799, 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.074659339000000005, 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.462155+0000", 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.462155+0000", 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.462155+0000", 2024-09-21T18:56:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7400000000000002e-07 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.462239+0000", 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3252999999999997e-05 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529271+0000", 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.067032674 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529285+0000", 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4086e-05 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529366+0000", 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0731999999999999e-05 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.529384+0000", 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7538e-05 2024-09-21T18:56:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.536775+0000", 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0073914109999999996 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:54:46.536814+0000", 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8871000000000001e-05 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:99371 7.d e102/82 7:b758ae26:::benchmark_data_smithi081_73391_object99370:head v 102'6268, mlcod=102'6266)", 2024-09-21T18:56:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:17.701918+0000", 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.230875660999999, 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.072328385999999995, 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.701918+0000", 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.701918+0000", 2024-09-21T18:56:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.701919+0000", 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3340000000000001e-06 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.702027+0000", 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010748599999999999 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763447+0000", 2024-09-21T18:56:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.061420332000000001 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763453+0000", 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0549999999999996e-06 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.767356+0000", 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039026400000000002 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.767375+0000", 2024-09-21T18:56:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9300000000000002e-05 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.774215+0000", 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0068405239999999997 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.774246+0000", 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0715e-05 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:99377 7.9 7:9f989195:::benchmark_data_smithi081_73391_object99376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.225186241999999, 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106765947, 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 99377 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:56:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707610+0000", 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.196e-06 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707638+0000", 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7880000000000001e-05 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763151+0000", 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055513121999999998 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763168+0000", 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6152000000000001e-05 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763470+0000", 2024-09-21T18:56:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030208399999999999 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763608+0000", 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013794999999999999 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763999+0000", 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000391691 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.771410+0000", 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074111979999999999 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.771550+0000", 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013909999999999999 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814280+0000", 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.042729984999999998 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814321+0000", 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1594999999999997e-05 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814373+0000", 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1993999999999997e-05 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.980 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101895 7.b 7:d6f8fed5:::benchmark_data_smithi081_73391_object101894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.022693894, 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10653628599999999, 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:17.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101895 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910101+0000", 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8109999999999999e-06 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910123+0000", 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1500000000000001e-05 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910138+0000", 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4798e-05 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910145+0000", 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1790000000000002e-06 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910195+0000", 2024-09-21T18:56:17.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9716000000000003e-05 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910299+0000", 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104134 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910439+0000", 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001403 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.912887+0000", 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024483080000000002 2024-09-21T18:56:17.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916379+0000", 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034916880000000002 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016537+0000", 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.100158393 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016591+0000", 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3170000000000001e-05 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016636+0000", 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5289e-05 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.987 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101904 7.b 7:d7f209bc:::benchmark_data_smithi081_73391_object101903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.015909501000003, 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10347793, 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101904 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916886+0000", 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916905+0000", 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9527000000000001e-05 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916915+0000", 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8130000000000007e-06 2024-09-21T18:56:17.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916920+0000", 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1420000000000001e-06 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916958+0000", 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8059000000000003e-05 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.917042+0000", 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4251000000000004e-05 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.917130+0000", 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7626000000000004e-05 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.919036+0000", 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00190617 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.919594+0000", 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055802599999999999 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020228+0000", 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.100633629 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020322+0000", 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4293e-05 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020362+0000", 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9777999999999998e-05 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:133492 7.d e111/82 7:b21cb62b:::benchmark_data_smithi081_73391_object133491:head v 111'8437, mlcod=111'8434)", 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:46.019499+0000", 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 31.913294426, 2024-09-21T18:56:17.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069682174, 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019499+0000", 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:17.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019499+0000", 2024-09-21T18:56:17.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:17.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019500+0000", 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.401e-06 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019532+0000", 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1999999999999999e-05 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019541+0000", 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1589999999999993e-06 2024-09-21T18:56:17.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019547+0000", 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.981e-06 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019616+0000", 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9534000000000002e-05 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019635+0000", 2024-09-21T18:56:17.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8783e-05 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.089141+0000", 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069506320999999996 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.089181+0000", 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9994999999999999e-05 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:17.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:133496 7.5 e111/82 7:a3e205bf:::benchmark_data_smithi081_73391_object133495:head v 111'8581, mlcod=111'8577)", 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:46.019941+0000", 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 31.912852217000001, 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069270692999999994, 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019941+0000", 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019941+0000", 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019943+0000", 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3179999999999999e-06 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019977+0000", 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4152000000000001e-05 2024-09-21T18:56:18.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019987+0000", 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.893e-06 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.019993+0000", 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4620000000000001e-06 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.022385+0000", 2024-09-21T18:56:18.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023920349999999998 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.022414+0000", 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9125999999999999e-05 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.089187+0000", 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.066773017000000004 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.089212+0000", 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4689999999999999e-05 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:133501 7.d e111/82 7:b318537a:::benchmark_data_smithi081_73391_object133500:head v 111'8439, mlcod=111'8435)", 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:46.025710+0000", 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 31.907083612000001, 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069381414000000002, 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.004 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.025710+0000", 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.025710+0000", 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.025711+0000", 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2190000000000001e-06 2024-09-21T18:56:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.025816+0000", 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104576 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.085545+0000", 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.059729628 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.085554+0000", 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0399999999999998e-06 2024-09-21T18:56:18.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.089235+0000", 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036810829999999999 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.089251+0000", 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6022e-05 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.095056+0000", 2024-09-21T18:56:18.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058047130000000004 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:46.095091+0000", 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5132999999999997e-05 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136125 7.b 7:df011645:::benchmark_data_smithi081_73391_object136124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.771813405, 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136450605, 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.008 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136125 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160983+0000", 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4490000000000002e-06 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161014+0000", 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1038000000000003e-05 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161026+0000", 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2753e-05 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161035+0000", 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0040000000000005e-06 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161146+0000", 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110518 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161292+0000", 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001461 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161445+0000", 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000153475 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170642+0000", 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0091963119999999999 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170656+0000", 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3947e-05 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297280+0000", 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12662463500000001 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297381+0000", 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101037 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297431+0000", 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9336999999999999e-05 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136126 7.b 7:d13dafa2:::benchmark_data_smithi081_73391_object136125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.771731546000002, 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139333067, 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.014 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136126 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:56:18.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161063+0000", 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3500000000000005e-07 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161084+0000", 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0755e-05 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161091+0000", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2169999999999997e-06 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161097+0000", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3230000000000003e-06 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164270+0000", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031733239999999999 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164391+0000", 2024-09-21T18:56:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120575 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164535+0000", 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014399899999999999 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170661+0000", 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061256419999999997 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170700+0000", 2024-09-21T18:56:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9578e-05 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300264+0000", 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12956358800000001 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300355+0000", 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1219999999999997e-05 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300395+0000", 2024-09-21T18:56:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0011e-05 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136128 7.b 7:df396259:::benchmark_data_smithi081_73391_object136127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.770088998999999, 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139089409, 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:18.020 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136128 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162707+0000", 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162731+0000", 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4485e-05 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162747+0000", 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5452999999999999e-05 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162755+0000", 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9920000000000007e-06 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167113+0000", 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043581749999999997 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167237+0000", 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012398699999999999 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167386+0000", 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149672 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170665+0000", 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032787960000000001 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170739+0000", 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4108000000000003e-05 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301654+0000", 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13091487199999999 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301749+0000", 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5187999999999997e-05 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301794+0000", 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4558000000000003e-05 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.025 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136146 7.b 7:da687baa:::benchmark_data_smithi081_73391_object136145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.757419287000001, 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12656247800000001, 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:18.026 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136146 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175376+0000", 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1040000000000001e-06 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175400+0000", 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4162e-05 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175412+0000", 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1802e-05 2024-09-21T18:56:18.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175420+0000", 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7149999999999998e-06 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175494+0000", 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3688000000000003e-05 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175609+0000", 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115519 2024-09-21T18:56:18.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175745+0000", 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013621299999999999 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177851+0000", 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021052779999999999 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177986+0000", 2024-09-21T18:56:18.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013539800000000001 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301842+0000", 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.123855964 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301899+0000", 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7426000000000002e-05 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301937+0000", 2024-09-21T18:56:18.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7209000000000002e-05 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136148 7.b 7:d9fea987:::benchmark_data_smithi081_73391_object136147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.75486527, 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12981583899999999, 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.032 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136148 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:56:18.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177930+0000", 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.545e-06 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177953+0000", 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.349e-05 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177962+0000", 2024-09-21T18:56:18.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8890000000000008e-06 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177969+0000", 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8759999999999999e-06 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178007+0000", 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8126000000000001e-05 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178088+0000", 2024-09-21T18:56:18.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:18.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:56:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0684000000000002e-05 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178175+0000", 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7643000000000001e-05 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.179852+0000", 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001676774 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.179997+0000", 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014443799999999999 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307615+0000", 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12761845299999999 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307693+0000", 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8103999999999996e-05 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307744+0000", 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0816999999999998e-05 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.039 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPushReply(7.9 117/114 [PushReplyOp(7:914913c5:::benchmark_data_smithi081_73391_object27593:head)])", 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.349493+0000", 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 18.583300194, 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.081388372, 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349493+0000", 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349493+0000", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349494+0000", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.65e-07 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349496+0000", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7460000000000001e-06 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349500+0000", 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0260000000000001e-06 2024-09-21T18:56:18.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349505+0000", 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8099999999999997e-06 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.430598+0000", 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.081093099000000002 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.430882+0000", 2024-09-21T18:56:18.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028332599999999998 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:145390 7.5 e117/114 7:a9dbf476:::benchmark_data_smithi081_73391_object6391:head v 117'9324, mlcod=117'9322)", 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.350452+0000", 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 18.582341293999999, 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.079042043000000006, 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:18.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.350452+0000", 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.350452+0000", 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.350453+0000", 2024-09-21T18:56:18.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1540000000000001e-06 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.350466+0000", 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3005e-05 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.425702+0000", 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.075235733999999999 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.425714+0000", 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2115e-05 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.425829+0000", 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011527 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.425856+0000", 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6169999999999998e-05 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.429444+0000", 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035880450000000002 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.429494+0000", 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0550000000000002e-05 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:18.047 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:56:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.085 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:18.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:18.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:18.218 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:18.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:18.350 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:18.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:18.484 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T18:56:18.499 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:18.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:18.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:18.640 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:18.741 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:18.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:18 smithi104 ceph-mon[32438]: pgmap v132: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 52122/350364 objects misplaced (14.877%); 22 MiB/s, 350 objects/s recovering 2024-09-21T18:56:18.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2913102091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T18:56:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:18 smithi104 ceph-mon[32438]: from='client.34925 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T18:56:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:18.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[39192]: pgmap v132: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 52122/350364 objects misplaced (14.877%); 22 MiB/s, 350 objects/s recovering 2024-09-21T18:56:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2913102091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T18:56:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[39192]: from='client.34925 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T18:56:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:18.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[29384]: pgmap v132: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 52122/350364 objects misplaced (14.877%); 22 MiB/s, 350 objects/s recovering 2024-09-21T18:56:18.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2913102091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T18:56:18.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[29384]: from='client.34925 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T18:56:18.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:18.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:18.904 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:18.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:19.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:19.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:19.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:19.169 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:19.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.298 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:19.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:19.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:19.424 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:19.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:19.546 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:19.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:19.685 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:19.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:19 smithi104 ceph-mon[32438]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T18:56:19.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:19 smithi104 ceph-mon[32438]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T18:56:19.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:19.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:19.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:19 smithi081 ceph-mon[39192]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T18:56:19.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:19 smithi081 ceph-mon[39192]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T18:56:19.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:19 smithi081 ceph-mon[29384]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T18:56:19.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:19 smithi081 ceph-mon[29384]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T18:56:19.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:19.948 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:19.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:20.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:20.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:20.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T18:56:20.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:20.222 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:20.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:20.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:20.478 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:20.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:20.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:20.745 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:20.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:20 smithi081 ceph-mon[39192]: pgmap v134: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 745 op/s; 52122/350364 objects misplaced (14.877%); 13 MiB/s, 205 objects/s recovering 2024-09-21T18:56:20.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:20 smithi081 ceph-mon[29384]: pgmap v134: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 745 op/s; 52122/350364 objects misplaced (14.877%); 13 MiB/s, 205 objects/s recovering 2024-09-21T18:56:20.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:20.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:20.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:21.003 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:21.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:20 smithi104 ceph-mon[32438]: pgmap v134: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 745 op/s; 52122/350364 objects misplaced (14.877%); 13 MiB/s, 205 objects/s recovering 2024-09-21T18:56:21.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:21.158 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:21.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:21.306 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:21.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:21.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:21.535 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:21.562 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:21.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T18:56:21.619 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:21.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:21.731 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:21.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:21.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:21.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:21.967 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:56:21.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:22.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2673078738' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:22.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:22.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:22.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2673078738' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:22.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:22.191 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2673078738' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:22.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:22.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:22.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:22.450 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:22.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:22.580 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:22.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:22.722 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:22.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:22.854 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:22.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:22.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:22.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:23.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:23.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:23.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:22 smithi104 ceph-mon[32438]: pgmap v135: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 600 op/s; 52122/350364 objects misplaced (14.877%); 10 MiB/s, 165 objects/s recovering 2024-09-21T18:56:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:22 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2673078738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:22 smithi104 ceph-mon[32438]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T18:56:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.259 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:23.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[29384]: pgmap v135: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 600 op/s; 52122/350364 objects misplaced (14.877%); 10 MiB/s, 165 objects/s recovering 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2673078738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[29384]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[39192]: pgmap v135: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 600 op/s; 52122/350364 objects misplaced (14.877%); 10 MiB/s, 165 objects/s recovering 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2673078738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:22 smithi081 ceph-mon[39192]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T18:56:23.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:23.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:23.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:23.532 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:23.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:23.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:23.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:23.801 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:23.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:23.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:24.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:24.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:24.162 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:24.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:24.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:23 smithi104 ceph-mon[32438]: pgmap v137: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 48775/336387 objects misplaced (14.500%); 20 MiB/s, 323 objects/s recovering 2024-09-21T18:56:24.285 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:23 smithi081 ceph-mon[29384]: pgmap v137: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 48775/336387 objects misplaced (14.500%); 20 MiB/s, 323 objects/s recovering 2024-09-21T18:56:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:23 smithi081 ceph-mon[39192]: pgmap v137: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 48775/336387 objects misplaced (14.500%); 20 MiB/s, 323 objects/s recovering 2024-09-21T18:56:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.328 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:24.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:24.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.451 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:24.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:24.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.595 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:24.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:24.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:24.717 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:24.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:24.847 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:24.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:24.973 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:24.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:25.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:25.080 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:56:25.080 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:25.124 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:56:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:56:25.124 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3870462143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:25.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3870462143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:25.168 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:56:25.168 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:56:25.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:25.223 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:56:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:56:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:25.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:25.265 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:56:25.265 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:56:25.302 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:56:25.303 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:56:25.341 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:56:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:56:25.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:25.377 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:56:25.377 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:56:25.392 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:25.494 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:25.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3870462143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:25.521 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:25.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:25.672 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:25.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:25.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:25.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:25.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:26.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:26.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T18:56:26.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:26.201 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:26.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:26.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:26.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:26.470 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:26.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:26 smithi104 ceph-mon[32438]: pgmap v138: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 582 op/s; 48775/336387 objects misplaced (14.500%); 9.9 MiB/s, 157 objects/s recovering 2024-09-21T18:56:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2558939916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T18:56:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:26 smithi104 ceph-mon[32438]: from='client.34940 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T18:56:26.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:26 smithi081 ceph-mon[29384]: pgmap v138: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 582 op/s; 48775/336387 objects misplaced (14.500%); 9.9 MiB/s, 157 objects/s recovering 2024-09-21T18:56:26.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2558939916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T18:56:26.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:26 smithi081 ceph-mon[29384]: from='client.34940 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T18:56:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:26 smithi081 ceph-mon[39192]: pgmap v138: 148 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 140 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 582 op/s; 48775/336387 objects misplaced (14.500%); 9.9 MiB/s, 157 objects/s recovering 2024-09-21T18:56:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2558939916' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T18:56:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:26 smithi081 ceph-mon[39192]: from='client.34940 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T18:56:26.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:26.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:26.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:26.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:26.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:27.014 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:27.116 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:27.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:27.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:27.272 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:27.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:27 smithi081 ceph-mon[29384]: from='client.34940 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T18:56:27.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:27 smithi081 ceph-mon[29384]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T18:56:27.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:27 smithi081 ceph-mon[39192]: from='client.34940 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T18:56:27.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:27 smithi081 ceph-mon[39192]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T18:56:27.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:56:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:56:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:56:27.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:27.403 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:27.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:27 smithi104 ceph-mon[32438]: from='client.34940 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T18:56:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:27 smithi104 ceph-mon[32438]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T18:56:27.531 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:27.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:27.666 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:27.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:27.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:27.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:27.948 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:27.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:28.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:56:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:56:28.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:28.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:28.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:28.227 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:28 smithi081 ceph-mon[39192]: pgmap v140: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 45583/322821 objects misplaced (14.120%); 20 MiB/s, 322 objects/s recovering 2024-09-21T18:56:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:28 smithi081 ceph-mon[29384]: pgmap v140: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 45583/322821 objects misplaced (14.120%); 20 MiB/s, 322 objects/s recovering 2024-09-21T18:56:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:28.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:28.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:28.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:28.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:28 smithi104 ceph-mon[32438]: pgmap v140: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 45583/322821 objects misplaced (14.120%); 20 MiB/s, 322 objects/s recovering 2024-09-21T18:56:28.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:28.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:28.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:28.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:28.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:28.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:29.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:29.048 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:29.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:29.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T18:56:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:29.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:29.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:29.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.477 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:29.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:29.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:29.630 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:29.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:56:29.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:29.801 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:29.837 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:56:29.837 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:56:29.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:29.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:29.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:30.055 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:30.083 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:30.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:30.143 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:30.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:30.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:30.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:30.373 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:30.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:30.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:56:30.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:30.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:30.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:30.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T18:56:30.624 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:30 smithi081 ceph-mon[29384]: pgmap v141: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 45583/322821 objects misplaced (14.120%); 20 MiB/s, 322 objects/s recovering 2024-09-21T18:56:30.624 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:30 smithi081 ceph-mon[39192]: pgmap v141: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 45583/322821 objects misplaced (14.120%); 20 MiB/s, 322 objects/s recovering 2024-09-21T18:56:30.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:30.721 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:30.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:30 smithi104 ceph-mon[32438]: pgmap v141: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 45583/322821 objects misplaced (14.120%); 20 MiB/s, 322 objects/s recovering 2024-09-21T18:56:30.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:30.878 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:30.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:30.905 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:30.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:31.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:31.179 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:31.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.330 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:31.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:31.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:31.474 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:31.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:31.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:31.711 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:31.738 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:31.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:31.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:31.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:31.999 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:32.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:32.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.285 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:32.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:32.386 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:32.418 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:32.442 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:32.498 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:56:32.502 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T18:56:32.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:32.526 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:32.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:32.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.730 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:32.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:32.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:32 smithi104 ceph-mon[32438]: pgmap v142: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 541 B/s rd, 0 B/s wr, 478 op/s; 45583/322821 objects misplaced (14.120%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T18:56:32.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:32 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/885307976' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T18:56:32.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:32 smithi104 ceph-mon[32438]: from='client.34946 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T18:56:32.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:32 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2370590516' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:56:32.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[29384]: pgmap v142: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 541 B/s rd, 0 B/s wr, 478 op/s; 45583/322821 objects misplaced (14.120%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T18:56:32.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/885307976' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T18:56:32.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[29384]: from='client.34946 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T18:56:32.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2370590516' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:56:32.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[39192]: pgmap v142: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 541 B/s rd, 0 B/s wr, 478 op/s; 45583/322821 objects misplaced (14.120%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T18:56:32.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/885307976' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T18:56:32.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[39192]: from='client.34946 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T18:56:32.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:32 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2370590516' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:56:32.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:32.857 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:32.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:32.988 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:32.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:33.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:33.135 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:33.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:33.263 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:33.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:33.394 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:33.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:33.522 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:33.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:33.662 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:33.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:33.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[29384]: from='client.34946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2370590516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[29384]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[39192]: from='client.34946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2370590516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:56:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:33 smithi081 ceph-mon[39192]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T18:56:33.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:33.923 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:33.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:34.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:33 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:34.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:33 smithi104 ceph-mon[32438]: from='client.34946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T18:56:34.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2370590516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:56:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:33 smithi104 ceph-mon[32438]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T18:56:34.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:34.052 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:34.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:34.179 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:34.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:34.287 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:56:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:56:34.314 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:34.322 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:56:34.323 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:56:34.358 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:56:34.358 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:56:34.393 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:56:34.393 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:56:34.416 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:34.425 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:56:34.426 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:56:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:34.463 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:56:34.463 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:56:34.507 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:56:34.507 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:56:34.541 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:56:34.541 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:56:34.546 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:34.586 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:34.612 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:56:34.612 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:56:34.612 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:56:34.612 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:56:34.612 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:56:34.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[29384]: pgmap v144: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 42262/308604 objects misplaced (13.695%); 20 MiB/s, 325 objects/s recovering 2024-09-21T18:56:34.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T18:56:34.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T18:56:34.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1239295405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:34.613 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[39192]: pgmap v144: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 42262/308604 objects misplaced (13.695%); 20 MiB/s, 325 objects/s recovering 2024-09-21T18:56:34.613 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T18:56:34.613 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T18:56:34.613 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1239295405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:34.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:34.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:34.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:34.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:35.006 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:35.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:34 smithi104 ceph-mon[32438]: pgmap v144: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 42262/308604 objects misplaced (13.695%); 20 MiB/s, 325 objects/s recovering 2024-09-21T18:56:35.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:34 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T18:56:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:34 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T18:56:35.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1239295405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:35.108 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:35.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:35.202 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:56:35.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:35.274 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:35.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.434 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:35.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:35.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:35.541 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:35 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T18:56:35.541 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:35 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T18:56:35.589 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:35.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:35.717 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:35.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T18:56:35.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:35 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T18:56:35.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:35 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T18:56:35.818 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:35.842 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:35.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:35.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:36.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:35 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T18:56:36.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:35 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T18:56:36.072 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:36.099 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:36.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:36.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:36.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:36.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:36.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:36.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T18:56:36.498 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:36.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:36.606 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:36.614 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:36.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:36.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:36.757 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:36.765 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[29384]: pgmap v145: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 994 op/s; 42262/308604 objects misplaced (13.695%); 17 MiB/s, 279 objects/s recovering 2024-09-21T18:56:36.765 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T18:56:36.765 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T18:56:36.765 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:36.765 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[39192]: pgmap v145: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 994 op/s; 42262/308604 objects misplaced (13.695%); 17 MiB/s, 279 objects/s recovering 2024-09-21T18:56:36.765 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T18:56:36.766 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T18:56:36.766 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:36.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:36.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:36.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:36 smithi104 ceph-mon[32438]: pgmap v145: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 994 op/s; 42262/308604 objects misplaced (13.695%); 17 MiB/s, 279 objects/s recovering 2024-09-21T18:56:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:36 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T18:56:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:36 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T18:56:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:37.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.056 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:37.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:37.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:37.190 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:37.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:56:37.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.329 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:37.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:37.332 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:56:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:56:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:56:37.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:37.455 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:37.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:37.599 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:56:37.600 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:56:37.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:37.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:37.759 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:37.858 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:37.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:37.877 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:37.886 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:37.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:37.988 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:38.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:56:38.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:56:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:56:38.017 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:38.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:38.146 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:38.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:38.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:38.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:38.410 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:38.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T18:56:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:56:38.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:38.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:38.750 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:56:38.750 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:56:38.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:56:38.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:56:38.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101583 7.8 7:14d2ad66:::benchmark_data_smithi081_73391_object101582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.626245+0000", 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 79.117725257999993, 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069010198999999994, 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101583 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626245+0000", 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626245+0000", 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626246+0000", 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3960000000000001e-06 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626266+0000", 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0313e-05 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626276+0000", 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0059e-05 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626283+0000", 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5540000000000003e-06 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631863+0000", 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005580097 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631967+0000", 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010382099999999999 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.632093+0000", 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012662 2024-09-21T18:56:38.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.641318+0000", 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0092242369999999997 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.641337+0000", 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9745000000000002e-05 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.695179+0000", 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.053842099999999997 2024-09-21T18:56:38.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.695221+0000", 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1560999999999997e-05 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.695255+0000", 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3695999999999997e-05 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101584 7.8 7:10573145:::benchmark_data_smithi081_73391_object101583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.626625+0000", 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 79.117345047000001, 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.068796866999999998, 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101584 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626625+0000", 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626625+0000", 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626626+0000", 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1999999999999999e-06 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626646+0000", 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9791000000000001e-05 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626655+0000", 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9299999999999992e-06 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.626661+0000", 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2339999999999999e-06 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.634980+0000", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0083187490000000003 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.635111+0000", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013168899999999999 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.635254+0000", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000142884 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.641346+0000", 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060915470000000001 2024-09-21T18:56:38.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.649547+0000", 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082014589999999995 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.695304+0000", 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.045756674999999997 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.695365+0000", 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1391999999999998e-05 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.695422+0000", 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6317000000000003e-05 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101590 7.8 7:1a915cc8:::benchmark_data_smithi081_73391_object101589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.631582+0000", 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 79.112387588999994, 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.075028945999999999, 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.762 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101590 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631582+0000", 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631582+0000", 2024-09-21T18:56:38.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631585+0000", 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4250000000000001e-06 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631610+0000", 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4927999999999999e-05 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631626+0000", 2024-09-21T18:56:38.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6364000000000001e-05 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.631634+0000", 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6599999999999995e-06 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.638422+0000", 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0067881449999999998 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.638572+0000", 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015004100000000001 2024-09-21T18:56:38.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.638732+0000", 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016025999999999999 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.641567+0000", 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028349299999999998 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.649598+0000", 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008031481 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.706271+0000", 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.056672608999999999 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.706560+0000", 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028898900000000002 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.706611+0000", 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1113999999999999e-05 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.767 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101894 7.8 7:1138a4d6:::benchmark_data_smithi081_73391_object101893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.909620+0000", 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 78.834349908999997, 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.087448464000000004, 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101894 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.768 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909620+0000", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909620+0000", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909622+0000", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.686e-06 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909640+0000", 2024-09-21T18:56:38.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8893999999999999e-05 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909651+0000", 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0949e-05 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909660+0000", 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1869999999999997e-06 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909727+0000", 2024-09-21T18:56:38.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7782999999999998e-05 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.909850+0000", 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000122287 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910020+0000", 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017054700000000001 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.912871+0000", 2024-09-21T18:56:38.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028508769999999999 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.996816+0000", 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.083944593999999997 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.996913+0000", 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7263999999999996e-05 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.997020+0000", 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107013 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.997068+0000", 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8383000000000001e-05 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114536 7.e 7:7856d85f:::benchmark_data_smithi081_73391_object114535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.722701+0000", 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.021268513999999, 2024-09-21T18:56:38.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11396537700000001, 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114536 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722701+0000", 2024-09-21T18:56:38.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722701+0000", 2024-09-21T18:56:38.775 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: pgmap v146: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s; 39256/294999 objects misplaced (13.307%); 16 MiB/s, 252 objects/s recovering 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2614988000' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: from='client.34961 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/870325963' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T18:56:38.776 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:38 smithi104 ceph-mon[32438]: from='client.24952 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T18:56:38.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722703+0000", 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5990000000000001e-06 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722722+0000", 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9514e-05 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722731+0000", 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9029999999999993e-06 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722737+0000", 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6740000000000002e-06 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722791+0000", 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.363e-05 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.722870+0000", 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9127000000000003e-05 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.723003+0000", 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013346399999999999 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.725130+0000", 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021263559999999998 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.727028+0000", 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018979870000000001 2024-09-21T18:56:38.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.836555+0000", 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.109527286 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.836607+0000", 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2500999999999998e-05 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.836667+0000", 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9336000000000002e-05 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.780 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:115637 7.e 7:72bc0032:::benchmark_data_smithi081_73391_object115636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:31.706292+0000", 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 67.037678193999994, 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.069453416000000004, 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.781 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 115637 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706292+0000", 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706292+0000", 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706294+0000", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1270000000000001e-06 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706317+0000", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3348000000000001e-05 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706330+0000", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2445e-05 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706337+0000", 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6240000000000002e-06 2024-09-21T18:56:38.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706393+0000", 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6221000000000003e-05 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706519+0000", 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012606 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.706652+0000", 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000132662 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.709544+0000", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002891682 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.713099+0000", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035556400000000001 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.775638+0000", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.062539044000000002 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.775701+0000", 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2456e-05 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:31.775745+0000", 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4107000000000003e-05 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:121048 7.e 7:7fe1c9d5:::benchmark_data_smithi081_73391_object121047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:36.421274+0000", 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.322695981999999, 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.084801724999999994, 2024-09-21T18:56:38.786 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 121048 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421274+0000", 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421274+0000", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421275+0000", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6750000000000001e-06 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421297+0000", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1458999999999999e-05 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421305+0000", 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7400000000000004e-06 2024-09-21T18:56:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421311+0000", 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7240000000000003e-06 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421377+0000", 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5149999999999998e-05 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421519+0000", 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014276600000000001 2024-09-21T18:56:38.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.421601+0000", 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1717000000000004e-05 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.423820+0000", 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002218879 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.426656+0000", 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028356689999999999 2024-09-21T18:56:38.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.505984+0000", 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.079328522999999998 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.506037+0000", 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2818999999999997e-05 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.506076+0000", 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8603999999999998e-05 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:121053 7.e 7:78c4748f:::benchmark_data_smithi081_73391_object121052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:36.427186+0000", 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.316783422999997, 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.084611028000000005, 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 121053 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427186+0000", 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427186+0000", 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427188+0000", 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3820000000000001e-06 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427205+0000", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7438000000000001e-05 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427213+0000", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.554e-06 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427217+0000", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3869999999999997e-06 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427250+0000", 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2840999999999999e-05 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427323+0000", 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2558000000000006e-05 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.427403+0000", 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0965999999999997e-05 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.429577+0000", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021733270000000001 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.430684+0000", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011075989999999999 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.511641+0000", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.080957030999999999 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.511747+0000", 2024-09-21T18:56:38.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010553299999999999 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:36.511797+0000", 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0411999999999998e-05 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145382 7.e 7:7cb7ae17:::benchmark_data_smithi081_73391_object6383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.342599+0000", 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 39.401371050999998, 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11885158, 2024-09-21T18:56:38.797 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145382 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342599+0000", 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342599+0000", 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342601+0000", 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3290000000000001e-06 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342608+0000", 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6830000000000002e-06 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342620+0000", 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2581e-05 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342629+0000", 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2460000000000003e-06 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342695+0000", 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6538999999999994e-05 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342831+0000", 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000136327 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343070+0000", 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000238965 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349422+0000", 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063518719999999997 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349469+0000", 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6894999999999997e-05 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.350298+0000", 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082869400000000005 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461236+0000", 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.110937963 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461333+0000", 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7094000000000002e-05 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461450+0000", 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117392 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145389 7.e 7:771830be:::benchmark_data_smithi081_73391_object6390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:56:38.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.344503+0000", 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 39.399467047000002, 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.117194543, 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145389 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.344503+0000", 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.344503+0000", 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.344505+0000", 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9039999999999999e-06 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.344510+0000", 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0089999999999998e-06 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.344520+0000", 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0197000000000001e-05 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.344527+0000", 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2019999999999998e-06 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.346201+0000", 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001673782 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.346327+0000", 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012578599999999999 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.346452+0000", 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000125129 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349490+0000", 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030383760000000002 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.349896+0000", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040587499999999999 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.350385+0000", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00048860400000000001 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461538+0000", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11115381100000001 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461593+0000", 2024-09-21T18:56:38.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4287999999999999e-05 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461697+0000", 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010458 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145392 7.8 7:145220fb:::benchmark_data_smithi081_73391_object6393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.353028+0000", 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 39.390941503000001, 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.073134613000000001, 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.809 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145392 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353028+0000", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353028+0000", 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353030+0000", 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.395e-06 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353033+0000", 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6959999999999998e-06 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353040+0000", 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6680000000000004e-06 2024-09-21T18:56:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353044+0000", 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2699999999999998e-06 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353078+0000", 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3247000000000002e-05 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353141+0000", 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3078999999999997e-05 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:56:38.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.353196+0000", 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5661999999999997e-05 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.354741+0000", 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001545076 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355049+0000", 2024-09-21T18:56:38.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030728999999999999 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355240+0000", 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019114800000000001 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426113+0000", 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.070873253999999997 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426163+0000", 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9827999999999998e-05 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:145392 7.8 e117/79 ondisk, result = 0)", 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.355196+0000", 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 39.388773372999999, 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.070973132999999994, 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.816 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355196+0000", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355196+0000", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355197+0000", 2024-09-21T18:56:38.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8000000000000005e-07 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355201+0000", 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.427e-06 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355207+0000", 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0889999999999996e-06 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355210+0000", 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7629999999999998e-06 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355233+0000", 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2852000000000001e-05 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.355238+0000", 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1749999999999998e-06 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426170+0000", 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.070932046999999998 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:147823 7.e 7:7ee3be05:::benchmark_data_smithi081_73391_object8824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.046516+0000", 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 36.697453834000001, 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.098133057999999995, 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 147823 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.046516+0000", 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.046516+0000", 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.046518+0000", 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6589999999999999e-06 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.046523+0000", 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8919999999999999e-06 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.046531+0000", 2024-09-21T18:56:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2269999999999993e-06 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.046537+0000", 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2720000000000003e-06 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.047033+0000", 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00049582900000000004 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.047138+0000", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104929 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.047239+0000", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010156 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.050925+0000", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036853060000000002 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.054500+0000", 2024-09-21T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035754760000000002 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.060717+0000", 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0062167630000000002 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.144319+0000", 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.083601987000000003 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.144539+0000", 2024-09-21T18:56:38.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022001400000000001 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.144649+0000", 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011014399999999999 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:147831 7.e 7:75950a91:::benchmark_data_smithi081_73391_object8832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:02.056856+0000", 2024-09-21T18:56:38.826 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 36.687113781999997, 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.088207245000000004, 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 147831 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056856+0000", 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056856+0000", 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056858+0000", 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056866+0000", 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.52e-06 2024-09-21T18:56:38.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056886+0000", 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9534999999999998e-05 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056895+0000", 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1060000000000006e-06 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.056977+0000", 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2307999999999999e-05 2024-09-21T18:56:38.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.057122+0000", 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000144678 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.057260+0000", 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138629 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.060940+0000", 2024-09-21T18:56:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003679795 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.061008+0000", 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8027000000000004e-05 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.061975+0000", 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00096679700000000003 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.144759+0000", 2024-09-21T18:56:38.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.082784464000000002 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.144959+0000", 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000199614 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:02.145063+0000", 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104318 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.832 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149070 7.e 7:7987acde:::benchmark_data_smithi081_73391_object10071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.593882+0000", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.150088269999998, 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10474167, 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149070 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593882+0000", 2024-09-21T18:56:38.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593882+0000", 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593884+0000", 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9920000000000002e-06 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593888+0000", 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9810000000000003e-06 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593901+0000", 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2218999999999999e-05 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593908+0000", 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.418e-06 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596642+0000", 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002733849 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596759+0000", 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116907 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596872+0000", 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112704 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606300+0000", 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0094281320000000005 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606619+0000", 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000319103 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606685+0000", 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6451999999999993e-05 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.698444+0000", 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.091759032000000004 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.698514+0000", 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9308999999999997e-05 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.698623+0000", 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109572 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149072 7.e 7:7e6a9180:::benchmark_data_smithi081_73391_object10073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.594183+0000", 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.149787256000003, 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.112788716, 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149072 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.839 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594183+0000", 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594183+0000", 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594184+0000", 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0100000000000001e-06 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594188+0000", 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0579999999999999e-06 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594197+0000", 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4229999999999994e-06 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594204+0000", 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0180000000000004e-06 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.599868+0000", 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0056638529999999999 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.599997+0000", 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012900300000000001 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.600822+0000", 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082521600000000004 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606588+0000", 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057662870000000001 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606639+0000", 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0371999999999998e-05 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606688+0000", 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9302999999999999e-05 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.704725+0000", 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.098036513000000006 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.704835+0000", 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110591 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.706971+0000", 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002136069 2024-09-21T18:56:38.844 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149074 7.e 7:76573c5e:::benchmark_data_smithi081_73391_object10075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.597401+0000", 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.146569233999998, 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.115087279, 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.845 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149074 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597401+0000", 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597401+0000", 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597402+0000", 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.689e-06 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597408+0000", 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4269999999999999e-06 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597416+0000", 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3780000000000001e-06 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597423+0000", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8279999999999997e-06 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.603403+0000", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059804869999999996 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.603514+0000", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111026 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.603591+0000", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6638999999999996e-05 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606658+0000", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030671729999999999 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606679+0000", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0398e-05 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.606691+0000", 2024-09-21T18:56:38.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2133000000000001e-05 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712273+0000", 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10558176399999999 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712371+0000", 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8194000000000002e-05 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712488+0000", 2024-09-21T18:56:38.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117143 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:166585 7.e 7:70738f38:::benchmark_data_smithi081_73391_object27586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:22.088599+0000", 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.655370797, 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.078505858999999997, 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.851 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 166585 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088599+0000", 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088599+0000", 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088601+0000", 2024-09-21T18:56:38.853 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: pgmap v146: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s; 39256/294999 objects misplaced (13.307%); 16 MiB/s, 252 objects/s recovering 2024-09-21T18:56:38.853 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:38.853 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2614988000' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: from='client.34961 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/870325963' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[39192]: from='client.24952 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: pgmap v146: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s; 39256/294999 objects misplaced (13.307%); 16 MiB/s, 252 objects/s recovering 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T18:56:38.854 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2614988000' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:38.855 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: from='client.34961 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:56:38.855 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/870325963' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T18:56:38.855 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:38 smithi081 ceph-mon[29384]: from='client.24952 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7349999999999999e-06 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088606+0000", 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3449999999999996e-06 2024-09-21T18:56:38.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088616+0000", 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0386e-05 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088624+0000", 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1189999999999999e-06 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088682+0000", 2024-09-21T18:56:38.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8653999999999998e-05 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088799+0000", 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116624 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.088916+0000", 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011749200000000001 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.092921+0000", 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040047590000000001 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.093032+0000", 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011069300000000001 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.093080+0000", 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8282000000000003e-05 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.166894+0000", 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.073813884999999996 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.166994+0000", 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9687999999999998e-05 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.167105+0000", 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111197 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.859 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:166929 7.e 7:703cd514:::benchmark_data_smithi081_73391_object27930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:22.456650+0000", 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.287320111, 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.075017931999999996, 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 166929 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.860 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.456650+0000", 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.456650+0000", 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.456652+0000", 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.131e-06 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.456658+0000", 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.801e-06 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.456676+0000", 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7886000000000001e-05 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.456683+0000", 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8520000000000004e-06 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.460997+0000", 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043137760000000001 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.461111+0000", 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011394400000000001 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.461234+0000", 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012314699999999999 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.465530+0000", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042957860000000002 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.466314+0000", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00078383499999999998 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.472188+0000", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058740279999999999 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.531007+0000", 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.058819171000000003 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.531171+0000", 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000163566 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:22.531668+0000", 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00049700899999999997 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:168572 7.e 7:72ed473a:::benchmark_data_smithi081_73391_object29573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.128924+0000", 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.615046091, 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.070236338999999995, 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 168572 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.866 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128924+0000", 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128924+0000", 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128926+0000", 2024-09-21T18:56:38.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9300000000000002e-06 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128932+0000", 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1720000000000004e-06 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128944+0000", 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1747e-05 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128951+0000", 2024-09-21T18:56:38.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1080000000000004e-06 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.129006+0000", 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4934e-05 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.129123+0000", 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116964 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.129242+0000", 2024-09-21T18:56:38.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011976000000000001 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.132732+0000", 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034893350000000001 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.134852+0000", 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021204319999999998 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.141311+0000", 2024-09-21T18:56:38.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064589419999999996 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.198953+0000", 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.05764205 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.199016+0000", 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2534999999999997e-05 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.199160+0000", 2024-09-21T18:56:38.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014443 2024-09-21T18:56:38.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.872 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:56:38.872 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:56:38.872 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:56:38.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:38.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:38.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:39.031 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:39.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:39.167 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:39.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:39.309 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:39.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:39.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:39.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:39.587 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:39.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:39.732 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:39.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:39.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:56:39.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:39.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: from='client.24952 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T18:56:39.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T18:56:39.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:56:39.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[29384]: 7.0 scrub starts 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: from='client.24952 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:56:39.845 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:39 smithi081 ceph-mon[39192]: 7.0 scrub starts 2024-09-21T18:56:39.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:39.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:39.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T18:56:39.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:40.000 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: from='client.24952 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:56:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:39 smithi104 ceph-mon[32438]: 7.0 scrub starts 2024-09-21T18:56:40.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:40.232 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:40.261 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:40.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:40.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:40.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:40.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:40.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.683 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:40.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:40.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:40.815 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:40.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:40.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:40.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:40.973 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:40 smithi081 ceph-mon[39192]: pgmap v148: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 39256/294999 objects misplaced (13.307%); 20 MiB/s, 315 objects/s recovering 2024-09-21T18:56:40.973 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:40 smithi081 ceph-mon[39192]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T18:56:40.973 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:40 smithi081 ceph-mon[29384]: pgmap v148: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 39256/294999 objects misplaced (13.307%); 20 MiB/s, 315 objects/s recovering 2024-09-21T18:56:40.973 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:40 smithi081 ceph-mon[29384]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T18:56:41.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:40 smithi104 ceph-mon[32438]: pgmap v148: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 39256/294999 objects misplaced (13.307%); 20 MiB/s, 315 objects/s recovering 2024-09-21T18:56:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:40 smithi104 ceph-mon[32438]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T18:56:41.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T18:56:41.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:41.112 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:41.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.254 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:41.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:41.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:41.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:41.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:41.645 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:41.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:41.790 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:41 smithi081 ceph-mon[39192]: 7.0 scrub ok 2024-09-21T18:56:41.791 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:41 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T18:56:41.791 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:41 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T18:56:41.791 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:41 smithi081 ceph-mon[29384]: 7.0 scrub ok 2024-09-21T18:56:41.791 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:41 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T18:56:41.791 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:41 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T18:56:41.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:41.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:56:41.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:56:41.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:56:41.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:56:41.837 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:56:41.837 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:56:41.837 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:56:41.837 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:56:41.865 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:56:41.865 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:56:41.898 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:56:41.898 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:56:41.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:41.957 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:41.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:41.960 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:56:41.960 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:56:41.993 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:56:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:56:42.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:41 smithi104 ceph-mon[32438]: 7.0 scrub ok 2024-09-21T18:56:42.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:41 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T18:56:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:41 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T18:56:42.058 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:42.085 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:42.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:42.211 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:42.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:42.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:42.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:42.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:42.631 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:42.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:42.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:42 smithi104 ceph-mon[32438]: pgmap v150: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 567 op/s; 39256/294999 objects misplaced (13.307%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T18:56:42.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4136916273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:42.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:42 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T18:56:42.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:42 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T18:56:42.770 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:42.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:42.915 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:42.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:43.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:43.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[39192]: pgmap v150: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 567 op/s; 39256/294999 objects misplaced (13.307%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4136916273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[29384]: pgmap v150: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 567 op/s; 39256/294999 objects misplaced (13.307%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4136916273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T18:56:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:42 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T18:56:43.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:43.191 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:43.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:43.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:43.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:43.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:43.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:43.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:43.900 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:43.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:43.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:56:44.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:44.020 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:43 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:44.020 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:43 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:44.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:43 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:44.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:43 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:44.172 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.199 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:44.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:44.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:43 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:43 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:44.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:44.325 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:44.367 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:44.381 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:44.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.486 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:44.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:44.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.639 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:44.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:44.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:44.768 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:44.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:44.910 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:44.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:45.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:45.036 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:45.138 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:45.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:45.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[39192]: pgmap v151: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 36215/281787 objects misplaced (12.852%); 20 MiB/s, 326 objects/s recovering 2024-09-21T18:56:45.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:56:45.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:56:45.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[39192]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T18:56:45.176 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[29384]: pgmap v151: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 36215/281787 objects misplaced (12.852%); 20 MiB/s, 326 objects/s recovering 2024-09-21T18:56:45.176 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:56:45.176 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:56:45.176 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:44 smithi081 ceph-mon[29384]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T18:56:45.234 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:56:45.234 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:56:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:44 smithi104 ceph-mon[32438]: pgmap v151: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 36215/281787 objects misplaced (12.852%); 20 MiB/s, 326 objects/s recovering 2024-09-21T18:56:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:44 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:56:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:44 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:56:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:44 smithi104 ceph-mon[32438]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T18:56:45.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.306 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:45.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:45.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:45.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:45.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:45.580 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:45.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:45.707 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:45.775 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:45.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:45.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:56:45.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:45.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:45.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:45.959 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:45.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:45.978 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T18:56:46.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:46.089 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:46.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:46.219 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:45 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/885383591' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T18:56:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:45 smithi104 ceph-mon[32438]: from='client.34979 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T18:56:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:45 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/885383591' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T18:56:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:45 smithi081 ceph-mon[29384]: from='client.34979 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T18:56:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:45 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/885383591' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T18:56:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:45 smithi081 ceph-mon[39192]: from='client.34979 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T18:56:46.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:46.360 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:46.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:46.500 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:46.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:46.622 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:46.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:46.754 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:46.857 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:46.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:46.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:46.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:47.023 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:47.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:47.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:46 smithi104 ceph-mon[32438]: pgmap v153: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 641 op/s; 36215/281787 objects misplaced (12.852%); 13 MiB/s, 200 objects/s recovering 2024-09-21T18:56:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:46 smithi104 ceph-mon[32438]: from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T18:56:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:46 smithi104 ceph-mon[32438]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T18:56:47.278 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:46 smithi081 ceph-mon[29384]: pgmap v153: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 641 op/s; 36215/281787 objects misplaced (12.852%); 13 MiB/s, 200 objects/s recovering 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:46 smithi081 ceph-mon[29384]: from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:46 smithi081 ceph-mon[29384]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:46 smithi081 ceph-mon[39192]: pgmap v153: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 641 op/s; 36215/281787 objects misplaced (12.852%); 13 MiB/s, 200 objects/s recovering 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:46 smithi081 ceph-mon[39192]: from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:46 smithi081 ceph-mon[39192]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T18:56:47.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:56:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:56:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:56:47.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:47.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:47.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:47.530 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:47.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:47.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:47.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:47.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:47.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:47.923 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:47.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:47.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:56:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:56:48.024 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:48.052 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:48.153 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:48.174 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:48.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:47 smithi104 ceph-mon[32438]: pgmap v155: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s; 33174/268032 objects misplaced (12.377%); 21 MiB/s, 339 objects/s recovering 2024-09-21T18:56:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:47 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:48.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:48.310 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:48.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:47 smithi081 ceph-mon[39192]: pgmap v155: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s; 33174/268032 objects misplaced (12.377%); 21 MiB/s, 339 objects/s recovering 2024-09-21T18:56:48.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:47 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:48.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:47 smithi081 ceph-mon[29384]: pgmap v155: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s; 33174/268032 objects misplaced (12.377%); 21 MiB/s, 339 objects/s recovering 2024-09-21T18:56:48.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:47 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:48.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:48.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:48.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:48.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:48.723 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:48.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:48.860 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:48.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:48.961 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:49.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:49.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:49.139 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:49.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:49.261 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:49.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:49.401 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:49.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:49.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:49.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:49.657 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:49.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:49.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:49.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:49.935 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:49.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:50.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:50.081 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:50.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.208 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:50.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:50.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:50.350 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:50.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.475 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:50.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:50.576 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:50.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:50.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.750 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:50.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:50.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:50 smithi104 ceph-mon[32438]: pgmap v156: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 33174/268032 objects misplaced (12.377%); 21 MiB/s, 332 objects/s recovering 2024-09-21T18:56:50.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:50 smithi081 ceph-mon[39192]: pgmap v156: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 33174/268032 objects misplaced (12.377%); 21 MiB/s, 332 objects/s recovering 2024-09-21T18:56:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:50 smithi081 ceph-mon[29384]: pgmap v156: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 33174/268032 objects misplaced (12.377%); 21 MiB/s, 332 objects/s recovering 2024-09-21T18:56:50.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:50.885 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:50.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:50.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:51.012 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:51.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:51.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:51.392 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:51.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:51.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:51.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:51.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T18:56:51.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:51.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:51.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:51.804 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:51.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:51.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:52.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.122 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:52.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:52.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:52.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:52 smithi104 ceph-mon[32438]: pgmap v157: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 573 op/s; 33174/268032 objects misplaced (12.377%); 10 MiB/s, 160 objects/s recovering 2024-09-21T18:56:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:52.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:52.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:52.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:52.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:52 smithi081 ceph-mon[39192]: pgmap v157: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 573 op/s; 33174/268032 objects misplaced (12.377%); 10 MiB/s, 160 objects/s recovering 2024-09-21T18:56:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:56:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:56:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:52 smithi081 ceph-mon[29384]: pgmap v157: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 573 op/s; 33174/268032 objects misplaced (12.377%); 10 MiB/s, 160 objects/s recovering 2024-09-21T18:56:52.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:52.404 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:52.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:52.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:52.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:52.661 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:52.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:52.790 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:52.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:52.929 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:52.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:52.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:53.020 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:56:53.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:53.034 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:53.037 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T18:56:53.055 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:53.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:53.297 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:53.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:53.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/585524356' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:53.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:53 smithi081 ceph-mon[39192]: from='client.24964 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/585524356' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:53 smithi081 ceph-mon[29384]: from='client.24964 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:53.320 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:53.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/585524356' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:53.511 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:53 smithi104 ceph-mon[32438]: from='client.24964 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:56:53.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:53.581 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:53.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:53.704 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:53.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:53.828 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:53.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:53.957 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:53.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:54.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:54.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[39192]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[39192]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[39192]: pgmap v159: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 30602/256902 objects misplaced (11.912%); 19 MiB/s, 308 objects/s recovering 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[29384]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[29384]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T18:56:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:54 smithi081 ceph-mon[29384]: pgmap v159: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 30602/256902 objects misplaced (11.912%); 19 MiB/s, 308 objects/s recovering 2024-09-21T18:56:54.100 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:54.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:54.229 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:54.331 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:54.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:54.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:54.502 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:54 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T18:56:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:54 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T18:56:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:54 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:56:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:54 smithi104 ceph-mon[32438]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:56:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:54 smithi104 ceph-mon[32438]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T18:56:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:54 smithi104 ceph-mon[32438]: pgmap v159: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 30602/256902 objects misplaced (11.912%); 19 MiB/s, 308 objects/s recovering 2024-09-21T18:56:54.603 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:54.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:54.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:54.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:54.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:54.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:55.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.036 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:55.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:55.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:55.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:55.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:55 smithi081 ceph-mon[29384]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T18:56:55.281 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:55 smithi081 ceph-mon[39192]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T18:56:55.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:55.313 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:55.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:55.468 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:55.490 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:56:55.490 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:56:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:55 smithi104 ceph-mon[32438]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T18:56:55.536 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:56:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:56:55.570 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:55.578 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:56:55.578 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:56:55.593 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:55.599 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:56:55.599 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:56:55.624 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:56:55.624 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:56:55.656 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:56:55.657 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:56:55.692 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:56:55.692 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:56:55.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:55.727 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:56:55.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:56:55.731 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:55.759 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:56:55.759 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:56:55.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:55.874 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:55.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:55.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.002 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:56.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:56.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:56.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:56.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:56:56.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/409688683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:56.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:56 smithi081 ceph-mon[29384]: pgmap v161: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 464 op/s; 30602/256902 objects misplaced (11.912%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T18:56:56.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/409688683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:56.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:56 smithi081 ceph-mon[39192]: pgmap v161: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 464 op/s; 30602/256902 objects misplaced (11.912%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T18:56:56.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:56.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:56.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:56.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/409688683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:56:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:56 smithi104 ceph-mon[32438]: pgmap v161: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 140 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 464 op/s; 30602/256902 objects misplaced (11.912%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T18:56:56.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:56.567 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:56.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:56.717 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:56.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:56.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:56.873 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:56.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:56.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:56.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:57.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:57.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T18:56:57.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:57.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.170 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:57.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:57.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:57.315 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:56:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:56:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:56:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.321 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:57.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:57.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:57.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:57.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:57.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:57.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:57.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:57.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:57.869 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:57.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:57.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:58.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:56:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:56:58.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:56:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.136 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:58.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:56:58.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:58.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:58.355 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:58.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:56:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:56:58.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:56:58.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:56:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:56:58.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:58.684 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:56:58.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:58 smithi104 ceph-mon[32438]: pgmap v162: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 27221/245715 objects misplaced (11.078%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:56:58.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:58.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:58.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:58 smithi081 ceph-mon[39192]: pgmap v162: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 27221/245715 objects misplaced (11.078%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:56:58.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:58 smithi081 ceph-mon[29384]: pgmap v162: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 27221/245715 objects misplaced (11.078%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:56:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:56:58.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:58.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:59.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:56:59.066 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:56:59.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:59.205 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:59.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.352 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:59.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:59.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:56:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.494 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:59.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:56:59.505 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T18:56:59.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:56:59.638 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:56:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:56:59.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:56:59.767 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:56:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/44478013' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:56:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:56:59 smithi081 ceph-mon[39192]: from='client.34994 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:56:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/44478013' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:56:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:56:59 smithi081 ceph-mon[29384]: from='client.34994 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:56:59.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:56:59.893 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:56:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:56:59.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:56:59.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:00.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/44478013' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:57:00.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:56:59 smithi104 ceph-mon[32438]: from='client.34994 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T18:57:00.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:00.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:00.200 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:00.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:00.346 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:00.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:00.473 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:00.518 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:57:00.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:00.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:00.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[29384]: pgmap v163: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 27221/245715 objects misplaced (11.078%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[29384]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[29384]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/685270019' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[29384]: from='client.35000 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[39192]: pgmap v163: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 27221/245715 objects misplaced (11.078%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[39192]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[39192]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/685270019' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:00.645 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:00 smithi081 ceph-mon[39192]: from='client.35000 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:00.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:00.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:00.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:00.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:00 smithi104 ceph-mon[32438]: pgmap v163: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 930 op/s; 27221/245715 objects misplaced (11.078%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:57:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:00 smithi104 ceph-mon[32438]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T18:57:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:00 smithi104 ceph-mon[32438]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T18:57:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/685270019' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:00 smithi104 ceph-mon[32438]: from='client.35000 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:00.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:01.024 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:01.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:01.158 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:01.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:01.290 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:01.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:01.433 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:01.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:01.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:01.635 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:01 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:01.635 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:01 smithi081 ceph-mon[39192]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:01.636 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:01 smithi081 ceph-mon[39192]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T18:57:01.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:01 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:01.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:01 smithi081 ceph-mon[29384]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:01.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:01 smithi081 ceph-mon[29384]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T18:57:01.636 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:57:01.636 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114524 7.f 7:f2ee5f2d:::benchmark_data_smithi081_73391_object114523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.711631+0000", 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.919754476999998, 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.110562246, 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114524 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711631+0000", 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711631+0000", 2024-09-21T18:57:01.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711634+0000", 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.475e-06 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711655+0000", 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1458999999999999e-05 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711666+0000", 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0806e-05 2024-09-21T18:57:01.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711673+0000", 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8789999999999997e-06 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711725+0000", 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2244999999999999e-05 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.711846+0000", 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012117600000000001 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712898+0000", 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00105174 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.716077+0000", 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031790619999999999 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.718871+0000", 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027943099999999999 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822098+0000", 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10322682 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822149+0000", 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1416999999999999e-05 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822193+0000", 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3856999999999997e-05 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114526 7.3 7:ceea1fa3:::benchmark_data_smithi081_73391_object114525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.712427+0000", 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.918958146999998, 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.112353493, 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.643 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114526 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712427+0000", 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712427+0000", 2024-09-21T18:57:01.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712429+0000", 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.096e-06 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712452+0000", 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3371e-05 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712484+0000", 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1866999999999998e-05 2024-09-21T18:57:01.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712491+0000", 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5120000000000002e-06 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712615+0000", 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012345300000000001 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.713325+0000", 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00071033000000000001 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.713476+0000", 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015074000000000001 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822133+0000", 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.108657399 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822260+0000", 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012636900000000001 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824687+0000", 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024277560000000001 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824739+0000", 2024-09-21T18:57:01.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1659999999999997e-05 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824781+0000", 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1940000000000002e-05 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114527 7.3 7:c96a6902:::benchmark_data_smithi081_73391_object114526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.712513+0000", 2024-09-21T18:57:01.649 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.918872132999994, 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.112339647, 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114527 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712513+0000", 2024-09-21T18:57:01.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712513+0000", 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712514+0000", 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0049999999999999e-06 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712538+0000", 2024-09-21T18:57:01.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3325000000000001e-05 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712545+0000", 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2849999999999997e-06 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.712551+0000", 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.789e-06 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715784+0000", 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003232904 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715898+0000", 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011421600000000001 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.716047+0000", 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014889800000000001 2024-09-21T18:57:01.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822184+0000", 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106136939 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822294+0000", 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109738 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824790+0000", 2024-09-21T18:57:01.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024968999999999998 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824825+0000", 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4564999999999999e-05 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824853+0000", 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8082999999999999e-05 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114530 7.3 7:c8e61aea:::benchmark_data_smithi081_73391_object114529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.715519+0000", 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.915866726999994, 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10940768100000001, 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114530 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.656 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715519+0000", 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715519+0000", 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715521+0000", 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1859999999999999e-06 2024-09-21T18:57:01.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715546+0000", 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4944e-05 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715557+0000", 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0604999999999999e-05 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.715566+0000", 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9819999999999997e-06 2024-09-21T18:57:01.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.717721+0000", 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021557759999999999 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.717838+0000", 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011623100000000001 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.717978+0000", 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014039400000000001 2024-09-21T18:57:01.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822222+0000", 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.104244529 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822327+0000", 2024-09-21T18:57:01.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104114 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824859+0000", 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025323580000000002 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824899+0000", 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0311000000000001e-05 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.824927+0000", 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7251e-05 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114534 7.2 7:4941a9a0:::benchmark_data_smithi081_73391_object114533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.719536+0000", 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.911849541999999, 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11323662299999999, 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.662 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114534 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719536+0000", 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.663 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719536+0000", 2024-09-21T18:57:01.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719538+0000", 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8139999999999999e-06 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719560+0000", 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1974999999999999e-05 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719569+0000", 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2769999999999994e-06 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719576+0000", 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5880000000000003e-06 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719629+0000", 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3726e-05 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.821775+0000", 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10214522700000001 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.821963+0000", 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018830999999999999 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.827952+0000", 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005988663 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.827963+0000", 2024-09-21T18:57:01.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1059e-05 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.832677+0000", 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047140059999999998 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.832730+0000", 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3473000000000001e-05 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.832773+0000", 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2505e-05 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:114519 7.6 e108/82 7:6b5e3486:::benchmark_data_smithi081_73391_object114518:head v 108'7134, mlcod=108'7128)", 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.719791+0000", 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.911594086999997, 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.115333081, 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719791+0000", 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719791+0000", 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719793+0000", 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1459999999999999e-06 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719823+0000", 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0361e-05 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.821763+0000", 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.101940091 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.821797+0000", 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3668999999999999e-05 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822326+0000", 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00052908499999999999 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822345+0000", 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8800999999999998e-05 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.835049+0000", 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012704595000000001 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.835125+0000", 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5333000000000005e-05 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:114535 7.f 7:f39e74d1:::benchmark_data_smithi081_73391_object114534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.719885+0000", 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.911500887000003, 2024-09-21T18:57:01.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10977445500000001, 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 114535 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719885+0000", 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719885+0000", 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719886+0000", 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.254e-06 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.719909+0000", 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2578999999999998e-05 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.821791+0000", 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10188272 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.821800+0000", 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0920000000000004e-06 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822222+0000", 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00042121200000000002 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822339+0000", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117545 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822468+0000", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012937300000000001 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.825012+0000", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025437260000000001 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.825714+0000", 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000701467 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.829567+0000", 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038530460000000002 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.829615+0000", 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8492999999999998e-05 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.829659+0000", 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3948e-05 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:114533 7.7 e108/82 7:ea70de9d:::benchmark_data_smithi081_73391_object114532:head v 108'7185, mlcod=108'7183)", 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:30.720224+0000", 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 90.911161316999994, 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11153162899999999, 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.679 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.720224+0000", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.720224+0000", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.720225+0000", 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.097e-06 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.720305+0000", 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9276000000000005e-05 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822558+0000", 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.102253618 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822565+0000", 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5919999999999997e-06 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822635+0000", 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0628000000000002e-05 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.822656+0000", 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0951e-05 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.831716+0000", 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0090599860000000008 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:30.831756+0000", 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9480999999999997e-05 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145372 7.f 7:f3db53f7:::benchmark_data_smithi081_73391_object6373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.333907+0000", 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.297478955999999, 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.132201821, 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.683 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145372 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333907+0000", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333907+0000", 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333908+0000", 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1289999999999999e-06 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333912+0000", 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7720000000000001e-06 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333918+0000", 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6059999999999999e-06 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333923+0000", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1220000000000003e-06 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431117+0000", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097193494000000005 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431239+0000", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012223400000000001 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:57:01.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431352+0000", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011341900000000001 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462090+0000", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030737417 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462135+0000", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5228999999999997e-05 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462174+0000", 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9218000000000003e-05 2024-09-21T18:57:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.465927+0000", 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037523449999999998 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.465992+0000", 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5376999999999994e-05 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.466108+0000", 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116459 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:145381 7.0 e117/79 7:07a4e617:::benchmark_data_smithi081_73391_object6382:head v 117'9241, mlcod=117'9239)", 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.342889+0000", 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.288496803000001, 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11207307599999999, 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.689 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342889+0000", 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342889+0000", 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342890+0000", 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6400000000000001e-07 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342895+0000", 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.22e-06 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342902+0000", 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.905e-06 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.342907+0000", 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1970000000000004e-06 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426517+0000", 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.083610209000000005 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426535+0000", 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8054999999999999e-05 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.454857+0000", 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.028321428999999999 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.454962+0000", 2024-09-21T18:57:01.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010529700000000001 2024-09-21T18:57:01.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.694 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145383 7.a 7:57f58b8d:::benchmark_data_smithi081_73391_object6384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.343222+0000", 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.288163253999997, 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.118700001, 2024-09-21T18:57:01.695 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145383 2024-09-21T18:57:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343222+0000", 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343222+0000", 2024-09-21T18:57:01.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343224+0000", 2024-09-21T18:57:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.742e-06 2024-09-21T18:57:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343229+0000", 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343239+0000", 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0029e-05 2024-09-21T18:57:01.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343246+0000", 2024-09-21T18:57:01.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3490000000000003e-06 2024-09-21T18:57:01.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.425961+0000", 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.082714855000000004 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426093+0000", 2024-09-21T18:57:01.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013234100000000001 2024-09-21T18:57:01.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:57:01.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426332+0000", 2024-09-21T18:57:01.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023874000000000001 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.428997+0000", 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002664783 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.432653+0000", 2024-09-21T18:57:01.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003656201 2024-09-21T18:57:01.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461845+0000", 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.029191814 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461885+0000", 2024-09-21T18:57:01.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0484000000000001e-05 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461922+0000", 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.684e-05 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.707 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145386 7.f 7:f7192f4f:::benchmark_data_smithi081_73391_object6387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.343264+0000", 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.288121740999998, 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12299823, 2024-09-21T18:57:01.708 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145386 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343264+0000", 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343264+0000", 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343265+0000", 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1999999999999999e-07 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343268+0000", 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1089999999999998e-06 2024-09-21T18:57:01.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343274+0000", 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.981e-06 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343279+0000", 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2870000000000004e-06 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.458772+0000", 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11549300999999999 2024-09-21T18:57:01.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.458894+0000", 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012238599999999999 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.459022+0000", 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012768100000000001 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462223+0000", 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032008380000000001 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462259+0000", 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6282000000000003e-05 2024-09-21T18:57:01.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462316+0000", 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7027999999999997e-05 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.466117+0000", 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003801221 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.466161+0000", 2024-09-21T18:57:01.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3640999999999998e-05 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.466262+0000", 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010104599999999999 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:145382 7.e e117/114 7:7cb7ae17:::benchmark_data_smithi081_73391_object6383:head v 117'8995, mlcod=117'8993)", 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.343294+0000", 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 62.288091522000002, 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.117731797, 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.716 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343294+0000", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343294+0000", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343295+0000", 2024-09-21T18:57:01.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0800000000000004e-07 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343388+0000", 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.31e-05 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343396+0000", 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5599999999999994e-06 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.343402+0000", 2024-09-21T18:57:01.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.648e-06 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426230+0000", 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.082827868999999998 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.426253+0000", 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3081000000000001e-05 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.460994+0000", 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.034741373999999998 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.461026+0000", 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1557000000000003e-05 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149056 7.a 7:50db3241:::benchmark_data_smithi081_73391_object10057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.580766+0000", 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.050620068999997, 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.113148713, 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.720 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149056 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.580766+0000", 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.580766+0000", 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.580767+0000", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9700000000000002e-06 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.580773+0000", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3159999999999996e-06 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.580783+0000", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0145e-05 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.580790+0000", 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5290000000000003e-06 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.592164+0000", 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011373322999999999 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.592320+0000", 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015669300000000001 2024-09-21T18:57:01.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.594372+0000", 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020516430000000001 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597097+0000", 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002724396 2024-09-21T18:57:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597298+0000", 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020098199999999999 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.693823+0000", 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096525330000000006 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.693872+0000", 2024-09-21T18:57:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9434000000000001e-05 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.693914+0000", 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1952000000000001e-05 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149057 7.2 7:45801893:::benchmark_data_smithi081_73391_object10058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.581500+0000", 2024-09-21T18:57:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.049885539999998, 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11109055800000001, 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149057 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.581500+0000", 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.581500+0000", 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.581501+0000", 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.093e-06 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.581506+0000", 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6689999999999997e-06 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.581516+0000", 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0315e-05 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.581523+0000", 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5019999999999998e-06 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.591753+0000", 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010230079 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.591859+0000", 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010671300000000001 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.592426+0000", 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00056682800000000001 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597202+0000", 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.004775736 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597293+0000", 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0576999999999997e-05 2024-09-21T18:57:01.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.692502+0000", 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.095209926 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.692548+0000", 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5500999999999997e-05 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.692591+0000", 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2619e-05 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.732 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149059 7.a 7:5ba07a57:::benchmark_data_smithi081_73391_object10060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.583214+0000", 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.048171619999998, 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12822613299999999, 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149059 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.583214+0000", 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.583214+0000", 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.583216+0000", 2024-09-21T18:57:01.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.652e-06 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.583221+0000", 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8980000000000004e-06 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.583273+0000", 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2191000000000002e-05 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.583280+0000", 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2089999999999999e-06 2024-09-21T18:57:01.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.690433+0000", 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.107152946 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.690578+0000", 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014538600000000001 2024-09-21T18:57:01.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.690700+0000", 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121555 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.701820+0000", 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011119736 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.711205+0000", 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093854720000000006 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.711370+0000", 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000164697 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.711408+0000", 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8231e-05 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.711440+0000", 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2159999999999997e-05 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149069 7.2 7:4e1d598e:::benchmark_data_smithi081_73391_object10070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.593667+0000", 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.037718081999998, 2024-09-21T18:57:01.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.118423156, 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149069 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593667+0000", 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593667+0000", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593669+0000", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.77e-06 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593674+0000", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7999999999999998e-06 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593684+0000", 2024-09-21T18:57:01.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6250000000000002e-06 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.593690+0000", 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4489999999999996e-06 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.597323+0000", 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036326599999999998 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.686989+0000", 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.089665759999999997 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.687120+0000", 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000131039 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.693775+0000", 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066556109999999996 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.711197+0000", 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017421972000000001 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712007+0000", 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00080989199999999999 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712051+0000", 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4286999999999998e-05 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712091+0000", 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9291e-05 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:149073 7.a 7:5f541013:::benchmark_data_smithi081_73391_object10074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:03.596920+0000", 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 58.034465982999997, 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.115292018, 2024-09-21T18:57:01.745 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 149073 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596920+0000", 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596920+0000", 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596921+0000", 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7129999999999999e-06 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596927+0000", 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6640000000000003e-06 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596936+0000", 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.596943+0000", 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0530000000000001e-06 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.693947+0000", 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097003621999999998 2024-09-21T18:57:01.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.694063+0000", 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115767 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.694194+0000", 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013116899999999999 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.702013+0000", 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078191860000000005 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.711210+0000", 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0091966659999999992 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712131+0000", 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092167599999999996 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712175+0000", 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3424000000000003e-05 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:03.712212+0000", 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6928000000000003e-05 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.751 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:168985 7.3 7:cc2d575f:::benchmark_data_smithi081_73391_object29986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.625487+0000", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 37.005898913999999, 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11587133500000001, 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 168985 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.625487+0000", 2024-09-21T18:57:01.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.625487+0000", 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.625488+0000", 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.167e-06 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.625492+0000", 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4479999999999996e-06 2024-09-21T18:57:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.625524+0000", 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1442000000000002e-05 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.625530+0000", 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9880000000000001e-06 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637008+0000", 2024-09-21T18:57:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01147813 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637115+0000", 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106665 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637231+0000", 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116134 2024-09-21T18:57:01.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.644611+0000", 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0073803530000000001 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.644709+0000", 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7595000000000003e-05 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739810+0000", 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.095101426000000003 2024-09-21T18:57:01.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.741303+0000", 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001493423 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.741358+0000", 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4564000000000001e-05 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.757 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169005 7.3 7:c935e317:::benchmark_data_smithi081_73391_object30006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637143+0000", 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 36.994242219999997, 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106118447, 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169005 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:01.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637143+0000", 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637143+0000", 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637145+0000", 2024-09-21T18:57:01.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8780000000000001e-06 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637150+0000", 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2429999999999996e-06 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637161+0000", 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0545e-05 2024-09-21T18:57:01.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637169+0000", 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7519999999999996e-06 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.641980+0000", 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048109340000000002 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:01.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.642096+0000", 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116086 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.642197+0000", 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010083 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.644749+0000", 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025521950000000002 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:01.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.741369+0000", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096620079999999997 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.743174+0000", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00180539 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.743225+0000", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0420999999999997e-05 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.743262+0000", 2024-09-21T18:57:01.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7092999999999999e-05 2024-09-21T18:57:01.764 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.764 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.764 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.764 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:01.764 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:01.764 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:57:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.765 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:01.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:01.766 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:01 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:01.766 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:01 smithi104 ceph-mon[32438]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:01.766 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:01 smithi104 ceph-mon[32438]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T18:57:01.866 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:01.900 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:01.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:02.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:02.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:02.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:02.276 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.303 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:02.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:02.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:02.442 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:02.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:02.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:02.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T18:57:02.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.741 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:02.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:02.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:02 smithi104 ceph-mon[32438]: pgmap v166: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 503 op/s; 27221/245715 objects misplaced (11.078%); 18 MiB/s, 282 objects/s recovering 2024-09-21T18:57:02.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:02 smithi081 ceph-mon[39192]: pgmap v166: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 503 op/s; 27221/245715 objects misplaced (11.078%); 18 MiB/s, 282 objects/s recovering 2024-09-21T18:57:02.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:02 smithi081 ceph-mon[29384]: pgmap v166: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 503 op/s; 27221/245715 objects misplaced (11.078%); 18 MiB/s, 282 objects/s recovering 2024-09-21T18:57:02.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:02.866 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:02.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:02.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:02.998 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:03.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:03.122 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:03.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:03.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:03.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:03.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:57:03.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:03.507 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:03.609 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:03.632 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:03.734 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:03.766 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:03 smithi081 ceph-mon[39192]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T18:57:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:03 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:03 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:03 smithi081 ceph-mon[29384]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T18:57:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:03 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:03 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:03.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:03.892 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:03.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:03.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T18:57:03.933 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:03.934 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:03.958 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:03 smithi104 ceph-mon[32438]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T18:57:03.958 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:03 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:03.958 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:03 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:03.993 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:04.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.177 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:04.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:04.271 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:04.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:04.308 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:04.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:04.446 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:04.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:04.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:04.616 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:04.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:04 smithi104 ceph-mon[32438]: pgmap v168: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 764 op/s; 24631/231960 objects misplaced (10.619%); 11 MiB/s, 177 objects/s recovering 2024-09-21T18:57:04.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:04 smithi104 ceph-mon[32438]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T18:57:04.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:04 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:57:04.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:04 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:57:04.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:04 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:04.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:04.698 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:04.698 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:04.733 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:04.734 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:04.768 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:04.769 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:04.802 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:04.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:04.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[29384]: pgmap v168: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 764 op/s; 24631/231960 objects misplaced (10.619%); 11 MiB/s, 177 objects/s recovering 2024-09-21T18:57:04.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[29384]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T18:57:04.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:57:04.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:57:04.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[39192]: pgmap v168: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 764 op/s; 24631/231960 objects misplaced (10.619%); 11 MiB/s, 177 objects/s recovering 2024-09-21T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[39192]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:04 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:04.822 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:202916 7.4 7:27bb09e1:::benchmark_data_smithi081_73391_object63917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:04.805615+0000", 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.013665451, 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013693234, 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:04.823 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 202916 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805615+0000", 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805615+0000", 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805617+0000", 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9460000000000001e-06 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805622+0000", 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0470000000000001e-06 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805693+0000", 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1730000000000006e-05 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805702+0000", 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7900000000000005e-06 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805776+0000", 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3478000000000004e-05 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.805908+0000", 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013244099999999999 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,6", 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.806052+0000", 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014358 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.808701+0000", 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026491660000000001 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.809642+0000", 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094110299999999999 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.812217+0000", 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025753130000000001 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:202925 7.4 7:2f64e186:::benchmark_data_smithi081_73391_object63926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:04.815190+0000", 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0040905389999999998, 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041982410000000001, 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:04.828 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 202925 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815190+0000", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815190+0000", 2024-09-21T18:57:04.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815192+0000", 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5600000000000001e-06 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815199+0000", 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6229999999999999e-06 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815219+0000", 2024-09-21T18:57:04.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9811999999999999e-05 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815228+0000", 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4199999999999996e-06 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815293+0000", 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5074999999999996e-05 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815429+0000", 2024-09-21T18:57:04.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013578 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,6", 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.815580+0000", 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015123599999999999 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:04.819030+0000", 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003450336 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:04.832 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T18:57:04.833 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 2 2024-09-21T18:57:04.833 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:57:04.835 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:04.836 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:04.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:04.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:04.870 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:04.870 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:04.902 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:04.902 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:04.934 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:04.934 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:04.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:05.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:05.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:05.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:05.269 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:05.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:05.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.537 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:05.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:05.577 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T18:57:05.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:05.671 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:05.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2467042934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:05.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:05 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:57:05.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:05 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:57:05.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:05.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:05.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:05.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:05.957 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:06.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:06.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2467042934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:06.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:05 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:57:06.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:05 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:57:06.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2467042934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:06.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:05 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:57:06.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:05 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:57:06.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:06.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:06.197 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:06.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:06.325 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:06.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:06.485 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:06.586 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.623 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:06.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:06.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:06 smithi104 ceph-mon[32438]: pgmap v170: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 862 B/s rd, 0 B/s wr, 773 op/s; 24631/231960 objects misplaced (10.619%); 11 MiB/s, 179 objects/s recovering 2024-09-21T18:57:06.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:06 smithi104 ceph-mon[32438]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T18:57:06.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:06.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:06.725 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:06.745 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:06.800 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:06.800 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[29384]: pgmap v170: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 862 B/s rd, 0 B/s wr, 773 op/s; 24631/231960 objects misplaced (10.619%); 11 MiB/s, 179 objects/s recovering 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[29384]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[39192]: pgmap v170: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 141 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 862 B/s rd, 0 B/s wr, 773 op/s; 24631/231960 objects misplaced (10.619%); 11 MiB/s, 179 objects/s recovering 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[39192]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:06.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:06.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:06.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:06.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:06.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:07.011 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:07.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.136 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:07.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:07.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.268 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:07.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:07.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:07.400 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:07.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:07.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:07.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:57:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:57:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:57:07.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:07.662 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:07.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:07.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:07.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:57:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:57:07.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T18:57:07.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:07.943 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:07.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:08.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:08.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:08.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:08.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:08.231 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:08.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:08.346 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:08.447 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.471 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:08.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:08.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.602 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:08.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:08.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:08 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:57:08.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:08 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:57:08.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:08 smithi104 ceph-mon[32438]: pgmap v172: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s; 21475/220752 objects misplaced (9.728%); 32 MiB/s, 516 objects/s recovering 2024-09-21T18:57:08.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:08 smithi104 ceph-mon[32438]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T18:57:08.670 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:08.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:08.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:08.802 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:08.802 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:08.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:08.871 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:08.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:08.911 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:57:08.911 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:57:08.911 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[39192]: pgmap v172: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s; 21475/220752 objects misplaced (9.728%); 32 MiB/s, 516 objects/s recovering 2024-09-21T18:57:08.911 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[39192]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T18:57:08.911 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:08.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:57:08.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:57:08.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[29384]: pgmap v172: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s; 21475/220752 objects misplaced (9.728%); 32 MiB/s, 516 objects/s recovering 2024-09-21T18:57:08.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[29384]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T18:57:08.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:08.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:09.034 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:09.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:09.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.304 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:09.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:09.405 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:09.435 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:09.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:09.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T18:57:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.581 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:09.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:09.613 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T18:57:09.617 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:57:09.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:09.710 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:09.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:09.857 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:09.874 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/272761034' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:09.875 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:09 smithi081 ceph-mon[39192]: from='client.24988 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:09.875 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/551368703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T18:57:09.875 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/272761034' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:09.875 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:09 smithi081 ceph-mon[29384]: from='client.24988 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:09.875 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/551368703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T18:57:09.907 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:09.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:09.996 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:09.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:10.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/272761034' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:10.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:09 smithi104 ceph-mon[32438]: from='client.24988 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:10.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/551368703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T18:57:10.098 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:10.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:10.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:10.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:10.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:10.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:10.508 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:10.508 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:10.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:10.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:10.701 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:10 smithi104 ceph-mon[32438]: pgmap v174: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 623 op/s; 21475/220752 objects misplaced (9.728%); 21 MiB/s, 338 objects/s recovering 2024-09-21T18:57:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:10 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:10 smithi104 ceph-mon[32438]: from='client.24988 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:57:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:10 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/551368703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T18:57:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:10 smithi104 ceph-mon[32438]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T18:57:10.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[39192]: pgmap v174: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 623 op/s; 21475/220752 objects misplaced (9.728%); 21 MiB/s, 338 objects/s recovering 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[39192]: from='client.24988 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/551368703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[39192]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[29384]: pgmap v174: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 623 op/s; 21475/220752 objects misplaced (9.728%); 21 MiB/s, 338 objects/s recovering 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[29384]: from='client.24988 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/551368703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T18:57:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:10 smithi081 ceph-mon[29384]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T18:57:10.834 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:10.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:10.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:57:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:10.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:10.976 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T18:57:10.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:57:11.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:11.137 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:11.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:11.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:11.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T18:57:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:11.307 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:11.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:11.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:11.472 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:11.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:11.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:11.522 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:11.581 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:11.601 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:11.610 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:11.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:11.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:11.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:11.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:11.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:11.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:11.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:11.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:12.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:12.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:12.139 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:12.139 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:12.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:12.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:12.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:12.349 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:12.349 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:12.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:12.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:12.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:12.426 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:12.427 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:12.438 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:12.473 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:12.505 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:12.505 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:12.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:12.541 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:12.541 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:12.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:57:12.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:12.582 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:12.583 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:12.619 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:12.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:12.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:12 smithi081 ceph-mon[39192]: pgmap v176: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 623 op/s; 21475/220752 objects misplaced (9.728%); 21 MiB/s, 338 objects/s recovering 2024-09-21T18:57:12.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/374957888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:12 smithi081 ceph-mon[29384]: pgmap v176: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 623 op/s; 21475/220752 objects misplaced (9.728%); 21 MiB/s, 338 objects/s recovering 2024-09-21T18:57:12.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/374957888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:12.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:12.862 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:12.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:12.891 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:12 smithi104 ceph-mon[32438]: pgmap v176: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 623 op/s; 21475/220752 objects misplaced (9.728%); 21 MiB/s, 338 objects/s recovering 2024-09-21T18:57:12.891 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/374957888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:12.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T18:57:12.957 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:12.960 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:57:12.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:13.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:13.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:13.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:13.290 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:13.320 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:13.392 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:13.420 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:13.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:13.562 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:13.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:13.694 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:13.727 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:13 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T18:57:13.727 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:13 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T18:57:13.727 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:13 smithi104 ceph-mon[32438]: from='client.15156 v1:172.21.15.81:0/2593183346' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:13.727 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:13 smithi104 ceph-mon[32438]: 5.1a scrub starts 2024-09-21T18:57:13.727 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:13 smithi104 ceph-mon[32438]: 5.1a scrub ok 2024-09-21T18:57:13.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:13.817 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:13.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:13.957 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:13.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54661 7.0 7:07cacdd3:::benchmark_data_smithi081_73391_object54660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.264263+0000", 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 147.78464221600001, 2024-09-21T18:57:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069566144999999996, 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54661 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264263+0000", 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264263+0000", 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264265+0000", 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8989999999999999e-06 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264329+0000", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3916000000000003e-05 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264354+0000", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5173000000000001e-05 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264361+0000", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7630000000000003e-06 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264406+0000", 2024-09-21T18:57:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4493999999999998e-05 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264494+0000", 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8230000000000001e-05 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264583+0000", 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9458999999999998e-05 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.268277+0000", 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036939899999999999 2024-09-21T18:57:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.268288+0000", 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0331e-05 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333729+0000", 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.065441705000000003 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333784+0000", 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5055000000000001e-05 2024-09-21T18:57:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333830+0000", 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5130000000000003e-05 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54662 7.0 7:0fe82cc8:::benchmark_data_smithi081_73391_object54661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.264375+0000", 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 147.78453099800001, 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.069597117999999999, 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54662 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264375+0000", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264375+0000", 2024-09-21T18:57:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264376+0000", 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5799999999999998e-07 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264404+0000", 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8065999999999999e-05 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264409+0000", 2024-09-21T18:57:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.665e-06 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.264413+0000", 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0319999999999997e-06 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.266408+0000", 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019946170000000002 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.266516+0000", 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108414 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.266658+0000", 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141827 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.268354+0000", 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016960689999999999 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.270100+0000", 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017457460000000001 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333874+0000", 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.063773683999999997 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333931+0000", 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6997999999999999e-05 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.333972+0000", 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1041999999999998e-05 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99372 7.0 7:018c5cb9:::benchmark_data_smithi081_73391_object99371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 116.347163663, 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100414906, 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99372 2024-09-21T18:57:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701744+0000", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3889999999999999e-06 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701794+0000", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0031000000000003e-05 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701809+0000", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4639e-05 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701817+0000", 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0090000000000007e-06 2024-09-21T18:57:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701879+0000", 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1820000000000001e-05 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.702011+0000", 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013177800000000001 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.702160+0000", 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014952100000000001 2024-09-21T18:57:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.704287+0000", 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021267270000000001 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.704646+0000", 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035917799999999998 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802063+0000", 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097417227999999995 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802117+0000", 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3488e-05 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802157+0000", 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0098000000000001e-05 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99378 7.0 7:0717396a:::benchmark_data_smithi081_73391_object99377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:57:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 116.341151665, 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094525928999999995, 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99378 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707756+0000", 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1110000000000002e-06 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707812+0000", 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6208000000000001e-05 2024-09-21T18:57:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707824+0000", 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1838000000000001e-05 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707832+0000", 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5560000000000002e-06 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707876+0000", 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4354000000000003e-05 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707962+0000", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5986e-05 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.708079+0000", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117478 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710054+0000", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019749530000000002 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710919+0000", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00086474499999999999 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802199+0000", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.091279896999999999 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802247+0000", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7550999999999999e-05 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802280+0000", 2024-09-21T18:57:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3251999999999999e-05 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99386 7.0 7:06617387:::benchmark_data_smithi081_73391_object99385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 116.33506785900001, 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088596121999999999, 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99386 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:57:14.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:14.080 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T18:57:14.080 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T18:57:14.080 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[39192]: from='client.15156 v1:172.21.15.81:0/2593183346' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:14.080 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[39192]: 5.1a scrub starts 2024-09-21T18:57:14.080 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[39192]: 5.1a scrub ok 2024-09-21T18:57:14.081 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T18:57:14.081 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T18:57:14.081 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[29384]: from='client.15156 v1:172.21.15.81:0/2593183346' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:14.081 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[29384]: 5.1a scrub starts 2024-09-21T18:57:14.081 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:13 smithi081 ceph-mon[29384]: 5.1a scrub ok 2024-09-21T18:57:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713840+0000", 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7990000000000001e-06 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713885+0000", 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5139999999999998e-05 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713895+0000", 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0193999999999999e-05 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713901+0000", 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7049999999999996e-06 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713952+0000", 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1641999999999998e-05 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.714080+0000", 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012742999999999999 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.714184+0000", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010401899999999999 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.718889+0000", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047058029999999997 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.718919+0000", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9291000000000001e-05 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802318+0000", 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083398876999999996 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802398+0000", 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0391999999999999e-05 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802434+0000", 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5830000000000001e-05 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99389 7.0 7:02d2e699:::benchmark_data_smithi081_73391_object99388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 116.33375121900001, 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087440860999999995, 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99389 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715156+0000", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6980000000000001e-06 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715227+0000", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0919000000000003e-05 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715243+0000", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5559000000000001e-05 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715250+0000", 2024-09-21T18:57:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6340000000000001e-06 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716563+0000", 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013130399999999999 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716688+0000", 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012472500000000001 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716876+0000", 2024-09-21T18:57:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000187586 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.719345+0000", 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024699539999999999 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720446+0000", 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011003219999999999 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802480+0000", 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082033990000000001 2024-09-21T18:57:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802566+0000", 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6119999999999995e-05 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802595+0000", 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9314000000000001e-05 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114537 7.0 7:04929e7d:::benchmark_data_smithi081_73391_object114536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 103.32550768500001, 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105603195, 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114537 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723400+0000", 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.119e-06 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723433+0000", 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3302000000000001e-05 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723444+0000", 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0523000000000001e-05 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723452+0000", 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8199999999999997e-06 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727483+0000", 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040313839999999998 2024-09-21T18:57:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727571+0000", 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7770999999999997e-05 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727655+0000", 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4322999999999999e-05 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730033+0000", 2024-09-21T18:57:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002377948 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730045+0000", 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2091e-05 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828840+0000", 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098794333999999998 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828919+0000", 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9752000000000005e-05 2024-09-21T18:57:14.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.829001+0000", 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1828000000000003e-05 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114544 7.0 7:016dd1d4:::benchmark_data_smithi081_73391_object114543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 103.317983889, 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101948043, 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114544 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730924+0000", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1280000000000002e-06 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730973+0000", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8687000000000003e-05 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730987+0000", 2024-09-21T18:57:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.448e-05 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730996+0000", 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9719999999999998e-06 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731081+0000", 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5292999999999998e-05 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731162+0000", 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0603000000000001e-05 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731260+0000", 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8259999999999998e-05 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.733069+0000", 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018090840000000001 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.780503+0000", 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.047434142999999998 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832639+0000", 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052135212 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832711+0000", 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2327000000000001e-05 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832870+0000", 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015885399999999999 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:115640 7.0 7:07e50b4e:::benchmark_data_smithi081_73391_object115639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:31.711141+0000", 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 102.33776446500001, 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070270899999999997, 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 115640 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711141+0000", 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711141+0000", 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711144+0000", 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4980000000000001e-06 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711178+0000", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4239999999999997e-05 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711190+0000", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2078e-05 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.711198+0000", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8010000000000004e-06 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.718891+0000", 2024-09-21T18:57:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076928630000000003 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.718976+0000", 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5426000000000005e-05 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.719071+0000", 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4860999999999999e-05 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.721212+0000", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021414110000000002 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.722474+0000", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012613839999999999 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.781300+0000", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058826330000000003 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.781356+0000", 2024-09-21T18:57:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5823999999999999e-05 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:31.781412+0000", 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6184000000000003e-05 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:116129 7.0 7:08f433c2:::benchmark_data_smithi081_73391_object116128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:32.139455+0000", 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 101.909450959, 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076617837999999994, 2024-09-21T18:57:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 116129 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139455+0000", 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139455+0000", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139457+0000", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9130000000000001e-06 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139488+0000", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1940000000000003e-05 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139497+0000", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1000000000000004e-06 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.139503+0000", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.981e-06 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.195546+0000", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056043530000000001 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.195659+0000", 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011290600000000001 2024-09-21T18:57:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.195796+0000", 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000136747 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.214061+0000", 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018265019 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.214100+0000", 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8948e-05 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.215953+0000", 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001852921 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216012+0000", 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8866000000000002e-05 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:32.216072+0000", 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0967000000000001e-05 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:133499 7.0 7:0892d353:::benchmark_data_smithi081_73391_object133498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:46.020272+0000", 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 88.028633626000001, 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.068801545000000006, 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133499 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020272+0000", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020272+0000", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020274+0000", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2330000000000001e-06 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020329+0000", 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4901000000000002e-05 2024-09-21T18:57:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020345+0000", 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5418e-05 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020352+0000", 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9300000000000003e-06 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020416+0000", 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3367000000000005e-05 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020552+0000", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013659100000000001 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.020689+0000", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000136964 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.025944+0000", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052550069999999999 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.071704+0000", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.045759936000000001 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.088941+0000", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017236853999999999 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089032+0000", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1247000000000002e-05 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089074+0000", 2024-09-21T18:57:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1097000000000003e-05 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:133500 7.0 7:083f69a6:::benchmark_data_smithi081_73391_object133499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:46.021327+0000", 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 88.027578578000004, 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.067899609, 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 133500 2024-09-21T18:57:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.021327+0000", 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.021327+0000", 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.021328+0000", 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3710000000000001e-06 2024-09-21T18:57:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.021400+0000", 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1582999999999995e-05 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.021413+0000", 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2729e-05 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.021420+0000", 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2359999999999998e-06 2024-09-21T18:57:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.023259+0000", 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018391309999999999 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.023368+0000", 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108959 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.023504+0000", 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013601800000000001 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.026124+0000", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026201570000000001 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.071740+0000", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.045615523999999998 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089120+0000", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017380448 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089193+0000", 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.271e-05 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:46.089227+0000", 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3742999999999999e-05 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145381 7.0 7:07a4e617:::benchmark_data_smithi081_73391_object6382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.342438+0000", 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.706468067000003, 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11277905000000001, 2024-09-21T18:57:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145381 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342438+0000", 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342438+0000", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342439+0000", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7179999999999999e-06 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342444+0000", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342456+0000", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2302e-05 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342464+0000", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.447e-06 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342510+0000", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5977999999999999e-05 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342606+0000", 2024-09-21T18:57:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6277e-05 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342712+0000", 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106092 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.348117+0000", 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005405371 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.349145+0000", 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010274399999999999 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455138+0000", 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105993714 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455174+0000", 2024-09-21T18:57:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5976999999999998e-05 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455217+0000", 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2103e-05 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145387 7.0 7:0b525978:::benchmark_data_smithi081_73391_object6388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.343230+0000", 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 74.705675284999998, 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112080033, 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145387 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343230+0000", 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343230+0000", 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343232+0000", 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.421e-06 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343237+0000", 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3650000000000003e-06 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343252+0000", 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4705000000000001e-05 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343260+0000", 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3629999999999994e-06 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345358+0000", 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020982319999999998 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345479+0000", 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012078699999999999 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345612+0000", 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013247700000000001 2024-09-21T18:57:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.348281+0000", 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002669186 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.349239+0000", 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095836999999999995 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455251+0000", 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10601202899999999 2024-09-21T18:57:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455284+0000", 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3090000000000003e-05 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455310+0000", 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6007999999999999e-05 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:147845 7.0 7:0886cefa:::benchmark_data_smithi081_73391_object8846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:02.071897+0000", 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 71.977008669, 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083496556, 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 147845 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071897+0000", 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071897+0000", 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071899+0000", 2024-09-21T18:57:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.244e-06 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071906+0000", 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8669999999999996e-06 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071920+0000", 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4124999999999999e-05 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071929+0000", 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6260000000000001e-06 2024-09-21T18:57:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071990+0000", 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1334000000000006e-05 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.072102+0000", 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111965 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.072174+0000", 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1947999999999996e-05 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.074271+0000", 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002097444 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.076397+0000", 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021253969999999998 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.155291+0000", 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.078894174999999997 2024-09-21T18:57:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.155349+0000", 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7510999999999999e-05 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.155393+0000", 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4919999999999997e-05 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:168574 7.0 7:00a13de8:::benchmark_data_smithi081_73391_object29575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.128928+0000", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.919978069000003, 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066923679, 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 168574 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.128928+0000", 2024-09-21T18:57:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.128928+0000", 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.128930+0000", 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.728e-06 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.128938+0000", 2024-09-21T18:57:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6990000000000003e-06 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.128959+0000", 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0644000000000001e-05 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.128968+0000", 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6800000000000005e-06 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.129029+0000", 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0934000000000003e-05 2024-09-21T18:57:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.129115+0000", 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6075999999999994e-05 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.129178+0000", 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3157999999999994e-05 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.131483+0000", 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002304427 2024-09-21T18:57:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.133165+0000", 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016820089999999999 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.195738+0000", 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.062572995000000006 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.195806+0000", 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7750999999999996e-05 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.195851+0000", 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5578000000000003e-05 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:168583 7.0 7:0243c025:::benchmark_data_smithi081_73391_object29584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.139820+0000", 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.909086013, 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085977963000000004, 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 168583 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139820+0000", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139820+0000", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139822+0000", 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.323e-06 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139829+0000", 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.725e-06 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139841+0000", 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1979e-05 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139849+0000", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9359999999999999e-06 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139915+0000", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6311999999999998e-05 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.140049+0000", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013385200000000001 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.140134+0000", 2024-09-21T18:57:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5255999999999998e-05 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.142437+0000", 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023029779999999998 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.199006+0000", 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056569341000000002 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.225678+0000", 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026671836000000001 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.225745+0000", 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6773000000000006e-05 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.225798+0000", 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2651999999999997e-05 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:169006 7.0 7:0bdc217b:::benchmark_data_smithi081_73391_object30007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.638054+0000", 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.410851491000003, 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101027187, 2024-09-21T18:57:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 169006 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638054+0000", 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638054+0000", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638056+0000", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.057e-06 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638062+0000", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1940000000000003e-06 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638079+0000", 2024-09-21T18:57:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6223999999999999e-05 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638086+0000", 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6199999999999999e-06 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638134+0000", 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7886000000000002e-05 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638238+0000", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104114 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638337+0000", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8924000000000004e-05 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.641212+0000", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028744550000000002 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.646975+0000", 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057632689999999997 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.738982+0000", 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092007007000000002 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739038+0000", 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6113999999999998e-05 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739081+0000", 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3322999999999998e-05 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:169009 7.0 7:01cb78f3:::benchmark_data_smithi081_73391_object30010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.644717+0000", 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.40418906, 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106867806, 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 169009 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644717+0000", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644717+0000", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644718+0000", 2024-09-21T18:57:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7910000000000001e-06 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644726+0000", 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6960000000000005e-06 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644736+0000", 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0375e-05 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644743+0000", 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3679999999999998e-06 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644795+0000", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2070999999999997e-05 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644903+0000", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108062 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644986+0000", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3425999999999997e-05 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.647116+0000", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021293950000000001 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739125+0000", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092009274000000002 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751496+0000", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012371312000000001 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751539+0000", 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2358999999999999e-05 2024-09-21T18:57:14.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751584+0000", 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5677000000000003e-05 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:210802 7.0 7:0129baa6:::benchmark_data_smithi081_73391_object71803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.279940+0000", 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.76896513899999996, 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.067450270000000007, 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:14.164 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 210802 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.279940+0000", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.279940+0000", 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.279943+0000", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.576e-06 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.279951+0000", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.279969+0000", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8110000000000001e-05 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:14.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.279977+0000", 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7219999999999999e-06 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.280053+0000", 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6567000000000001e-05 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.280217+0000", 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016383200000000001 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.280297+0000", 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9979000000000001e-05 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.282908+0000", 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026111120000000001 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.282929+0000", 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0312000000000001e-05 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.347266+0000", 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.064337746000000001 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.347346+0000", 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9451999999999998e-05 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.347391+0000", 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4991999999999999e-05 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:14.170 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:57:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:14.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:14.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:14.309 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:14.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:14.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:14.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.565 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:14.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:14.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:14.710 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:14.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:14.840 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:14.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:14.955 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:14 smithi104 ceph-mon[32438]: 6.1 scrub starts 2024-09-21T18:57:14.955 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:14 smithi104 ceph-mon[32438]: 6.1 scrub ok 2024-09-21T18:57:14.955 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:14 smithi104 ceph-mon[32438]: pgmap v177: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 18957/206529 objects misplaced (9.179%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:57:14.955 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:14 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T18:57:14.956 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:14 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T18:57:14.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:14.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:14.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[39192]: 6.1 scrub starts 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[39192]: 6.1 scrub ok 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[39192]: pgmap v177: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 18957/206529 objects misplaced (9.179%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[29384]: 6.1 scrub starts 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[29384]: 6.1 scrub ok 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[29384]: pgmap v177: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 18957/206529 objects misplaced (9.179%); 26 MiB/s, 409 objects/s recovering 2024-09-21T18:57:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T18:57:15.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:14 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T18:57:15.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:15.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:15.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:15.240 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:15.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:15.379 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:15.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:15.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:15.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:15.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:15 smithi104 ceph-mon[32438]: 2.0 scrub starts 2024-09-21T18:57:15.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:15 smithi104 ceph-mon[32438]: 2.0 scrub ok 2024-09-21T18:57:15.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:15.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:15.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:15.923 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:15.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:212982 7.4 e138/114)", 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.934317+0000", 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6621000000000001e-05, 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6501e-05, 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:15.937 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934317+0000", 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934317+0000", 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:15.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934318+0000", 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4020000000000001e-06 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934326+0000", 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9179999999999994e-06 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934336+0000", 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0161e-05 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934344+0000", 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2089999999999999e-06 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:212976 7.c 7:3307a717:::benchmark_data_smithi081_73391_object73977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.864295+0000", 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.070058069000000001, 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070162805999999994, 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:15.940 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 212976 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.864295+0000", 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.864295+0000", 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.864297+0000", 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4190000000000001e-06 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.864301+0000", 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4279999999999998e-06 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.864310+0000", 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1030000000000008e-06 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.864317+0000", 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5780000000000004e-06 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.866383+0000", 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002065562 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.866489+0000", 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106443 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.866628+0000", 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013945200000000001 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.869459+0000", 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028309030000000001 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.870450+0000", 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0009907570000000001 2024-09-21T18:57:15.944 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:212973 7.c 7:37ce032e:::benchmark_data_smithi081_73391_object73974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.862897+0000", 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.071456465999999996, 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071654585000000007, 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:15.945 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 212973 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862897+0000", 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862897+0000", 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862899+0000", 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.917e-06 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862904+0000", 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1950000000000002e-06 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862915+0000", 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0380999999999999e-05 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862922+0000", 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3370000000000002e-06 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862978+0000", 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6320000000000003e-05 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.863105+0000", 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000126439 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.863227+0000", 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000122638 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.869261+0000", 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006034017 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.870223+0000", 2024-09-21T18:57:15.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00096162899999999998 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:212984 7.7 7:e417361f:::benchmark_data_smithi081_73391_object73985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.934033+0000", 2024-09-21T18:57:15.950 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00032005599999999999, 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056688599999999995, 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 212984 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934033+0000", 2024-09-21T18:57:15.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934033+0000", 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934035+0000", 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9819999999999998e-06 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934042+0000", 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1190000000000002e-06 2024-09-21T18:57:15.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934055+0000", 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3901e-05 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934064+0000", 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4840000000000004e-06 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934141+0000", 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7402000000000002e-05 2024-09-21T18:57:15.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934275+0000", 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013354799999999999 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934362+0000", 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6848999999999995e-05 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.954 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPushReply(7.7 138/114 [PushReplyOp(7:e6515956:::benchmark_data_smithi081_73391_object111001:head)])", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.934117+0000", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.000236758, 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00051589300000000002, 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934117+0000", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934117+0000", 2024-09-21T18:57:15.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934117+0000", 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6899999999999998e-07 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934121+0000", 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0029999999999996e-06 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934141+0000", 2024-09-21T18:57:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0285999999999999e-05 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.934149+0000", 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5730000000000002e-06 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 5 2024-09-21T18:57:15.957 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:57:16.025 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:15 smithi081 ceph-mon[29384]: 2.0 scrub starts 2024-09-21T18:57:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:15 smithi081 ceph-mon[29384]: 2.0 scrub ok 2024-09-21T18:57:16.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:15 smithi081 ceph-mon[39192]: 2.0 scrub starts 2024-09-21T18:57:16.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:15 smithi081 ceph-mon[39192]: 2.0 scrub ok 2024-09-21T18:57:16.067 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:16.168 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:16.197 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:16.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:16.339 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:16.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:16.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:16.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:16.609 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:16.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:16.744 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:16.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:16.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:16.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:16.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:16.993 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T18:57:16.993 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T18:57:16.993 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: pgmap v178: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 593 op/s; 18957/206529 objects misplaced (9.179%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T18:57:16.994 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: 3.1d scrub starts 2024-09-21T18:57:16.994 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: 3.1d scrub ok 2024-09-21T18:57:16.994 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: 3.13 scrub starts 2024-09-21T18:57:16.994 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:16 smithi104 ceph-mon[32438]: 3.13 scrub ok 2024-09-21T18:57:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.036 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:17.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:17.039 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: pgmap v178: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 593 op/s; 18957/206529 objects misplaced (9.179%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: 3.1d scrub starts 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: 3.1d scrub ok 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: 3.13 scrub starts 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[39192]: 3.13 scrub ok 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T18:57:17.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T18:57:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: pgmap v178: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 593 op/s; 18957/206529 objects misplaced (9.179%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T18:57:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: 3.1d scrub starts 2024-09-21T18:57:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: 3.1d scrub ok 2024-09-21T18:57:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: 3.13 scrub starts 2024-09-21T18:57:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:16 smithi081 ceph-mon[29384]: 3.13 scrub ok 2024-09-21T18:57:17.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:17.176 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:17.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:17.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:17.379 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:17.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:17.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:17.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:57:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:57:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:57:17.592 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:17.694 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.726 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:17.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:17.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:17.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:17.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:17.945 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:57:17.945 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:57:17.945 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T18:57:17.945 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T18:57:17.945 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:57:17.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:17.987 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-21T18:57:11.436521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4809890355,"num_objects":73584,"num_object_clones":0,"num_object_copies":220752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21475,"num_objects_unfound":0,"num_objects_dirty":73584,"num_whiteouts":0,"num_read":1072,"num_read_kb":929,"num_write":205158,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21272,"num_bytes_recovered":1387404762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85865,"ondisk_log_size":85865,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15993208,"kb_used_data":14874012,"kb_used_omap":12,"kb_used_meta":1118387,"kb_avail":733968008,"statfs":{"total":767960285184,"available":751583240192,"internally_reserved":0,"allocated":15230988288,"data_stored":15228052407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1145228852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-244842496,"num_objects":-3736,"num_object_clones":0,"num_object_copies":-11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3156,"num_objects_unfound":0,"num_objects_dirty":-3736,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3736,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2033,"num_bytes_recovered":133234688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001464"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.650607+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:57:05.650607+0000","last_peered":"2024-09-21T18:57:05.650607+0000","last_clean":"2024-09-21T18:57:05.650607+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:57:05.650607+0000","last_undegraded":"2024-09-21T18:57:05.650607+0000","last_fullsized":"2024-09-21T18:57:05.650607+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":205,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592966+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:57:05.592966+0000","last_peered":"2024-09-21T18:57:05.592966+0000","last_clean":"2024-09-21T18:57:05.592966+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:57:05.592966+0000","last_undegraded":"2024-09-21T18:57:05.592966+0000","last_fullsized":"2024-09-21T18:57:05.592966+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586650+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:57:05.586650+0000","last_peered":"2024-09-21T18:57:05.586650+0000","last_clean":"2024-09-21T18:57:05.586650+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:57:05.586650+0000","last_undegraded":"2024-09-21T18:57:05.586650+0000","last_fullsized":"2024-09-21T18:57:05.586650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":156,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593260+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:57:05.593260+0000","last_peered":"2024-09-21T18:57:05.593260+0000","last_clean":"2024-09-21T18:57:05.593260+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:57:05.593260+0000","last_undegraded":"2024-09-21T18:57:05.593260+0000","last_fullsized":"2024-09-21T18:57:05.593260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":165,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.654051+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:57:05.654051+0000","last_peered":"2024-09-21T18:57:05.654051+0000","last_clean":"2024-09-21T18:57:05.654051+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:57:05.654051+0000","last_undegraded":"2024-09-21T18:57:05.654051+0000","last_fullsized":"2024-09-21T18:57:05.654051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.658309+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:57:05.658309+0000","last_peered":"2024-09-21T18:57:05.658309+0000","last_clean":"2024-09-21T18:57:05.658309+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:57:05.658309+0000","last_undegraded":"2024-09-21T18:57:05.658309+0000","last_fullsized":"2024-09-21T18:57:05.658309+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":236,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653686+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:57:05.653686+0000","last_peered":"2024-09-21T18:57:05.653686+0000","last_clean":"2024-09-21T18:57:05.653686+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:57:05.653686+0000","last_undegraded":"2024-09-21T18:57:05.653686+0000","last_fullsized":"2024-09-21T18:57:05.653686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592868+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:57:05.592868+0000","last_peered":"2024-09-21T18:57:05.592868+0000","last_clean":"2024-09-21T18:57:05.592868+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:57:05.592868+0000","last_undegraded":"2024-09-21T18:57:05.592868+0000","last_fullsized":"2024-09-21T18:57:05.592868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590610+0000","last_change":"2024-09-21T18:53:07.400730+0000","last_active":"2024-09-21T18:57:05.590610+0000","last_peered":"2024-09-21T18:57:05.590610+0000","last_clean":"2024-09-21T18:57:05.590610+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:57:05.590610+0000","last_undegraded":"2024-09-21T18:57:05.590610+0000","last_fullsized":"2024-09-21T18:57:05.590610+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:15.037400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":230,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593596+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:57:05.593596+0000","last_peered":"2024-09-21T18:57:05.593596+0000","last_clean":"2024-09-21T18:57:05.593596+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:57:05.593596+0000","last_undegraded":"2024-09-21T18:57:05.593596+0000","last_fullsized":"2024-09-21T18:57:05.593596+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:05.704741+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":190,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.591208+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:57:05.591208+0000","last_peered":"2024-09-21T18:57:05.591208+0000","last_clean":"2024-09-21T18:57:05.591208+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:57:05.591208+0000","last_undegraded":"2024-09-21T18:57:05.591208+0000","last_fullsized":"2024-09-21T18:57:05.591208+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.948408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.640529+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:57:05.640529+0000","last_peered":"2024-09-21T18:57:05.640529+0000","last_clean":"2024-09-21T18:57:05.640529+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:57:05.640529+0000","last_undegraded":"2024-09-21T18:57:05.640529+0000","last_fullsized":"2024-09-21T18:57:05.640529+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590015+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:57:05.590015+0000","last_peered":"2024-09-21T18:57:05.590015+0000","last_clean":"2024-09-21T18:57:05.590015+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:57:05.590015+0000","last_undegraded":"2024-09-21T18:57:05.590015+0000","last_fullsized":"2024-09-21T18:57:05.590015+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:15.988901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593159+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:57:05.593159+0000","last_peered":"2024-09-21T18:57:05.593159+0000","last_clean":"2024-09-21T18:57:05.593159+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:57:05.593159+0000","last_undegraded":"2024-09-21T18:57:05.593159+0000","last_fullsized":"2024-09-21T18:57:05.593159+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":167,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585536+0000","last_change":"2024-09-21T18:54:53.035653+0000","last_active":"2024-09-21T18:57:05.585536+0000","last_peered":"2024-09-21T18:57:05.585536+0000","last_clean":"2024-09-21T18:57:05.585536+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:57:05.585536+0000","last_undegraded":"2024-09-21T18:57:05.585536+0000","last_fullsized":"2024-09-21T18:57:05.585536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.035514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.035514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":172,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.589744+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:57:05.589744+0000","last_peered":"2024-09-21T18:57:05.589744+0000","last_clean":"2024-09-21T18:57:05.589744+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:57:05.589744+0000","last_undegraded":"2024-09-21T18:57:05.589744+0000","last_fullsized":"2024-09-21T18:57:05.589744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:46.401461+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593947+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:57:05.593947+0000","last_peered":"2024-09-21T18:57:05.593947+0000","last_clean":"2024-09-21T18:57:05.593947+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:57:05.593947+0000","last_undegraded":"2024-09-21T18:57:05.593947+0000","last_fullsized":"2024-09-21T18:57:05.593947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.650307+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:57:05.650307+0000","last_peered":"2024-09-21T18:57:05.650307+0000","last_clean":"2024-09-21T18:57:05.650307+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:57:05.650307+0000","last_undegraded":"2024-09-21T18:57:05.650307+0000","last_fullsized":"2024-09-21T18:57:05.650307+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":197,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641504+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:57:05.641504+0000","last_peered":"2024-09-21T18:57:05.641504+0000","last_clean":"2024-09-21T18:57:05.641504+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:57:05.641504+0000","last_undegraded":"2024-09-21T18:57:05.641504+0000","last_fullsized":"2024-09-21T18:57:05.641504+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.648243+0000","last_change":"2024-09-21T18:55:51.536009+0000","last_active":"2024-09-21T18:57:05.648243+0000","last_peered":"2024-09-21T18:57:05.648243+0000","last_clean":"2024-09-21T18:57:05.648243+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:57:05.648243+0000","last_undegraded":"2024-09-21T18:57:05.648243+0000","last_fullsized":"2024-09-21T18:57:05.648243+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.655847+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:57:05.655847+0000","last_peered":"2024-09-21T18:57:05.655847+0000","last_clean":"2024-09-21T18:57:05.655847+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:57:05.655847+0000","last_undegraded":"2024-09-21T18:57:05.655847+0000","last_fullsized":"2024-09-21T18:57:05.655847+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593767+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:57:05.593767+0000","last_peered":"2024-09-21T18:57:05.593767+0000","last_clean":"2024-09-21T18:57:05.593767+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:57:05.593767+0000","last_undegraded":"2024-09-21T18:57:05.593767+0000","last_fullsized":"2024-09-21T18:57:05.593767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:19.550236+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":196,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653620+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:57:05.653620+0000","last_peered":"2024-09-21T18:57:05.653620+0000","last_clean":"2024-09-21T18:57:05.653620+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:57:05.653620+0000","last_undegraded":"2024-09-21T18:57:05.653620+0000","last_fullsized":"2024-09-21T18:57:05.653620+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593562+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:57:05.593562+0000","last_peered":"2024-09-21T18:57:05.593562+0000","last_clean":"2024-09-21T18:57:05.593562+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:57:05.593562+0000","last_undegraded":"2024-09-21T18:57:05.593562+0000","last_fullsized":"2024-09-21T18:57:05.593562+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586204+0000","last_change":"2024-09-21T18:54:55.039104+0000","last_active":"2024-09-21T18:57:05.586204+0000","last_peered":"2024-09-21T18:57:05.586204+0000","last_clean":"2024-09-21T18:57:05.586204+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:57:05.586204+0000","last_undegraded":"2024-09-21T18:57:05.586204+0000","last_fullsized":"2024-09-21T18:57:05.586204+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:54:55.039055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:55.039055+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586728+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:57:05.586728+0000","last_peered":"2024-09-21T18:57:05.586728+0000","last_clean":"2024-09-21T18:57:05.586728+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:57:05.586728+0000","last_undegraded":"2024-09-21T18:57:05.586728+0000","last_fullsized":"2024-09-21T18:57:05.586728+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":68,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.643839+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:57:05.643839+0000","last_peered":"2024-09-21T18:57:05.643839+0000","last_clean":"2024-09-21T18:57:05.643839+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:57:05.643839+0000","last_undegraded":"2024-09-21T18:57:05.643839+0000","last_fullsized":"2024-09-21T18:57:05.643839+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":226,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.594436+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:57:05.594436+0000","last_peered":"2024-09-21T18:57:05.594436+0000","last_clean":"2024-09-21T18:57:05.594436+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:57:05.594436+0000","last_undegraded":"2024-09-21T18:57:05.594436+0000","last_fullsized":"2024-09-21T18:57:05.594436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"136'12635","reported_seq":12762,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.440189+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:57:06.440189+0000","last_peered":"2024-09-21T18:57:06.440189+0000","last_clean":"2024-09-21T18:57:06.440189+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:57:06.440189+0000","last_undegraded":"2024-09-21T18:57:06.440189+0000","last_fullsized":"2024-09-21T18:57:06.440189+0000","mapping_epoch":79,"log_start":"105'6500","ondisk_log_start":"105'6500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293797888,"num_objects":4483,"num_object_clones":0,"num_object_copies":13449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12635,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":189,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.646165+0000","last_change":"2024-09-21T18:53:09.411275+0000","last_active":"2024-09-21T18:57:05.646165+0000","last_peered":"2024-09-21T18:57:05.646165+0000","last_clean":"2024-09-21T18:57:05.646165+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:57:05.646165+0000","last_undegraded":"2024-09-21T18:57:05.646165+0000","last_fullsized":"2024-09-21T18:57:05.646165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653324+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:57:05.653324+0000","last_peered":"2024-09-21T18:57:05.653324+0000","last_clean":"2024-09-21T18:57:05.653324+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:57:05.653324+0000","last_undegraded":"2024-09-21T18:57:05.653324+0000","last_fullsized":"2024-09-21T18:57:05.653324+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":246,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593994+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:57:05.593994+0000","last_peered":"2024-09-21T18:57:05.593994+0000","last_clean":"2024-09-21T18:57:05.593994+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:57:05.593994+0000","last_undegraded":"2024-09-21T18:57:05.593994+0000","last_fullsized":"2024-09-21T18:57:05.593994+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:12.350077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":186,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590234+0000","last_change":"2024-09-21T18:56:37.863087+0000","last_active":"2024-09-21T18:57:05.590234+0000","last_peered":"2024-09-21T18:57:05.590234+0000","last_clean":"2024-09-21T18:57:05.590234+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:57:05.590234+0000","last_undegraded":"2024-09-21T18:57:05.590234+0000","last_fullsized":"2024-09-21T18:57:05.590234+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:37.863026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:37.863026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.972736+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"136'12684","reported_seq":12811,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.983332+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:57:06.983332+0000","last_peered":"2024-09-21T18:57:06.983332+0000","last_clean":"2024-09-21T18:57:06.983332+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:57:06.983332+0000","last_undegraded":"2024-09-21T18:57:06.983332+0000","last_fullsized":"2024-09-21T18:57:06.983332+0000","mapping_epoch":79,"log_start":"111'8200","ondisk_log_start":"111'8200","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296091648,"num_objects":4518,"num_object_clones":0,"num_object_copies":13554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12684,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":255,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.649609+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:57:05.649609+0000","last_peered":"2024-09-21T18:57:05.649609+0000","last_clean":"2024-09-21T18:57:05.649609+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:57:05.649609+0000","last_undegraded":"2024-09-21T18:57:05.649609+0000","last_fullsized":"2024-09-21T18:57:05.649609+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":191,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.657603+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:57:05.657603+0000","last_peered":"2024-09-21T18:57:05.657603+0000","last_clean":"2024-09-21T18:57:05.657603+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:57:05.657603+0000","last_undegraded":"2024-09-21T18:57:05.657603+0000","last_fullsized":"2024-09-21T18:57:05.657603+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593146+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:57:05.593146+0000","last_peered":"2024-09-21T18:57:05.593146+0000","last_clean":"2024-09-21T18:57:05.593146+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:57:05.593146+0000","last_undegraded":"2024-09-21T18:57:05.593146+0000","last_fullsized":"2024-09-21T18:57:05.593146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:32.288577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":156,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593485+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:57:05.593485+0000","last_peered":"2024-09-21T18:57:05.593485+0000","last_clean":"2024-09-21T18:57:05.593485+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:57:05.593485+0000","last_undegraded":"2024-09-21T18:57:05.593485+0000","last_fullsized":"2024-09-21T18:57:05.593485+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:15.982856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"136'12929","reported_seq":13056,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.994018+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:57:06.994018+0000","last_peered":"2024-09-21T18:57:06.994018+0000","last_clean":"2024-09-21T18:57:06.994018+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:57:06.994018+0000","last_undegraded":"2024-09-21T18:57:06.994018+0000","last_fullsized":"2024-09-21T18:57:06.994018+0000","mapping_epoch":79,"log_start":"111'8400","ondisk_log_start":"111'8400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299040768,"num_objects":4563,"num_object_clones":0,"num_object_copies":13689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":193,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593268+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:57:05.593268+0000","last_peered":"2024-09-21T18:57:05.593268+0000","last_clean":"2024-09-21T18:57:05.593268+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:57:05.593268+0000","last_undegraded":"2024-09-21T18:57:05.593268+0000","last_fullsized":"2024-09-21T18:57:05.593268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:21.938392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":60,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.644105+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:57:05.644105+0000","last_peered":"2024-09-21T18:57:05.644105+0000","last_clean":"2024-09-21T18:57:05.644105+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:57:05.644105+0000","last_undegraded":"2024-09-21T18:57:05.644105+0000","last_fullsized":"2024-09-21T18:57:05.644105+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":217,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.649310+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:57:05.649310+0000","last_peered":"2024-09-21T18:57:05.649310+0000","last_clean":"2024-09-21T18:57:05.649310+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:57:05.649310+0000","last_undegraded":"2024-09-21T18:57:05.649310+0000","last_fullsized":"2024-09-21T18:57:05.649310+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585699+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:57:05.585699+0000","last_peered":"2024-09-21T18:57:05.585699+0000","last_clean":"2024-09-21T18:57:05.585699+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:57:05.585699+0000","last_undegraded":"2024-09-21T18:57:05.585699+0000","last_fullsized":"2024-09-21T18:57:05.585699+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.105131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"136'12670","reported_seq":12888,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.425666+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:57:06.425666+0000","last_peered":"2024-09-21T18:57:06.425666+0000","last_clean":"2024-09-21T18:57:06.425666+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:57:06.425666+0000","last_undegraded":"2024-09-21T18:57:06.425666+0000","last_fullsized":"2024-09-21T18:57:06.425666+0000","mapping_epoch":79,"log_start":"111'7900","ondisk_log_start":"111'7900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:25.602584+0000","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300941312,"num_objects":4592,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":179,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586802+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:57:05.586802+0000","last_peered":"2024-09-21T18:57:05.586802+0000","last_clean":"2024-09-21T18:57:05.586802+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:57:05.586802+0000","last_undegraded":"2024-09-21T18:57:05.586802+0000","last_fullsized":"2024-09-21T18:57:05.586802+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590727+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:57:05.590727+0000","last_peered":"2024-09-21T18:57:05.590727+0000","last_clean":"2024-09-21T18:57:05.590727+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:57:05.590727+0000","last_undegraded":"2024-09-21T18:57:05.590727+0000","last_fullsized":"2024-09-21T18:57:05.590727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":75,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.650397+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:57:05.650397+0000","last_peered":"2024-09-21T18:57:05.650397+0000","last_clean":"2024-09-21T18:57:05.650397+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:57:05.650397+0000","last_undegraded":"2024-09-21T18:57:05.650397+0000","last_fullsized":"2024-09-21T18:57:05.650397+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.640701+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:57:05.640701+0000","last_peered":"2024-09-21T18:57:05.640701+0000","last_clean":"2024-09-21T18:57:05.640701+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:57:05.640701+0000","last_undegraded":"2024-09-21T18:57:05.640701+0000","last_fullsized":"2024-09-21T18:57:05.640701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"136'12996","reported_seq":13272,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.440505+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:57:06.440505+0000","last_peered":"2024-09-21T18:57:06.440505+0000","last_clean":"2024-09-21T18:57:06.440505+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:57:06.440505+0000","last_undegraded":"2024-09-21T18:57:06.440505+0000","last_fullsized":"2024-09-21T18:57:06.440505+0000","mapping_epoch":79,"log_start":"109'7400","ondisk_log_start":"109'7400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:43.426801+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305528832,"num_objects":4662,"num_object_clones":0,"num_object_copies":13986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12996,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":74,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.651283+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:57:05.651283+0000","last_peered":"2024-09-21T18:57:05.651283+0000","last_clean":"2024-09-21T18:57:05.651283+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:57:05.651283+0000","last_undegraded":"2024-09-21T18:57:05.651283+0000","last_fullsized":"2024-09-21T18:57:05.651283+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":176,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.648940+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:57:05.648940+0000","last_peered":"2024-09-21T18:57:05.648940+0000","last_clean":"2024-09-21T18:57:05.648940+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:57:05.648940+0000","last_undegraded":"2024-09-21T18:57:05.648940+0000","last_fullsized":"2024-09-21T18:57:05.648940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592017+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:57:05.592017+0000","last_peered":"2024-09-21T18:57:05.592017+0000","last_clean":"2024-09-21T18:57:05.592017+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:57:05.592017+0000","last_undegraded":"2024-09-21T18:57:05.592017+0000","last_fullsized":"2024-09-21T18:57:05.592017+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":156,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586569+0000","last_change":"2024-09-21T18:53:13.657330+0000","last_active":"2024-09-21T18:57:05.586569+0000","last_peered":"2024-09-21T18:57:05.586569+0000","last_clean":"2024-09-21T18:57:05.586569+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:57:05.586569+0000","last_undegraded":"2024-09-21T18:57:05.586569+0000","last_fullsized":"2024-09-21T18:57:05.586569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"136'12749","reported_seq":14384,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:06.459887+0000","last_change":"2024-09-21T18:57:02.353922+0000","last_active":"2024-09-21T18:57:06.459887+0000","last_peered":"2024-09-21T18:57:06.459887+0000","last_clean":"2024-09-21T18:55:50.489495+0000","last_became_active":"2024-09-21T18:55:52.881338+0000","last_became_peered":"2024-09-21T18:55:52.881338+0000","last_unstale":"2024-09-21T18:57:06.459887+0000","last_undegraded":"2024-09-21T18:57:06.459887+0000","last_fullsized":"2024-09-21T18:57:06.459887+0000","mapping_epoch":114,"log_start":"105'6600","ondisk_log_start":"105'6600","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300089344,"num_objects":4579,"num_object_clones":0,"num_object_copies":13737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3854,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12749,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":748,"num_bytes_recovered":49020928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,3,1],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":4579}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":268,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.648802+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:57:05.648802+0000","last_peered":"2024-09-21T18:57:05.648802+0000","last_clean":"2024-09-21T18:57:05.648802+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:57:05.648802+0000","last_undegraded":"2024-09-21T18:57:05.648802+0000","last_fullsized":"2024-09-21T18:57:05.648802+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":198,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593600+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:57:05.593600+0000","last_peered":"2024-09-21T18:57:05.593600+0000","last_clean":"2024-09-21T18:57:05.593600+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:57:05.593600+0000","last_undegraded":"2024-09-21T18:57:05.593600+0000","last_fullsized":"2024-09-21T18:57:05.593600+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:12.823290+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":194,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590002+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:57:05.590002+0000","last_peered":"2024-09-21T18:57:05.590002+0000","last_clean":"2024-09-21T18:57:05.590002+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:57:05.590002+0000","last_undegraded":"2024-09-21T18:57:05.590002+0000","last_fullsized":"2024-09-21T18:57:05.590002+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.643087+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:57:05.643087+0000","last_peered":"2024-09-21T18:57:05.643087+0000","last_clean":"2024-09-21T18:57:05.643087+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:57:05.643087+0000","last_undegraded":"2024-09-21T18:57:05.643087+0000","last_fullsized":"2024-09-21T18:57:05.643087+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":165,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.658040+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:57:05.658040+0000","last_peered":"2024-09-21T18:57:05.658040+0000","last_clean":"2024-09-21T18:57:05.658040+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:57:05.658040+0000","last_undegraded":"2024-09-21T18:57:05.658040+0000","last_fullsized":"2024-09-21T18:57:05.658040+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"136'12930","reported_seq":13067,"reported_epoch":136,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T18:57:06.420362+0000","last_change":"2024-09-21T18:56:22.889265+0000","last_active":"2024-09-21T18:57:06.420362+0000","last_peered":"2024-09-21T18:57:06.420362+0000","last_clean":"2024-09-21T18:55:50.469777+0000","last_became_active":"2024-09-21T18:55:52.866581+0000","last_became_peered":"2024-09-21T18:55:52.866581+0000","last_unstale":"2024-09-21T18:57:06.420362+0000","last_undegraded":"2024-09-21T18:57:06.420362+0000","last_fullsized":"2024-09-21T18:57:06.420362+0000","mapping_epoch":114,"log_start":"111'8200","ondisk_log_start":"111'8200","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4730,"log_dups_size":0,"ondisk_log_size":4730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300154880,"num_objects":4580,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4580,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12930,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":4580}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641740+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:57:05.641740+0000","last_peered":"2024-09-21T18:57:05.641740+0000","last_clean":"2024-09-21T18:57:05.641740+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:57:05.641740+0000","last_undegraded":"2024-09-21T18:57:05.641740+0000","last_fullsized":"2024-09-21T18:57:05.641740+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586366+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:57:05.586366+0000","last_peered":"2024-09-21T18:57:05.586366+0000","last_clean":"2024-09-21T18:57:05.586366+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:57:05.586366+0000","last_undegraded":"2024-09-21T18:57:05.586366+0000","last_fullsized":"2024-09-21T18:57:05.586366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":207,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.640837+0000","last_change":"2024-09-21T18:55:51.541888+0000","last_active":"2024-09-21T18:57:05.640837+0000","last_peered":"2024-09-21T18:57:05.640837+0000","last_clean":"2024-09-21T18:57:05.640837+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:57:05.640837+0000","last_undegraded":"2024-09-21T18:57:05.640837+0000","last_fullsized":"2024-09-21T18:57:05.640837+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:25.237173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.237173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.643573+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:57:05.643573+0000","last_peered":"2024-09-21T18:57:05.643573+0000","last_clean":"2024-09-21T18:57:05.643573+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:57:05.643573+0000","last_undegraded":"2024-09-21T18:57:05.643573+0000","last_fullsized":"2024-09-21T18:57:05.643573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":167,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585658+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:57:05.585658+0000","last_peered":"2024-09-21T18:57:05.585658+0000","last_clean":"2024-09-21T18:57:05.585658+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:57:05.585658+0000","last_undegraded":"2024-09-21T18:57:05.585658+0000","last_fullsized":"2024-09-21T18:57:05.585658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"136'13061","reported_seq":27331,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.456372+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:57:06.456372+0000","last_peered":"2024-09-21T18:57:06.456372+0000","last_clean":"2024-09-21T18:57:06.456372+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:57:06.456372+0000","last_undegraded":"2024-09-21T18:57:06.456372+0000","last_fullsized":"2024-09-21T18:57:06.456372+0000","mapping_epoch":130,"log_start":"110'7900","ondisk_log_start":"110'7900","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312016896,"num_objects":4761,"num_object_clones":0,"num_object_copies":14283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13061,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":70,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.643933+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:57:05.643933+0000","last_peered":"2024-09-21T18:57:05.643933+0000","last_clean":"2024-09-21T18:57:05.643933+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:57:05.643933+0000","last_undegraded":"2024-09-21T18:57:05.643933+0000","last_fullsized":"2024-09-21T18:57:05.643933+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641072+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:57:05.641072+0000","last_peered":"2024-09-21T18:57:05.641072+0000","last_clean":"2024-09-21T18:57:05.641072+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:57:05.641072+0000","last_undegraded":"2024-09-21T18:57:05.641072+0000","last_fullsized":"2024-09-21T18:57:05.641072+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":441,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585324+0000","last_change":"2024-09-21T18:55:51.563629+0000","last_active":"2024-09-21T18:57:05.585324+0000","last_peered":"2024-09-21T18:57:05.585324+0000","last_clean":"2024-09-21T18:57:05.585324+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:57:05.585324+0000","last_undegraded":"2024-09-21T18:57:05.585324+0000","last_fullsized":"2024-09-21T18:57:05.585324+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:41.110776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.110776+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":228,"num_read_kb":223,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":190,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590962+0000","last_change":"2024-09-21T18:55:53.079160+0000","last_active":"2024-09-21T18:57:05.590962+0000","last_peered":"2024-09-21T18:57:05.590962+0000","last_clean":"2024-09-21T18:57:05.590962+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:57:05.590962+0000","last_undegraded":"2024-09-21T18:57:05.590962+0000","last_fullsized":"2024-09-21T18:57:05.590962+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:53.079108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.079108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":156,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.647037+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:57:05.647037+0000","last_peered":"2024-09-21T18:57:05.647037+0000","last_clean":"2024-09-21T18:57:05.647037+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:57:05.647037+0000","last_undegraded":"2024-09-21T18:57:05.647037+0000","last_fullsized":"2024-09-21T18:57:05.647037+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"136'12763","reported_seq":23333,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:07.035395+0000","last_change":"2024-09-21T18:55:52.898089+0000","last_active":"2024-09-21T18:57:07.035395+0000","last_peered":"2024-09-21T18:57:07.035395+0000","last_clean":"2024-09-21T18:55:50.465179+0000","last_became_active":"2024-09-21T18:55:52.872405+0000","last_became_peered":"2024-09-21T18:55:52.872405+0000","last_unstale":"2024-09-21T18:57:07.035395+0000","last_undegraded":"2024-09-21T18:57:07.035395+0000","last_fullsized":"2024-09-21T18:57:07.035395+0000","mapping_epoch":114,"log_start":"111'8202","ondisk_log_start":"111'8202","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301268992,"num_objects":4597,"num_object_clones":0,"num_object_copies":13791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":772,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12763,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5223,"num_bytes_recovered":342294528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":4597}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":241,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592129+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:57:05.592129+0000","last_peered":"2024-09-21T18:57:05.592129+0000","last_clean":"2024-09-21T18:57:05.592129+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:57:05.592129+0000","last_undegraded":"2024-09-21T18:57:05.592129+0000","last_fullsized":"2024-09-21T18:57:05.592129+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.654959+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:57:05.654959+0000","last_peered":"2024-09-21T18:57:05.654959+0000","last_clean":"2024-09-21T18:57:05.654959+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:57:05.654959+0000","last_undegraded":"2024-09-21T18:57:05.654959+0000","last_fullsized":"2024-09-21T18:57:05.654959+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":239,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585415+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:57:05.585415+0000","last_peered":"2024-09-21T18:57:05.585415+0000","last_clean":"2024-09-21T18:57:05.585415+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:57:05.585415+0000","last_undegraded":"2024-09-21T18:57:05.585415+0000","last_fullsized":"2024-09-21T18:57:05.585415+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:05.121733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586007+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:57:05.586007+0000","last_peered":"2024-09-21T18:57:05.586007+0000","last_clean":"2024-09-21T18:57:05.586007+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:57:05.586007+0000","last_undegraded":"2024-09-21T18:57:05.586007+0000","last_fullsized":"2024-09-21T18:57:05.586007+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:01.123772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590889+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:57:05.590889+0000","last_peered":"2024-09-21T18:57:05.590889+0000","last_clean":"2024-09-21T18:57:05.590889+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:57:05.590889+0000","last_undegraded":"2024-09-21T18:57:05.590889+0000","last_fullsized":"2024-09-21T18:57:05.590889+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:26.445821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"136'12919","reported_seq":13148,"reported_epoch":136,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T18:57:06.438028+0000","last_change":"2024-09-21T18:56:22.889684+0000","last_active":"2024-09-21T18:57:06.438028+0000","last_peered":"2024-09-21T18:57:06.438028+0000","last_clean":"2024-09-21T18:55:50.477883+0000","last_became_active":"2024-09-21T18:55:52.879827+0000","last_became_peered":"2024-09-21T18:55:52.879827+0000","last_unstale":"2024-09-21T18:57:06.438028+0000","last_undegraded":"2024-09-21T18:57:06.438028+0000","last_fullsized":"2024-09-21T18:57:06.438028+0000","mapping_epoch":114,"log_start":"105'6700","ondisk_log_start":"105'6700","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6219,"log_dups_size":0,"ondisk_log_size":6219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308019224,"num_objects":4701,"num_object_clones":0,"num_object_copies":14103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4701,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12919,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":4701}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":191,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.594031+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:57:05.594031+0000","last_peered":"2024-09-21T18:57:05.594031+0000","last_clean":"2024-09-21T18:57:05.594031+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:57:05.594031+0000","last_undegraded":"2024-09-21T18:57:05.594031+0000","last_fullsized":"2024-09-21T18:57:05.594031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593584+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:57:05.593584+0000","last_peered":"2024-09-21T18:57:05.593584+0000","last_clean":"2024-09-21T18:57:05.593584+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:57:05.593584+0000","last_undegraded":"2024-09-21T18:57:05.593584+0000","last_fullsized":"2024-09-21T18:57:05.593584+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":176,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590817+0000","last_change":"2024-09-21T18:56:33.913317+0000","last_active":"2024-09-21T18:57:05.590817+0000","last_peered":"2024-09-21T18:57:05.590817+0000","last_clean":"2024-09-21T18:57:05.590817+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:57:05.590817+0000","last_undegraded":"2024-09-21T18:57:05.590817+0000","last_fullsized":"2024-09-21T18:57:05.590817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:33.913250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:33.913250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:52.937683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653186+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:57:05.653186+0000","last_peered":"2024-09-21T18:57:05.653186+0000","last_clean":"2024-09-21T18:57:05.653186+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:57:05.653186+0000","last_undegraded":"2024-09-21T18:57:05.653186+0000","last_fullsized":"2024-09-21T18:57:05.653186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"136'12645","reported_seq":16114,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:06.485343+0000","last_change":"2024-09-21T18:56:52.172568+0000","last_active":"2024-09-21T18:57:06.485343+0000","last_peered":"2024-09-21T18:57:06.485343+0000","last_clean":"2024-09-21T18:55:50.490306+0000","last_became_active":"2024-09-21T18:55:52.880630+0000","last_became_peered":"2024-09-21T18:55:52.880630+0000","last_unstale":"2024-09-21T18:57:06.485343+0000","last_undegraded":"2024-09-21T18:57:06.485343+0000","last_fullsized":"2024-09-21T18:57:06.485343+0000","mapping_epoch":114,"log_start":"102'5900","ondisk_log_start":"102'5900","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294322176,"num_objects":4491,"num_object_clones":0,"num_object_copies":13473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2997,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":106496000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":4491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":200,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.594042+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:57:05.594042+0000","last_peered":"2024-09-21T18:57:05.594042+0000","last_clean":"2024-09-21T18:57:05.594042+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:57:05.594042+0000","last_undegraded":"2024-09-21T18:57:05.594042+0000","last_fullsized":"2024-09-21T18:57:05.594042+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:21.119260+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592318+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:57:05.592318+0000","last_peered":"2024-09-21T18:57:05.592318+0000","last_clean":"2024-09-21T18:57:05.592318+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:57:05.592318+0000","last_undegraded":"2024-09-21T18:57:05.592318+0000","last_fullsized":"2024-09-21T18:57:05.592318+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.647462+0000","last_change":"2024-09-21T18:55:51.535384+0000","last_active":"2024-09-21T18:57:05.647462+0000","last_peered":"2024-09-21T18:57:05.647462+0000","last_clean":"2024-09-21T18:57:05.647462+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:57:05.647462+0000","last_undegraded":"2024-09-21T18:57:05.647462+0000","last_fullsized":"2024-09-21T18:57:05.647462+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.042616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641912+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:57:05.641912+0000","last_peered":"2024-09-21T18:57:05.641912+0000","last_clean":"2024-09-21T18:57:05.641912+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:57:05.641912+0000","last_undegraded":"2024-09-21T18:57:05.641912+0000","last_fullsized":"2024-09-21T18:57:05.641912+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"136'12767","reported_seq":12899,"reported_epoch":136,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T18:57:07.034402+0000","last_change":"2024-09-21T18:55:52.897819+0000","last_active":"2024-09-21T18:57:07.034402+0000","last_peered":"2024-09-21T18:57:07.034402+0000","last_clean":"2024-09-21T18:55:50.495661+0000","last_became_active":"2024-09-21T18:55:52.870612+0000","last_became_peered":"2024-09-21T18:55:52.870612+0000","last_unstale":"2024-09-21T18:57:07.034402+0000","last_undegraded":"2024-09-21T18:57:07.034402+0000","last_fullsized":"2024-09-21T18:57:07.034402+0000","mapping_epoch":114,"log_start":"111'8200","ondisk_log_start":"111'8200","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299565056,"num_objects":4571,"num_object_clones":0,"num_object_copies":13713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4571,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":4571}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":187,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593691+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:57:05.593691+0000","last_peered":"2024-09-21T18:57:05.593691+0000","last_clean":"2024-09-21T18:57:05.593691+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:57:05.593691+0000","last_undegraded":"2024-09-21T18:57:05.593691+0000","last_fullsized":"2024-09-21T18:57:05.593691+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:09.960146+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":205,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.654539+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:57:05.654539+0000","last_peered":"2024-09-21T18:57:05.654539+0000","last_clean":"2024-09-21T18:57:05.654539+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:57:05.654539+0000","last_undegraded":"2024-09-21T18:57:05.654539+0000","last_fullsized":"2024-09-21T18:57:05.654539+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":224,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641380+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:57:05.641380+0000","last_peered":"2024-09-21T18:57:05.641380+0000","last_clean":"2024-09-21T18:57:05.641380+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:57:05.641380+0000","last_undegraded":"2024-09-21T18:57:05.641380+0000","last_fullsized":"2024-09-21T18:57:05.641380+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593865+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:57:05.593865+0000","last_peered":"2024-09-21T18:57:05.593865+0000","last_clean":"2024-09-21T18:57:05.593865+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:57:05.593865+0000","last_undegraded":"2024-09-21T18:57:05.593865+0000","last_fullsized":"2024-09-21T18:57:05.593865+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:18.009607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"136'12794","reported_seq":13001,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.477476+0000","last_change":"2024-09-21T18:55:30.532749+0000","last_active":"2024-09-21T18:57:06.477476+0000","last_peered":"2024-09-21T18:57:06.477476+0000","last_clean":"2024-09-21T18:57:06.477476+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:57:06.477476+0000","last_undegraded":"2024-09-21T18:57:06.477476+0000","last_fullsized":"2024-09-21T18:57:06.477476+0000","mapping_epoch":79,"log_start":"102'6100","ondisk_log_start":"102'6100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7201","last_scrub_stamp":"2024-09-21T18:55:30.532526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.532526+0000","objects_scrubbed":7133,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305659904,"num_objects":4664,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12794,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":66,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.651380+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:57:05.651380+0000","last_peered":"2024-09-21T18:57:05.651380+0000","last_clean":"2024-09-21T18:57:05.651380+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:57:05.651380+0000","last_undegraded":"2024-09-21T18:57:05.651380+0000","last_fullsized":"2024-09-21T18:57:05.651380+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590139+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:57:05.590139+0000","last_peered":"2024-09-21T18:57:05.590139+0000","last_clean":"2024-09-21T18:57:05.590139+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:57:05.590139+0000","last_undegraded":"2024-09-21T18:57:05.590139+0000","last_fullsized":"2024-09-21T18:57:05.590139+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.889938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.642887+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:57:05.642887+0000","last_peered":"2024-09-21T18:57:05.642887+0000","last_clean":"2024-09-21T18:57:05.642887+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:57:05.642887+0000","last_undegraded":"2024-09-21T18:57:05.642887+0000","last_fullsized":"2024-09-21T18:57:05.642887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.640995+0000","last_change":"2024-09-21T18:55:51.540541+0000","last_active":"2024-09-21T18:57:05.640995+0000","last_peered":"2024-09-21T18:57:05.640995+0000","last_clean":"2024-09-21T18:57:05.640995+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:57:05.640995+0000","last_undegraded":"2024-09-21T18:57:05.640995+0000","last_fullsized":"2024-09-21T18:57:05.640995+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"136'12878","reported_seq":25985,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.456634+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:57:06.456634+0000","last_peered":"2024-09-21T18:57:06.456634+0000","last_clean":"2024-09-21T18:57:06.456634+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:57:06.456634+0000","last_undegraded":"2024-09-21T18:57:06.456634+0000","last_fullsized":"2024-09-21T18:57:06.456634+0000","mapping_epoch":134,"log_start":"110'7600","ondisk_log_start":"110'7600","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298450944,"num_objects":4554,"num_object_clones":0,"num_object_copies":13662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":202,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592108+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:57:05.592108+0000","last_peered":"2024-09-21T18:57:05.592108+0000","last_clean":"2024-09-21T18:57:05.592108+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:57:05.592108+0000","last_undegraded":"2024-09-21T18:57:05.592108+0000","last_fullsized":"2024-09-21T18:57:05.592108+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585916+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:57:05.585916+0000","last_peered":"2024-09-21T18:57:05.585916+0000","last_clean":"2024-09-21T18:57:05.585916+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:57:05.585916+0000","last_undegraded":"2024-09-21T18:57:05.585916+0000","last_fullsized":"2024-09-21T18:57:05.585916+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.646823+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:57:05.646823+0000","last_peered":"2024-09-21T18:57:05.646823+0000","last_clean":"2024-09-21T18:57:05.646823+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:57:05.646823+0000","last_undegraded":"2024-09-21T18:57:05.646823+0000","last_fullsized":"2024-09-21T18:57:05.646823+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585311+0000","last_change":"2024-09-21T18:55:24.049546+0000","last_active":"2024-09-21T18:57:05.585311+0000","last_peered":"2024-09-21T18:57:05.585311+0000","last_clean":"2024-09-21T18:57:05.585311+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:57:05.585311+0000","last_undegraded":"2024-09-21T18:57:05.585311+0000","last_fullsized":"2024-09-21T18:57:05.585311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.049407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.049407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"136'12620","reported_seq":12747,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:07.023074+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:57:07.023074+0000","last_peered":"2024-09-21T18:57:07.023074+0000","last_clean":"2024-09-21T18:57:07.023074+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:57:07.023074+0000","last_undegraded":"2024-09-21T18:57:07.023074+0000","last_fullsized":"2024-09-21T18:57:07.023074+0000","mapping_epoch":79,"log_start":"111'8100","ondisk_log_start":"111'8100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":214,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593436+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:57:05.593436+0000","last_peered":"2024-09-21T18:57:05.593436+0000","last_clean":"2024-09-21T18:57:05.593436+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:57:05.593436+0000","last_undegraded":"2024-09-21T18:57:05.593436+0000","last_fullsized":"2024-09-21T18:57:05.593436+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:16.969327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586393+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:57:05.586393+0000","last_peered":"2024-09-21T18:57:05.586393+0000","last_clean":"2024-09-21T18:57:05.586393+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:57:05.586393+0000","last_undegraded":"2024-09-21T18:57:05.586393+0000","last_fullsized":"2024-09-21T18:57:05.586393+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":190,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641874+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:57:05.641874+0000","last_peered":"2024-09-21T18:57:05.641874+0000","last_clean":"2024-09-21T18:57:05.641874+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:57:05.641874+0000","last_undegraded":"2024-09-21T18:57:05.641874+0000","last_fullsized":"2024-09-21T18:57:05.641874+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653412+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:57:05.653412+0000","last_peered":"2024-09-21T18:57:05.653412+0000","last_clean":"2024-09-21T18:57:05.653412+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:57:05.653412+0000","last_undegraded":"2024-09-21T18:57:05.653412+0000","last_fullsized":"2024-09-21T18:57:05.653412+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"136'12569","reported_seq":12696,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:06.662808+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:57:06.662808+0000","last_peered":"2024-09-21T18:57:06.662808+0000","last_clean":"2024-09-21T18:57:06.662808+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:57:06.662808+0000","last_undegraded":"2024-09-21T18:57:06.662808+0000","last_fullsized":"2024-09-21T18:57:06.662808+0000","mapping_epoch":79,"log_start":"110'7400","ondisk_log_start":"110'7400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298123264,"num_objects":4549,"num_object_clones":0,"num_object_copies":13647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":205,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590245+0000","last_change":"2024-09-21T18:56:35.846693+0000","last_active":"2024-09-21T18:57:05.590245+0000","last_peered":"2024-09-21T18:57:05.590245+0000","last_clean":"2024-09-21T18:57:05.590245+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:57:05.590245+0000","last_undegraded":"2024-09-21T18:57:05.590245+0000","last_fullsized":"2024-09-21T18:57:05.590245+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:56:35.846605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:35.846605+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:51.214847+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":205,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586624+0000","last_change":"2024-09-21T18:55:51.567353+0000","last_active":"2024-09-21T18:57:05.586624+0000","last_peered":"2024-09-21T18:57:05.586624+0000","last_clean":"2024-09-21T18:57:05.586624+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:57:05.586624+0000","last_undegraded":"2024-09-21T18:57:05.586624+0000","last_fullsized":"2024-09-21T18:57:05.586624+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:55.125124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":226,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585471+0000","last_change":"2024-09-21T18:55:39.995074+0000","last_active":"2024-09-21T18:57:05.585471+0000","last_peered":"2024-09-21T18:57:05.585471+0000","last_clean":"2024-09-21T18:57:05.585471+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:57:05.585471+0000","last_undegraded":"2024-09-21T18:57:05.585471+0000","last_fullsized":"2024-09-21T18:57:05.585471+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:39.994892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.994892+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.589692+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:57:05.589692+0000","last_peered":"2024-09-21T18:57:05.589692+0000","last_clean":"2024-09-21T18:57:05.589692+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:57:05.589692+0000","last_undegraded":"2024-09-21T18:57:05.589692+0000","last_fullsized":"2024-09-21T18:57:05.589692+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:24.875223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":189,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.591858+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:57:05.591858+0000","last_peered":"2024-09-21T18:57:05.591858+0000","last_clean":"2024-09-21T18:57:05.591858+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:57:05.591858+0000","last_undegraded":"2024-09-21T18:57:05.591858+0000","last_fullsized":"2024-09-21T18:57:05.591858+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.657478+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:57:05.657478+0000","last_peered":"2024-09-21T18:57:05.657478+0000","last_clean":"2024-09-21T18:57:05.657478+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:57:05.657478+0000","last_undegraded":"2024-09-21T18:57:05.657478+0000","last_fullsized":"2024-09-21T18:57:05.657478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":176,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.591172+0000","last_change":"2024-09-21T18:56:40.848060+0000","last_active":"2024-09-21T18:57:05.591172+0000","last_peered":"2024-09-21T18:57:05.591172+0000","last_clean":"2024-09-21T18:57:05.591172+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:57:05.591172+0000","last_undegraded":"2024-09-21T18:57:05.591172+0000","last_fullsized":"2024-09-21T18:57:05.591172+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:40.847976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.847976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:57.531530+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585732+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:57:05.585732+0000","last_peered":"2024-09-21T18:57:05.585732+0000","last_clean":"2024-09-21T18:57:05.585732+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:57:05.585732+0000","last_undegraded":"2024-09-21T18:57:05.585732+0000","last_fullsized":"2024-09-21T18:57:05.585732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:00.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":66,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.644019+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:57:05.644019+0000","last_peered":"2024-09-21T18:57:05.644019+0000","last_clean":"2024-09-21T18:57:05.644019+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:57:05.644019+0000","last_undegraded":"2024-09-21T18:57:05.644019+0000","last_fullsized":"2024-09-21T18:57:05.644019+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585686+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:57:05.585686+0000","last_peered":"2024-09-21T18:57:05.585686+0000","last_clean":"2024-09-21T18:57:05.585686+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:57:05.585686+0000","last_undegraded":"2024-09-21T18:57:05.585686+0000","last_fullsized":"2024-09-21T18:57:05.585686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.657330+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:57:05.657330+0000","last_peered":"2024-09-21T18:57:05.657330+0000","last_clean":"2024-09-21T18:57:05.657330+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:57:05.657330+0000","last_undegraded":"2024-09-21T18:57:05.657330+0000","last_fullsized":"2024-09-21T18:57:05.657330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641098+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:57:05.641098+0000","last_peered":"2024-09-21T18:57:05.641098+0000","last_clean":"2024-09-21T18:57:05.641098+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:57:05.641098+0000","last_undegraded":"2024-09-21T18:57:05.641098+0000","last_fullsized":"2024-09-21T18:57:05.641098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":180,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590526+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:57:05.590526+0000","last_peered":"2024-09-21T18:57:05.590526+0000","last_clean":"2024-09-21T18:57:05.590526+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:57:05.590526+0000","last_undegraded":"2024-09-21T18:57:05.590526+0000","last_fullsized":"2024-09-21T18:57:05.590526+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:22.890902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.641899+0000","last_change":"2024-09-21T18:55:51.523039+0000","last_active":"2024-09-21T18:57:05.641899+0000","last_peered":"2024-09-21T18:57:05.641899+0000","last_clean":"2024-09-21T18:57:05.641899+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:57:05.641899+0000","last_undegraded":"2024-09-21T18:57:05.641899+0000","last_fullsized":"2024-09-21T18:57:05.641899+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":225,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653286+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:57:05.653286+0000","last_peered":"2024-09-21T18:57:05.653286+0000","last_clean":"2024-09-21T18:57:05.653286+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:57:05.653286+0000","last_undegraded":"2024-09-21T18:57:05.653286+0000","last_fullsized":"2024-09-21T18:57:05.653286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.594353+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:57:05.594353+0000","last_peered":"2024-09-21T18:57:05.594353+0000","last_clean":"2024-09-21T18:57:05.594353+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:57:05.594353+0000","last_undegraded":"2024-09-21T18:57:05.594353+0000","last_fullsized":"2024-09-21T18:57:05.594353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":180,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.658479+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:57:05.658479+0000","last_peered":"2024-09-21T18:57:05.658479+0000","last_clean":"2024-09-21T18:57:05.658479+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:57:05.658479+0000","last_undegraded":"2024-09-21T18:57:05.658479+0000","last_fullsized":"2024-09-21T18:57:05.658479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586007+0000","last_change":"2024-09-21T18:54:40.089802+0000","last_active":"2024-09-21T18:57:05.586007+0000","last_peered":"2024-09-21T18:57:05.586007+0000","last_clean":"2024-09-21T18:57:05.586007+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:57:05.586007+0000","last_undegraded":"2024-09-21T18:57:05.586007+0000","last_fullsized":"2024-09-21T18:57:05.586007+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:40.089723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.089723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":238,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.590365+0000","last_change":"2024-09-21T18:55:51.543525+0000","last_active":"2024-09-21T18:57:05.590365+0000","last_peered":"2024-09-21T18:57:05.590365+0000","last_clean":"2024-09-21T18:57:05.590365+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:57:05.590365+0000","last_undegraded":"2024-09-21T18:57:05.590365+0000","last_fullsized":"2024-09-21T18:57:05.590365+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.924485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.644341+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:57:05.644341+0000","last_peered":"2024-09-21T18:57:05.644341+0000","last_clean":"2024-09-21T18:57:05.644341+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:57:05.644341+0000","last_undegraded":"2024-09-21T18:57:05.644341+0000","last_fullsized":"2024-09-21T18:57:05.644341+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":210,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593324+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:57:05.593324+0000","last_peered":"2024-09-21T18:57:05.593324+0000","last_clean":"2024-09-21T18:57:05.593324+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:57:05.593324+0000","last_undegraded":"2024-09-21T18:57:05.593324+0000","last_fullsized":"2024-09-21T18:57:05.593324+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:14.030615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.593812+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:57:05.593812+0000","last_peered":"2024-09-21T18:57:05.593812+0000","last_clean":"2024-09-21T18:57:05.593812+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:57:05.593812+0000","last_undegraded":"2024-09-21T18:57:05.593812+0000","last_fullsized":"2024-09-21T18:57:05.593812+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:19.966099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":160,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.586667+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:57:05.586667+0000","last_peered":"2024-09-21T18:57:05.586667+0000","last_clean":"2024-09-21T18:57:05.586667+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:57:05.586667+0000","last_undegraded":"2024-09-21T18:57:05.586667+0000","last_fullsized":"2024-09-21T18:57:05.586667+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.649554+0000","last_change":"2024-09-21T18:55:51.516509+0000","last_active":"2024-09-21T18:57:05.649554+0000","last_peered":"2024-09-21T18:57:05.649554+0000","last_clean":"2024-09-21T18:57:05.649554+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:57:05.649554+0000","last_undegraded":"2024-09-21T18:57:05.649554+0000","last_fullsized":"2024-09-21T18:57:05.649554+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":188,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.646218+0000","last_change":"2024-09-21T18:55:40.664189+0000","last_active":"2024-09-21T18:57:05.646218+0000","last_peered":"2024-09-21T18:57:05.646218+0000","last_clean":"2024-09-21T18:57:05.646218+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:57:05.646218+0000","last_undegraded":"2024-09-21T18:57:05.646218+0000","last_fullsized":"2024-09-21T18:57:05.646218+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:40.664110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.664110+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585825+0000","last_change":"2024-09-21T18:54:52.054723+0000","last_active":"2024-09-21T18:57:05.585825+0000","last_peered":"2024-09-21T18:57:05.585825+0000","last_clean":"2024-09-21T18:57:05.585825+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:57:05.585825+0000","last_undegraded":"2024-09-21T18:57:05.585825+0000","last_fullsized":"2024-09-21T18:57:05.585825+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:52.054655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.054655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":183,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.654330+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:57:05.654330+0000","last_peered":"2024-09-21T18:57:05.654330+0000","last_clean":"2024-09-21T18:57:05.654330+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:57:05.654330+0000","last_undegraded":"2024-09-21T18:57:05.654330+0000","last_fullsized":"2024-09-21T18:57:05.654330+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.640796+0000","last_change":"2024-09-21T18:55:51.536412+0000","last_active":"2024-09-21T18:57:05.640796+0000","last_peered":"2024-09-21T18:57:05.640796+0000","last_clean":"2024-09-21T18:57:05.640796+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:57:05.640796+0000","last_undegraded":"2024-09-21T18:57:05.640796+0000","last_fullsized":"2024-09-21T18:57:05.640796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653188+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:57:05.653188+0000","last_peered":"2024-09-21T18:57:05.653188+0000","last_clean":"2024-09-21T18:57:05.653188+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:57:05.653188+0000","last_undegraded":"2024-09-21T18:57:05.653188+0000","last_fullsized":"2024-09-21T18:57:05.653188+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":176,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.591174+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:57:05.591174+0000","last_peered":"2024-09-21T18:57:05.591174+0000","last_clean":"2024-09-21T18:57:05.591174+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:57:05.591174+0000","last_undegraded":"2024-09-21T18:57:05.591174+0000","last_fullsized":"2024-09-21T18:57:05.591174+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.912165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":167,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585082+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:57:05.585082+0000","last_peered":"2024-09-21T18:57:05.585082+0000","last_clean":"2024-09-21T18:57:05.585082+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:57:05.585082+0000","last_undegraded":"2024-09-21T18:57:05.585082+0000","last_fullsized":"2024-09-21T18:57:05.585082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.654455+0000","last_change":"2024-09-21T18:55:51.529791+0000","last_active":"2024-09-21T18:57:05.654455+0000","last_peered":"2024-09-21T18:57:05.654455+0000","last_clean":"2024-09-21T18:57:05.654455+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:57:05.654455+0000","last_undegraded":"2024-09-21T18:57:05.654455+0000","last_fullsized":"2024-09-21T18:57:05.654455+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.649520+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:57:05.649520+0000","last_peered":"2024-09-21T18:57:05.649520+0000","last_clean":"2024-09-21T18:57:05.649520+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:57:05.649520+0000","last_undegraded":"2024-09-21T18:57:05.649520+0000","last_fullsized":"2024-09-21T18:57:05.649520+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.650467+0000","last_change":"2024-09-21T18:55:51.529323+0000","last_active":"2024-09-21T18:57:05.650467+0000","last_peered":"2024-09-21T18:57:05.650467+0000","last_clean":"2024-09-21T18:57:05.650467+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:57:05.650467+0000","last_undegraded":"2024-09-21T18:57:05.650467+0000","last_fullsized":"2024-09-21T18:57:05.650467+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.592495+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:57:05.592495+0000","last_peered":"2024-09-21T18:57:05.592495+0000","last_clean":"2024-09-21T18:57:05.592495+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:57:05.592495+0000","last_undegraded":"2024-09-21T18:57:05.592495+0000","last_fullsized":"2024-09-21T18:57:05.592495+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.585439+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:57:05.585439+0000","last_peered":"2024-09-21T18:57:05.585439+0000","last_clean":"2024-09-21T18:57:05.585439+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:57:05.585439+0000","last_undegraded":"2024-09-21T18:57:05.585439+0000","last_fullsized":"2024-09-21T18:57:05.585439+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.192235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.648024+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:57:05.648024+0000","last_peered":"2024-09-21T18:57:05.648024+0000","last_clean":"2024-09-21T18:57:05.648024+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:57:05.648024+0000","last_undegraded":"2024-09-21T18:57:05.648024+0000","last_fullsized":"2024-09-21T18:57:05.648024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":191,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.653240+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:57:05.653240+0000","last_peered":"2024-09-21T18:57:05.653240+0000","last_clean":"2024-09-21T18:57:05.653240+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:57:05.653240+0000","last_undegraded":"2024-09-21T18:57:05.653240+0000","last_fullsized":"2024-09-21T18:57:05.653240+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.594525+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:57:05.594525+0000","last_peered":"2024-09-21T18:57:05.594525+0000","last_clean":"2024-09-21T18:57:05.594525+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:57:05.594525+0000","last_undegraded":"2024-09-21T18:57:05.594525+0000","last_fullsized":"2024-09-21T18:57:05.594525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":56,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T18:57:05.644259+0000","last_change":"2024-09-21T18:55:51.545494+0000","last_active":"2024-09-21T18:57:05.644259+0000","last_peered":"2024-09-21T18:57:05.644259+0000","last_clean":"2024-09-21T18:57:05.644259+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:57:05.644259+0000","last_undegraded":"2024-09-21T18:57:05.644259+0000","last_fullsized":"2024-09-21T18:57:05.644259+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4809293848,"num_objects":73385,"num_object_clones":0,"num_object_copies":220155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21475,"num_objects_unfound":0,"num_objects_dirty":73385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204609,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21143,"num_bytes_recovered":1385627648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15128342528,"data_stored":15128330312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85307,"ondisk_log_size":85307,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":228,"num_read_kb":223,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266740,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1915132,"kb_used_data":1785388,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91830020,"statfs":{"total":95995035648,"available":94033940480,"internally_reserved":0,"allocated":1828237312,"data_stored":1827870890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528377,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1906404,"kb_used_data":1780756,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91838748,"statfs":{"total":95995035648,"available":94042877952,"internally_reserved":0,"allocated":1823494144,"data_stored":1823144511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790012,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574192,"kb_used_data":1474972,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92170960,"statfs":{"total":95995035648,"available":94383063040,"internally_reserved":0,"allocated":1510371328,"data_stored":1510020227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018946,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1901000,"kb_used_data":1775408,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91844152,"statfs":{"total":95995035648,"available":94048411648,"internally_reserved":0,"allocated":1818017792,"data_stored":1817647230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3459604,"kb_used_data":3233728,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":90285548,"statfs":{"total":95995035648,"available":92452401152,"internally_reserved":0,"allocated":3311337472,"data_stored":3310950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":230947274},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1883640,"kb_used_data":1771876,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91861512,"statfs":{"total":95995035648,"available":94066188288,"internally_reserved":0,"allocated":1814401024,"data_stored":1814042286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2230160,"kb_used_data":2072572,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91514992,"statfs":{"total":95995035648,"available":93711351808,"internally_reserved":0,"allocated":2122313728,"data_stored":2121930755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1123076,"kb_used_data":979312,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92622076,"statfs":{"total":95995035648,"available":94845005824,"internally_reserved":0,"allocated":1002815488,"data_stored":1002445659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1006567424,"data_stored":1006567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2106068992,"data_stored":2106064920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1807351808,"data_stored":1807351808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3283030016,"data_stored":3283025944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1804730368,"data_stored":1804730368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1499463680,"data_stored":1499463680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1801392128,"data_stored":1801388056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1819738112,"data_stored":1819738112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:57:17.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:57:18.003 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:18.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.152 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:18.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:18.253 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:18.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:57:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:57:18.281 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:18.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:18.408 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:18.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:18.537 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:18.638 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:18.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:18.746 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:18.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:18 smithi081 ceph-mon[39192]: pgmap v179: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 922 op/s; 16606/193554 objects misplaced (8.580%); 15 MiB/s, 245 objects/s recovering 2024-09-21T18:57:18.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:18.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:18 smithi081 ceph-mon[29384]: pgmap v179: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 922 op/s; 16606/193554 objects misplaced (8.580%); 15 MiB/s, 245 objects/s recovering 2024-09-21T18:57:18.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:18.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:18.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:18.949 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:18 smithi104 ceph-mon[32438]: pgmap v179: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 922 op/s; 16606/193554 objects misplaced (8.580%); 15 MiB/s, 245 objects/s recovering 2024-09-21T18:57:18.950 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:18.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T18:57:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:18.978 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:18.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:19.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:19.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:19.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:19.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:57:19.240 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:19.293 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:19.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:19.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:19.505 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:57:19.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:19.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:19.662 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:19.763 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:19.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T18:57:19.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T18:57:19.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:57:19.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:57:19.772 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T18:57:19.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T18:57:19.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3805180183' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:19.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: from='client.25009 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:19.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:57:19.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: from='client.25009 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:19.773 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[39192]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T18:57:19.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3805180183' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: from='client.25009 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: from='client.25009 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:19.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:19 smithi081 ceph-mon[29384]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T18:57:19.780 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:57:19.800 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102355 7.4 e105/82 7:2a73ca6f:::benchmark_data_smithi081_73391_object102354:head v 105'6448, mlcod=105'6446)", 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.327165+0000", 2024-09-21T18:57:19.839 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.506551085, 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16628193299999999, 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327165+0000", 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327165+0000", 2024-09-21T18:57:19.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327166+0000", 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5990000000000001e-06 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327197+0000", 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1355e-05 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327209+0000", 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1109e-05 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327217+0000", 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1960000000000008e-06 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327305+0000", 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7937999999999997e-05 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327323+0000", 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8034999999999999e-05 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493402+0000", 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.166079438 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493446+0000", 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4263e-05 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102359 7.c 7:3c684ab9:::benchmark_data_smithi081_73391_object102358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.329664+0000", 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.504051471, 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16395372499999999, 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102359 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329664+0000", 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329664+0000", 2024-09-21T18:57:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329666+0000", 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.477e-06 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329703+0000", 2024-09-21T18:57:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7696e-05 2024-09-21T18:57:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329715+0000", 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1430000000000001e-05 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329722+0000", 2024-09-21T18:57:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4100000000000002e-06 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329782+0000", 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9558000000000001e-05 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329907+0000", 2024-09-21T18:57:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554600000000001 2024-09-21T18:57:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330061+0000", 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015371300000000001 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337366+0000", 2024-09-21T18:57:19.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073049380000000004 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493489+0000", 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15612330799999999 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493524+0000", 2024-09-21T18:57:19.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4567999999999998e-05 2024-09-21T18:57:19.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493575+0000", 2024-09-21T18:57:19.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1141999999999999e-05 2024-09-21T18:57:19.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493618+0000", 2024-09-21T18:57:19.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2938999999999997e-05 2024-09-21T18:57:19.855 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.855 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.855 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.856 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102362 7.d 7:b4fc79ed:::benchmark_data_smithi081_73391_object102361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:57:19.856 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.330143+0000", 2024-09-21T18:57:19.856 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.503572803, 2024-09-21T18:57:19.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16063384, 2024-09-21T18:57:19.856 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.856 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102362 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.857 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330143+0000", 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330143+0000", 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330144+0000", 2024-09-21T18:57:19.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.361e-06 2024-09-21T18:57:19.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330179+0000", 2024-09-21T18:57:19.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4944000000000003e-05 2024-09-21T18:57:19.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330189+0000", 2024-09-21T18:57:19.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9639999999999998e-06 2024-09-21T18:57:19.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330196+0000", 2024-09-21T18:57:19.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5259999999999999e-06 2024-09-21T18:57:19.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333790+0000", 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035948120000000002 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333921+0000", 2024-09-21T18:57:19.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000131051 2024-09-21T18:57:19.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:19.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.334318+0000", 2024-09-21T18:57:19.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000396463 2024-09-21T18:57:19.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337942+0000", 2024-09-21T18:57:19.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003624132 2024-09-21T18:57:19.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489748+0000", 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15180602900000001 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490678+0000", 2024-09-21T18:57:19.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093001499999999996 2024-09-21T18:57:19.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490730+0000", 2024-09-21T18:57:19.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1604000000000003e-05 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490777+0000", 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6938999999999999e-05 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.868 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102363 7.c 7:396ad605:::benchmark_data_smithi081_73391_object102362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.503508048, 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.165470859, 2024-09-21T18:57:19.869 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102363 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.870 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3300000000000002e-07 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330233+0000", 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5241000000000001e-05 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330239+0000", 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9950000000000002e-06 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330244+0000", 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8119999999999998e-06 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333287+0000", 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030428870000000002 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333401+0000", 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113414 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333571+0000", 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017085599999999999 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.492806+0000", 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15923499299999999 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493628+0000", 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082127600000000002 2024-09-21T18:57:19.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495595+0000", 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001967445 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495642+0000", 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7024000000000002e-05 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495678+0000", 2024-09-21T18:57:19.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6282999999999998e-05 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102361 7.4 e105/82 7:2b82beac:::benchmark_data_smithi081_73391_object102360:head v 105'6449, mlcod=105'6446)", 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333737+0000", 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.499978394, 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.162036767, 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333737+0000", 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333737+0000", 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333739+0000", 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333760+0000", 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1755000000000001e-05 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333770+0000", 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9699999999999994e-06 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333777+0000", 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3740000000000003e-06 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335178+0000", 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014012409999999999 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335197+0000", 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8695e-05 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495739+0000", 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16054214 2024-09-21T18:57:19.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495774+0000", 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5228000000000002e-05 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102367 7.4 e105/82 7:2bc3d420:::benchmark_data_smithi081_73391_object102366:head v 105'6450, mlcod=105'6446)", 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333834+0000", 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.499881317, 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16196540800000001, 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.881 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333834+0000", 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333834+0000", 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333835+0000", 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6399999999999998e-07 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333854+0000", 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9420999999999999e-05 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333859+0000", 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1490000000000002e-06 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333863+0000", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.16e-06 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337397+0000", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00353382 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337414+0000", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7051e-05 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495777+0000", 2024-09-21T18:57:19.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.158363108 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495800+0000", 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2334999999999999e-05 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102368 7.4 e105/82 7:29e74c0a:::benchmark_data_smithi081_73391_object102367:head v 105'6451, mlcod=105'6446)", 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.336323+0000", 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.49739298199999, 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15951227800000001, 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.885 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336323+0000", 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336323+0000", 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336324+0000", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3260000000000001e-06 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336346+0000", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1977e-05 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336355+0000", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3090000000000001e-06 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336361+0000", 2024-09-21T18:57:19.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1689999999999997e-06 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.491302+0000", 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.154940193 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.491320+0000", 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8844000000000001e-05 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495810+0000", 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004489537 2024-09-21T18:57:19.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495835+0000", 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4922999999999998e-05 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102371 7.7 7:e4277e62:::benchmark_data_smithi081_73391_object102370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.337854+0000", 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.49586167699999, 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15779875900000001, 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102371 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337854+0000", 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337854+0000", 2024-09-21T18:57:19.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337855+0000", 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5239999999999999e-06 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337887+0000", 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1814999999999999e-05 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337895+0000", 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6639999999999998e-06 2024-09-21T18:57:19.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337900+0000", 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.48e-06 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338644+0000", 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074344799999999996 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338755+0000", 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011141099999999999 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489524+0000", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15076826500000001 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495470+0000", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059459880000000001 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495536+0000", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6666000000000002e-05 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495551+0000", 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5211000000000001e-05 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495608+0000", 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6668e-05 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495653+0000", 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4619000000000001e-05 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102375 7.c 7:3f5265de:::benchmark_data_smithi081_73391_object102374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.338287+0000", 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.49542836000001, 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15767785000000001, 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102375 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338287+0000", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338287+0000", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338288+0000", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.029e-06 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338354+0000", 2024-09-21T18:57:19.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5264000000000004e-05 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338362+0000", 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.524e-06 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338369+0000", 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.421e-06 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489653+0000", 2024-09-21T18:57:19.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15128423099999999 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489791+0000", 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013800800000000001 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489995+0000", 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000203773 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495713+0000", 2024-09-21T18:57:19.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057185710000000004 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495809+0000", 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5754000000000005e-05 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495889+0000", 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9679999999999996e-05 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495934+0000", 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5609000000000003e-05 2024-09-21T18:57:19.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495965+0000", 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0985999999999998e-05 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop_reply(client.34742.0:102362 7.d e105/82 ondisk, result = 0)", 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.338914+0000", 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.49480149999999, 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15084166600000001, 2024-09-21T18:57:19.900 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338914+0000", 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338914+0000", 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338915+0000", 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2810000000000001e-06 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338920+0000", 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3749999999999996e-06 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489604+0000", 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150684605 2024-09-21T18:57:19.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489671+0000", 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6698000000000004e-05 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489735+0000", 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4158999999999993e-05 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489742+0000", 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2150000000000004e-06 2024-09-21T18:57:19.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489756+0000", 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3332999999999999e-05 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102370 7.4 e105/82 7:261d1db0:::benchmark_data_smithi081_73391_object102369:head v 105'6452, mlcod=105'6446)", 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.339637+0000", 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 119.49407907699999, 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15622318700000001, 2024-09-21T18:57:19.904 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339637+0000", 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339637+0000", 2024-09-21T18:57:19.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339638+0000", 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1790000000000001e-06 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339663+0000", 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5381999999999999e-05 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489911+0000", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15024772 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489917+0000", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1090000000000003e-06 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493494+0000", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035770260000000001 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493514+0000", 2024-09-21T18:57:19.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9647999999999999e-05 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495837+0000", 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002323215 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495860+0000", 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2908000000000002e-05 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136121 7.c 7:3d398460:::benchmark_data_smithi081_73391_object136120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.155866+0000", 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.677849369, 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13614505299999999, 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136121 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155866+0000", 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155866+0000", 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155868+0000", 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9180000000000001e-06 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155929+0000", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0381999999999999e-05 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155939+0000", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0815999999999999e-05 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155946+0000", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9920000000000001e-06 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155991+0000", 2024-09-21T18:57:19.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4379999999999999e-05 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.156091+0000", 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010033 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.156182+0000", 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1057999999999995e-05 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.158936+0000", 2024-09-21T18:57:19.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027542299999999999 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159467+0000", 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053071599999999998 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291914+0000", 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13244737400000001 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291965+0000", 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0878999999999999e-05 2024-09-21T18:57:19.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292011+0000", 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5977999999999999e-05 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136123 7.c 7:3f91197c:::benchmark_data_smithi081_73391_object136122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.159407+0000", 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.674309034999993, 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13406175400000001, 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.914 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136123 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159407+0000", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159407+0000", 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159408+0000", 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.35e-06 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159468+0000", 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0106999999999999e-05 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159476+0000", 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7999999999999999e-06 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159480+0000", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6140000000000002e-06 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159515+0000", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4264000000000003e-05 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159606+0000", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1194999999999997e-05 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159691+0000", 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4628000000000004e-05 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162301+0000", 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002610804 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162478+0000", 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017648300000000001 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293380+0000", 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.130902345 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293430+0000", 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9713999999999999e-05 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293468+0000", 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8449999999999999e-05 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.919 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:136125 7.b e112/79 7:df011645:::benchmark_data_smithi081_73391_object136124:head v 112'8371, mlcod=112'8369)", 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.161637+0000", 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.672078876, 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.135163383, 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161637+0000", 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161637+0000", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161638+0000", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161761+0000", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012348200000000001 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161767+0000", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8560000000000003e-06 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161771+0000", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4909999999999999e-06 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161867+0000", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5877000000000004e-05 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.161885+0000", 2024-09-21T18:57:19.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7322000000000001e-05 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.296758+0000", 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13487303 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.296800+0000", 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2428000000000001e-05 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.923 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136131 7.c 7:3956c63c:::benchmark_data_smithi081_73391_object136130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.162650+0000", 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.671065630000001, 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13093136899999999, 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136131 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162650+0000", 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162650+0000", 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162651+0000", 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.451e-06 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162698+0000", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6567000000000003e-05 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162714+0000", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5764e-05 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162720+0000", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1920000000000002e-06 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162776+0000", 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6233999999999997e-05 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162882+0000", 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106031 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163012+0000", 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000129596 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291836+0000", 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12882446 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293318+0000", 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001481771 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293500+0000", 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018201400000000001 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293551+0000", 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0498000000000003e-05 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293581+0000", 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0790999999999997e-05 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136133 7.c 7:3f5a1e61:::benchmark_data_smithi081_73391_object136132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.162758+0000", 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.670957951999995, 2024-09-21T18:57:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.130927449, 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136133 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162758+0000", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162758+0000", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162759+0000", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4900000000000005e-07 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162788+0000", 2024-09-21T18:57:19.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9621999999999999e-05 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162795+0000", 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8109999999999997e-06 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162800+0000", 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1889999999999998e-06 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.165817+0000", 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003016838 2024-09-21T18:57:19.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.165906+0000", 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8929000000000002e-05 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166071+0000", 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016516 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292161+0000", 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12608985 2024-09-21T18:57:19.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293351+0000", 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00118983 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293614+0000", 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026280200000000002 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293656+0000", 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2642000000000003e-05 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293685+0000", 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8926999999999999e-05 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136135 7.7 7:e5c64189:::benchmark_data_smithi081_73391_object136134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.162823+0000", 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.670892551999998, 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.130175026, 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.935 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136135 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162823+0000", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162823+0000", 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162824+0000", 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7499999999999999e-07 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162900+0000", 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5754999999999996e-05 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162906+0000", 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8390000000000001e-06 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162912+0000", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1850000000000003e-06 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162967+0000", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5174999999999999e-05 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163078+0000", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111599 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:57:19.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163214+0000", 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013555800000000001 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291915+0000", 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12870121700000001 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292205+0000", 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028952699999999998 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292908+0000", 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00070294699999999999 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292958+0000", 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0649999999999998e-05 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292998+0000", 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9799000000000003e-05 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop_reply(client.34742.0:136121 7.c e112/79 ondisk, result = 0)", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.163903+0000", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.669812293000007, 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12811572700000001, 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163903+0000", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163903+0000", 2024-09-21T18:57:19.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163904+0000", 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2900000000000002e-07 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163908+0000", 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0339999999999998e-06 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163917+0000", 2024-09-21T18:57:19.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0049999999999993e-06 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163923+0000", 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1929999999999998e-06 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291902+0000", 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12797826700000001 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291909+0000", 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2429999999999999e-06 2024-09-21T18:57:19.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292019+0000", 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011015599999999999 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:136126 7.b e112/79 7:d13dafa2:::benchmark_data_smithi081_73391_object136125:head v 112'8372, mlcod=112'8369)", 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.166046+0000", 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.667669515, 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.133113181, 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.944 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166046+0000", 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166046+0000", 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166047+0000", 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.223e-06 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166095+0000", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7735000000000003e-05 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166104+0000", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8650000000000007e-06 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166110+0000", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2099999999999998e-06 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166196+0000", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5686999999999995e-05 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291935+0000", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12573884900000001 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.299128+0000", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0071933220000000003 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.299159+0000", 2024-09-21T18:57:19.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.129e-05 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:136128 7.b e112/79 7:df396259:::benchmark_data_smithi081_73391_object136127:head v 112'8373, mlcod=112'8369)", 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.167598+0000", 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 91.666117919000001, 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13380571999999999, 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.167598+0000", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.167598+0000", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.167599+0000", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1349999999999999e-06 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.167658+0000", 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9074999999999999e-05 2024-09-21T18:57:19.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291853+0000", 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.124195026 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291861+0000", 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7209999999999994e-06 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.294558+0000", 2024-09-21T18:57:19.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026976909999999999 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.294580+0000", 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1407999999999999e-05 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.301316+0000", 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067362350000000001 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.301403+0000", 2024-09-21T18:57:19.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7429000000000006e-05 2024-09-21T18:57:19.952 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.952 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.952 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.952 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:19.952 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:19.952 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:57:19.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:19.974 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:19.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:20.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3805180183' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: from='client.25009 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: from='client.25009 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:19 smithi104 ceph-mon[32438]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T18:57:20.076 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:20.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:20.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:20.227 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:20.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:20.353 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:20.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:20.476 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:20.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:20.599 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:20.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:20.727 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:20.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:20 smithi104 ceph-mon[32438]: pgmap v180: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 907 op/s; 16606/193554 objects misplaced (8.580%); 15 MiB/s, 241 objects/s recovering 2024-09-21T18:57:20.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:20 smithi104 ceph-mon[32438]: from='client.35039 v1:172.21.15.81:0/2393362502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:20.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:20 smithi104 ceph-mon[32438]: 3.1c scrub starts 2024-09-21T18:57:20.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:20 smithi104 ceph-mon[32438]: 3.1c scrub ok 2024-09-21T18:57:20.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:20 smithi104 ceph-mon[32438]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T18:57:20.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:20.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:20.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T18:57:20.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:21.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[29384]: pgmap v180: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 907 op/s; 16606/193554 objects misplaced (8.580%); 15 MiB/s, 241 objects/s recovering 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[29384]: from='client.35039 v1:172.21.15.81:0/2393362502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[29384]: 3.1c scrub starts 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[29384]: 3.1c scrub ok 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[29384]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[39192]: pgmap v180: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 907 op/s; 16606/193554 objects misplaced (8.580%); 15 MiB/s, 241 objects/s recovering 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[39192]: from='client.35039 v1:172.21.15.81:0/2393362502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[39192]: 3.1c scrub starts 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[39192]: 3.1c scrub ok 2024-09-21T18:57:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:20 smithi081 ceph-mon[39192]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T18:57:21.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:21.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:21.294 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:21.300 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:21.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:21.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:21.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:21.460 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-21T18:57:19.495624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4215740979,"num_objects":64518,"num_object_clones":0,"num_object_copies":193554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16606,"num_objects_unfound":0,"num_objects_dirty":64518,"num_whiteouts":0,"num_read":1082,"num_read_kb":939,"num_write":214224,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23688,"num_bytes_recovered":1545739738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86231,"ondisk_log_size":86231,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14379112,"kb_used_data":13218728,"kb_used_omap":12,"kb_used_meta":1160051,"kb_avail":735582104,"statfs":{"total":767960285184,"available":753236074496,"internally_reserved":0,"allocated":13535977472,"data_stored":13532954113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1187892788},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-594149376,"num_objects":-9066,"num_object_clones":0,"num_object_copies":-27198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4869,"num_objects_unfound":0,"num_objects_dirty":-9066,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9066,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":158334976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.886901"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625562+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:57:09.625562+0000","last_peered":"2024-09-21T18:57:09.625562+0000","last_clean":"2024-09-21T18:57:09.625562+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:57:09.625562+0000","last_undegraded":"2024-09-21T18:57:09.625562+0000","last_fullsized":"2024-09-21T18:57:09.625562+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":209,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628378+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:57:09.628378+0000","last_peered":"2024-09-21T18:57:09.628378+0000","last_clean":"2024-09-21T18:57:09.628378+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:57:09.628378+0000","last_undegraded":"2024-09-21T18:57:09.628378+0000","last_fullsized":"2024-09-21T18:57:09.628378+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623743+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:57:09.623743+0000","last_peered":"2024-09-21T18:57:09.623743+0000","last_clean":"2024-09-21T18:57:09.623743+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:57:09.623743+0000","last_undegraded":"2024-09-21T18:57:09.623743+0000","last_fullsized":"2024-09-21T18:57:09.623743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":160,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629225+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:57:09.629225+0000","last_peered":"2024-09-21T18:57:09.629225+0000","last_clean":"2024-09-21T18:57:09.629225+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:57:09.629225+0000","last_undegraded":"2024-09-21T18:57:09.629225+0000","last_fullsized":"2024-09-21T18:57:09.629225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":169,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622446+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:57:09.622446+0000","last_peered":"2024-09-21T18:57:09.622446+0000","last_clean":"2024-09-21T18:57:09.622446+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:57:09.622446+0000","last_undegraded":"2024-09-21T18:57:09.622446+0000","last_fullsized":"2024-09-21T18:57:09.622446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":182,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.642080+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:57:09.642080+0000","last_peered":"2024-09-21T18:57:09.642080+0000","last_clean":"2024-09-21T18:57:09.642080+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:57:09.642080+0000","last_undegraded":"2024-09-21T18:57:09.642080+0000","last_fullsized":"2024-09-21T18:57:09.642080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":240,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624083+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:57:09.624083+0000","last_peered":"2024-09-21T18:57:09.624083+0000","last_clean":"2024-09-21T18:57:09.624083+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:57:09.624083+0000","last_undegraded":"2024-09-21T18:57:09.624083+0000","last_fullsized":"2024-09-21T18:57:09.624083+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628232+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:57:09.628232+0000","last_peered":"2024-09-21T18:57:09.628232+0000","last_clean":"2024-09-21T18:57:09.628232+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:57:09.628232+0000","last_undegraded":"2024-09-21T18:57:09.628232+0000","last_fullsized":"2024-09-21T18:57:09.628232+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":180,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:15.761415+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:57:15.761415+0000","last_peered":"2024-09-21T18:57:15.761415+0000","last_clean":"2024-09-21T18:57:15.761415+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:57:15.761415+0000","last_undegraded":"2024-09-21T18:57:15.761415+0000","last_fullsized":"2024-09-21T18:57:15.761415+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:22.885755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":234,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629550+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:57:09.629550+0000","last_peered":"2024-09-21T18:57:09.629550+0000","last_clean":"2024-09-21T18:57:09.629550+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:57:09.629550+0000","last_undegraded":"2024-09-21T18:57:09.629550+0000","last_fullsized":"2024-09-21T18:57:09.629550+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":194,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623389+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:57:09.623389+0000","last_peered":"2024-09-21T18:57:09.623389+0000","last_clean":"2024-09-21T18:57:09.623389+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:57:09.623389+0000","last_undegraded":"2024-09-21T18:57:09.623389+0000","last_fullsized":"2024-09-21T18:57:09.623389+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.948408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.610677+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:57:09.610677+0000","last_peered":"2024-09-21T18:57:09.610677+0000","last_clean":"2024-09-21T18:57:09.610677+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:57:09.610677+0000","last_undegraded":"2024-09-21T18:57:09.610677+0000","last_fullsized":"2024-09-21T18:57:09.610677+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623134+0000","last_change":"2024-09-21T18:53:07.401768+0000","last_active":"2024-09-21T18:57:09.623134+0000","last_peered":"2024-09-21T18:57:09.623134+0000","last_clean":"2024-09-21T18:57:09.623134+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:57:09.623134+0000","last_undegraded":"2024-09-21T18:57:09.623134+0000","last_fullsized":"2024-09-21T18:57:09.623134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:15.988901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":186,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629559+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:57:09.629559+0000","last_peered":"2024-09-21T18:57:09.629559+0000","last_clean":"2024-09-21T18:57:09.629559+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:57:09.629559+0000","last_undegraded":"2024-09-21T18:57:09.629559+0000","last_fullsized":"2024-09-21T18:57:09.629559+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:13.008471+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:57:13.008471+0000","last_peered":"2024-09-21T18:57:13.008471+0000","last_clean":"2024-09-21T18:57:13.008471+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:57:13.008471+0000","last_undegraded":"2024-09-21T18:57:13.008471+0000","last_fullsized":"2024-09-21T18:57:13.008471+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:16.165478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":176,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622928+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:57:09.622928+0000","last_peered":"2024-09-21T18:57:09.622928+0000","last_clean":"2024-09-21T18:57:09.622928+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:57:09.622928+0000","last_undegraded":"2024-09-21T18:57:09.622928+0000","last_fullsized":"2024-09-21T18:57:09.622928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:46.401461+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":169,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628442+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:57:09.628442+0000","last_peered":"2024-09-21T18:57:09.628442+0000","last_clean":"2024-09-21T18:57:09.628442+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:57:09.628442+0000","last_undegraded":"2024-09-21T18:57:09.628442+0000","last_fullsized":"2024-09-21T18:57:09.628442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.615952+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:57:09.615952+0000","last_peered":"2024-09-21T18:57:09.615952+0000","last_clean":"2024-09-21T18:57:09.615952+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:57:09.615952+0000","last_undegraded":"2024-09-21T18:57:09.615952+0000","last_fullsized":"2024-09-21T18:57:09.615952+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":201,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625843+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:57:09.625843+0000","last_peered":"2024-09-21T18:57:09.625843+0000","last_clean":"2024-09-21T18:57:09.625843+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:57:09.625843+0000","last_undegraded":"2024-09-21T18:57:09.625843+0000","last_fullsized":"2024-09-21T18:57:09.625843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":178,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624238+0000","last_change":"2024-09-21T18:55:51.536009+0000","last_active":"2024-09-21T18:57:09.624238+0000","last_peered":"2024-09-21T18:57:09.624238+0000","last_clean":"2024-09-21T18:57:09.624238+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:57:09.624238+0000","last_undegraded":"2024-09-21T18:57:09.624238+0000","last_fullsized":"2024-09-21T18:57:09.624238+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":161,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622149+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:57:09.622149+0000","last_peered":"2024-09-21T18:57:09.622149+0000","last_clean":"2024-09-21T18:57:09.622149+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:57:09.622149+0000","last_undegraded":"2024-09-21T18:57:09.622149+0000","last_fullsized":"2024-09-21T18:57:09.622149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":188,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629696+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:57:09.629696+0000","last_peered":"2024-09-21T18:57:09.629696+0000","last_clean":"2024-09-21T18:57:09.629696+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:57:09.629696+0000","last_undegraded":"2024-09-21T18:57:09.629696+0000","last_fullsized":"2024-09-21T18:57:09.629696+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:19.550236+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":200,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623188+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:57:09.623188+0000","last_peered":"2024-09-21T18:57:09.623188+0000","last_clean":"2024-09-21T18:57:09.623188+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:57:09.623188+0000","last_undegraded":"2024-09-21T18:57:09.623188+0000","last_fullsized":"2024-09-21T18:57:09.623188+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628180+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:57:09.628180+0000","last_peered":"2024-09-21T18:57:09.628180+0000","last_clean":"2024-09-21T18:57:09.628180+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:57:09.628180+0000","last_undegraded":"2024-09-21T18:57:09.628180+0000","last_fullsized":"2024-09-21T18:57:09.628180+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":176,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:14.035912+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:57:14.035912+0000","last_peered":"2024-09-21T18:57:14.035912+0000","last_clean":"2024-09-21T18:57:14.035912+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:57:14.035912+0000","last_undegraded":"2024-09-21T18:57:14.035912+0000","last_fullsized":"2024-09-21T18:57:14.035912+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":181,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623824+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:57:09.623824+0000","last_peered":"2024-09-21T18:57:09.623824+0000","last_clean":"2024-09-21T18:57:09.623824+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:57:09.623824+0000","last_undegraded":"2024-09-21T18:57:09.623824+0000","last_fullsized":"2024-09-21T18:57:09.623824+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":72,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.618212+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:57:09.618212+0000","last_peered":"2024-09-21T18:57:09.618212+0000","last_clean":"2024-09-21T18:57:09.618212+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:57:09.618212+0000","last_undegraded":"2024-09-21T18:57:09.618212+0000","last_fullsized":"2024-09-21T18:57:09.618212+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":232,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.620220+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:57:09.620220+0000","last_peered":"2024-09-21T18:57:09.620220+0000","last_clean":"2024-09-21T18:57:09.620220+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:57:09.620220+0000","last_undegraded":"2024-09-21T18:57:09.620220+0000","last_fullsized":"2024-09-21T18:57:09.620220+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"138'13182","reported_seq":13313,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.457063+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:57:16.457063+0000","last_peered":"2024-09-21T18:57:16.457063+0000","last_clean":"2024-09-21T18:57:16.457063+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:57:16.457063+0000","last_undegraded":"2024-09-21T18:57:16.457063+0000","last_fullsized":"2024-09-21T18:57:16.457063+0000","mapping_epoch":79,"log_start":"107'7000","ondisk_log_start":"107'7000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257949696,"num_objects":3936,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13182,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":201,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:15.119607+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:57:15.119607+0000","last_peered":"2024-09-21T18:57:15.119607+0000","last_clean":"2024-09-21T18:57:15.119607+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:57:15.119607+0000","last_undegraded":"2024-09-21T18:57:15.119607+0000","last_fullsized":"2024-09-21T18:57:15.119607+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.223966+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":181,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623483+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:57:09.623483+0000","last_peered":"2024-09-21T18:57:09.623483+0000","last_clean":"2024-09-21T18:57:09.623483+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:57:09.623483+0000","last_undegraded":"2024-09-21T18:57:09.623483+0000","last_fullsized":"2024-09-21T18:57:09.623483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":252,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628552+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:57:09.628552+0000","last_peered":"2024-09-21T18:57:09.628552+0000","last_clean":"2024-09-21T18:57:09.628552+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:57:09.628552+0000","last_undegraded":"2024-09-21T18:57:09.628552+0000","last_fullsized":"2024-09-21T18:57:09.628552+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":190,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623152+0000","last_change":"2024-09-21T18:56:37.863087+0000","last_active":"2024-09-21T18:57:09.623152+0000","last_peered":"2024-09-21T18:57:09.623152+0000","last_clean":"2024-09-21T18:57:09.623152+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:57:09.623152+0000","last_undegraded":"2024-09-21T18:57:09.623152+0000","last_fullsized":"2024-09-21T18:57:09.623152+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:37.863026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:37.863026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.972736+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"138'13258","reported_seq":13389,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:17.038499+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:57:17.038499+0000","last_peered":"2024-09-21T18:57:17.038499+0000","last_clean":"2024-09-21T18:57:17.038499+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:57:17.038499+0000","last_undegraded":"2024-09-21T18:57:17.038499+0000","last_fullsized":"2024-09-21T18:57:17.038499+0000","mapping_epoch":79,"log_start":"116'8700","ondisk_log_start":"116'8700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258473984,"num_objects":3944,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13258,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":260,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623307+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:57:09.623307+0000","last_peered":"2024-09-21T18:57:09.623307+0000","last_clean":"2024-09-21T18:57:09.623307+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:57:09.623307+0000","last_undegraded":"2024-09-21T18:57:09.623307+0000","last_fullsized":"2024-09-21T18:57:09.623307+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":195,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623007+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:57:09.623007+0000","last_peered":"2024-09-21T18:57:09.623007+0000","last_clean":"2024-09-21T18:57:09.623007+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:57:09.623007+0000","last_undegraded":"2024-09-21T18:57:09.623007+0000","last_fullsized":"2024-09-21T18:57:09.623007+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628300+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:57:09.628300+0000","last_peered":"2024-09-21T18:57:09.628300+0000","last_clean":"2024-09-21T18:57:09.628300+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:57:09.628300+0000","last_undegraded":"2024-09-21T18:57:09.628300+0000","last_fullsized":"2024-09-21T18:57:09.628300+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:32.288577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":160,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629452+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:57:09.629452+0000","last_peered":"2024-09-21T18:57:09.629452+0000","last_clean":"2024-09-21T18:57:09.629452+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:57:09.629452+0000","last_undegraded":"2024-09-21T18:57:09.629452+0000","last_fullsized":"2024-09-21T18:57:09.629452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:15.982856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"138'13504","reported_seq":13635,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.958799+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:57:16.958799+0000","last_peered":"2024-09-21T18:57:16.958799+0000","last_clean":"2024-09-21T18:57:16.958799+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:57:16.958799+0000","last_undegraded":"2024-09-21T18:57:16.958799+0000","last_fullsized":"2024-09-21T18:57:16.958799+0000","mapping_epoch":79,"log_start":"117'9000","ondisk_log_start":"117'9000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261357568,"num_objects":3988,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13504,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":198,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628557+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:57:09.628557+0000","last_peered":"2024-09-21T18:57:09.628557+0000","last_clean":"2024-09-21T18:57:09.628557+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:57:09.628557+0000","last_undegraded":"2024-09-21T18:57:09.628557+0000","last_fullsized":"2024-09-21T18:57:09.628557+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:21.938392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":64,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.618471+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:57:09.618471+0000","last_peered":"2024-09-21T18:57:09.618471+0000","last_clean":"2024-09-21T18:57:09.618471+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:57:09.618471+0000","last_undegraded":"2024-09-21T18:57:09.618471+0000","last_fullsized":"2024-09-21T18:57:09.618471+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":224,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:12.040328+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:57:12.040328+0000","last_peered":"2024-09-21T18:57:12.040328+0000","last_clean":"2024-09-21T18:57:12.040328+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:57:12.040328+0000","last_undegraded":"2024-09-21T18:57:12.040328+0000","last_fullsized":"2024-09-21T18:57:12.040328+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":159,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627391+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:57:09.627391+0000","last_peered":"2024-09-21T18:57:09.627391+0000","last_clean":"2024-09-21T18:57:09.627391+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:57:09.627391+0000","last_undegraded":"2024-09-21T18:57:09.627391+0000","last_fullsized":"2024-09-21T18:57:09.627391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.105131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"138'13281","reported_seq":13502,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.444863+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:57:16.444863+0000","last_peered":"2024-09-21T18:57:16.444863+0000","last_clean":"2024-09-21T18:57:16.444863+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:57:16.444863+0000","last_undegraded":"2024-09-21T18:57:16.444863+0000","last_fullsized":"2024-09-21T18:57:16.444863+0000","mapping_epoch":79,"log_start":"112'8500","ondisk_log_start":"112'8500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:25.602584+0000","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260898816,"num_objects":3981,"num_object_clones":0,"num_object_copies":11943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13281,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":183,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624257+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:57:09.624257+0000","last_peered":"2024-09-21T18:57:09.624257+0000","last_clean":"2024-09-21T18:57:09.624257+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:57:09.624257+0000","last_undegraded":"2024-09-21T18:57:09.624257+0000","last_fullsized":"2024-09-21T18:57:09.624257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.091213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":188,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623609+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:57:09.623609+0000","last_peered":"2024-09-21T18:57:09.623609+0000","last_clean":"2024-09-21T18:57:09.623609+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:57:09.623609+0000","last_undegraded":"2024-09-21T18:57:09.623609+0000","last_fullsized":"2024-09-21T18:57:09.623609+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.895558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":81,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.615909+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:57:09.615909+0000","last_peered":"2024-09-21T18:57:09.615909+0000","last_clean":"2024-09-21T18:57:09.615909+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:57:09.615909+0000","last_undegraded":"2024-09-21T18:57:09.615909+0000","last_fullsized":"2024-09-21T18:57:09.615909+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.610496+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:57:09.610496+0000","last_peered":"2024-09-21T18:57:09.610496+0000","last_clean":"2024-09-21T18:57:09.610496+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:57:09.610496+0000","last_undegraded":"2024-09-21T18:57:09.610496+0000","last_fullsized":"2024-09-21T18:57:09.610496+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"138'13567","reported_seq":13847,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.447697+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:57:16.447697+0000","last_peered":"2024-09-21T18:57:16.447697+0000","last_clean":"2024-09-21T18:57:16.447697+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:57:16.447697+0000","last_undegraded":"2024-09-21T18:57:16.447697+0000","last_fullsized":"2024-09-21T18:57:16.447697+0000","mapping_epoch":79,"log_start":"110'8000","ondisk_log_start":"110'8000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:43.426801+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268107776,"num_objects":4091,"num_object_clones":0,"num_object_copies":12273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13567,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":78,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.616516+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:57:09.616516+0000","last_peered":"2024-09-21T18:57:09.616516+0000","last_clean":"2024-09-21T18:57:09.616516+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:57:09.616516+0000","last_undegraded":"2024-09-21T18:57:09.616516+0000","last_fullsized":"2024-09-21T18:57:09.616516+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":180,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624991+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:57:09.624991+0000","last_peered":"2024-09-21T18:57:09.624991+0000","last_clean":"2024-09-21T18:57:09.624991+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:57:09.624991+0000","last_undegraded":"2024-09-21T18:57:09.624991+0000","last_fullsized":"2024-09-21T18:57:09.624991+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":185,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.619561+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:57:09.619561+0000","last_peered":"2024-09-21T18:57:09.619561+0000","last_clean":"2024-09-21T18:57:09.619561+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:57:09.619561+0000","last_undegraded":"2024-09-21T18:57:09.619561+0000","last_fullsized":"2024-09-21T18:57:09.619561+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:13.102974+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:57:13.102974+0000","last_peered":"2024-09-21T18:57:13.102974+0000","last_clean":"2024-09-21T18:57:13.102974+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:57:13.102974+0000","last_undegraded":"2024-09-21T18:57:13.102974+0000","last_fullsized":"2024-09-21T18:57:13.102974+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.637336+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"138'13321","reported_seq":17078,"reported_epoch":138,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:16.468548+0000","last_change":"2024-09-21T18:57:02.353922+0000","last_active":"2024-09-21T18:57:16.468548+0000","last_peered":"2024-09-21T18:57:16.468548+0000","last_clean":"2024-09-21T18:55:50.489495+0000","last_became_active":"2024-09-21T18:55:52.881338+0000","last_became_peered":"2024-09-21T18:55:52.881338+0000","last_unstale":"2024-09-21T18:57:16.468548+0000","last_undegraded":"2024-09-21T18:57:16.468548+0000","last_fullsized":"2024-09-21T18:57:16.468548+0000","mapping_epoch":114,"log_start":"107'7100","ondisk_log_start":"107'7100","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262602752,"num_objects":4007,"num_object_clones":0,"num_object_copies":12021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2388,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13321,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1808,"num_bytes_recovered":118489088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,3,1],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":4007}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":274,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624969+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:57:09.624969+0000","last_peered":"2024-09-21T18:57:09.624969+0000","last_clean":"2024-09-21T18:57:09.624969+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:57:09.624969+0000","last_undegraded":"2024-09-21T18:57:09.624969+0000","last_fullsized":"2024-09-21T18:57:09.624969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":202,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629912+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:57:09.629912+0000","last_peered":"2024-09-21T18:57:09.629912+0000","last_clean":"2024-09-21T18:57:09.629912+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:57:09.629912+0000","last_undegraded":"2024-09-21T18:57:09.629912+0000","last_fullsized":"2024-09-21T18:57:09.629912+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:12.823290+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":198,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623181+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:57:09.623181+0000","last_peered":"2024-09-21T18:57:09.623181+0000","last_clean":"2024-09-21T18:57:09.623181+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:57:09.623181+0000","last_undegraded":"2024-09-21T18:57:09.623181+0000","last_fullsized":"2024-09-21T18:57:09.623181+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.854902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.617271+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:57:09.617271+0000","last_peered":"2024-09-21T18:57:09.617271+0000","last_clean":"2024-09-21T18:57:09.617271+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:57:09.617271+0000","last_undegraded":"2024-09-21T18:57:09.617271+0000","last_fullsized":"2024-09-21T18:57:09.617271+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":169,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.641819+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:57:09.641819+0000","last_peered":"2024-09-21T18:57:09.641819+0000","last_clean":"2024-09-21T18:57:09.641819+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:57:09.641819+0000","last_undegraded":"2024-09-21T18:57:09.641819+0000","last_fullsized":"2024-09-21T18:57:09.641819+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"138'13472","reported_seq":13613,"reported_epoch":138,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T18:57:16.376250+0000","last_change":"2024-09-21T18:56:22.889265+0000","last_active":"2024-09-21T18:57:16.376250+0000","last_peered":"2024-09-21T18:57:16.376250+0000","last_clean":"2024-09-21T18:55:50.469777+0000","last_became_active":"2024-09-21T18:55:52.866581+0000","last_became_peered":"2024-09-21T18:55:52.866581+0000","last_unstale":"2024-09-21T18:57:16.376250+0000","last_undegraded":"2024-09-21T18:57:16.376250+0000","last_fullsized":"2024-09-21T18:57:16.376250+0000","mapping_epoch":114,"log_start":"112'8700","ondisk_log_start":"112'8700","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264634368,"num_objects":4038,"num_object_clones":0,"num_object_copies":12114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4038,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13472,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":4038}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":148,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.626020+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:57:09.626020+0000","last_peered":"2024-09-21T18:57:09.626020+0000","last_clean":"2024-09-21T18:57:09.626020+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:57:09.626020+0000","last_undegraded":"2024-09-21T18:57:09.626020+0000","last_fullsized":"2024-09-21T18:57:09.626020+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623501+0000","last_change":"2024-09-21T18:53:07.399334+0000","last_active":"2024-09-21T18:57:09.623501+0000","last_peered":"2024-09-21T18:57:09.623501+0000","last_clean":"2024-09-21T18:57:09.623501+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:57:09.623501+0000","last_undegraded":"2024-09-21T18:57:09.623501+0000","last_fullsized":"2024-09-21T18:57:09.623501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":212,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625709+0000","last_change":"2024-09-21T18:55:51.541888+0000","last_active":"2024-09-21T18:57:09.625709+0000","last_peered":"2024-09-21T18:57:09.625709+0000","last_clean":"2024-09-21T18:57:09.625709+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:57:09.625709+0000","last_undegraded":"2024-09-21T18:57:09.625709+0000","last_fullsized":"2024-09-21T18:57:09.625709+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:25.237173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.237173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627237+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:57:09.627237+0000","last_peered":"2024-09-21T18:57:09.627237+0000","last_clean":"2024-09-21T18:57:09.627237+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:57:09.627237+0000","last_undegraded":"2024-09-21T18:57:09.627237+0000","last_fullsized":"2024-09-21T18:57:09.627237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":171,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.630804+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:57:09.630804+0000","last_peered":"2024-09-21T18:57:09.630804+0000","last_clean":"2024-09-21T18:57:09.630804+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:57:09.630804+0000","last_undegraded":"2024-09-21T18:57:09.630804+0000","last_fullsized":"2024-09-21T18:57:09.630804+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"138'13636","reported_seq":27910,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.384297+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:57:16.384297+0000","last_peered":"2024-09-21T18:57:16.384297+0000","last_clean":"2024-09-21T18:57:16.384297+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:57:16.384297+0000","last_undegraded":"2024-09-21T18:57:16.384297+0000","last_fullsized":"2024-09-21T18:57:16.384297+0000","mapping_epoch":130,"log_start":"111'8500","ondisk_log_start":"111'8500","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.065509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274333696,"num_objects":4186,"num_object_clones":0,"num_object_copies":12558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":74,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.618300+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:57:09.618300+0000","last_peered":"2024-09-21T18:57:09.618300+0000","last_clean":"2024-09-21T18:57:09.618300+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:57:09.618300+0000","last_undegraded":"2024-09-21T18:57:09.618300+0000","last_fullsized":"2024-09-21T18:57:09.618300+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.616947+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:57:09.616947+0000","last_peered":"2024-09-21T18:57:09.616947+0000","last_clean":"2024-09-21T18:57:09.616947+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:57:09.616947+0000","last_undegraded":"2024-09-21T18:57:09.616947+0000","last_fullsized":"2024-09-21T18:57:09.616947+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":464,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:15.070648+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:57:15.070648+0000","last_peered":"2024-09-21T18:57:15.070648+0000","last_clean":"2024-09-21T18:57:15.070648+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:57:15.070648+0000","last_undegraded":"2024-09-21T18:57:15.070648+0000","last_fullsized":"2024-09-21T18:57:15.070648+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:17.496871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":238,"num_read_kb":233,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":202,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:12.834964+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:57:12.834964+0000","last_peered":"2024-09-21T18:57:12.834964+0000","last_clean":"2024-09-21T18:57:12.834964+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:57:12.834964+0000","last_undegraded":"2024-09-21T18:57:12.834964+0000","last_fullsized":"2024-09-21T18:57:12.834964+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.378290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":160,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624607+0000","last_change":"2024-09-21T18:53:13.432008+0000","last_active":"2024-09-21T18:57:09.624607+0000","last_peered":"2024-09-21T18:57:09.624607+0000","last_clean":"2024-09-21T18:57:09.624607+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:57:09.624607+0000","last_undegraded":"2024-09-21T18:57:09.624607+0000","last_fullsized":"2024-09-21T18:57:09.624607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"138'13302","reported_seq":25074,"reported_epoch":138,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:17.039329+0000","last_change":"2024-09-21T18:55:52.898089+0000","last_active":"2024-09-21T18:57:17.039329+0000","last_peered":"2024-09-21T18:57:17.039329+0000","last_clean":"2024-09-21T18:55:50.465179+0000","last_became_active":"2024-09-21T18:55:52.872405+0000","last_became_peered":"2024-09-21T18:55:52.872405+0000","last_unstale":"2024-09-21T18:57:17.039329+0000","last_undegraded":"2024-09-21T18:57:17.039329+0000","last_fullsized":"2024-09-21T18:57:17.039329+0000","mapping_epoch":114,"log_start":"116'8802","ondisk_log_start":"116'8802","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265945088,"num_objects":4058,"num_object_clones":0,"num_object_copies":12174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5822,"num_bytes_recovered":381550592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":4058}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":247,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.620510+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:57:09.620510+0000","last_peered":"2024-09-21T18:57:09.620510+0000","last_clean":"2024-09-21T18:57:09.620510+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:57:09.620510+0000","last_undegraded":"2024-09-21T18:57:09.620510+0000","last_fullsized":"2024-09-21T18:57:09.620510+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":181,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.626436+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:57:09.626436+0000","last_peered":"2024-09-21T18:57:09.626436+0000","last_clean":"2024-09-21T18:57:09.626436+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:57:09.626436+0000","last_undegraded":"2024-09-21T18:57:09.626436+0000","last_fullsized":"2024-09-21T18:57:09.626436+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":243,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628011+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:57:09.628011+0000","last_peered":"2024-09-21T18:57:09.628011+0000","last_clean":"2024-09-21T18:57:09.628011+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:57:09.628011+0000","last_undegraded":"2024-09-21T18:57:09.628011+0000","last_fullsized":"2024-09-21T18:57:09.628011+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:05.121733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":163,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627822+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:57:09.627822+0000","last_peered":"2024-09-21T18:57:09.627822+0000","last_clean":"2024-09-21T18:57:09.627822+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:57:09.627822+0000","last_undegraded":"2024-09-21T18:57:09.627822+0000","last_fullsized":"2024-09-21T18:57:09.627822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:01.123772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623715+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:57:09.623715+0000","last_peered":"2024-09-21T18:57:09.623715+0000","last_clean":"2024-09-21T18:57:09.623715+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:57:09.623715+0000","last_undegraded":"2024-09-21T18:57:09.623715+0000","last_fullsized":"2024-09-21T18:57:09.623715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:26.445821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"138'13509","reported_seq":13742,"reported_epoch":138,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T18:57:16.432954+0000","last_change":"2024-09-21T18:56:22.889684+0000","last_active":"2024-09-21T18:57:16.432954+0000","last_peered":"2024-09-21T18:57:16.432954+0000","last_clean":"2024-09-21T18:55:50.477883+0000","last_became_active":"2024-09-21T18:55:52.879827+0000","last_became_peered":"2024-09-21T18:55:52.879827+0000","last_unstale":"2024-09-21T18:57:16.432954+0000","last_undegraded":"2024-09-21T18:57:16.432954+0000","last_fullsized":"2024-09-21T18:57:16.432954+0000","mapping_epoch":114,"log_start":"109'7300","ondisk_log_start":"109'7300","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352984,"num_objects":4111,"num_object_clones":0,"num_object_copies":12333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13509,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":4111}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":195,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628603+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:57:09.628603+0000","last_peered":"2024-09-21T18:57:09.628603+0000","last_clean":"2024-09-21T18:57:09.628603+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:57:09.628603+0000","last_undegraded":"2024-09-21T18:57:09.628603+0000","last_fullsized":"2024-09-21T18:57:09.628603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629286+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:57:09.629286+0000","last_peered":"2024-09-21T18:57:09.629286+0000","last_clean":"2024-09-21T18:57:09.629286+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:57:09.629286+0000","last_undegraded":"2024-09-21T18:57:09.629286+0000","last_fullsized":"2024-09-21T18:57:09.629286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":180,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623668+0000","last_change":"2024-09-21T18:56:33.913317+0000","last_active":"2024-09-21T18:57:09.623668+0000","last_peered":"2024-09-21T18:57:09.623668+0000","last_clean":"2024-09-21T18:57:09.623668+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:57:09.623668+0000","last_undegraded":"2024-09-21T18:57:09.623668+0000","last_fullsized":"2024-09-21T18:57:09.623668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:33.913250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:33.913250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:52.937683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":161,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622149+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:57:09.622149+0000","last_peered":"2024-09-21T18:57:09.622149+0000","last_clean":"2024-09-21T18:57:09.622149+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:57:09.622149+0000","last_undegraded":"2024-09-21T18:57:09.622149+0000","last_fullsized":"2024-09-21T18:57:09.622149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"138'13162","reported_seq":18149,"reported_epoch":138,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:16.488781+0000","last_change":"2024-09-21T18:56:52.172568+0000","last_active":"2024-09-21T18:57:16.488781+0000","last_peered":"2024-09-21T18:57:16.488781+0000","last_clean":"2024-09-21T18:55:50.490306+0000","last_became_active":"2024-09-21T18:55:52.880630+0000","last_became_peered":"2024-09-21T18:55:52.880630+0000","last_unstale":"2024-09-21T18:57:16.488781+0000","last_undegraded":"2024-09-21T18:57:16.488781+0000","last_fullsized":"2024-09-21T18:57:16.488781+0000","mapping_epoch":114,"log_start":"105'6400","ondisk_log_start":"105'6400","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1960,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13162,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2382,"num_bytes_recovered":156106752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":3974}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":204,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628640+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:57:09.628640+0000","last_peered":"2024-09-21T18:57:09.628640+0000","last_clean":"2024-09-21T18:57:09.628640+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:57:09.628640+0000","last_undegraded":"2024-09-21T18:57:09.628640+0000","last_fullsized":"2024-09-21T18:57:09.628640+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:21.119260+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.619831+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:57:09.619831+0000","last_peered":"2024-09-21T18:57:09.619831+0000","last_clean":"2024-09-21T18:57:09.619831+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:57:09.619831+0000","last_undegraded":"2024-09-21T18:57:09.619831+0000","last_fullsized":"2024-09-21T18:57:09.619831+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":178,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624725+0000","last_change":"2024-09-21T18:55:51.535384+0000","last_active":"2024-09-21T18:57:09.624725+0000","last_peered":"2024-09-21T18:57:09.624725+0000","last_clean":"2024-09-21T18:57:09.624725+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:57:09.624725+0000","last_undegraded":"2024-09-21T18:57:09.624725+0000","last_fullsized":"2024-09-21T18:57:09.624725+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.042616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.626109+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:57:09.626109+0000","last_peered":"2024-09-21T18:57:09.626109+0000","last_clean":"2024-09-21T18:57:09.626109+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:57:09.626109+0000","last_undegraded":"2024-09-21T18:57:09.626109+0000","last_fullsized":"2024-09-21T18:57:09.626109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"138'13343","reported_seq":13479,"reported_epoch":138,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T18:57:17.031606+0000","last_change":"2024-09-21T18:55:52.897819+0000","last_active":"2024-09-21T18:57:17.031606+0000","last_peered":"2024-09-21T18:57:17.031606+0000","last_clean":"2024-09-21T18:55:50.495661+0000","last_became_active":"2024-09-21T18:55:52.870612+0000","last_became_peered":"2024-09-21T18:55:52.870612+0000","last_unstale":"2024-09-21T18:57:17.031606+0000","last_undegraded":"2024-09-21T18:57:17.031606+0000","last_fullsized":"2024-09-21T18:57:17.031606+0000","mapping_epoch":114,"log_start":"116'8800","ondisk_log_start":"116'8800","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261816320,"num_objects":3995,"num_object_clones":0,"num_object_copies":11985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3995,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13343,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":3995}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":191,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628902+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:57:09.628902+0000","last_peered":"2024-09-21T18:57:09.628902+0000","last_clean":"2024-09-21T18:57:09.628902+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:57:09.628902+0000","last_undegraded":"2024-09-21T18:57:09.628902+0000","last_fullsized":"2024-09-21T18:57:09.628902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:09.960146+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":209,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623081+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:57:09.623081+0000","last_peered":"2024-09-21T18:57:09.623081+0000","last_clean":"2024-09-21T18:57:09.623081+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:57:09.623081+0000","last_undegraded":"2024-09-21T18:57:09.623081+0000","last_fullsized":"2024-09-21T18:57:09.623081+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":230,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625949+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:57:09.625949+0000","last_peered":"2024-09-21T18:57:09.625949+0000","last_clean":"2024-09-21T18:57:09.625949+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:57:09.625949+0000","last_undegraded":"2024-09-21T18:57:09.625949+0000","last_fullsized":"2024-09-21T18:57:09.625949+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628630+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:57:09.628630+0000","last_peered":"2024-09-21T18:57:09.628630+0000","last_clean":"2024-09-21T18:57:09.628630+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:57:09.628630+0000","last_undegraded":"2024-09-21T18:57:09.628630+0000","last_fullsized":"2024-09-21T18:57:09.628630+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:18.009607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"138'13373","reported_seq":13584,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.483281+0000","last_change":"2024-09-21T18:55:30.532749+0000","last_active":"2024-09-21T18:57:16.483281+0000","last_peered":"2024-09-21T18:57:16.483281+0000","last_clean":"2024-09-21T18:57:16.483281+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:57:16.483281+0000","last_undegraded":"2024-09-21T18:57:16.483281+0000","last_fullsized":"2024-09-21T18:57:16.483281+0000","mapping_epoch":79,"log_start":"105'6700","ondisk_log_start":"105'6700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7201","last_scrub_stamp":"2024-09-21T18:55:30.532526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.532526+0000","objects_scrubbed":7133,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267714560,"num_objects":4085,"num_object_clones":0,"num_object_copies":12255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13373,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":70,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.616223+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:57:09.616223+0000","last_peered":"2024-09-21T18:57:09.616223+0000","last_clean":"2024-09-21T18:57:09.616223+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:57:09.616223+0000","last_undegraded":"2024-09-21T18:57:09.616223+0000","last_fullsized":"2024-09-21T18:57:09.616223+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":188,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623268+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:57:09.623268+0000","last_peered":"2024-09-21T18:57:09.623268+0000","last_clean":"2024-09-21T18:57:09.623268+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:57:09.623268+0000","last_undegraded":"2024-09-21T18:57:09.623268+0000","last_fullsized":"2024-09-21T18:57:09.623268+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.889938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.626301+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:57:09.626301+0000","last_peered":"2024-09-21T18:57:09.626301+0000","last_clean":"2024-09-21T18:57:09.626301+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:57:09.626301+0000","last_undegraded":"2024-09-21T18:57:09.626301+0000","last_fullsized":"2024-09-21T18:57:09.626301+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.616868+0000","last_change":"2024-09-21T18:55:51.540541+0000","last_active":"2024-09-21T18:57:09.616868+0000","last_peered":"2024-09-21T18:57:09.616868+0000","last_clean":"2024-09-21T18:57:09.616868+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:57:09.616868+0000","last_undegraded":"2024-09-21T18:57:09.616868+0000","last_fullsized":"2024-09-21T18:57:09.616868+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"138'13463","reported_seq":26574,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.437751+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:57:16.437751+0000","last_peered":"2024-09-21T18:57:16.437751+0000","last_clean":"2024-09-21T18:57:16.437751+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:57:16.437751+0000","last_undegraded":"2024-09-21T18:57:16.437751+0000","last_fullsized":"2024-09-21T18:57:16.437751+0000","mapping_epoch":134,"log_start":"110'7900","ondisk_log_start":"110'7900","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":11907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13463,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":206,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.619673+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:57:09.619673+0000","last_peered":"2024-09-21T18:57:09.619673+0000","last_clean":"2024-09-21T18:57:09.619673+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:57:09.619673+0000","last_undegraded":"2024-09-21T18:57:09.619673+0000","last_fullsized":"2024-09-21T18:57:09.619673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.631152+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:57:09.631152+0000","last_peered":"2024-09-21T18:57:09.631152+0000","last_clean":"2024-09-21T18:57:09.631152+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:57:09.631152+0000","last_undegraded":"2024-09-21T18:57:09.631152+0000","last_fullsized":"2024-09-21T18:57:09.631152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":186,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625762+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:57:09.625762+0000","last_peered":"2024-09-21T18:57:09.625762+0000","last_clean":"2024-09-21T18:57:09.625762+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:57:09.625762+0000","last_undegraded":"2024-09-21T18:57:09.625762+0000","last_fullsized":"2024-09-21T18:57:09.625762+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":167,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627925+0000","last_change":"2024-09-21T18:55:24.049546+0000","last_active":"2024-09-21T18:57:09.627925+0000","last_peered":"2024-09-21T18:57:09.627925+0000","last_clean":"2024-09-21T18:57:09.627925+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:57:09.627925+0000","last_undegraded":"2024-09-21T18:57:09.627925+0000","last_fullsized":"2024-09-21T18:57:09.627925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.049407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.049407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"138'13157","reported_seq":13288,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:17.039535+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:57:17.039535+0000","last_peered":"2024-09-21T18:57:17.039535+0000","last_clean":"2024-09-21T18:57:17.039535+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:57:17.039535+0000","last_undegraded":"2024-09-21T18:57:17.039535+0000","last_fullsized":"2024-09-21T18:57:17.039535+0000","mapping_epoch":79,"log_start":"116'8600","ondisk_log_start":"116'8600","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261029888,"num_objects":3983,"num_object_clones":0,"num_object_copies":11949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13157,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":218,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628670+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:57:09.628670+0000","last_peered":"2024-09-21T18:57:09.628670+0000","last_clean":"2024-09-21T18:57:09.628670+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:57:09.628670+0000","last_undegraded":"2024-09-21T18:57:09.628670+0000","last_fullsized":"2024-09-21T18:57:09.628670+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:16.969327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627981+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:57:09.627981+0000","last_peered":"2024-09-21T18:57:09.627981+0000","last_clean":"2024-09-21T18:57:09.627981+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:57:09.627981+0000","last_undegraded":"2024-09-21T18:57:09.627981+0000","last_fullsized":"2024-09-21T18:57:09.627981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.074325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":194,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.626093+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:57:09.626093+0000","last_peered":"2024-09-21T18:57:09.626093+0000","last_clean":"2024-09-21T18:57:09.626093+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:57:09.626093+0000","last_undegraded":"2024-09-21T18:57:09.626093+0000","last_fullsized":"2024-09-21T18:57:09.626093+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":161,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622320+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:57:09.622320+0000","last_peered":"2024-09-21T18:57:09.622320+0000","last_clean":"2024-09-21T18:57:09.622320+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:57:09.622320+0000","last_undegraded":"2024-09-21T18:57:09.622320+0000","last_fullsized":"2024-09-21T18:57:09.622320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"138'13145","reported_seq":13276,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.680660+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:57:16.680660+0000","last_peered":"2024-09-21T18:57:16.680660+0000","last_clean":"2024-09-21T18:57:16.680660+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:57:16.680660+0000","last_undegraded":"2024-09-21T18:57:16.680660+0000","last_fullsized":"2024-09-21T18:57:16.680660+0000","mapping_epoch":79,"log_start":"111'8000","ondisk_log_start":"111'8000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260374528,"num_objects":3973,"num_object_clones":0,"num_object_copies":11919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13145,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":209,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622915+0000","last_change":"2024-09-21T18:56:35.846693+0000","last_active":"2024-09-21T18:57:09.622915+0000","last_peered":"2024-09-21T18:57:09.622915+0000","last_clean":"2024-09-21T18:57:09.622915+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:57:09.622915+0000","last_undegraded":"2024-09-21T18:57:09.622915+0000","last_fullsized":"2024-09-21T18:57:09.622915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:56:35.846605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:35.846605+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:51.214847+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":209,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627899+0000","last_change":"2024-09-21T18:55:51.567353+0000","last_active":"2024-09-21T18:57:09.627899+0000","last_peered":"2024-09-21T18:57:09.627899+0000","last_clean":"2024-09-21T18:57:09.627899+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:57:09.627899+0000","last_undegraded":"2024-09-21T18:57:09.627899+0000","last_fullsized":"2024-09-21T18:57:09.627899+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:55.125124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":232,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627521+0000","last_change":"2024-09-21T18:55:39.995074+0000","last_active":"2024-09-21T18:57:09.627521+0000","last_peered":"2024-09-21T18:57:09.627521+0000","last_clean":"2024-09-21T18:57:09.627521+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:57:09.627521+0000","last_undegraded":"2024-09-21T18:57:09.627521+0000","last_fullsized":"2024-09-21T18:57:09.627521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:39.994892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.994892+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":182,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622880+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:57:09.622880+0000","last_peered":"2024-09-21T18:57:09.622880+0000","last_clean":"2024-09-21T18:57:09.622880+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:57:09.622880+0000","last_undegraded":"2024-09-21T18:57:09.622880+0000","last_fullsized":"2024-09-21T18:57:09.622880+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:24.875223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":193,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.619552+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:57:09.619552+0000","last_peered":"2024-09-21T18:57:09.619552+0000","last_clean":"2024-09-21T18:57:09.619552+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:57:09.619552+0000","last_undegraded":"2024-09-21T18:57:09.619552+0000","last_fullsized":"2024-09-21T18:57:09.619552+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":178,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622854+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:57:09.622854+0000","last_peered":"2024-09-21T18:57:09.622854+0000","last_clean":"2024-09-21T18:57:09.622854+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:57:09.622854+0000","last_undegraded":"2024-09-21T18:57:09.622854+0000","last_fullsized":"2024-09-21T18:57:09.622854+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":180,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623269+0000","last_change":"2024-09-21T18:56:40.848060+0000","last_active":"2024-09-21T18:57:09.623269+0000","last_peered":"2024-09-21T18:57:09.623269+0000","last_clean":"2024-09-21T18:57:09.623269+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:57:09.623269+0000","last_undegraded":"2024-09-21T18:57:09.623269+0000","last_fullsized":"2024-09-21T18:57:09.623269+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:40.847976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.847976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:57.531530+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":159,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.630708+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:57:09.630708+0000","last_peered":"2024-09-21T18:57:09.630708+0000","last_clean":"2024-09-21T18:57:09.630708+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:57:09.630708+0000","last_undegraded":"2024-09-21T18:57:09.630708+0000","last_fullsized":"2024-09-21T18:57:09.630708+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:00.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":70,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.618383+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:57:09.618383+0000","last_peered":"2024-09-21T18:57:09.618383+0000","last_clean":"2024-09-21T18:57:09.618383+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:57:09.618383+0000","last_undegraded":"2024-09-21T18:57:09.618383+0000","last_fullsized":"2024-09-21T18:57:09.618383+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627680+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:57:09.627680+0000","last_peered":"2024-09-21T18:57:09.627680+0000","last_clean":"2024-09-21T18:57:09.627680+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:57:09.627680+0000","last_undegraded":"2024-09-21T18:57:09.627680+0000","last_fullsized":"2024-09-21T18:57:09.627680+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:08.070910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622252+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:57:09.622252+0000","last_peered":"2024-09-21T18:57:09.622252+0000","last_clean":"2024-09-21T18:57:09.622252+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:57:09.622252+0000","last_undegraded":"2024-09-21T18:57:09.622252+0000","last_fullsized":"2024-09-21T18:57:09.622252+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625510+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:57:09.625510+0000","last_peered":"2024-09-21T18:57:09.625510+0000","last_clean":"2024-09-21T18:57:09.625510+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:57:09.625510+0000","last_undegraded":"2024-09-21T18:57:09.625510+0000","last_fullsized":"2024-09-21T18:57:09.625510+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":184,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623503+0000","last_change":"2024-09-21T18:53:09.720036+0000","last_active":"2024-09-21T18:57:09.623503+0000","last_peered":"2024-09-21T18:57:09.623503+0000","last_clean":"2024-09-21T18:57:09.623503+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:57:09.623503+0000","last_undegraded":"2024-09-21T18:57:09.623503+0000","last_fullsized":"2024-09-21T18:57:09.623503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:22.890902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.616418+0000","last_change":"2024-09-21T18:55:51.523039+0000","last_active":"2024-09-21T18:57:09.616418+0000","last_peered":"2024-09-21T18:57:09.616418+0000","last_clean":"2024-09-21T18:57:09.616418+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:57:09.616418+0000","last_undegraded":"2024-09-21T18:57:09.616418+0000","last_fullsized":"2024-09-21T18:57:09.616418+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":231,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622589+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:57:09.622589+0000","last_peered":"2024-09-21T18:57:09.622589+0000","last_clean":"2024-09-21T18:57:09.622589+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:57:09.622589+0000","last_undegraded":"2024-09-21T18:57:09.622589+0000","last_fullsized":"2024-09-21T18:57:09.622589+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.628688+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:57:09.628688+0000","last_peered":"2024-09-21T18:57:09.628688+0000","last_clean":"2024-09-21T18:57:09.628688+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:57:09.628688+0000","last_undegraded":"2024-09-21T18:57:09.628688+0000","last_fullsized":"2024-09-21T18:57:09.628688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":184,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622834+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:57:09.622834+0000","last_peered":"2024-09-21T18:57:09.622834+0000","last_clean":"2024-09-21T18:57:09.622834+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:57:09.622834+0000","last_undegraded":"2024-09-21T18:57:09.622834+0000","last_fullsized":"2024-09-21T18:57:09.622834+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:16.078610+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:57:16.078610+0000","last_peered":"2024-09-21T18:57:16.078610+0000","last_clean":"2024-09-21T18:57:16.078610+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:57:16.078610+0000","last_undegraded":"2024-09-21T18:57:16.078610+0000","last_fullsized":"2024-09-21T18:57:16.078610+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.429722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":244,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623369+0000","last_change":"2024-09-21T18:55:51.543525+0000","last_active":"2024-09-21T18:57:09.623369+0000","last_peered":"2024-09-21T18:57:09.623369+0000","last_clean":"2024-09-21T18:57:09.623369+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:57:09.623369+0000","last_undegraded":"2024-09-21T18:57:09.623369+0000","last_fullsized":"2024-09-21T18:57:09.623369+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.924485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.618732+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:57:09.618732+0000","last_peered":"2024-09-21T18:57:09.618732+0000","last_clean":"2024-09-21T18:57:09.618732+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:57:09.618732+0000","last_undegraded":"2024-09-21T18:57:09.618732+0000","last_fullsized":"2024-09-21T18:57:09.618732+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":214,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629666+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:57:09.629666+0000","last_peered":"2024-09-21T18:57:09.629666+0000","last_clean":"2024-09-21T18:57:09.629666+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:57:09.629666+0000","last_undegraded":"2024-09-21T18:57:09.629666+0000","last_fullsized":"2024-09-21T18:57:09.629666+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:14.030615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":172,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.629118+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:57:09.629118+0000","last_peered":"2024-09-21T18:57:09.629118+0000","last_clean":"2024-09-21T18:57:09.629118+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:57:09.629118+0000","last_undegraded":"2024-09-21T18:57:09.629118+0000","last_fullsized":"2024-09-21T18:57:09.629118+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:19.966099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":164,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624171+0000","last_change":"2024-09-21T18:53:11.413375+0000","last_active":"2024-09-21T18:57:09.624171+0000","last_peered":"2024-09-21T18:57:09.624171+0000","last_clean":"2024-09-21T18:57:09.624171+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:57:09.624171+0000","last_undegraded":"2024-09-21T18:57:09.624171+0000","last_fullsized":"2024-09-21T18:57:09.624171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":175,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623679+0000","last_change":"2024-09-21T18:55:51.516509+0000","last_active":"2024-09-21T18:57:09.623679+0000","last_peered":"2024-09-21T18:57:09.623679+0000","last_clean":"2024-09-21T18:57:09.623679+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:57:09.623679+0000","last_undegraded":"2024-09-21T18:57:09.623679+0000","last_fullsized":"2024-09-21T18:57:09.623679+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":192,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.624486+0000","last_change":"2024-09-21T18:55:40.664189+0000","last_active":"2024-09-21T18:57:09.624486+0000","last_peered":"2024-09-21T18:57:09.624486+0000","last_clean":"2024-09-21T18:57:09.624486+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:57:09.624486+0000","last_undegraded":"2024-09-21T18:57:09.624486+0000","last_fullsized":"2024-09-21T18:57:09.624486+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:40.664110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.664110+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.631006+0000","last_change":"2024-09-21T18:54:52.054723+0000","last_active":"2024-09-21T18:57:09.631006+0000","last_peered":"2024-09-21T18:57:09.631006+0000","last_clean":"2024-09-21T18:57:09.631006+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:57:09.631006+0000","last_undegraded":"2024-09-21T18:57:09.631006+0000","last_fullsized":"2024-09-21T18:57:09.631006+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:52.054655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.054655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622400+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:57:09.622400+0000","last_peered":"2024-09-21T18:57:09.622400+0000","last_clean":"2024-09-21T18:57:09.622400+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:57:09.622400+0000","last_undegraded":"2024-09-21T18:57:09.622400+0000","last_fullsized":"2024-09-21T18:57:09.622400+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.610596+0000","last_change":"2024-09-21T18:55:51.536412+0000","last_active":"2024-09-21T18:57:09.610596+0000","last_peered":"2024-09-21T18:57:09.610596+0000","last_clean":"2024-09-21T18:57:09.610596+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:57:09.610596+0000","last_undegraded":"2024-09-21T18:57:09.610596+0000","last_fullsized":"2024-09-21T18:57:09.610596+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":188,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623826+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:57:09.623826+0000","last_peered":"2024-09-21T18:57:09.623826+0000","last_clean":"2024-09-21T18:57:09.623826+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:57:09.623826+0000","last_undegraded":"2024-09-21T18:57:09.623826+0000","last_fullsized":"2024-09-21T18:57:09.623826+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":180,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623308+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:57:09.623308+0000","last_peered":"2024-09-21T18:57:09.623308+0000","last_clean":"2024-09-21T18:57:09.623308+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:57:09.623308+0000","last_undegraded":"2024-09-21T18:57:09.623308+0000","last_fullsized":"2024-09-21T18:57:09.623308+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.912165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":171,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627245+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:57:09.627245+0000","last_peered":"2024-09-21T18:57:09.627245+0000","last_clean":"2024-09-21T18:57:09.627245+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:57:09.627245+0000","last_undegraded":"2024-09-21T18:57:09.627245+0000","last_fullsized":"2024-09-21T18:57:09.627245+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":175,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.622544+0000","last_change":"2024-09-21T18:55:51.529791+0000","last_active":"2024-09-21T18:57:09.622544+0000","last_peered":"2024-09-21T18:57:09.622544+0000","last_clean":"2024-09-21T18:57:09.622544+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:57:09.622544+0000","last_undegraded":"2024-09-21T18:57:09.622544+0000","last_fullsized":"2024-09-21T18:57:09.622544+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":188,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623178+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:57:09.623178+0000","last_peered":"2024-09-21T18:57:09.623178+0000","last_clean":"2024-09-21T18:57:09.623178+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:57:09.623178+0000","last_undegraded":"2024-09-21T18:57:09.623178+0000","last_fullsized":"2024-09-21T18:57:09.623178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.625462+0000","last_change":"2024-09-21T18:55:51.529323+0000","last_active":"2024-09-21T18:57:09.625462+0000","last_peered":"2024-09-21T18:57:09.625462+0000","last_clean":"2024-09-21T18:57:09.625462+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:57:09.625462+0000","last_undegraded":"2024-09-21T18:57:09.625462+0000","last_fullsized":"2024-09-21T18:57:09.625462+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":163,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.620460+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:57:09.620460+0000","last_peered":"2024-09-21T18:57:09.620460+0000","last_clean":"2024-09-21T18:57:09.620460+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:57:09.620460+0000","last_undegraded":"2024-09-21T18:57:09.620460+0000","last_fullsized":"2024-09-21T18:57:09.620460+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":159,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.627244+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:57:09.627244+0000","last_peered":"2024-09-21T18:57:09.627244+0000","last_clean":"2024-09-21T18:57:09.627244+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:57:09.627244+0000","last_undegraded":"2024-09-21T18:57:09.627244+0000","last_fullsized":"2024-09-21T18:57:09.627244+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.192235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623997+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:57:09.623997+0000","last_peered":"2024-09-21T18:57:09.623997+0000","last_clean":"2024-09-21T18:57:09.623997+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:57:09.623997+0000","last_undegraded":"2024-09-21T18:57:09.623997+0000","last_fullsized":"2024-09-21T18:57:09.623997+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":195,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.623404+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:57:09.623404+0000","last_peered":"2024-09-21T18:57:09.623404+0000","last_clean":"2024-09-21T18:57:09.623404+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:57:09.623404+0000","last_undegraded":"2024-09-21T18:57:09.623404+0000","last_fullsized":"2024-09-21T18:57:09.623404+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.620296+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:57:09.620296+0000","last_peered":"2024-09-21T18:57:09.620296+0000","last_clean":"2024-09-21T18:57:09.620296+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:57:09.620296+0000","last_undegraded":"2024-09-21T18:57:09.620296+0000","last_fullsized":"2024-09-21T18:57:09.620296+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":60,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-21T18:57:09.618651+0000","last_change":"2024-09-21T18:55:51.545494+0000","last_active":"2024-09-21T18:57:09.618651+0000","last_peered":"2024-09-21T18:57:09.618651+0000","last_clean":"2024-09-21T18:57:09.618651+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:57:09.618651+0000","last_undegraded":"2024-09-21T18:57:09.618651+0000","last_fullsized":"2024-09-21T18:57:09.618651+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4215144472,"num_objects":64319,"num_object_clones":0,"num_object_copies":192957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16606,"num_objects_unfound":0,"num_objects_dirty":64319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213675,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23559,"num_bytes_recovered":1543962624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13428338688,"data_stored":13428326472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85673,"ondisk_log_size":85673,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":238,"num_read_kb":233,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266742,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707540,"kb_used_data":1586116,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92037612,"statfs":{"total":95995035648,"available":94246514688,"internally_reserved":0,"allocated":1624182784,"data_stored":1623806068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528379,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1683464,"kb_used_data":1562040,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92061688,"statfs":{"total":95995035648,"available":94271168512,"internally_reserved":0,"allocated":1599528960,"data_stored":1599163907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790014,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1411204,"kb_used_data":1303668,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92333948,"statfs":{"total":95995035648,"available":94549962752,"internally_reserved":0,"allocated":1334956032,"data_stored":1334594637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018948,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680536,"kb_used_data":1554952,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92064616,"statfs":{"total":95995035648,"available":94274166784,"internally_reserved":0,"allocated":1592270848,"data_stored":1591889992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3063720,"kb_used_data":2833816,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":90681432,"statfs":{"total":95995035648,"available":92857786368,"internally_reserved":0,"allocated":2901827584,"data_stored":2901430667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1682508,"kb_used_data":1556924,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92062644,"statfs":{"total":95995035648,"available":94272147456,"internally_reserved":0,"allocated":1594290176,"data_stored":1593921144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1973476,"kb_used_data":1815956,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91771676,"statfs":{"total":95995035648,"available":93974196224,"internally_reserved":0,"allocated":1859538944,"data_stored":1859145677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1176664,"kb_used_data":1005256,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92568488,"statfs":{"total":95995035648,"available":94790131712,"internally_reserved":0,"allocated":1029382144,"data_stored":1029002021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1030684672,"data_stored":1030684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1841958912,"data_stored":1841954840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1590165504,"data_stored":1590165504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2873364480,"data_stored":2873360408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1579155456,"data_stored":1579155456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1320878080,"data_stored":1320878080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1576341504,"data_stored":1576337432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1615790080,"data_stored":1615790080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:57:21.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T18:57:21.527 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:21.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:21.588 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:21.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.735 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:21.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:21.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:21.894 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:21.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:21.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:22.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:21 smithi104 ceph-mon[32438]: 6.8 scrub starts 2024-09-21T18:57:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:21 smithi104 ceph-mon[32438]: 6.8 scrub ok 2024-09-21T18:57:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:21 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T18:57:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:21 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T18:57:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[39192]: 6.8 scrub starts 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[39192]: 6.8 scrub ok 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:22.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[29384]: 6.8 scrub starts 2024-09-21T18:57:22.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[29384]: 6.8 scrub ok 2024-09-21T18:57:22.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T18:57:22.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T18:57:22.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:22.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:22.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:22.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:22.185 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:22.215 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:22.215 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:22.256 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:22.256 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:22.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:22.296 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:22.296 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:22.314 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:22.319 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:22.328 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:22.328 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:22.364 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:22.396 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:22.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:22.428 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:22.428 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:22.462 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:22.462 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:22.466 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:22.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:22.678 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:57:22.678 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:57:22.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.678 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:217698 7.5 7:aecafcd1:::benchmark_data_smithi081_73391_object78699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-21T18:57:22.678 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:21.893349+0000", 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0083833889999999998, 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0084107120000000007, 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 217698 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893349+0000", 2024-09-21T18:57:22.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893349+0000", 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893349+0000", 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7899999999999998e-07 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893352+0000", 2024-09-21T18:57:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.57e-06 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893359+0000", 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8159999999999996e-06 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893362+0000", 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.709e-06 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.895469+0000", 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021069019999999999 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.895555+0000", 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5575000000000007e-05 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.895657+0000", 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000101795 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:22.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.898113+0000", 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002456275 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:217702 7.8 e140/79 7:1ab17a83:::benchmark_data_smithi081_73391_object78703:head v 140'13611, mlcod=140'13609)", 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:21.899561+0000", 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0021713190000000001, 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002269259, 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T18:57:22.683 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899561+0000", 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899561+0000", 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899561+0000", 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6400000000000002e-07 2024-09-21T18:57:22.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899566+0000", 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7790000000000003e-06 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899580+0000", 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3359e-05 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899585+0000", 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6280000000000002e-06 2024-09-21T18:57:22.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899635+0000", 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0133999999999998e-05 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899657+0000", 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1576000000000001e-05 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:217705 7.5 7:a0c35fc6:::benchmark_data_smithi081_73391_object78706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:21.900602+0000", 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.001130287, 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001276868, 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 217705 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900602+0000", 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900602+0000", 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900605+0000", 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8080000000000001e-06 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900610+0000", 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5910000000000003e-06 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900624+0000", 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.377e-05 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900632+0000", 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3440000000000001e-06 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900683+0000", 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0386000000000002e-05 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900809+0000", 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012580800000000001 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.900924+0000", 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115393 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:217697 7.5 7:ae134c9e:::benchmark_data_smithi081_73391_object78698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:21.893315+0000", 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0084175729999999994, 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0086232419999999997, 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 217697 2024-09-21T18:57:22.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893315+0000", 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893315+0000", 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893316+0000", 2024-09-21T18:57:22.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.68e-06 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893321+0000", 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4399999999999998e-06 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893330+0000", 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3149999999999998e-06 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893335+0000", 2024-09-21T18:57:22.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2549999999999997e-06 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893376+0000", 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0932e-05 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893449+0000", 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2540999999999995e-05 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.893517+0000", 2024-09-21T18:57:22.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8194000000000005e-05 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.898054+0000", 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045371179999999997 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:21.899616+0000", 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015614649999999999 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:57:22.695 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T18:57:22.696 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 4 2024-09-21T18:57:22.696 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:57:22.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:57:22.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:22.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:22.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:22.907 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:22.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:23.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:22 smithi104 ceph-mon[32438]: pgmap v183: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 541 op/s; 16606/193554 objects misplaced (8.580%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T18:57:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3339841551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:23.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:23.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:22 smithi081 ceph-mon[29384]: pgmap v183: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 541 op/s; 16606/193554 objects misplaced (8.580%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T18:57:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3339841551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:22 smithi081 ceph-mon[39192]: pgmap v183: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 541 op/s; 16606/193554 objects misplaced (8.580%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T18:57:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3339841551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:23.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:23.163 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:23.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:23.307 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:23.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:23.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:23.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:23.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:23.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:23.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:23.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:23.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:23.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:23.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:23.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:23 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T18:57:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:23 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T18:57:24.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:24.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:23 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T18:57:24.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:23 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T18:57:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:23 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T18:57:24.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:23 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T18:57:24.093 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:24.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:24.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:24.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:24.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:24.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:24.540 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:24.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:24.668 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:24.710 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T18:57:24.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:24.816 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:24.854 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:24.918 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:24.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:24.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:24.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:24 smithi081 ceph-mon[29384]: 5.9 scrub starts 2024-09-21T18:57:24.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:24 smithi081 ceph-mon[29384]: 5.9 scrub ok 2024-09-21T18:57:24.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:24 smithi081 ceph-mon[29384]: pgmap v184: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 13220/182148 objects misplaced (7.258%); 29 MiB/s, 458 objects/s recovering 2024-09-21T18:57:24.995 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:24 smithi081 ceph-mon[39192]: 5.9 scrub starts 2024-09-21T18:57:24.995 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:24 smithi081 ceph-mon[39192]: 5.9 scrub ok 2024-09-21T18:57:24.996 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:24 smithi081 ceph-mon[39192]: pgmap v184: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 13220/182148 objects misplaced (7.258%); 29 MiB/s, 458 objects/s recovering 2024-09-21T18:57:25.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:24 smithi104 ceph-mon[32438]: 5.9 scrub starts 2024-09-21T18:57:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:24 smithi104 ceph-mon[32438]: 5.9 scrub ok 2024-09-21T18:57:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:24 smithi104 ceph-mon[32438]: pgmap v184: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 13220/182148 objects misplaced (7.258%); 29 MiB/s, 458 objects/s recovering 2024-09-21T18:57:25.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:25.096 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:25.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:25.204 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:25.204 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":476,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":140,"num_osds":8,"num_up_osds":8,"osd_up_since":1726944778,"num_in_osds":8,"osd_in_since":1726944950,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":143},{"state_name":"active+remapped+backfilling","count":3},{"state_name":"active+remapped+backfill_wait+backfill_toofull","count":2}],"num_pgs":148,"num_pools":7,"num_objects":60716,"data_bytes":3966573107,"bytes_used":14152114176,"bytes_avail":753808171008,"bytes_total":767960285184,"misplaced_objects":13220,"misplaced_total":182148,"misplaced_ratio":0.072578342886004787,"recovering_objects_per_sec":458,"recovering_bytes_per_sec":30047420,"recovering_keys_per_sec":0,"num_objects_recovered":3670,"num_bytes_recovered":240451584,"num_keys_recovered":0,"read_bytes_sec":1279,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":1015},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":13,"modified":"2024-09-21T18:54:37.384117+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T18:57:25.224 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:25.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:25.369 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:25.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:25.510 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:25.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:25.637 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:25.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:25.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3037142489' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T18:57:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:25 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:57:25.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:25 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:57:25.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:25.785 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:25.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3037142489' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T18:57:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:25 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:57:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:25 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:57:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3037142489' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T18:57:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:25 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:57:25.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:25 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:57:25.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:25.916 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:25.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:26.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:26.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.039 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:26.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:26.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:26.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:26.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:26.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:57:26.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:26.433 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:26.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T18:57:26.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.574 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:26.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:26.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:26.708 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:26.787 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:57:26.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:26.840 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:57:26.850 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:26.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:26.981 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:26.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T18:57:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T18:57:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: pgmap v185: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 475 op/s; 13220/182148 objects misplaced (7.258%); 19 MiB/s, 304 objects/s recovering 2024-09-21T18:57:27.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1835449248' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: from='client.35057 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: 4.15 scrub starts 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[29384]: 4.15 scrub ok 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: pgmap v185: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 475 op/s; 13220/182148 objects misplaced (7.258%); 19 MiB/s, 304 objects/s recovering 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1835449248' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: from='client.35057 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:57:27.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: 4.15 scrub starts 2024-09-21T18:57:27.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:26 smithi081 ceph-mon[39192]: 4.15 scrub ok 2024-09-21T18:57:27.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:27.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: pgmap v185: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 475 op/s; 13220/182148 objects misplaced (7.258%); 19 MiB/s, 304 objects/s recovering 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1835449248' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: from='client.35057 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: 4.15 scrub starts 2024-09-21T18:57:27.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:26 smithi104 ceph-mon[32438]: 4.15 scrub ok 2024-09-21T18:57:27.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:27.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:27.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:27.397 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:27.488 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:57:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:57:27.489 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T18:57:27.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:57:27.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:27.533 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:27.612 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:27.613 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:27.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:27.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:27.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:27.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:27.891 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:27.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:27.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:28.040 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:28.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:57:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:57:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[29384]: from='client.35057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:57:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[29384]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T18:57:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:57:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:57:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[39192]: from='client.35057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:57:28.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:27 smithi081 ceph-mon[39192]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T18:57:28.105 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:28.117 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:28.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:28.235 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:28.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:57:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:57:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:27 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:57:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:27 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:57:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:27 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:27 smithi104 ceph-mon[32438]: from='client.35057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:57:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:27 smithi104 ceph-mon[32438]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T18:57:28.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:28.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T18:57:28.486 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:28.512 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:28.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:28.638 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:28.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:28.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:28.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:28.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:29.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:29.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:28 smithi081 ceph-mon[29384]: pgmap v188: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s; 11149/168585 objects misplaced (6.613%); 32 MiB/s, 510 objects/s recovering 2024-09-21T18:57:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:28 smithi081 ceph-mon[39192]: pgmap v188: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s; 11149/168585 objects misplaced (6.613%); 32 MiB/s, 510 objects/s recovering 2024-09-21T18:57:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:29.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:29.186 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:28 smithi104 ceph-mon[32438]: pgmap v188: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.20k op/s; 11149/168585 objects misplaced (6.613%); 32 MiB/s, 510 objects/s recovering 2024-09-21T18:57:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:29.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:29.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:29.385 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:29.394 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:29.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:29.426 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:29.431 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:29.434 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:57:29.476 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:29.482 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:29.482 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:29.531 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:29.532 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:29.569 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:29.569 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:29.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:29.603 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:29.604 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.627 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:29.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:29.639 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:29.640 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:29.675 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:29.675 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:29.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.755 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:29.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:29.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:29.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:29.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:29.975 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2769559421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:29.975 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2769559421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:29.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:57:29.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:30.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:30.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:30.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2769559421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.192 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:30.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:30.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:30.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:30.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:30.449 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225485 7.7 e142/114 7:ecdcf442:::benchmark_data_smithi081_73391_object86486:head v 142'14087, mlcod=142'14084)", 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.433593+0000", 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.012407533, 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012432528, 2024-09-21T18:57:30.450 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433593+0000", 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433593+0000", 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433605+0000", 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433616+0000", 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1146e-05 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433639+0000", 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3144999999999999e-05 2024-09-21T18:57:30.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433646+0000", 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.708e-06 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433730+0000", 2024-09-21T18:57:30.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4129999999999996e-05 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433767+0000", 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6933e-05 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225486 7.7 e142/114 7:eec4f258:::benchmark_data_smithi081_73391_object86487:head v 142'14088, mlcod=142'14084)", 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.436835+0000", 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0091651179999999999, 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0092444329999999998, 2024-09-21T18:57:30.454 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.436835+0000", 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.436835+0000", 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.436836+0000", 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4e-07 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.436841+0000", 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0669999999999999e-06 2024-09-21T18:57:30.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.436849+0000", 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5809999999999999e-06 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.436855+0000", 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6489999999999996e-06 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.442160+0000", 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053053650000000003 2024-09-21T18:57:30.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.442179+0000", 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9165e-05 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225492 7.f e142/114 7:f9eb6bc7:::benchmark_data_smithi081_73391_object86493:head v 142'14080, mlcod=142'14077)", 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.439196+0000", 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0068047230000000004, 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0069304409999999999, 2024-09-21T18:57:30.458 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.439196+0000", 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.439196+0000", 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.439197+0000", 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.37e-06 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.439204+0000", 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6699999999999997e-06 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.439214+0000", 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8689999999999999e-06 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.439220+0000", 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8519999999999998e-06 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445151+0000", 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059301149999999997 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445244+0000", 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3499999999999996e-05 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225491 7.7 e142/114 7:eb081df6:::benchmark_data_smithi081_73391_object86492:head v 142'14089, mlcod=142'14084)", 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.440773+0000", 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0052275530000000002, 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053976390000000001, 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.440773+0000", 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.440773+0000", 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.440774+0000", 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.457e-06 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.440781+0000", 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2190000000000001e-06 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.440790+0000", 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5179999999999993e-06 2024-09-21T18:57:30.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.440796+0000", 2024-09-21T18:57:30.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0209999999999996e-06 2024-09-21T18:57:30.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445337+0000", 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045404800000000004 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445355+0000", 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8898000000000001e-05 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225496 7.f e142/114)", 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.444410+0000", 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0015901159999999999, 2024-09-21T18:57:30.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018033109999999999, 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444410+0000", 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444410+0000", 2024-09-21T18:57:30.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444412+0000", 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6020000000000001e-06 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444419+0000", 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6610000000000003e-06 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444434+0000", 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5569e-05 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444442+0000", 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7200000000000006e-06 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225493 7.7 e142/114)", 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.444470+0000", 2024-09-21T18:57:30.470 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001530843, 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001777477, 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444470+0000", 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444470+0000", 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444470+0000", 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6700000000000003e-07 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444475+0000", 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5879999999999999e-06 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444481+0000", 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5139999999999998e-06 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444486+0000", 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9910000000000002e-06 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225490 7.e e142/114 7:7f86adf9:::benchmark_data_smithi081_73391_object86491:head v 142'13906, mlcod=142'13902)", 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.444557+0000", 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0014435380000000001, 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017269430000000001, 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.474 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444557+0000", 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444557+0000", 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444557+0000", 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1199999999999998e-07 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444562+0000", 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2270000000000004e-06 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444566+0000", 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9599999999999999e-06 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444571+0000", 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3490000000000002e-06 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444900+0000", 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00032916899999999998 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444918+0000", 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7666999999999999e-05 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(7.e 142/114 [PushOp(7:7e5b309b:::benchmark_data_smithi081_73391_object89648:head, version: 100'5541, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:7e5b309b:::benchmark_data_smithi081_73391_object89648:head@100'5541, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.444577+0000", 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0014236870000000001, 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017845580000000001, 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.478 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444577+0000", 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444577+0000", 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444577+0000", 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8000000000000002e-07 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444610+0000", 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3361000000000001e-05 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444629+0000", 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8386999999999999e-05 2024-09-21T18:57:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444642+0000", 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3472000000000001e-05 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:225500 7.6 7.f6f94416 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.445607+0000", 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00039313800000000002, 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079276200000000002, 2024-09-21T18:57:30.481 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 225500 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445607+0000", 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445607+0000", 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445609+0000", 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1500000000000002e-06 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445615+0000", 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4720000000000003e-06 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445626+0000", 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0778000000000001e-05 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.445632+0000", 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7429999999999996e-06 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.484 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:225488 7.6 7:62d98bcd:::benchmark_data_smithi081_73391_object86489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.429128+0000", 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.016872872000000001, 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017326332999999999, 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.485 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 225488 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.429128+0000", 2024-09-21T18:57:30.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.429128+0000", 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.429130+0000", 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1260000000000001e-06 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.429135+0000", 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3839999999999996e-06 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.429149+0000", 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4314999999999999e-05 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.429157+0000", 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9219999999999997e-06 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433146+0000", 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039884040000000001 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433335+0000", 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018953000000000001 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.433443+0000", 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108211 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.443544+0000", 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010100913 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.444469+0000", 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092469099999999995 2024-09-21T18:57:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:225487 7.f e142/114 7:f5f1b946:::benchmark_data_smithi081_73391_object86488:head v 142'14079, mlcod=142'14077)", 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:30.430289+0000", 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.015711184, 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016236606000000001, 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430289+0000", 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430289+0000", 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430290+0000", 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9800000000000003e-07 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430294+0000", 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7510000000000002e-06 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430299+0000", 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.631e-06 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430304+0000", 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.357e-06 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430679+0000", 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037544599999999998 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:30.430699+0000", 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0251999999999999e-05 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 11 2024-09-21T18:57:30.494 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:57:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.495 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:30.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:30.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.629 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:30.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:30.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:30.769 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:30.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:30 smithi081 ceph-mon[29384]: from='client.25030 v1:172.21.15.81:0/689652296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:30 smithi081 ceph-mon[29384]: pgmap v189: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 11149/168585 objects misplaced (6.613%); 28 MiB/s, 442 objects/s recovering 2024-09-21T18:57:30.907 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:30 smithi081 ceph-mon[39192]: from='client.25030 v1:172.21.15.81:0/689652296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:30.907 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:30 smithi081 ceph-mon[39192]: pgmap v189: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 11149/168585 objects misplaced (6.613%); 28 MiB/s, 442 objects/s recovering 2024-09-21T18:57:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:30.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:30.941 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:30 smithi104 ceph-mon[32438]: from='client.25030 v1:172.21.15.81:0/689652296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:30.941 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:30 smithi104 ceph-mon[32438]: pgmap v189: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 11149/168585 objects misplaced (6.613%); 28 MiB/s, 442 objects/s recovering 2024-09-21T18:57:31.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:31.045 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:31.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:31.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:31.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:31.327 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:31.419 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-21T18:57:27.441511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3670284851,"num_objects":56195,"num_object_clones":0,"num_object_copies":168585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11149,"num_objects_unfound":0,"num_objects_dirty":56195,"num_whiteouts":0,"num_read":1092,"num_read_kb":949,"num_write":222547,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27233,"num_bytes_recovered":1777999322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86354,"ondisk_log_size":86354,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12896288,"kb_used_data":11734536,"kb_used_omap":12,"kb_used_meta":1161459,"kb_avail":737064928,"statfs":{"total":767960285184,"available":754754486272,"internally_reserved":0,"allocated":12016164864,"data_stored":12013017585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1189334580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-545456128,"num_objects":-8323,"num_object_clones":0,"num_object_copies":-24969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5457,"num_objects_unfound":0,"num_objects_dirty":-8323,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3545,"num_bytes_recovered":232259584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.944811"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.787603+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:57:25.787603+0000","last_peered":"2024-09-21T18:57:25.787603+0000","last_clean":"2024-09-21T18:57:25.787603+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:57:25.787603+0000","last_undegraded":"2024-09-21T18:57:25.787603+0000","last_fullsized":"2024-09-21T18:57:25.787603+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":215,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764128+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:57:25.764128+0000","last_peered":"2024-09-21T18:57:25.764128+0000","last_clean":"2024-09-21T18:57:25.764128+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:57:25.764128+0000","last_undegraded":"2024-09-21T18:57:25.764128+0000","last_fullsized":"2024-09-21T18:57:25.764128+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771827+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:57:25.771827+0000","last_peered":"2024-09-21T18:57:25.771827+0000","last_clean":"2024-09-21T18:57:25.771827+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:57:25.771827+0000","last_undegraded":"2024-09-21T18:57:25.771827+0000","last_fullsized":"2024-09-21T18:57:25.771827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.763934+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:57:25.763934+0000","last_peered":"2024-09-21T18:57:25.763934+0000","last_clean":"2024-09-21T18:57:25.763934+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:57:25.763934+0000","last_undegraded":"2024-09-21T18:57:25.763934+0000","last_fullsized":"2024-09-21T18:57:25.763934+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":177,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801402+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:57:26.801402+0000","last_peered":"2024-09-21T18:57:26.801402+0000","last_clean":"2024-09-21T18:57:26.801402+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:57:26.801402+0000","last_undegraded":"2024-09-21T18:57:26.801402+0000","last_fullsized":"2024-09-21T18:57:26.801402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":190,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800481+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:57:26.800481+0000","last_peered":"2024-09-21T18:57:26.800481+0000","last_clean":"2024-09-21T18:57:26.800481+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:57:26.800481+0000","last_undegraded":"2024-09-21T18:57:26.800481+0000","last_fullsized":"2024-09-21T18:57:26.800481+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":248,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800456+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:57:26.800456+0000","last_peered":"2024-09-21T18:57:26.800456+0000","last_clean":"2024-09-21T18:57:26.800456+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:57:26.800456+0000","last_undegraded":"2024-09-21T18:57:26.800456+0000","last_fullsized":"2024-09-21T18:57:26.800456+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.763966+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:57:25.763966+0000","last_peered":"2024-09-21T18:57:25.763966+0000","last_clean":"2024-09-21T18:57:25.763966+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:57:25.763966+0000","last_undegraded":"2024-09-21T18:57:25.763966+0000","last_fullsized":"2024-09-21T18:57:25.763966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778182+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:57:25.778182+0000","last_peered":"2024-09-21T18:57:25.778182+0000","last_clean":"2024-09-21T18:57:25.778182+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:57:25.778182+0000","last_undegraded":"2024-09-21T18:57:25.778182+0000","last_fullsized":"2024-09-21T18:57:25.778182+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:22.885755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":240,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764892+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:57:25.764892+0000","last_peered":"2024-09-21T18:57:25.764892+0000","last_clean":"2024-09-21T18:57:25.764892+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:57:25.764892+0000","last_undegraded":"2024-09-21T18:57:25.764892+0000","last_fullsized":"2024-09-21T18:57:25.764892+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":200,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777807+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:57:25.777807+0000","last_peered":"2024-09-21T18:57:25.777807+0000","last_clean":"2024-09-21T18:57:25.777807+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:57:25.777807+0000","last_undegraded":"2024-09-21T18:57:25.777807+0000","last_fullsized":"2024-09-21T18:57:25.777807+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.948408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778022+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:57:25.778022+0000","last_peered":"2024-09-21T18:57:25.778022+0000","last_clean":"2024-09-21T18:57:25.778022+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:57:25.778022+0000","last_undegraded":"2024-09-21T18:57:25.778022+0000","last_fullsized":"2024-09-21T18:57:25.778022+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777822+0000","last_change":"2024-09-21T18:57:19.821897+0000","last_active":"2024-09-21T18:57:25.777822+0000","last_peered":"2024-09-21T18:57:25.777822+0000","last_clean":"2024-09-21T18:57:25.777822+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:57:25.777822+0000","last_undegraded":"2024-09-21T18:57:25.777822+0000","last_fullsized":"2024-09-21T18:57:25.777822+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.821849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.129486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":192,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764210+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:57:25.764210+0000","last_peered":"2024-09-21T18:57:25.764210+0000","last_clean":"2024-09-21T18:57:25.764210+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:57:25.764210+0000","last_undegraded":"2024-09-21T18:57:25.764210+0000","last_fullsized":"2024-09-21T18:57:25.764210+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771564+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:57:25.771564+0000","last_peered":"2024-09-21T18:57:25.771564+0000","last_clean":"2024-09-21T18:57:25.771564+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:57:25.771564+0000","last_undegraded":"2024-09-21T18:57:25.771564+0000","last_fullsized":"2024-09-21T18:57:25.771564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:16.165478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777589+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:57:25.777589+0000","last_peered":"2024-09-21T18:57:25.777589+0000","last_clean":"2024-09-21T18:57:25.777589+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:57:25.777589+0000","last_undegraded":"2024-09-21T18:57:25.777589+0000","last_fullsized":"2024-09-21T18:57:25.777589+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:46.401461+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765342+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:57:25.765342+0000","last_peered":"2024-09-21T18:57:25.765342+0000","last_clean":"2024-09-21T18:57:25.765342+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:57:25.765342+0000","last_undegraded":"2024-09-21T18:57:25.765342+0000","last_fullsized":"2024-09-21T18:57:25.765342+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778631+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:57:25.778631+0000","last_peered":"2024-09-21T18:57:25.778631+0000","last_clean":"2024-09-21T18:57:25.778631+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:57:25.778631+0000","last_undegraded":"2024-09-21T18:57:25.778631+0000","last_fullsized":"2024-09-21T18:57:25.778631+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":207,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.775130+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:57:25.775130+0000","last_peered":"2024-09-21T18:57:25.775130+0000","last_clean":"2024-09-21T18:57:25.775130+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:57:25.775130+0000","last_undegraded":"2024-09-21T18:57:25.775130+0000","last_fullsized":"2024-09-21T18:57:25.775130+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":192,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778957+0000","last_change":"2024-09-21T18:57:20.229156+0000","last_active":"2024-09-21T18:57:25.778957+0000","last_peered":"2024-09-21T18:57:25.778957+0000","last_clean":"2024-09-21T18:57:25.778957+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:57:25.778957+0000","last_undegraded":"2024-09-21T18:57:25.778957+0000","last_fullsized":"2024-09-21T18:57:25.778957+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.229088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.229088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.406792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":170,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801060+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:57:26.801060+0000","last_peered":"2024-09-21T18:57:26.801060+0000","last_clean":"2024-09-21T18:57:26.801060+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:57:26.801060+0000","last_undegraded":"2024-09-21T18:57:26.801060+0000","last_fullsized":"2024-09-21T18:57:26.801060+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":194,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764945+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:57:25.764945+0000","last_peered":"2024-09-21T18:57:25.764945+0000","last_clean":"2024-09-21T18:57:25.764945+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:57:25.764945+0000","last_undegraded":"2024-09-21T18:57:25.764945+0000","last_fullsized":"2024-09-21T18:57:25.764945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":208,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.792738+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:57:26.792738+0000","last_peered":"2024-09-21T18:57:26.792738+0000","last_clean":"2024-09-21T18:57:26.792738+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:57:26.792738+0000","last_undegraded":"2024-09-21T18:57:26.792738+0000","last_fullsized":"2024-09-21T18:57:26.792738+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765230+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:57:25.765230+0000","last_peered":"2024-09-21T18:57:25.765230+0000","last_clean":"2024-09-21T18:57:25.765230+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:57:25.765230+0000","last_undegraded":"2024-09-21T18:57:25.765230+0000","last_fullsized":"2024-09-21T18:57:25.765230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771831+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:57:25.771831+0000","last_peered":"2024-09-21T18:57:25.771831+0000","last_clean":"2024-09-21T18:57:25.771831+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:57:25.771831+0000","last_undegraded":"2024-09-21T18:57:25.771831+0000","last_fullsized":"2024-09-21T18:57:25.771831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":187,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771911+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:57:25.771911+0000","last_peered":"2024-09-21T18:57:25.771911+0000","last_clean":"2024-09-21T18:57:25.771911+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:57:25.771911+0000","last_undegraded":"2024-09-21T18:57:25.771911+0000","last_fullsized":"2024-09-21T18:57:25.771911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":78,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.779872+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:57:25.779872+0000","last_peered":"2024-09-21T18:57:25.779872+0000","last_clean":"2024-09-21T18:57:25.779872+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:57:25.779872+0000","last_undegraded":"2024-09-21T18:57:25.779872+0000","last_fullsized":"2024-09-21T18:57:25.779872+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":241,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.769670+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:57:25.769670+0000","last_peered":"2024-09-21T18:57:25.769670+0000","last_clean":"2024-09-21T18:57:25.769670+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:57:25.769670+0000","last_undegraded":"2024-09-21T18:57:25.769670+0000","last_fullsized":"2024-09-21T18:57:25.769670+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"141'13676","reported_seq":13813,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.400620+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:57:26.400620+0000","last_peered":"2024-09-21T18:57:26.400620+0000","last_clean":"2024-09-21T18:57:26.400620+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:57:26.400620+0000","last_undegraded":"2024-09-21T18:57:26.400620+0000","last_fullsized":"2024-09-21T18:57:26.400620+0000","mapping_epoch":79,"log_start":"110'7500","ondisk_log_start":"110'7500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225574912,"num_objects":3442,"num_object_clones":0,"num_object_copies":10326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13676,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":207,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.779357+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:57:25.779357+0000","last_peered":"2024-09-21T18:57:25.779357+0000","last_clean":"2024-09-21T18:57:25.779357+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:57:25.779357+0000","last_undegraded":"2024-09-21T18:57:25.779357+0000","last_fullsized":"2024-09-21T18:57:25.779357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.223966+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":189,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801196+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:57:26.801196+0000","last_peered":"2024-09-21T18:57:26.801196+0000","last_clean":"2024-09-21T18:57:26.801196+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:57:26.801196+0000","last_undegraded":"2024-09-21T18:57:26.801196+0000","last_fullsized":"2024-09-21T18:57:26.801196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":260,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765431+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:57:25.765431+0000","last_peered":"2024-09-21T18:57:25.765431+0000","last_clean":"2024-09-21T18:57:25.765431+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:57:25.765431+0000","last_undegraded":"2024-09-21T18:57:25.765431+0000","last_fullsized":"2024-09-21T18:57:25.765431+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":196,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777900+0000","last_change":"2024-09-21T18:56:37.863087+0000","last_active":"2024-09-21T18:57:25.777900+0000","last_peered":"2024-09-21T18:57:25.777900+0000","last_clean":"2024-09-21T18:57:25.777900+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:57:25.777900+0000","last_undegraded":"2024-09-21T18:57:25.777900+0000","last_fullsized":"2024-09-21T18:57:25.777900+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:37.863026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:37.863026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.972736+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"142'13782","reported_seq":13921,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:27.057194+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:57:27.057194+0000","last_peered":"2024-09-21T18:57:27.057194+0000","last_clean":"2024-09-21T18:57:27.057194+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:57:27.057194+0000","last_undegraded":"2024-09-21T18:57:27.057194+0000","last_fullsized":"2024-09-21T18:57:27.057194+0000","mapping_epoch":79,"log_start":"119'9300","ondisk_log_start":"119'9300","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224133120,"num_objects":3420,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13782,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":268,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800898+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:57:26.800898+0000","last_peered":"2024-09-21T18:57:26.800898+0000","last_clean":"2024-09-21T18:57:26.800898+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:57:26.800898+0000","last_undegraded":"2024-09-21T18:57:26.800898+0000","last_fullsized":"2024-09-21T18:57:26.800898+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":203,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801296+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:57:26.801296+0000","last_peered":"2024-09-21T18:57:26.801296+0000","last_clean":"2024-09-21T18:57:26.801296+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:57:26.801296+0000","last_undegraded":"2024-09-21T18:57:26.801296+0000","last_fullsized":"2024-09-21T18:57:26.801296+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":179,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.767254+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:57:25.767254+0000","last_peered":"2024-09-21T18:57:25.767254+0000","last_clean":"2024-09-21T18:57:25.767254+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:57:25.767254+0000","last_undegraded":"2024-09-21T18:57:25.767254+0000","last_fullsized":"2024-09-21T18:57:25.767254+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:32.288577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764191+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:57:25.764191+0000","last_peered":"2024-09-21T18:57:25.764191+0000","last_clean":"2024-09-21T18:57:25.764191+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:57:25.764191+0000","last_undegraded":"2024-09-21T18:57:25.764191+0000","last_fullsized":"2024-09-21T18:57:25.764191+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"142'14047","reported_seq":14186,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:27.062205+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:57:27.062205+0000","last_peered":"2024-09-21T18:57:27.062205+0000","last_clean":"2024-09-21T18:57:27.062205+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:57:27.062205+0000","last_undegraded":"2024-09-21T18:57:27.062205+0000","last_fullsized":"2024-09-21T18:57:27.062205+0000","mapping_epoch":79,"log_start":"119'9500","ondisk_log_start":"119'9500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225771520,"num_objects":3445,"num_object_clones":0,"num_object_copies":10335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14047,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":204,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.767367+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:57:25.767367+0000","last_peered":"2024-09-21T18:57:25.767367+0000","last_clean":"2024-09-21T18:57:25.767367+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:57:25.767367+0000","last_undegraded":"2024-09-21T18:57:25.767367+0000","last_fullsized":"2024-09-21T18:57:25.767367+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":70,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.780172+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:57:25.780172+0000","last_peered":"2024-09-21T18:57:25.780172+0000","last_clean":"2024-09-21T18:57:25.780172+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:57:25.780172+0000","last_undegraded":"2024-09-21T18:57:25.780172+0000","last_fullsized":"2024-09-21T18:57:25.780172+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":234,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800707+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:57:26.800707+0000","last_peered":"2024-09-21T18:57:26.800707+0000","last_clean":"2024-09-21T18:57:26.800707+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:57:26.800707+0000","last_undegraded":"2024-09-21T18:57:26.800707+0000","last_fullsized":"2024-09-21T18:57:26.800707+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771919+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:57:25.771919+0000","last_peered":"2024-09-21T18:57:25.771919+0000","last_clean":"2024-09-21T18:57:25.771919+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:57:25.771919+0000","last_undegraded":"2024-09-21T18:57:25.771919+0000","last_fullsized":"2024-09-21T18:57:25.771919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.105131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"141'13786","reported_seq":14014,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.405687+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:57:26.405687+0000","last_peered":"2024-09-21T18:57:26.405687+0000","last_clean":"2024-09-21T18:57:26.405687+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:57:26.405687+0000","last_undegraded":"2024-09-21T18:57:26.405687+0000","last_fullsized":"2024-09-21T18:57:26.405687+0000","mapping_epoch":79,"log_start":"117'9000","ondisk_log_start":"117'9000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13786,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.783296+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:57:25.783296+0000","last_peered":"2024-09-21T18:57:25.783296+0000","last_clean":"2024-09-21T18:57:25.783296+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:57:25.783296+0000","last_undegraded":"2024-09-21T18:57:25.783296+0000","last_fullsized":"2024-09-21T18:57:25.783296+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.091213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778223+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:57:25.778223+0000","last_peered":"2024-09-21T18:57:25.778223+0000","last_clean":"2024-09-21T18:57:25.778223+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:57:25.778223+0000","last_undegraded":"2024-09-21T18:57:25.778223+0000","last_fullsized":"2024-09-21T18:57:25.778223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.895558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":89,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778898+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:57:25.778898+0000","last_peered":"2024-09-21T18:57:25.778898+0000","last_clean":"2024-09-21T18:57:25.778898+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:57:25.778898+0000","last_undegraded":"2024-09-21T18:57:25.778898+0000","last_fullsized":"2024-09-21T18:57:25.778898+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778154+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:57:25.778154+0000","last_peered":"2024-09-21T18:57:25.778154+0000","last_clean":"2024-09-21T18:57:25.778154+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:57:25.778154+0000","last_undegraded":"2024-09-21T18:57:25.778154+0000","last_fullsized":"2024-09-21T18:57:25.778154+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"141'14089","reported_seq":14375,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.428054+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:57:26.428054+0000","last_peered":"2024-09-21T18:57:26.428054+0000","last_clean":"2024-09-21T18:57:26.428054+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:57:26.428054+0000","last_undegraded":"2024-09-21T18:57:26.428054+0000","last_fullsized":"2024-09-21T18:57:26.428054+0000","mapping_epoch":79,"log_start":"111'8500","ondisk_log_start":"111'8500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:43.426801+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233897984,"num_objects":3569,"num_object_clones":0,"num_object_copies":10707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14089,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":84,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778713+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:57:25.778713+0000","last_peered":"2024-09-21T18:57:25.778713+0000","last_clean":"2024-09-21T18:57:25.778713+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:57:25.778713+0000","last_undegraded":"2024-09-21T18:57:25.778713+0000","last_fullsized":"2024-09-21T18:57:25.778713+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":187,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.783512+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:57:25.783512+0000","last_peered":"2024-09-21T18:57:25.783512+0000","last_clean":"2024-09-21T18:57:25.783512+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:57:25.783512+0000","last_undegraded":"2024-09-21T18:57:25.783512+0000","last_fullsized":"2024-09-21T18:57:25.783512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":192,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.769000+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:57:25.769000+0000","last_peered":"2024-09-21T18:57:25.769000+0000","last_clean":"2024-09-21T18:57:25.769000+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:57:25.769000+0000","last_undegraded":"2024-09-21T18:57:25.769000+0000","last_fullsized":"2024-09-21T18:57:25.769000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771783+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:57:25.771783+0000","last_peered":"2024-09-21T18:57:25.771783+0000","last_clean":"2024-09-21T18:57:25.771783+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:57:25.771783+0000","last_undegraded":"2024-09-21T18:57:25.771783+0000","last_fullsized":"2024-09-21T18:57:25.771783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.637336+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"141'13849","reported_seq":20473,"reported_epoch":141,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:26.466499+0000","last_change":"2024-09-21T18:57:02.353922+0000","last_active":"2024-09-21T18:57:26.466499+0000","last_peered":"2024-09-21T18:57:26.466499+0000","last_clean":"2024-09-21T18:55:50.489495+0000","last_became_active":"2024-09-21T18:55:52.881338+0000","last_became_peered":"2024-09-21T18:55:52.881338+0000","last_unstale":"2024-09-21T18:57:26.466499+0000","last_undegraded":"2024-09-21T18:57:26.466499+0000","last_fullsized":"2024-09-21T18:57:26.466499+0000","mapping_epoch":114,"log_start":"110'7700","ondisk_log_start":"110'7700","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227999744,"num_objects":3479,"num_object_clones":0,"num_object_copies":10437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":744,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13849,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3238,"num_bytes_recovered":212205568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,3,1],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":3479}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":283,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.783349+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:57:25.783349+0000","last_peered":"2024-09-21T18:57:25.783349+0000","last_clean":"2024-09-21T18:57:25.783349+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:57:25.783349+0000","last_undegraded":"2024-09-21T18:57:25.783349+0000","last_fullsized":"2024-09-21T18:57:25.783349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":209,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764599+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:57:25.764599+0000","last_peered":"2024-09-21T18:57:25.764599+0000","last_clean":"2024-09-21T18:57:25.764599+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:57:25.764599+0000","last_undegraded":"2024-09-21T18:57:25.764599+0000","last_fullsized":"2024-09-21T18:57:25.764599+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":205,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778040+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:57:25.778040+0000","last_peered":"2024-09-21T18:57:25.778040+0000","last_clean":"2024-09-21T18:57:25.778040+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:57:25.778040+0000","last_undegraded":"2024-09-21T18:57:25.778040+0000","last_fullsized":"2024-09-21T18:57:25.778040+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.854902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.786625+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:57:25.786625+0000","last_peered":"2024-09-21T18:57:25.786625+0000","last_clean":"2024-09-21T18:57:25.786625+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:57:25.786625+0000","last_undegraded":"2024-09-21T18:57:25.786625+0000","last_fullsized":"2024-09-21T18:57:25.786625+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":177,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800235+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:57:26.800235+0000","last_peered":"2024-09-21T18:57:26.800235+0000","last_clean":"2024-09-21T18:57:26.800235+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:57:26.800235+0000","last_undegraded":"2024-09-21T18:57:26.800235+0000","last_fullsized":"2024-09-21T18:57:26.800235+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"141'14009","reported_seq":14156,"reported_epoch":141,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T18:57:26.416508+0000","last_change":"2024-09-21T18:56:22.889265+0000","last_active":"2024-09-21T18:57:26.416508+0000","last_peered":"2024-09-21T18:57:26.416508+0000","last_clean":"2024-09-21T18:55:50.469777+0000","last_became_active":"2024-09-21T18:55:52.866581+0000","last_became_peered":"2024-09-21T18:55:52.866581+0000","last_unstale":"2024-09-21T18:57:26.416508+0000","last_undegraded":"2024-09-21T18:57:26.416508+0000","last_fullsized":"2024-09-21T18:57:26.416508+0000","mapping_epoch":114,"log_start":"118'9300","ondisk_log_start":"118'9300","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229441536,"num_objects":3501,"num_object_clones":0,"num_object_copies":10503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3501,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14009,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":3501}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.775331+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:57:25.775331+0000","last_peered":"2024-09-21T18:57:25.775331+0000","last_clean":"2024-09-21T18:57:25.775331+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:57:25.775331+0000","last_undegraded":"2024-09-21T18:57:25.775331+0000","last_fullsized":"2024-09-21T18:57:25.775331+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.769892+0000","last_change":"2024-09-21T18:57:22.224988+0000","last_active":"2024-09-21T18:57:25.769892+0000","last_peered":"2024-09-21T18:57:25.769892+0000","last_clean":"2024-09-21T18:57:25.769892+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:57:25.769892+0000","last_undegraded":"2024-09-21T18:57:25.769892+0000","last_fullsized":"2024-09-21T18:57:25.769892+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:22.220182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:22.220182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.617863+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":218,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.774585+0000","last_change":"2024-09-21T18:55:51.541888+0000","last_active":"2024-09-21T18:57:25.774585+0000","last_peered":"2024-09-21T18:57:25.774585+0000","last_clean":"2024-09-21T18:57:25.774585+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:57:25.774585+0000","last_undegraded":"2024-09-21T18:57:25.774585+0000","last_fullsized":"2024-09-21T18:57:25.774585+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:25.237173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.237173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.776993+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:57:25.776993+0000","last_peered":"2024-09-21T18:57:25.776993+0000","last_clean":"2024-09-21T18:57:25.776993+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:57:25.776993+0000","last_undegraded":"2024-09-21T18:57:25.776993+0000","last_fullsized":"2024-09-21T18:57:25.776993+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772282+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:57:25.772282+0000","last_peered":"2024-09-21T18:57:25.772282+0000","last_clean":"2024-09-21T18:57:25.772282+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:57:25.772282+0000","last_undegraded":"2024-09-21T18:57:25.772282+0000","last_fullsized":"2024-09-21T18:57:25.772282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"141'14205","reported_seq":28485,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.440995+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:57:26.440995+0000","last_peered":"2024-09-21T18:57:26.440995+0000","last_clean":"2024-09-21T18:57:26.440995+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:57:26.440995+0000","last_undegraded":"2024-09-21T18:57:26.440995+0000","last_fullsized":"2024-09-21T18:57:26.440995+0000","mapping_epoch":130,"log_start":"117'9100","ondisk_log_start":"117'9100","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.065509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237043712,"num_objects":3617,"num_object_clones":0,"num_object_copies":10851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14205,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":80,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.779975+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:57:25.779975+0000","last_peered":"2024-09-21T18:57:25.779975+0000","last_clean":"2024-09-21T18:57:25.779975+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:57:25.779975+0000","last_undegraded":"2024-09-21T18:57:25.779975+0000","last_fullsized":"2024-09-21T18:57:25.779975+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.779159+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:57:25.779159+0000","last_peered":"2024-09-21T18:57:25.779159+0000","last_clean":"2024-09-21T18:57:25.779159+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:57:25.779159+0000","last_undegraded":"2024-09-21T18:57:25.779159+0000","last_fullsized":"2024-09-21T18:57:25.779159+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":480,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771525+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:57:25.771525+0000","last_peered":"2024-09-21T18:57:25.771525+0000","last_clean":"2024-09-21T18:57:25.771525+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:57:25.771525+0000","last_undegraded":"2024-09-21T18:57:25.771525+0000","last_fullsized":"2024-09-21T18:57:25.771525+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:17.496871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":248,"num_read_kb":243,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":208,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778709+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:57:25.778709+0000","last_peered":"2024-09-21T18:57:25.778709+0000","last_clean":"2024-09-21T18:57:25.778709+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:57:25.778709+0000","last_undegraded":"2024-09-21T18:57:25.778709+0000","last_fullsized":"2024-09-21T18:57:25.778709+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.378290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.782971+0000","last_change":"2024-09-21T18:57:18.176297+0000","last_active":"2024-09-21T18:57:25.782971+0000","last_peered":"2024-09-21T18:57:25.782971+0000","last_clean":"2024-09-21T18:57:25.782971+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:57:25.782971+0000","last_undegraded":"2024-09-21T18:57:25.782971+0000","last_fullsized":"2024-09-21T18:57:25.782971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:18.176233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:18.176233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:24.204170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"141'13778","reported_seq":25795,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.440677+0000","last_change":"2024-09-21T18:57:20.837816+0000","last_active":"2024-09-21T18:57:26.440677+0000","last_peered":"2024-09-21T18:57:26.440677+0000","last_clean":"2024-09-21T18:57:26.440677+0000","last_became_active":"2024-09-21T18:57:20.750225+0000","last_became_peered":"2024-09-21T18:57:20.750225+0000","last_unstale":"2024-09-21T18:57:26.440677+0000","last_undegraded":"2024-09-21T18:57:26.440677+0000","last_fullsized":"2024-09-21T18:57:26.440677+0000","mapping_epoch":139,"log_start":"117'8902","ondisk_log_start":"117'8902","created":79,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234749952,"num_objects":3582,"num_object_clones":0,"num_object_copies":10746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13778,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":256,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.770222+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:57:25.770222+0000","last_peered":"2024-09-21T18:57:25.770222+0000","last_clean":"2024-09-21T18:57:25.770222+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:57:25.770222+0000","last_undegraded":"2024-09-21T18:57:25.770222+0000","last_fullsized":"2024-09-21T18:57:25.770222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":190,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.792254+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:57:26.792254+0000","last_peered":"2024-09-21T18:57:26.792254+0000","last_clean":"2024-09-21T18:57:26.792254+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:57:26.792254+0000","last_undegraded":"2024-09-21T18:57:26.792254+0000","last_fullsized":"2024-09-21T18:57:26.792254+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":249,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771694+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:57:25.771694+0000","last_peered":"2024-09-21T18:57:25.771694+0000","last_clean":"2024-09-21T18:57:25.771694+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:57:25.771694+0000","last_undegraded":"2024-09-21T18:57:25.771694+0000","last_fullsized":"2024-09-21T18:57:25.771694+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:05.121733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772117+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:57:25.772117+0000","last_peered":"2024-09-21T18:57:25.772117+0000","last_clean":"2024-09-21T18:57:25.772117+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:57:25.772117+0000","last_undegraded":"2024-09-21T18:57:25.772117+0000","last_fullsized":"2024-09-21T18:57:25.772117+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:01.123772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778658+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:57:25.778658+0000","last_peered":"2024-09-21T18:57:25.778658+0000","last_clean":"2024-09-21T18:57:25.778658+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:57:25.778658+0000","last_undegraded":"2024-09-21T18:57:25.778658+0000","last_fullsized":"2024-09-21T18:57:25.778658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:26.445821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"141'14010","reported_seq":14249,"reported_epoch":141,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T18:57:26.431697+0000","last_change":"2024-09-21T18:56:22.889684+0000","last_active":"2024-09-21T18:57:26.431697+0000","last_peered":"2024-09-21T18:57:26.431697+0000","last_clean":"2024-09-21T18:55:50.477883+0000","last_became_active":"2024-09-21T18:55:52.879827+0000","last_became_peered":"2024-09-21T18:55:52.879827+0000","last_unstale":"2024-09-21T18:57:26.431697+0000","last_undegraded":"2024-09-21T18:57:26.431697+0000","last_fullsized":"2024-09-21T18:57:26.431697+0000","mapping_epoch":114,"log_start":"110'7800","ondisk_log_start":"110'7800","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6210,"log_dups_size":0,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236519448,"num_objects":3610,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3610,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14010,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":3610}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":201,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765543+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:57:25.765543+0000","last_peered":"2024-09-21T18:57:25.765543+0000","last_clean":"2024-09-21T18:57:25.765543+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:57:25.765543+0000","last_undegraded":"2024-09-21T18:57:25.765543+0000","last_fullsized":"2024-09-21T18:57:25.765543+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.767617+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:57:25.767617+0000","last_peered":"2024-09-21T18:57:25.767617+0000","last_clean":"2024-09-21T18:57:25.767617+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:57:25.767617+0000","last_undegraded":"2024-09-21T18:57:25.767617+0000","last_fullsized":"2024-09-21T18:57:25.767617+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778550+0000","last_change":"2024-09-21T18:56:33.913317+0000","last_active":"2024-09-21T18:57:25.778550+0000","last_peered":"2024-09-21T18:57:25.778550+0000","last_clean":"2024-09-21T18:57:25.778550+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:57:25.778550+0000","last_undegraded":"2024-09-21T18:57:25.778550+0000","last_fullsized":"2024-09-21T18:57:25.778550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:33.913250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:33.913250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:52.937683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":169,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.792865+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:57:26.792865+0000","last_peered":"2024-09-21T18:57:26.792865+0000","last_clean":"2024-09-21T18:57:26.792865+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:57:26.792865+0000","last_undegraded":"2024-09-21T18:57:26.792865+0000","last_fullsized":"2024-09-21T18:57:26.792865+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"141'13689","reported_seq":20929,"reported_epoch":141,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:26.512947+0000","last_change":"2024-09-21T18:56:52.172568+0000","last_active":"2024-09-21T18:57:26.512947+0000","last_peered":"2024-09-21T18:57:26.512947+0000","last_clean":"2024-09-21T18:55:50.490306+0000","last_became_active":"2024-09-21T18:55:52.880630+0000","last_became_peered":"2024-09-21T18:55:52.880630+0000","last_unstale":"2024-09-21T18:57:26.512947+0000","last_undegraded":"2024-09-21T18:57:26.512947+0000","last_fullsized":"2024-09-21T18:57:26.512947+0000","mapping_epoch":114,"log_start":"107'7000","ondisk_log_start":"107'7000","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902592,"num_objects":3447,"num_object_clones":0,"num_object_copies":10341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":650,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13689,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3505,"num_bytes_recovered":229703680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":3447}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":210,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765724+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:57:25.765724+0000","last_peered":"2024-09-21T18:57:25.765724+0000","last_clean":"2024-09-21T18:57:25.765724+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:57:25.765724+0000","last_undegraded":"2024-09-21T18:57:25.765724+0000","last_fullsized":"2024-09-21T18:57:25.765724+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.769154+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:57:25.769154+0000","last_peered":"2024-09-21T18:57:25.769154+0000","last_clean":"2024-09-21T18:57:25.769154+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:57:25.769154+0000","last_undegraded":"2024-09-21T18:57:25.769154+0000","last_fullsized":"2024-09-21T18:57:25.769154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.783044+0000","last_change":"2024-09-21T18:55:51.535384+0000","last_active":"2024-09-21T18:57:25.783044+0000","last_peered":"2024-09-21T18:57:25.783044+0000","last_clean":"2024-09-21T18:57:25.783044+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:57:25.783044+0000","last_undegraded":"2024-09-21T18:57:25.783044+0000","last_fullsized":"2024-09-21T18:57:25.783044+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.042616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.775486+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:57:25.775486+0000","last_peered":"2024-09-21T18:57:25.775486+0000","last_clean":"2024-09-21T18:57:25.775486+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:57:25.775486+0000","last_undegraded":"2024-09-21T18:57:25.775486+0000","last_fullsized":"2024-09-21T18:57:25.775486+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"142'13888","reported_seq":15793,"reported_epoch":142,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:27.088442+0000","last_change":"2024-09-21T18:57:18.832498+0000","last_active":"2024-09-21T18:57:27.088442+0000","last_peered":"2024-09-21T18:57:27.088442+0000","last_clean":"2024-09-21T18:55:50.495661+0000","last_became_active":"2024-09-21T18:55:52.870612+0000","last_became_peered":"2024-09-21T18:55:52.870612+0000","last_unstale":"2024-09-21T18:57:27.088442+0000","last_undegraded":"2024-09-21T18:57:27.088442+0000","last_fullsized":"2024-09-21T18:57:27.088442+0000","mapping_epoch":114,"log_start":"119'9400","ondisk_log_start":"119'9400","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226099200,"num_objects":3450,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2644,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13888,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":881,"num_bytes_recovered":57737216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":3450}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":197,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765432+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:57:25.765432+0000","last_peered":"2024-09-21T18:57:25.765432+0000","last_clean":"2024-09-21T18:57:25.765432+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:57:25.765432+0000","last_undegraded":"2024-09-21T18:57:25.765432+0000","last_fullsized":"2024-09-21T18:57:25.765432+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":217,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801794+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:57:26.801794+0000","last_peered":"2024-09-21T18:57:26.801794+0000","last_clean":"2024-09-21T18:57:26.801794+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:57:26.801794+0000","last_undegraded":"2024-09-21T18:57:26.801794+0000","last_fullsized":"2024-09-21T18:57:26.801794+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":238,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.776158+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:57:25.776158+0000","last_peered":"2024-09-21T18:57:25.776158+0000","last_clean":"2024-09-21T18:57:25.776158+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:57:25.776158+0000","last_undegraded":"2024-09-21T18:57:25.776158+0000","last_fullsized":"2024-09-21T18:57:25.776158+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765253+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:57:25.765253+0000","last_peered":"2024-09-21T18:57:25.765253+0000","last_clean":"2024-09-21T18:57:25.765253+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:57:25.765253+0000","last_undegraded":"2024-09-21T18:57:25.765253+0000","last_fullsized":"2024-09-21T18:57:25.765253+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"141'13890","reported_seq":14107,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.488239+0000","last_change":"2024-09-21T18:55:30.532749+0000","last_active":"2024-09-21T18:57:26.488239+0000","last_peered":"2024-09-21T18:57:26.488239+0000","last_clean":"2024-09-21T18:57:26.488239+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:57:26.488239+0000","last_undegraded":"2024-09-21T18:57:26.488239+0000","last_fullsized":"2024-09-21T18:57:26.488239+0000","mapping_epoch":79,"log_start":"108'7200","ondisk_log_start":"108'7200","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7201","last_scrub_stamp":"2024-09-21T18:55:30.532526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.532526+0000","objects_scrubbed":7133,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233832448,"num_objects":3568,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13890,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":76,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778803+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:57:25.778803+0000","last_peered":"2024-09-21T18:57:25.778803+0000","last_clean":"2024-09-21T18:57:25.778803+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:57:25.778803+0000","last_undegraded":"2024-09-21T18:57:25.778803+0000","last_fullsized":"2024-09-21T18:57:25.778803+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":194,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777966+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:57:25.777966+0000","last_peered":"2024-09-21T18:57:25.777966+0000","last_clean":"2024-09-21T18:57:25.777966+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:57:25.777966+0000","last_undegraded":"2024-09-21T18:57:25.777966+0000","last_fullsized":"2024-09-21T18:57:25.777966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.889938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.774769+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:57:25.774769+0000","last_peered":"2024-09-21T18:57:25.774769+0000","last_clean":"2024-09-21T18:57:25.774769+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:57:25.774769+0000","last_undegraded":"2024-09-21T18:57:25.774769+0000","last_fullsized":"2024-09-21T18:57:25.774769+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.779083+0000","last_change":"2024-09-21T18:55:51.540541+0000","last_active":"2024-09-21T18:57:25.779083+0000","last_peered":"2024-09-21T18:57:25.779083+0000","last_clean":"2024-09-21T18:57:25.779083+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:57:25.779083+0000","last_undegraded":"2024-09-21T18:57:25.779083+0000","last_fullsized":"2024-09-21T18:57:25.779083+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"141'13980","reported_seq":27097,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.467491+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:57:26.467491+0000","last_peered":"2024-09-21T18:57:26.467491+0000","last_clean":"2024-09-21T18:57:26.467491+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:57:26.467491+0000","last_undegraded":"2024-09-21T18:57:26.467491+0000","last_fullsized":"2024-09-21T18:57:26.467491+0000","mapping_epoch":134,"log_start":"111'8400","ondisk_log_start":"111'8400","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5580,"log_dups_size":0,"ondisk_log_size":5580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226230272,"num_objects":3452,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13980,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":212,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.768919+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:57:25.768919+0000","last_peered":"2024-09-21T18:57:25.768919+0000","last_clean":"2024-09-21T18:57:25.768919+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:57:25.768919+0000","last_undegraded":"2024-09-21T18:57:25.768919+0000","last_fullsized":"2024-09-21T18:57:25.768919+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772445+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:57:25.772445+0000","last_peered":"2024-09-21T18:57:25.772445+0000","last_clean":"2024-09-21T18:57:25.772445+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:57:25.772445+0000","last_undegraded":"2024-09-21T18:57:25.772445+0000","last_fullsized":"2024-09-21T18:57:25.772445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":192,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.775860+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:57:25.775860+0000","last_peered":"2024-09-21T18:57:25.775860+0000","last_clean":"2024-09-21T18:57:25.775860+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:57:25.775860+0000","last_undegraded":"2024-09-21T18:57:25.775860+0000","last_fullsized":"2024-09-21T18:57:25.775860+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771587+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T18:57:25.771587+0000","last_peered":"2024-09-21T18:57:25.771587+0000","last_clean":"2024-09-21T18:57:25.771587+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:57:25.771587+0000","last_undegraded":"2024-09-21T18:57:25.771587+0000","last_fullsized":"2024-09-21T18:57:25.771587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.190036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"142'13652","reported_seq":13791,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:27.075684+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:57:27.075684+0000","last_peered":"2024-09-21T18:57:27.075684+0000","last_clean":"2024-09-21T18:57:27.075684+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:57:27.075684+0000","last_undegraded":"2024-09-21T18:57:27.075684+0000","last_fullsized":"2024-09-21T18:57:27.075684+0000","mapping_epoch":79,"log_start":"118'9100","ondisk_log_start":"118'9100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228589568,"num_objects":3488,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13652,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":224,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.767485+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:57:25.767485+0000","last_peered":"2024-09-21T18:57:25.767485+0000","last_clean":"2024-09-21T18:57:25.767485+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:57:25.767485+0000","last_undegraded":"2024-09-21T18:57:25.767485+0000","last_fullsized":"2024-09-21T18:57:25.767485+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772512+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:57:25.772512+0000","last_peered":"2024-09-21T18:57:25.772512+0000","last_clean":"2024-09-21T18:57:25.772512+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:57:25.772512+0000","last_undegraded":"2024-09-21T18:57:25.772512+0000","last_fullsized":"2024-09-21T18:57:25.772512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.074325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":200,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.774410+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:57:25.774410+0000","last_peered":"2024-09-21T18:57:25.774410+0000","last_clean":"2024-09-21T18:57:25.774410+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:57:25.774410+0000","last_undegraded":"2024-09-21T18:57:25.774410+0000","last_fullsized":"2024-09-21T18:57:25.774410+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":169,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801488+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:57:26.801488+0000","last_peered":"2024-09-21T18:57:26.801488+0000","last_clean":"2024-09-21T18:57:26.801488+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:57:26.801488+0000","last_undegraded":"2024-09-21T18:57:26.801488+0000","last_fullsized":"2024-09-21T18:57:26.801488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"141'13668","reported_seq":13805,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.655964+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:57:26.655964+0000","last_peered":"2024-09-21T18:57:26.655964+0000","last_clean":"2024-09-21T18:57:26.655964+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:57:26.655964+0000","last_undegraded":"2024-09-21T18:57:26.655964+0000","last_fullsized":"2024-09-21T18:57:26.655964+0000","mapping_epoch":79,"log_start":"112'8500","ondisk_log_start":"112'8500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226099200,"num_objects":3450,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13668,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":215,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777560+0000","last_change":"2024-09-21T18:56:35.846693+0000","last_active":"2024-09-21T18:57:25.777560+0000","last_peered":"2024-09-21T18:57:25.777560+0000","last_clean":"2024-09-21T18:57:25.777560+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:57:25.777560+0000","last_undegraded":"2024-09-21T18:57:25.777560+0000","last_fullsized":"2024-09-21T18:57:25.777560+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:56:35.846605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:35.846605+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:51.214847+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":215,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772480+0000","last_change":"2024-09-21T18:55:51.567353+0000","last_active":"2024-09-21T18:57:25.772480+0000","last_peered":"2024-09-21T18:57:25.772480+0000","last_clean":"2024-09-21T18:57:25.772480+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:57:25.772480+0000","last_undegraded":"2024-09-21T18:57:25.772480+0000","last_fullsized":"2024-09-21T18:57:25.772480+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:55.125124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":248,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771739+0000","last_change":"2024-09-21T18:57:23.116600+0000","last_active":"2024-09-21T18:57:25.771739+0000","last_peered":"2024-09-21T18:57:25.771739+0000","last_clean":"2024-09-21T18:57:25.771739+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:57:25.771739+0000","last_undegraded":"2024-09-21T18:57:25.771739+0000","last_fullsized":"2024-09-21T18:57:25.771739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:57:23.116520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:23.116520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.889890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":188,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.777707+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:57:25.777707+0000","last_peered":"2024-09-21T18:57:25.777707+0000","last_clean":"2024-09-21T18:57:25.777707+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:57:25.777707+0000","last_undegraded":"2024-09-21T18:57:25.777707+0000","last_fullsized":"2024-09-21T18:57:25.777707+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":199,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.768390+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:57:25.768390+0000","last_peered":"2024-09-21T18:57:25.768390+0000","last_clean":"2024-09-21T18:57:25.768390+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:57:25.768390+0000","last_undegraded":"2024-09-21T18:57:25.768390+0000","last_fullsized":"2024-09-21T18:57:25.768390+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":186,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801174+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:57:26.801174+0000","last_peered":"2024-09-21T18:57:26.801174+0000","last_clean":"2024-09-21T18:57:26.801174+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:57:26.801174+0000","last_undegraded":"2024-09-21T18:57:26.801174+0000","last_fullsized":"2024-09-21T18:57:26.801174+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778427+0000","last_change":"2024-09-21T18:56:40.848060+0000","last_active":"2024-09-21T18:57:25.778427+0000","last_peered":"2024-09-21T18:57:25.778427+0000","last_clean":"2024-09-21T18:57:25.778427+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:57:25.778427+0000","last_undegraded":"2024-09-21T18:57:25.778427+0000","last_fullsized":"2024-09-21T18:57:25.778427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:40.847976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.847976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:57.531530+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":165,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772154+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:57:25.772154+0000","last_peered":"2024-09-21T18:57:25.772154+0000","last_clean":"2024-09-21T18:57:25.772154+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:57:25.772154+0000","last_undegraded":"2024-09-21T18:57:25.772154+0000","last_fullsized":"2024-09-21T18:57:25.772154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:00.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":76,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.780068+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:57:25.780068+0000","last_peered":"2024-09-21T18:57:25.780068+0000","last_clean":"2024-09-21T18:57:25.780068+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:57:25.780068+0000","last_undegraded":"2024-09-21T18:57:25.780068+0000","last_fullsized":"2024-09-21T18:57:25.780068+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771830+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:57:25.771830+0000","last_peered":"2024-09-21T18:57:25.771830+0000","last_clean":"2024-09-21T18:57:25.771830+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:57:25.771830+0000","last_undegraded":"2024-09-21T18:57:25.771830+0000","last_fullsized":"2024-09-21T18:57:25.771830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:08.070910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":181,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801080+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:57:26.801080+0000","last_peered":"2024-09-21T18:57:26.801080+0000","last_clean":"2024-09-21T18:57:26.801080+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:57:26.801080+0000","last_undegraded":"2024-09-21T18:57:26.801080+0000","last_fullsized":"2024-09-21T18:57:26.801080+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.774709+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:57:25.774709+0000","last_peered":"2024-09-21T18:57:25.774709+0000","last_clean":"2024-09-21T18:57:25.774709+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:57:25.774709+0000","last_undegraded":"2024-09-21T18:57:25.774709+0000","last_fullsized":"2024-09-21T18:57:25.774709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":198,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.890791+0000","last_change":"2024-09-21T18:57:25.890791+0000","last_active":"2024-09-21T18:57:25.890791+0000","last_peered":"2024-09-21T18:57:25.890791+0000","last_clean":"2024-09-21T18:57:25.890791+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:57:25.890791+0000","last_undegraded":"2024-09-21T18:57:25.890791+0000","last_fullsized":"2024-09-21T18:57:25.890791+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:25.890710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.890710+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:51.251415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778095+0000","last_change":"2024-09-21T18:55:51.523039+0000","last_active":"2024-09-21T18:57:25.778095+0000","last_peered":"2024-09-21T18:57:25.778095+0000","last_clean":"2024-09-21T18:57:25.778095+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:57:25.778095+0000","last_undegraded":"2024-09-21T18:57:25.778095+0000","last_fullsized":"2024-09-21T18:57:25.778095+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":241,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.792961+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:57:26.792961+0000","last_peered":"2024-09-21T18:57:26.792961+0000","last_clean":"2024-09-21T18:57:26.792961+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:57:26.792961+0000","last_undegraded":"2024-09-21T18:57:26.792961+0000","last_fullsized":"2024-09-21T18:57:26.792961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765754+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:57:25.765754+0000","last_peered":"2024-09-21T18:57:25.765754+0000","last_clean":"2024-09-21T18:57:25.765754+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:57:25.765754+0000","last_undegraded":"2024-09-21T18:57:25.765754+0000","last_fullsized":"2024-09-21T18:57:25.765754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":192,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.793345+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:57:26.793345+0000","last_peered":"2024-09-21T18:57:26.793345+0000","last_clean":"2024-09-21T18:57:26.793345+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:57:26.793345+0000","last_undegraded":"2024-09-21T18:57:26.793345+0000","last_fullsized":"2024-09-21T18:57:26.793345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771983+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:57:25.771983+0000","last_peered":"2024-09-21T18:57:25.771983+0000","last_clean":"2024-09-21T18:57:25.771983+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:57:25.771983+0000","last_undegraded":"2024-09-21T18:57:25.771983+0000","last_fullsized":"2024-09-21T18:57:25.771983+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.429722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":252,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778050+0000","last_change":"2024-09-21T18:55:51.543525+0000","last_active":"2024-09-21T18:57:25.778050+0000","last_peered":"2024-09-21T18:57:25.778050+0000","last_clean":"2024-09-21T18:57:25.778050+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:57:25.778050+0000","last_undegraded":"2024-09-21T18:57:25.778050+0000","last_fullsized":"2024-09-21T18:57:25.778050+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.924485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.780499+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:57:25.780499+0000","last_peered":"2024-09-21T18:57:25.780499+0000","last_clean":"2024-09-21T18:57:25.780499+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:57:25.780499+0000","last_undegraded":"2024-09-21T18:57:25.780499+0000","last_fullsized":"2024-09-21T18:57:25.780499+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":220,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.764380+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:57:25.764380+0000","last_peered":"2024-09-21T18:57:25.764380+0000","last_clean":"2024-09-21T18:57:25.764380+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:57:25.764380+0000","last_undegraded":"2024-09-21T18:57:25.764380+0000","last_fullsized":"2024-09-21T18:57:25.764380+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.765915+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:57:25.765915+0000","last_peered":"2024-09-21T18:57:25.765915+0000","last_clean":"2024-09-21T18:57:25.765915+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:57:25.765915+0000","last_undegraded":"2024-09-21T18:57:25.765915+0000","last_fullsized":"2024-09-21T18:57:25.765915+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.783162+0000","last_change":"2024-09-21T18:57:25.184491+0000","last_active":"2024-09-21T18:57:25.783162+0000","last_peered":"2024-09-21T18:57:25.783162+0000","last_clean":"2024-09-21T18:57:25.783162+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:57:25.783162+0000","last_undegraded":"2024-09-21T18:57:25.783162+0000","last_fullsized":"2024-09-21T18:57:25.783162+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:25.180690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.180690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:31.159574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.791639+0000","last_change":"2024-09-21T18:55:51.516509+0000","last_active":"2024-09-21T18:57:26.791639+0000","last_peered":"2024-09-21T18:57:26.791639+0000","last_clean":"2024-09-21T18:57:26.791639+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:57:26.791639+0000","last_undegraded":"2024-09-21T18:57:26.791639+0000","last_fullsized":"2024-09-21T18:57:26.791639+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":198,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.783423+0000","last_change":"2024-09-21T18:55:40.664189+0000","last_active":"2024-09-21T18:57:25.783423+0000","last_peered":"2024-09-21T18:57:25.783423+0000","last_clean":"2024-09-21T18:57:25.783423+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:57:25.783423+0000","last_undegraded":"2024-09-21T18:57:25.783423+0000","last_fullsized":"2024-09-21T18:57:25.783423+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:40.664110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.664110+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.772403+0000","last_change":"2024-09-21T18:57:19.082962+0000","last_active":"2024-09-21T18:57:25.772403+0000","last_peered":"2024-09-21T18:57:25.772403+0000","last_clean":"2024-09-21T18:57:25.772403+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:57:25.772403+0000","last_undegraded":"2024-09-21T18:57:25.772403+0000","last_fullsized":"2024-09-21T18:57:25.772403+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.082873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.082873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.905947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":195,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801567+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:57:26.801567+0000","last_peered":"2024-09-21T18:57:26.801567+0000","last_clean":"2024-09-21T18:57:26.801567+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:57:26.801567+0000","last_undegraded":"2024-09-21T18:57:26.801567+0000","last_fullsized":"2024-09-21T18:57:26.801567+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778235+0000","last_change":"2024-09-21T18:55:51.536412+0000","last_active":"2024-09-21T18:57:25.778235+0000","last_peered":"2024-09-21T18:57:25.778235+0000","last_clean":"2024-09-21T18:57:25.778235+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:57:25.778235+0000","last_undegraded":"2024-09-21T18:57:25.778235+0000","last_fullsized":"2024-09-21T18:57:25.778235+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":196,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800197+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:57:26.800197+0000","last_peered":"2024-09-21T18:57:26.800197+0000","last_clean":"2024-09-21T18:57:26.800197+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:57:26.800197+0000","last_undegraded":"2024-09-21T18:57:26.800197+0000","last_fullsized":"2024-09-21T18:57:26.800197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778487+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:57:25.778487+0000","last_peered":"2024-09-21T18:57:25.778487+0000","last_clean":"2024-09-21T18:57:25.778487+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:57:25.778487+0000","last_undegraded":"2024-09-21T18:57:25.778487+0000","last_fullsized":"2024-09-21T18:57:25.778487+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.912165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771367+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:57:25.771367+0000","last_peered":"2024-09-21T18:57:25.771367+0000","last_clean":"2024-09-21T18:57:25.771367+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:57:25.771367+0000","last_undegraded":"2024-09-21T18:57:25.771367+0000","last_fullsized":"2024-09-21T18:57:25.771367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801708+0000","last_change":"2024-09-21T18:55:51.529791+0000","last_active":"2024-09-21T18:57:26.801708+0000","last_peered":"2024-09-21T18:57:26.801708+0000","last_clean":"2024-09-21T18:57:26.801708+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:57:26.801708+0000","last_undegraded":"2024-09-21T18:57:26.801708+0000","last_fullsized":"2024-09-21T18:57:26.801708+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":196,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.800807+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:57:26.800807+0000","last_peered":"2024-09-21T18:57:26.800807+0000","last_clean":"2024-09-21T18:57:26.800807+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:57:26.800807+0000","last_undegraded":"2024-09-21T18:57:26.800807+0000","last_fullsized":"2024-09-21T18:57:26.800807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.787309+0000","last_change":"2024-09-21T18:55:51.529323+0000","last_active":"2024-09-21T18:57:25.787309+0000","last_peered":"2024-09-21T18:57:25.787309+0000","last_clean":"2024-09-21T18:57:25.787309+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:57:25.787309+0000","last_undegraded":"2024-09-21T18:57:25.787309+0000","last_fullsized":"2024-09-21T18:57:25.787309+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":169,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.768734+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:57:25.768734+0000","last_peered":"2024-09-21T18:57:25.768734+0000","last_clean":"2024-09-21T18:57:25.768734+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:57:25.768734+0000","last_undegraded":"2024-09-21T18:57:25.768734+0000","last_fullsized":"2024-09-21T18:57:25.768734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":165,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.771525+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:57:25.771525+0000","last_peered":"2024-09-21T18:57:25.771525+0000","last_clean":"2024-09-21T18:57:25.771525+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:57:25.771525+0000","last_undegraded":"2024-09-21T18:57:25.771525+0000","last_fullsized":"2024-09-21T18:57:25.771525+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.192235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.778695+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:57:25.778695+0000","last_peered":"2024-09-21T18:57:25.778695+0000","last_clean":"2024-09-21T18:57:25.778695+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:57:25.778695+0000","last_undegraded":"2024-09-21T18:57:25.778695+0000","last_fullsized":"2024-09-21T18:57:25.778695+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":203,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-21T18:57:26.801116+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:57:26.801116+0000","last_peered":"2024-09-21T18:57:26.801116+0000","last_clean":"2024-09-21T18:57:26.801116+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:57:26.801116+0000","last_undegraded":"2024-09-21T18:57:26.801116+0000","last_fullsized":"2024-09-21T18:57:26.801116+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.769713+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:57:25.769713+0000","last_peered":"2024-09-21T18:57:25.769713+0000","last_clean":"2024-09-21T18:57:25.769713+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:57:25.769713+0000","last_undegraded":"2024-09-21T18:57:25.769713+0000","last_fullsized":"2024-09-21T18:57:25.769713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":66,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-21T18:57:25.780423+0000","last_change":"2024-09-21T18:55:51.545494+0000","last_active":"2024-09-21T18:57:25.780423+0000","last_peered":"2024-09-21T18:57:25.780423+0000","last_clean":"2024-09-21T18:57:25.780423+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:57:25.780423+0000","last_undegraded":"2024-09-21T18:57:25.780423+0000","last_fullsized":"2024-09-21T18:57:25.780423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3669688344,"num_objects":55996,"num_object_clones":0,"num_object_copies":167988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11149,"num_objects_unfound":0,"num_objects_dirty":55996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221998,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27104,"num_bytes_recovered":1776222208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11931561984,"data_stored":11931549768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85796,"ondisk_log_size":85796,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":243,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516348,"kb_used_data":1399080,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92228804,"statfs":{"total":95995035648,"available":94442295296,"internally_reserved":0,"allocated":1432657920,"data_stored":1432265714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528381,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1465456,"kb_used_data":1344028,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92279696,"statfs":{"total":95995035648,"available":94494408704,"internally_reserved":0,"allocated":1376284672,"data_stored":1375904129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1249132,"kb_used_data":1145752,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92496020,"statfs":{"total":95995035648,"available":94715924480,"internally_reserved":0,"allocated":1173250048,"data_stored":1172873163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018950,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1474432,"kb_used_data":1353004,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92270720,"statfs":{"total":95995035648,"available":94485217280,"internally_reserved":0,"allocated":1385476096,"data_stored":1385079750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2715920,"kb_used_data":2468156,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91029232,"statfs":{"total":95995035648,"available":93213933568,"internally_reserved":0,"allocated":2527391744,"data_stored":2526979337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472816,"kb_used_data":1351264,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92272336,"statfs":{"total":95995035648,"available":94486872064,"internally_reserved":0,"allocated":1383694336,"data_stored":1383309814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735240,"kb_used_data":1581880,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92009912,"statfs":{"total":95995035648,"available":94218149888,"internally_reserved":0,"allocated":1619845120,"data_stored":1619436363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266944,"kb_used_data":1091372,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92478208,"statfs":{"total":95995035648,"available":94697684992,"internally_reserved":0,"allocated":1117564928,"data_stored":1117169315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1112408064,"data_stored":1112408064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1602424832,"data_stored":1602420760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1381040128,"data_stored":1381040128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2499743744,"data_stored":2499739672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1374093312,"data_stored":1374093312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1162674176,"data_stored":1162674176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1373704192,"data_stored":1373700120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1425473536,"data_stored":1425473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:57:31.429 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:31.457 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:31.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:31.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:31.701 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:31.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:31.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:31.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:31.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:32.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:32.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:32.154 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:32.255 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:32.280 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:32.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T18:57:32.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:32.411 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:32.462 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:32.564 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:32.591 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:32.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:32.723 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:32.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:32.852 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:32.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:32.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:57:32.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:33.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:32 smithi081 ceph-mon[29384]: pgmap v190: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 565 op/s; 11149/168585 objects misplaced (6.613%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T18:57:33.000 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:32 smithi081 ceph-mon[39192]: pgmap v190: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 565 op/s; 11149/168585 objects misplaced (6.613%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T18:57:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.001 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:33.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:33.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:33.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:32 smithi104 ceph-mon[32438]: pgmap v190: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 565 op/s; 11149/168585 objects misplaced (6.613%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T18:57:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:33.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:33.420 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:33.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:33.496 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:33.575 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:33.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.638 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:33.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:33.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:33.763 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:33.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:33.889 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:33.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:33.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:34.019 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:34.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:34.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:34.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:34.268 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:34.275 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:34.276 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:34.299 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:34.320 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:34.320 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:34.343 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:34.343 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:34.364 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:34.364 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:34.370 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:34.385 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:34.385 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:34.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:34.405 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:34.406 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:34.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:57:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:34.426 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:34.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:34.499 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:57:34.520 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:34.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:34.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:34.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:34.921 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:34.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:35.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:35.045 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:35.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:35.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: pgmap v191: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 9175/154497 objects misplaced (5.939%); 17 MiB/s, 277 objects/s recovering 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/944766696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: from='client.25042 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/384348019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:34 smithi104 ceph-mon[32438]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T18:57:35.273 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:35.299 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: pgmap v191: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 9175/154497 objects misplaced (5.939%); 17 MiB/s, 277 objects/s recovering 2024-09-21T18:57:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/944766696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T18:57:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: from='client.25042 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/384348019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[39192]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T18:57:35.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: pgmap v191: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 9175/154497 objects misplaced (5.939%); 17 MiB/s, 277 objects/s recovering 2024-09-21T18:57:35.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/944766696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:35.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T18:57:35.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: from='client.25042 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:35.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/384348019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:35.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:35.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:35.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:34 smithi081 ceph-mon[29384]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T18:57:35.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:35.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.553 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:35.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:35.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:35.689 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:35.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:35.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:35.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:35.969 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:35.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:36.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:36.117 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:36.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:36.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:36.273 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:36.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:36.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:36.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:36.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:36.556 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:36.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:36.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:36.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:36.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T18:57:36.783 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:36.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[29384]: pgmap v194: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s; 9175/154497 objects misplaced (5.939%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[29384]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[39192]: pgmap v194: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s; 9175/154497 objects misplaced (5.939%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[39192]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:36.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:36.856 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:36.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:36.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:36 smithi104 ceph-mon[32438]: pgmap v194: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 587 op/s; 9175/154497 objects misplaced (5.939%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T18:57:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:36 smithi104 ceph-mon[32438]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T18:57:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:37.011 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:37.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:37.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:37.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:37.406 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:57:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:57:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:57:37.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:37.445 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:37.528 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:37.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:37.573 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:37.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:37 smithi081 ceph-mon[29384]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T18:57:37.666 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:37 smithi081 ceph-mon[39192]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T18:57:37.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:37.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:37.792 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:57:37.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:37.812 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:37.817 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:57:37.847 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:37.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:37.992 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:37.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:38.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:37 smithi104 ceph-mon[32438]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T18:57:38.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:57:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:57:38.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:38.145 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:38.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:38.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:38.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:57:38.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:38.432 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:38.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:38.560 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:38.605 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:38.605 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:38.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:38.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:38.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:38.847 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:38.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:38.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:38.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:39.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:38 smithi104 ceph-mon[32438]: pgmap v196: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.37k op/s; 5555/143955 objects misplaced (3.859%); 44 MiB/s, 699 objects/s recovering 2024-09-21T18:57:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:38 smithi104 ceph-mon[32438]: from='client.15210 v1:172.21.15.81:0/1609854982' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:38 smithi081 ceph-mon[39192]: pgmap v196: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.37k op/s; 5555/143955 objects misplaced (3.859%); 44 MiB/s, 699 objects/s recovering 2024-09-21T18:57:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:38 smithi081 ceph-mon[39192]: from='client.15210 v1:172.21.15.81:0/1609854982' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:38 smithi081 ceph-mon[29384]: pgmap v196: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.37k op/s; 5555/143955 objects misplaced (3.859%); 44 MiB/s, 699 objects/s recovering 2024-09-21T18:57:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:38 smithi081 ceph-mon[29384]: from='client.15210 v1:172.21.15.81:0/1609854982' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:39.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:39.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:39.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:39.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:39.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:39.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:39.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:39.503 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:39.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:39.626 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:39.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:39.753 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:39.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:39.880 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:39.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:39.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:40.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:40.111 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.136 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:40.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:40.265 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:40.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:40.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:40.526 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:40.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:40.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:40.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:40.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:40.908 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:40.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:40 smithi104 ceph-mon[32438]: pgmap v197: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 586 op/s; 5555/143955 objects misplaced (3.859%); 32 MiB/s, 514 objects/s recovering 2024-09-21T18:57:41.010 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:41.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:40 smithi081 ceph-mon[39192]: pgmap v197: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 586 op/s; 5555/143955 objects misplaced (3.859%); 32 MiB/s, 514 objects/s recovering 2024-09-21T18:57:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:40 smithi081 ceph-mon[29384]: pgmap v197: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 586 op/s; 5555/143955 objects misplaced (3.859%); 32 MiB/s, 514 objects/s recovering 2024-09-21T18:57:41.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:41.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:41.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:41.297 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:41.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:41.429 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:41.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:41.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:41.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:41.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:41.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:41.849 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:41.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:41.979 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:41.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:42.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:42.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:42.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:42.253 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:42.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:42.382 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:42.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:42.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:42.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:42.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:42.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:42.778 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:42.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:42.909 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:42.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:42.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:42 smithi104 ceph-mon[32438]: pgmap v198: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 670 B/s rd, 0 B/s wr, 461 op/s; 5555/143955 objects misplaced (3.859%); 25 MiB/s, 404 objects/s recovering 2024-09-21T18:57:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:57:43.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:57:43.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:43.052 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:42 smithi081 ceph-mon[39192]: pgmap v198: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 670 B/s rd, 0 B/s wr, 461 op/s; 5555/143955 objects misplaced (3.859%); 25 MiB/s, 404 objects/s recovering 2024-09-21T18:57:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:57:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:57:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:42 smithi081 ceph-mon[29384]: pgmap v198: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 670 B/s rd, 0 B/s wr, 461 op/s; 5555/143955 objects misplaced (3.859%); 25 MiB/s, 404 objects/s recovering 2024-09-21T18:57:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:57:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:57:43.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:43.189 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:43.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:43.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:43.456 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:43.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:43.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:43.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:43.721 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:43.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:43.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:43.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:43.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:44.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:43 smithi104 ceph-mon[32438]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T18:57:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:43 smithi081 ceph-mon[39192]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T18:57:44.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:43 smithi081 ceph-mon[29384]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T18:57:44.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:44.127 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:44.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:44.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:44.273 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-21T18:57:37.445116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3132234291,"num_objects":47985,"num_object_clones":0,"num_object_copies":143955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5555,"num_objects_unfound":0,"num_objects_dirty":47985,"num_whiteouts":0,"num_read":1102,"num_read_kb":959,"num_write":230757,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31431,"num_bytes_recovered":2053119450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87164,"ondisk_log_size":87164,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":451,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11259092,"kb_used_data":9994608,"kb_used_omap":12,"kb_used_meta":1264115,"kb_avail":738702124,"statfs":{"total":767960285184,"available":756430974976,"internally_reserved":0,"allocated":10234478592,"data_stored":10231174831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1294454324},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,10,1,2,3,4,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-538050560,"num_objects":-8210,"num_object_clones":0,"num_object_copies":-24630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5594,"num_objects_unfound":0,"num_objects_dirty":-8210,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8210,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4198,"num_bytes_recovered":275120128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002328"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516732+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:57:35.516732+0000","last_peered":"2024-09-21T18:57:35.516732+0000","last_clean":"2024-09-21T18:57:35.516732+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:57:35.516732+0000","last_undegraded":"2024-09-21T18:57:35.516732+0000","last_fullsized":"2024-09-21T18:57:35.516732+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":223,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509107+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:57:35.509107+0000","last_peered":"2024-09-21T18:57:35.509107+0000","last_clean":"2024-09-21T18:57:35.509107+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:57:35.509107+0000","last_undegraded":"2024-09-21T18:57:35.509107+0000","last_fullsized":"2024-09-21T18:57:35.509107+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":186,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.566927+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:57:35.566927+0000","last_peered":"2024-09-21T18:57:35.566927+0000","last_clean":"2024-09-21T18:57:35.566927+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:57:35.566927+0000","last_undegraded":"2024-09-21T18:57:35.566927+0000","last_fullsized":"2024-09-21T18:57:35.566927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.239093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":174,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509115+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:57:35.509115+0000","last_peered":"2024-09-21T18:57:35.509115+0000","last_clean":"2024-09-21T18:57:35.509115+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:57:35.509115+0000","last_undegraded":"2024-09-21T18:57:35.509115+0000","last_fullsized":"2024-09-21T18:57:35.509115+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":183,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517488+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:57:35.517488+0000","last_peered":"2024-09-21T18:57:35.517488+0000","last_clean":"2024-09-21T18:57:35.517488+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:57:35.517488+0000","last_undegraded":"2024-09-21T18:57:35.517488+0000","last_fullsized":"2024-09-21T18:57:35.517488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":196,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518772+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:57:35.518772+0000","last_peered":"2024-09-21T18:57:35.518772+0000","last_clean":"2024-09-21T18:57:35.518772+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:57:35.518772+0000","last_undegraded":"2024-09-21T18:57:35.518772+0000","last_fullsized":"2024-09-21T18:57:35.518772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":254,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516854+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:57:35.516854+0000","last_peered":"2024-09-21T18:57:35.516854+0000","last_clean":"2024-09-21T18:57:35.516854+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:57:35.516854+0000","last_undegraded":"2024-09-21T18:57:35.516854+0000","last_fullsized":"2024-09-21T18:57:35.516854+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":186,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.508989+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:57:35.508989+0000","last_peered":"2024-09-21T18:57:35.508989+0000","last_clean":"2024-09-21T18:57:35.508989+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:57:35.508989+0000","last_undegraded":"2024-09-21T18:57:35.508989+0000","last_fullsized":"2024-09-21T18:57:35.508989+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520830+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:57:35.520830+0000","last_peered":"2024-09-21T18:57:35.520830+0000","last_clean":"2024-09-21T18:57:35.520830+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:57:35.520830+0000","last_undegraded":"2024-09-21T18:57:35.520830+0000","last_fullsized":"2024-09-21T18:57:35.520830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:22.885755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":248,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509424+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:57:35.509424+0000","last_peered":"2024-09-21T18:57:35.509424+0000","last_clean":"2024-09-21T18:57:35.509424+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:57:35.509424+0000","last_undegraded":"2024-09-21T18:57:35.509424+0000","last_fullsized":"2024-09-21T18:57:35.509424+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":208,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520791+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:57:35.520791+0000","last_peered":"2024-09-21T18:57:35.520791+0000","last_clean":"2024-09-21T18:57:35.520791+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:57:35.520791+0000","last_undegraded":"2024-09-21T18:57:35.520791+0000","last_fullsized":"2024-09-21T18:57:35.520791+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.948408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516042+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:57:35.516042+0000","last_peered":"2024-09-21T18:57:35.516042+0000","last_clean":"2024-09-21T18:57:35.516042+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:57:35.516042+0000","last_undegraded":"2024-09-21T18:57:35.516042+0000","last_fullsized":"2024-09-21T18:57:35.516042+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520581+0000","last_change":"2024-09-21T18:57:19.821897+0000","last_active":"2024-09-21T18:57:35.520581+0000","last_peered":"2024-09-21T18:57:35.520581+0000","last_clean":"2024-09-21T18:57:35.520581+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:57:35.520581+0000","last_undegraded":"2024-09-21T18:57:35.520581+0000","last_fullsized":"2024-09-21T18:57:35.520581+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.821849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.129486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":200,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509325+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:57:35.509325+0000","last_peered":"2024-09-21T18:57:35.509325+0000","last_clean":"2024-09-21T18:57:35.509325+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:57:35.509325+0000","last_undegraded":"2024-09-21T18:57:35.509325+0000","last_fullsized":"2024-09-21T18:57:35.509325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":193,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567409+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:57:35.567409+0000","last_peered":"2024-09-21T18:57:35.567409+0000","last_clean":"2024-09-21T18:57:35.567409+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:57:35.567409+0000","last_undegraded":"2024-09-21T18:57:35.567409+0000","last_fullsized":"2024-09-21T18:57:35.567409+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:16.165478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":190,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520259+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:57:35.520259+0000","last_peered":"2024-09-21T18:57:35.520259+0000","last_clean":"2024-09-21T18:57:35.520259+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:57:35.520259+0000","last_undegraded":"2024-09-21T18:57:35.520259+0000","last_fullsized":"2024-09-21T18:57:35.520259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:46.401461+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":183,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509609+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:57:35.509609+0000","last_peered":"2024-09-21T18:57:35.509609+0000","last_clean":"2024-09-21T18:57:35.509609+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:57:35.509609+0000","last_undegraded":"2024-09-21T18:57:35.509609+0000","last_fullsized":"2024-09-21T18:57:35.509609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516649+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:57:35.516649+0000","last_peered":"2024-09-21T18:57:35.516649+0000","last_clean":"2024-09-21T18:57:35.516649+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:57:35.516649+0000","last_undegraded":"2024-09-21T18:57:35.516649+0000","last_fullsized":"2024-09-21T18:57:35.516649+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":215,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519184+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:57:35.519184+0000","last_peered":"2024-09-21T18:57:35.519184+0000","last_clean":"2024-09-21T18:57:35.519184+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:57:35.519184+0000","last_undegraded":"2024-09-21T18:57:35.519184+0000","last_fullsized":"2024-09-21T18:57:35.519184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":200,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568394+0000","last_change":"2024-09-21T18:57:20.229156+0000","last_active":"2024-09-21T18:57:35.568394+0000","last_peered":"2024-09-21T18:57:35.568394+0000","last_clean":"2024-09-21T18:57:35.568394+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:57:35.568394+0000","last_undegraded":"2024-09-21T18:57:35.568394+0000","last_fullsized":"2024-09-21T18:57:35.568394+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.229088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.229088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.406792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":176,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519491+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:57:35.519491+0000","last_peered":"2024-09-21T18:57:35.519491+0000","last_clean":"2024-09-21T18:57:35.519491+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:57:35.519491+0000","last_undegraded":"2024-09-21T18:57:35.519491+0000","last_fullsized":"2024-09-21T18:57:35.519491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":202,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509582+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:57:35.509582+0000","last_peered":"2024-09-21T18:57:35.509582+0000","last_clean":"2024-09-21T18:57:35.509582+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:57:35.509582+0000","last_undegraded":"2024-09-21T18:57:35.509582+0000","last_fullsized":"2024-09-21T18:57:35.509582+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":214,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518232+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:57:35.518232+0000","last_peered":"2024-09-21T18:57:35.518232+0000","last_clean":"2024-09-21T18:57:35.518232+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:57:35.518232+0000","last_undegraded":"2024-09-21T18:57:35.518232+0000","last_fullsized":"2024-09-21T18:57:35.518232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":186,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509516+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:57:35.509516+0000","last_peered":"2024-09-21T18:57:35.509516+0000","last_clean":"2024-09-21T18:57:35.509516+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:57:35.509516+0000","last_undegraded":"2024-09-21T18:57:35.509516+0000","last_fullsized":"2024-09-21T18:57:35.509516+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":190,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567756+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:57:35.567756+0000","last_peered":"2024-09-21T18:57:35.567756+0000","last_clean":"2024-09-21T18:57:35.567756+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:57:35.567756+0000","last_undegraded":"2024-09-21T18:57:35.567756+0000","last_fullsized":"2024-09-21T18:57:35.567756+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":195,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567016+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:57:35.567016+0000","last_peered":"2024-09-21T18:57:35.567016+0000","last_clean":"2024-09-21T18:57:35.567016+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:57:35.567016+0000","last_undegraded":"2024-09-21T18:57:35.567016+0000","last_fullsized":"2024-09-21T18:57:35.567016+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":86,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517151+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:57:35.517151+0000","last_peered":"2024-09-21T18:57:35.517151+0000","last_clean":"2024-09-21T18:57:35.517151+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:57:35.517151+0000","last_undegraded":"2024-09-21T18:57:35.517151+0000","last_fullsized":"2024-09-21T18:57:35.517151+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":251,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521446+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:57:35.521446+0000","last_peered":"2024-09-21T18:57:35.521446+0000","last_clean":"2024-09-21T18:57:35.521446+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:57:35.521446+0000","last_undegraded":"2024-09-21T18:57:35.521446+0000","last_fullsized":"2024-09-21T18:57:35.521446+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"145'14205","reported_seq":14350,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.472876+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:57:36.472876+0000","last_peered":"2024-09-21T18:57:36.472876+0000","last_clean":"2024-09-21T18:57:36.472876+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:57:36.472876+0000","last_undegraded":"2024-09-21T18:57:36.472876+0000","last_fullsized":"2024-09-21T18:57:36.472876+0000","mapping_epoch":79,"log_start":"111'8000","ondisk_log_start":"111'8000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190906368,"num_objects":2913,"num_object_clones":0,"num_object_copies":8739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14205,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":215,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568772+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:57:35.568772+0000","last_peered":"2024-09-21T18:57:35.568772+0000","last_clean":"2024-09-21T18:57:35.568772+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:57:35.568772+0000","last_undegraded":"2024-09-21T18:57:35.568772+0000","last_fullsized":"2024-09-21T18:57:35.568772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.223966+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":196,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516859+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:57:35.516859+0000","last_peered":"2024-09-21T18:57:35.516859+0000","last_clean":"2024-09-21T18:57:35.516859+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:57:35.516859+0000","last_undegraded":"2024-09-21T18:57:35.516859+0000","last_fullsized":"2024-09-21T18:57:35.516859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":270,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509699+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:57:35.509699+0000","last_peered":"2024-09-21T18:57:35.509699+0000","last_clean":"2024-09-21T18:57:35.509699+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:57:35.509699+0000","last_undegraded":"2024-09-21T18:57:35.509699+0000","last_fullsized":"2024-09-21T18:57:35.509699+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":204,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520370+0000","last_change":"2024-09-21T18:56:37.863087+0000","last_active":"2024-09-21T18:57:35.520370+0000","last_peered":"2024-09-21T18:57:35.520370+0000","last_clean":"2024-09-21T18:57:35.520370+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:57:35.520370+0000","last_undegraded":"2024-09-21T18:57:35.520370+0000","last_fullsized":"2024-09-21T18:57:35.520370+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:37.863026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:37.863026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.972736+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"145'14288","reported_seq":14433,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:37.110793+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:57:37.110793+0000","last_peered":"2024-09-21T18:57:37.110793+0000","last_clean":"2024-09-21T18:57:37.110793+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:57:37.110793+0000","last_undegraded":"2024-09-21T18:57:37.110793+0000","last_fullsized":"2024-09-21T18:57:37.110793+0000","mapping_epoch":79,"log_start":"119'9700","ondisk_log_start":"119'9700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14288,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":274,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516629+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:57:35.516629+0000","last_peered":"2024-09-21T18:57:35.516629+0000","last_clean":"2024-09-21T18:57:35.516629+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:57:35.516629+0000","last_undegraded":"2024-09-21T18:57:35.516629+0000","last_fullsized":"2024-09-21T18:57:35.516629+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":210,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519892+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:57:35.519892+0000","last_peered":"2024-09-21T18:57:35.519892+0000","last_clean":"2024-09-21T18:57:35.519892+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:57:35.519892+0000","last_undegraded":"2024-09-21T18:57:35.519892+0000","last_fullsized":"2024-09-21T18:57:35.519892+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509132+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:57:35.509132+0000","last_peered":"2024-09-21T18:57:35.509132+0000","last_clean":"2024-09-21T18:57:35.509132+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:57:35.509132+0000","last_undegraded":"2024-09-21T18:57:35.509132+0000","last_fullsized":"2024-09-21T18:57:35.509132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":175,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509372+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:57:35.509372+0000","last_peered":"2024-09-21T18:57:35.509372+0000","last_clean":"2024-09-21T18:57:35.509372+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:57:35.509372+0000","last_undegraded":"2024-09-21T18:57:35.509372+0000","last_fullsized":"2024-09-21T18:57:35.509372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"145'14582","reported_seq":14727,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:37.081597+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:57:37.081597+0000","last_peered":"2024-09-21T18:57:37.081597+0000","last_clean":"2024-09-21T18:57:37.081597+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:57:37.081597+0000","last_undegraded":"2024-09-21T18:57:37.081597+0000","last_fullsized":"2024-09-21T18:57:37.081597+0000","mapping_epoch":79,"log_start":"120'10000","ondisk_log_start":"120'10000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14582,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":212,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509196+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:57:35.509196+0000","last_peered":"2024-09-21T18:57:35.509196+0000","last_clean":"2024-09-21T18:57:35.509196+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:57:35.509196+0000","last_undegraded":"2024-09-21T18:57:35.509196+0000","last_fullsized":"2024-09-21T18:57:35.509196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":78,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517470+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:57:35.517470+0000","last_peered":"2024-09-21T18:57:35.517470+0000","last_clean":"2024-09-21T18:57:35.517470+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:57:35.517470+0000","last_undegraded":"2024-09-21T18:57:35.517470+0000","last_fullsized":"2024-09-21T18:57:35.517470+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":242,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516301+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:57:35.516301+0000","last_peered":"2024-09-21T18:57:35.516301+0000","last_clean":"2024-09-21T18:57:35.516301+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:57:35.516301+0000","last_undegraded":"2024-09-21T18:57:35.516301+0000","last_fullsized":"2024-09-21T18:57:35.516301+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":174,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567458+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:57:35.567458+0000","last_peered":"2024-09-21T18:57:35.567458+0000","last_clean":"2024-09-21T18:57:35.567458+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:57:35.567458+0000","last_undegraded":"2024-09-21T18:57:35.567458+0000","last_fullsized":"2024-09-21T18:57:35.567458+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.105131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"145'14316","reported_seq":14552,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.443253+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:57:36.443253+0000","last_peered":"2024-09-21T18:57:36.443253+0000","last_clean":"2024-09-21T18:57:36.443253+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:57:36.443253+0000","last_undegraded":"2024-09-21T18:57:36.443253+0000","last_fullsized":"2024-09-21T18:57:36.443253+0000","mapping_epoch":79,"log_start":"119'9600","ondisk_log_start":"119'9600","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14316,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567143+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:57:35.567143+0000","last_peered":"2024-09-21T18:57:35.567143+0000","last_clean":"2024-09-21T18:57:35.567143+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:57:35.567143+0000","last_undegraded":"2024-09-21T18:57:35.567143+0000","last_fullsized":"2024-09-21T18:57:35.567143+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.091213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":203,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520912+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:57:35.520912+0000","last_peered":"2024-09-21T18:57:35.520912+0000","last_clean":"2024-09-21T18:57:35.520912+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:57:35.520912+0000","last_undegraded":"2024-09-21T18:57:35.520912+0000","last_fullsized":"2024-09-21T18:57:35.520912+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.895558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":99,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516642+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:57:35.516642+0000","last_peered":"2024-09-21T18:57:35.516642+0000","last_clean":"2024-09-21T18:57:35.516642+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:57:35.516642+0000","last_undegraded":"2024-09-21T18:57:35.516642+0000","last_fullsized":"2024-09-21T18:57:35.516642+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516205+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:57:35.516205+0000","last_peered":"2024-09-21T18:57:35.516205+0000","last_clean":"2024-09-21T18:57:35.516205+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:57:35.516205+0000","last_undegraded":"2024-09-21T18:57:35.516205+0000","last_fullsized":"2024-09-21T18:57:35.516205+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"145'14604","reported_seq":14899,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.456498+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:57:36.456498+0000","last_peered":"2024-09-21T18:57:36.456498+0000","last_clean":"2024-09-21T18:57:36.456498+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:57:36.456498+0000","last_undegraded":"2024-09-21T18:57:36.456498+0000","last_fullsized":"2024-09-21T18:57:36.456498+0000","mapping_epoch":79,"log_start":"117'9000","ondisk_log_start":"117'9000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:43.426801+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200146944,"num_objects":3054,"num_object_clones":0,"num_object_copies":9162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14604,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":92,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516746+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:57:35.516746+0000","last_peered":"2024-09-21T18:57:35.516746+0000","last_clean":"2024-09-21T18:57:35.516746+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:57:35.516746+0000","last_undegraded":"2024-09-21T18:57:35.516746+0000","last_fullsized":"2024-09-21T18:57:35.516746+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":195,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568024+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:57:35.568024+0000","last_peered":"2024-09-21T18:57:35.568024+0000","last_clean":"2024-09-21T18:57:35.568024+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:57:35.568024+0000","last_undegraded":"2024-09-21T18:57:35.568024+0000","last_fullsized":"2024-09-21T18:57:35.568024+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":200,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519706+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:57:35.519706+0000","last_peered":"2024-09-21T18:57:35.519706+0000","last_clean":"2024-09-21T18:57:35.519706+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:57:35.519706+0000","last_undegraded":"2024-09-21T18:57:35.519706+0000","last_fullsized":"2024-09-21T18:57:35.519706+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":183,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.566895+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:57:35.566895+0000","last_peered":"2024-09-21T18:57:35.566895+0000","last_clean":"2024-09-21T18:57:35.566895+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:57:35.566895+0000","last_undegraded":"2024-09-21T18:57:35.566895+0000","last_fullsized":"2024-09-21T18:57:35.566895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.637336+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"145'14361","reported_seq":22394,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.459026+0000","last_change":"2024-09-21T18:57:34.539837+0000","last_active":"2024-09-21T18:57:36.459026+0000","last_peered":"2024-09-21T18:57:36.459026+0000","last_clean":"2024-09-21T18:57:36.459026+0000","last_became_active":"2024-09-21T18:57:34.506455+0000","last_became_peered":"2024-09-21T18:57:34.506455+0000","last_unstale":"2024-09-21T18:57:36.459026+0000","last_undegraded":"2024-09-21T18:57:36.459026+0000","last_fullsized":"2024-09-21T18:57:36.459026+0000","mapping_epoch":143,"log_start":"111'8200","ondisk_log_start":"111'8200","created":79,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194445312,"num_objects":2967,"num_object_clones":0,"num_object_copies":8901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14361,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3933,"num_bytes_recovered":257753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":293,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567896+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:57:35.567896+0000","last_peered":"2024-09-21T18:57:35.567896+0000","last_clean":"2024-09-21T18:57:35.567896+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:57:35.567896+0000","last_undegraded":"2024-09-21T18:57:35.567896+0000","last_fullsized":"2024-09-21T18:57:35.567896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":217,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509713+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:57:35.509713+0000","last_peered":"2024-09-21T18:57:35.509713+0000","last_clean":"2024-09-21T18:57:35.509713+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:57:35.509713+0000","last_undegraded":"2024-09-21T18:57:35.509713+0000","last_fullsized":"2024-09-21T18:57:35.509713+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":213,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521169+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:57:35.521169+0000","last_peered":"2024-09-21T18:57:35.521169+0000","last_clean":"2024-09-21T18:57:35.521169+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:57:35.521169+0000","last_undegraded":"2024-09-21T18:57:35.521169+0000","last_fullsized":"2024-09-21T18:57:35.521169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.854902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517586+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:57:35.517586+0000","last_peered":"2024-09-21T18:57:35.517586+0000","last_clean":"2024-09-21T18:57:35.517586+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:57:35.517586+0000","last_undegraded":"2024-09-21T18:57:35.517586+0000","last_fullsized":"2024-09-21T18:57:35.517586+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":184,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518563+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:57:35.518563+0000","last_peered":"2024-09-21T18:57:35.518563+0000","last_clean":"2024-09-21T18:57:35.518563+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:57:35.518563+0000","last_undegraded":"2024-09-21T18:57:35.518563+0000","last_fullsized":"2024-09-21T18:57:35.518563+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"145'14496","reported_seq":16560,"reported_epoch":145,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:36.455112+0000","last_change":"2024-09-21T18:57:33.623733+0000","last_active":"2024-09-21T18:57:36.455112+0000","last_peered":"2024-09-21T18:57:36.455112+0000","last_clean":"2024-09-21T18:55:50.469777+0000","last_became_active":"2024-09-21T18:55:52.866581+0000","last_became_peered":"2024-09-21T18:55:52.866581+0000","last_unstale":"2024-09-21T18:57:36.455112+0000","last_undegraded":"2024-09-21T18:57:36.455112+0000","last_fullsized":"2024-09-21T18:57:36.455112+0000","mapping_epoch":114,"log_start":"119'9800","ondisk_log_start":"119'9800","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197525504,"num_objects":3014,"num_object_clones":0,"num_object_copies":9042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2079,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14496,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":956,"num_bytes_recovered":62652416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":3014}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":162,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518947+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:57:35.518947+0000","last_peered":"2024-09-21T18:57:35.518947+0000","last_clean":"2024-09-21T18:57:35.518947+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:57:35.518947+0000","last_undegraded":"2024-09-21T18:57:35.518947+0000","last_fullsized":"2024-09-21T18:57:35.518947+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568108+0000","last_change":"2024-09-21T18:57:22.224988+0000","last_active":"2024-09-21T18:57:35.568108+0000","last_peered":"2024-09-21T18:57:35.568108+0000","last_clean":"2024-09-21T18:57:35.568108+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:57:35.568108+0000","last_undegraded":"2024-09-21T18:57:35.568108+0000","last_fullsized":"2024-09-21T18:57:35.568108+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:22.220182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:22.220182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.617863+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":226,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518554+0000","last_change":"2024-09-21T18:55:51.541888+0000","last_active":"2024-09-21T18:57:35.518554+0000","last_peered":"2024-09-21T18:57:35.518554+0000","last_clean":"2024-09-21T18:57:35.518554+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:57:35.518554+0000","last_undegraded":"2024-09-21T18:57:35.518554+0000","last_fullsized":"2024-09-21T18:57:35.518554+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:25.237173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.237173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518684+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:57:35.518684+0000","last_peered":"2024-09-21T18:57:35.518684+0000","last_clean":"2024-09-21T18:57:35.518684+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:57:35.518684+0000","last_undegraded":"2024-09-21T18:57:35.518684+0000","last_fullsized":"2024-09-21T18:57:35.518684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":186,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567815+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:57:35.567815+0000","last_peered":"2024-09-21T18:57:35.567815+0000","last_clean":"2024-09-21T18:57:35.567815+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:57:35.567815+0000","last_undegraded":"2024-09-21T18:57:35.567815+0000","last_fullsized":"2024-09-21T18:57:35.567815+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"145'14710","reported_seq":28998,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.457591+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:57:36.457591+0000","last_peered":"2024-09-21T18:57:36.457591+0000","last_clean":"2024-09-21T18:57:36.457591+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:57:36.457591+0000","last_undegraded":"2024-09-21T18:57:36.457591+0000","last_fullsized":"2024-09-21T18:57:36.457591+0000","mapping_epoch":130,"log_start":"119'9600","ondisk_log_start":"119'9600","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.065509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14710,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":88,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517245+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:57:35.517245+0000","last_peered":"2024-09-21T18:57:35.517245+0000","last_clean":"2024-09-21T18:57:35.517245+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:57:35.517245+0000","last_undegraded":"2024-09-21T18:57:35.517245+0000","last_fullsized":"2024-09-21T18:57:35.517245+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516259+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:57:35.516259+0000","last_peered":"2024-09-21T18:57:35.516259+0000","last_clean":"2024-09-21T18:57:35.516259+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:57:35.516259+0000","last_undegraded":"2024-09-21T18:57:35.516259+0000","last_fullsized":"2024-09-21T18:57:35.516259+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":498,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567207+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:57:35.567207+0000","last_peered":"2024-09-21T18:57:35.567207+0000","last_clean":"2024-09-21T18:57:35.567207+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:57:35.567207+0000","last_undegraded":"2024-09-21T18:57:35.567207+0000","last_fullsized":"2024-09-21T18:57:35.567207+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:17.496871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":258,"num_read_kb":253,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":217,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521582+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:57:35.521582+0000","last_peered":"2024-09-21T18:57:35.521582+0000","last_clean":"2024-09-21T18:57:35.521582+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:57:35.521582+0000","last_undegraded":"2024-09-21T18:57:35.521582+0000","last_fullsized":"2024-09-21T18:57:35.521582+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.378290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":183,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.566935+0000","last_change":"2024-09-21T18:57:18.176297+0000","last_active":"2024-09-21T18:57:35.566935+0000","last_peered":"2024-09-21T18:57:35.566935+0000","last_clean":"2024-09-21T18:57:35.566935+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:57:35.566935+0000","last_undegraded":"2024-09-21T18:57:35.566935+0000","last_fullsized":"2024-09-21T18:57:35.566935+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:18.176233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:18.176233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:24.204170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"145'14320","reported_seq":26345,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.485566+0000","last_change":"2024-09-21T18:57:20.837816+0000","last_active":"2024-09-21T18:57:36.485566+0000","last_peered":"2024-09-21T18:57:36.485566+0000","last_clean":"2024-09-21T18:57:36.485566+0000","last_became_active":"2024-09-21T18:57:20.750225+0000","last_became_peered":"2024-09-21T18:57:20.750225+0000","last_unstale":"2024-09-21T18:57:36.485566+0000","last_undegraded":"2024-09-21T18:57:36.485566+0000","last_fullsized":"2024-09-21T18:57:36.485566+0000","mapping_epoch":139,"log_start":"117'8902","ondisk_log_start":"117'8902","created":79,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199229440,"num_objects":3040,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14320,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":266,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521271+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:57:35.521271+0000","last_peered":"2024-09-21T18:57:35.521271+0000","last_clean":"2024-09-21T18:57:35.521271+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:57:35.521271+0000","last_undegraded":"2024-09-21T18:57:35.521271+0000","last_fullsized":"2024-09-21T18:57:35.521271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":196,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517349+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:57:35.517349+0000","last_peered":"2024-09-21T18:57:35.517349+0000","last_clean":"2024-09-21T18:57:35.517349+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:57:35.517349+0000","last_undegraded":"2024-09-21T18:57:35.517349+0000","last_fullsized":"2024-09-21T18:57:35.517349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":257,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567047+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:57:35.567047+0000","last_peered":"2024-09-21T18:57:35.567047+0000","last_clean":"2024-09-21T18:57:35.567047+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:57:35.567047+0000","last_undegraded":"2024-09-21T18:57:35.567047+0000","last_fullsized":"2024-09-21T18:57:35.567047+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:05.121733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":178,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567990+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:57:35.567990+0000","last_peered":"2024-09-21T18:57:35.567990+0000","last_clean":"2024-09-21T18:57:35.567990+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:57:35.567990+0000","last_undegraded":"2024-09-21T18:57:35.567990+0000","last_fullsized":"2024-09-21T18:57:35.567990+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:01.123772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":183,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521431+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:57:35.521431+0000","last_peered":"2024-09-21T18:57:35.521431+0000","last_clean":"2024-09-21T18:57:35.521431+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:57:35.521431+0000","last_undegraded":"2024-09-21T18:57:35.521431+0000","last_fullsized":"2024-09-21T18:57:35.521431+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:28.894465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"145'14538","reported_seq":16353,"reported_epoch":145,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:36.470226+0000","last_change":"2024-09-21T18:57:34.219340+0000","last_active":"2024-09-21T18:57:36.470226+0000","last_peered":"2024-09-21T18:57:36.470226+0000","last_clean":"2024-09-21T18:55:50.477883+0000","last_became_active":"2024-09-21T18:55:52.879827+0000","last_became_peered":"2024-09-21T18:55:52.879827+0000","last_unstale":"2024-09-21T18:57:36.470226+0000","last_undegraded":"2024-09-21T18:57:36.470226+0000","last_fullsized":"2024-09-21T18:57:36.470226+0000","mapping_epoch":114,"log_start":"111'8400","ondisk_log_start":"111'8400","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201916440,"num_objects":3082,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2307,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14538,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":3082}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":209,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509754+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:57:35.509754+0000","last_peered":"2024-09-21T18:57:35.509754+0000","last_clean":"2024-09-21T18:57:35.509754+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:57:35.509754+0000","last_undegraded":"2024-09-21T18:57:35.509754+0000","last_fullsized":"2024-09-21T18:57:35.509754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509349+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:57:35.509349+0000","last_peered":"2024-09-21T18:57:35.509349+0000","last_clean":"2024-09-21T18:57:35.509349+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:57:35.509349+0000","last_undegraded":"2024-09-21T18:57:35.509349+0000","last_fullsized":"2024-09-21T18:57:35.509349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521251+0000","last_change":"2024-09-21T18:56:33.913317+0000","last_active":"2024-09-21T18:57:35.521251+0000","last_peered":"2024-09-21T18:57:35.521251+0000","last_clean":"2024-09-21T18:57:35.521251+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:57:35.521251+0000","last_undegraded":"2024-09-21T18:57:35.521251+0000","last_fullsized":"2024-09-21T18:57:35.521251+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:33.913250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:33.913250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:52.937683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":175,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517835+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:57:35.517835+0000","last_peered":"2024-09-21T18:57:35.517835+0000","last_clean":"2024-09-21T18:57:35.517835+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:57:35.517835+0000","last_undegraded":"2024-09-21T18:57:35.517835+0000","last_fullsized":"2024-09-21T18:57:35.517835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"145'14211","reported_seq":22679,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.522509+0000","last_change":"2024-09-21T18:57:35.557746+0000","last_active":"2024-09-21T18:57:36.522509+0000","last_peered":"2024-09-21T18:57:36.522509+0000","last_clean":"2024-09-21T18:57:36.522509+0000","last_became_active":"2024-09-21T18:57:35.521875+0000","last_became_peered":"2024-09-21T18:57:35.521875+0000","last_unstale":"2024-09-21T18:57:36.522509+0000","last_undegraded":"2024-09-21T18:57:36.522509+0000","last_fullsized":"2024-09-21T18:57:36.522509+0000","mapping_epoch":144,"log_start":"110'7500","ondisk_log_start":"110'7500","created":79,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191692800,"num_objects":2925,"num_object_clones":0,"num_object_copies":8775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14211,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4110,"num_bytes_recovered":269352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":218,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509973+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:57:35.509973+0000","last_peered":"2024-09-21T18:57:35.509973+0000","last_clean":"2024-09-21T18:57:35.509973+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:57:35.509973+0000","last_undegraded":"2024-09-21T18:57:35.509973+0000","last_fullsized":"2024-09-21T18:57:35.509973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520446+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:57:35.520446+0000","last_peered":"2024-09-21T18:57:35.520446+0000","last_clean":"2024-09-21T18:57:35.520446+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:57:35.520446+0000","last_undegraded":"2024-09-21T18:57:35.520446+0000","last_fullsized":"2024-09-21T18:57:35.520446+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":192,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567648+0000","last_change":"2024-09-21T18:55:51.535384+0000","last_active":"2024-09-21T18:57:35.567648+0000","last_peered":"2024-09-21T18:57:35.567648+0000","last_clean":"2024-09-21T18:57:35.567648+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:57:35.567648+0000","last_undegraded":"2024-09-21T18:57:35.567648+0000","last_fullsized":"2024-09-21T18:57:35.567648+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.042616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":182,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519098+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:57:35.519098+0000","last_peered":"2024-09-21T18:57:35.519098+0000","last_clean":"2024-09-21T18:57:35.519098+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:57:35.519098+0000","last_undegraded":"2024-09-21T18:57:35.519098+0000","last_fullsized":"2024-09-21T18:57:35.519098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"145'14384","reported_seq":18611,"reported_epoch":145,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:37.093742+0000","last_change":"2024-09-21T18:57:18.832498+0000","last_active":"2024-09-21T18:57:37.093742+0000","last_peered":"2024-09-21T18:57:37.093742+0000","last_clean":"2024-09-21T18:55:50.495661+0000","last_became_active":"2024-09-21T18:55:52.870612+0000","last_became_peered":"2024-09-21T18:55:52.870612+0000","last_unstale":"2024-09-21T18:57:37.093742+0000","last_undegraded":"2024-09-21T18:57:37.093742+0000","last_fullsized":"2024-09-21T18:57:37.093742+0000","mapping_epoch":114,"log_start":"119'9800","ondisk_log_start":"119'9800","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193593344,"num_objects":2954,"num_object_clones":0,"num_object_copies":8862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1169,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14384,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":133627904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":2954}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":206,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509593+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:57:35.509593+0000","last_peered":"2024-09-21T18:57:35.509593+0000","last_clean":"2024-09-21T18:57:35.509593+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:57:35.509593+0000","last_undegraded":"2024-09-21T18:57:35.509593+0000","last_fullsized":"2024-09-21T18:57:35.509593+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":223,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517798+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:57:35.517798+0000","last_peered":"2024-09-21T18:57:35.517798+0000","last_clean":"2024-09-21T18:57:35.517798+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:57:35.517798+0000","last_undegraded":"2024-09-21T18:57:35.517798+0000","last_fullsized":"2024-09-21T18:57:35.517798+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":248,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518726+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:57:35.518726+0000","last_peered":"2024-09-21T18:57:35.518726+0000","last_clean":"2024-09-21T18:57:35.518726+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:57:35.518726+0000","last_undegraded":"2024-09-21T18:57:35.518726+0000","last_fullsized":"2024-09-21T18:57:35.518726+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":182,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509666+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:57:35.509666+0000","last_peered":"2024-09-21T18:57:35.509666+0000","last_clean":"2024-09-21T18:57:35.509666+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:57:35.509666+0000","last_undegraded":"2024-09-21T18:57:35.509666+0000","last_fullsized":"2024-09-21T18:57:35.509666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"145'14418","reported_seq":14643,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.500136+0000","last_change":"2024-09-21T18:55:30.532749+0000","last_active":"2024-09-21T18:57:36.500136+0000","last_peered":"2024-09-21T18:57:36.500136+0000","last_clean":"2024-09-21T18:57:36.500136+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:57:36.500136+0000","last_undegraded":"2024-09-21T18:57:36.500136+0000","last_fullsized":"2024-09-21T18:57:36.500136+0000","mapping_epoch":79,"log_start":"110'7700","ondisk_log_start":"110'7700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7201","last_scrub_stamp":"2024-09-21T18:55:30.532526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.532526+0000","objects_scrubbed":7133,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199229440,"num_objects":3040,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14418,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":84,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516838+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:57:35.516838+0000","last_peered":"2024-09-21T18:57:35.516838+0000","last_clean":"2024-09-21T18:57:35.516838+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:57:35.516838+0000","last_undegraded":"2024-09-21T18:57:35.516838+0000","last_fullsized":"2024-09-21T18:57:35.516838+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":202,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520781+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:57:35.520781+0000","last_peered":"2024-09-21T18:57:35.520781+0000","last_clean":"2024-09-21T18:57:35.520781+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:57:35.520781+0000","last_undegraded":"2024-09-21T18:57:35.520781+0000","last_fullsized":"2024-09-21T18:57:35.520781+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.889938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519051+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:57:35.519051+0000","last_peered":"2024-09-21T18:57:35.519051+0000","last_clean":"2024-09-21T18:57:35.519051+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:57:35.519051+0000","last_undegraded":"2024-09-21T18:57:35.519051+0000","last_fullsized":"2024-09-21T18:57:35.519051+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.513174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516167+0000","last_change":"2024-09-21T18:55:51.540541+0000","last_active":"2024-09-21T18:57:35.516167+0000","last_peered":"2024-09-21T18:57:35.516167+0000","last_clean":"2024-09-21T18:57:35.516167+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:57:35.516167+0000","last_undegraded":"2024-09-21T18:57:35.516167+0000","last_fullsized":"2024-09-21T18:57:35.516167+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"145'14431","reported_seq":27556,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.475146+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:57:36.475146+0000","last_peered":"2024-09-21T18:57:36.475146+0000","last_clean":"2024-09-21T18:57:36.475146+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:57:36.475146+0000","last_undegraded":"2024-09-21T18:57:36.475146+0000","last_fullsized":"2024-09-21T18:57:36.475146+0000","mapping_epoch":134,"log_start":"116'8800","ondisk_log_start":"116'8800","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196673536,"num_objects":3001,"num_object_clones":0,"num_object_copies":9003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14431,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":221,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520004+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:57:35.520004+0000","last_peered":"2024-09-21T18:57:35.520004+0000","last_clean":"2024-09-21T18:57:35.520004+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:57:35.520004+0000","last_undegraded":"2024-09-21T18:57:35.520004+0000","last_fullsized":"2024-09-21T18:57:35.520004+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":193,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568057+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:57:35.568057+0000","last_peered":"2024-09-21T18:57:35.568057+0000","last_clean":"2024-09-21T18:57:35.568057+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:57:35.568057+0000","last_undegraded":"2024-09-21T18:57:35.568057+0000","last_fullsized":"2024-09-21T18:57:35.568057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.169542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":200,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519613+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:57:35.519613+0000","last_peered":"2024-09-21T18:57:35.519613+0000","last_clean":"2024-09-21T18:57:35.519613+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:57:35.519613+0000","last_undegraded":"2024-09-21T18:57:35.519613+0000","last_fullsized":"2024-09-21T18:57:35.519613+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":190,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567542+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T18:57:35.567542+0000","last_peered":"2024-09-21T18:57:35.567542+0000","last_clean":"2024-09-21T18:57:35.567542+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:57:35.567542+0000","last_undegraded":"2024-09-21T18:57:35.567542+0000","last_fullsized":"2024-09-21T18:57:35.567542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.190036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"145'14167","reported_seq":14312,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:37.098279+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:57:37.098279+0000","last_peered":"2024-09-21T18:57:37.098279+0000","last_clean":"2024-09-21T18:57:37.098279+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:57:37.098279+0000","last_undegraded":"2024-09-21T18:57:37.098279+0000","last_fullsized":"2024-09-21T18:57:37.098279+0000","mapping_epoch":79,"log_start":"119'9600","ondisk_log_start":"119'9600","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194838528,"num_objects":2973,"num_object_clones":0,"num_object_copies":8919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":233,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509311+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:57:35.509311+0000","last_peered":"2024-09-21T18:57:35.509311+0000","last_clean":"2024-09-21T18:57:35.509311+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:57:35.509311+0000","last_undegraded":"2024-09-21T18:57:35.509311+0000","last_fullsized":"2024-09-21T18:57:35.509311+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":193,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568946+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:57:35.568946+0000","last_peered":"2024-09-21T18:57:35.568946+0000","last_clean":"2024-09-21T18:57:35.568946+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:57:35.568946+0000","last_undegraded":"2024-09-21T18:57:35.568946+0000","last_fullsized":"2024-09-21T18:57:35.568946+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.074325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":209,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518381+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:57:35.518381+0000","last_peered":"2024-09-21T18:57:35.518381+0000","last_clean":"2024-09-21T18:57:35.518381+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:57:35.518381+0000","last_undegraded":"2024-09-21T18:57:35.518381+0000","last_fullsized":"2024-09-21T18:57:35.518381+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":175,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517574+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:57:35.517574+0000","last_peered":"2024-09-21T18:57:35.517574+0000","last_clean":"2024-09-21T18:57:35.517574+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:57:35.517574+0000","last_undegraded":"2024-09-21T18:57:35.517574+0000","last_fullsized":"2024-09-21T18:57:35.517574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"145'14177","reported_seq":14322,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:36.685363+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:57:36.685363+0000","last_peered":"2024-09-21T18:57:36.685363+0000","last_clean":"2024-09-21T18:57:36.685363+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:57:36.685363+0000","last_undegraded":"2024-09-21T18:57:36.685363+0000","last_fullsized":"2024-09-21T18:57:36.685363+0000","mapping_epoch":79,"log_start":"118'9000","ondisk_log_start":"118'9000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:29.184091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192741376,"num_objects":2941,"num_object_clones":0,"num_object_copies":8823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":224,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520553+0000","last_change":"2024-09-21T18:56:35.846693+0000","last_active":"2024-09-21T18:57:35.520553+0000","last_peered":"2024-09-21T18:57:35.520553+0000","last_clean":"2024-09-21T18:57:35.520553+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:57:35.520553+0000","last_undegraded":"2024-09-21T18:57:35.520553+0000","last_fullsized":"2024-09-21T18:57:35.520553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:56:35.846605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:35.846605+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:51.214847+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":223,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567093+0000","last_change":"2024-09-21T18:55:51.567353+0000","last_active":"2024-09-21T18:57:35.567093+0000","last_peered":"2024-09-21T18:57:35.567093+0000","last_clean":"2024-09-21T18:57:35.567093+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:57:35.567093+0000","last_undegraded":"2024-09-21T18:57:35.567093+0000","last_fullsized":"2024-09-21T18:57:35.567093+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:55.125124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":259,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567608+0000","last_change":"2024-09-21T18:57:23.116600+0000","last_active":"2024-09-21T18:57:35.567608+0000","last_peered":"2024-09-21T18:57:35.567608+0000","last_clean":"2024-09-21T18:57:35.567608+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:57:35.567608+0000","last_undegraded":"2024-09-21T18:57:35.567608+0000","last_fullsized":"2024-09-21T18:57:35.567608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:57:23.116520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:23.116520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.889890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":197,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520975+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:57:35.520975+0000","last_peered":"2024-09-21T18:57:35.520975+0000","last_clean":"2024-09-21T18:57:35.520975+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:57:35.520975+0000","last_undegraded":"2024-09-21T18:57:35.520975+0000","last_fullsized":"2024-09-21T18:57:35.520975+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:26.930637+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":207,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519754+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:57:35.519754+0000","last_peered":"2024-09-21T18:57:35.519754+0000","last_clean":"2024-09-21T18:57:35.519754+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:57:35.519754+0000","last_undegraded":"2024-09-21T18:57:35.519754+0000","last_fullsized":"2024-09-21T18:57:35.519754+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":192,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519736+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:57:35.519736+0000","last_peered":"2024-09-21T18:57:35.519736+0000","last_clean":"2024-09-21T18:57:35.519736+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:57:35.519736+0000","last_undegraded":"2024-09-21T18:57:35.519736+0000","last_fullsized":"2024-09-21T18:57:35.519736+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521347+0000","last_change":"2024-09-21T18:56:40.848060+0000","last_active":"2024-09-21T18:57:35.521347+0000","last_peered":"2024-09-21T18:57:35.521347+0000","last_clean":"2024-09-21T18:57:35.521347+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:57:35.521347+0000","last_undegraded":"2024-09-21T18:57:35.521347+0000","last_fullsized":"2024-09-21T18:57:35.521347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:40.847976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.847976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:57.531530+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":173,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567709+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:57:35.567709+0000","last_peered":"2024-09-21T18:57:35.567709+0000","last_clean":"2024-09-21T18:57:35.567709+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:57:35.567709+0000","last_undegraded":"2024-09-21T18:57:35.567709+0000","last_fullsized":"2024-09-21T18:57:35.567709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:00.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":84,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517336+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:57:35.517336+0000","last_peered":"2024-09-21T18:57:35.517336+0000","last_clean":"2024-09-21T18:57:35.517336+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:57:35.517336+0000","last_undegraded":"2024-09-21T18:57:35.517336+0000","last_fullsized":"2024-09-21T18:57:35.517336+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":193,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567735+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:57:35.567735+0000","last_peered":"2024-09-21T18:57:35.567735+0000","last_clean":"2024-09-21T18:57:35.567735+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:57:35.567735+0000","last_undegraded":"2024-09-21T18:57:35.567735+0000","last_fullsized":"2024-09-21T18:57:35.567735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:08.070910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519590+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:57:35.519590+0000","last_peered":"2024-09-21T18:57:35.519590+0000","last_clean":"2024-09-21T18:57:35.519590+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:57:35.519590+0000","last_undegraded":"2024-09-21T18:57:35.519590+0000","last_fullsized":"2024-09-21T18:57:35.519590+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":182,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518827+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:57:35.518827+0000","last_peered":"2024-09-21T18:57:35.518827+0000","last_clean":"2024-09-21T18:57:35.518827+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:57:35.518827+0000","last_undegraded":"2024-09-21T18:57:35.518827+0000","last_fullsized":"2024-09-21T18:57:35.518827+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":206,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520740+0000","last_change":"2024-09-21T18:57:25.890791+0000","last_active":"2024-09-21T18:57:35.520740+0000","last_peered":"2024-09-21T18:57:35.520740+0000","last_clean":"2024-09-21T18:57:35.520740+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:57:35.520740+0000","last_undegraded":"2024-09-21T18:57:35.520740+0000","last_fullsized":"2024-09-21T18:57:35.520740+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:25.890710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.890710+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:51.251415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516843+0000","last_change":"2024-09-21T18:55:51.523039+0000","last_active":"2024-09-21T18:57:35.516843+0000","last_peered":"2024-09-21T18:57:35.516843+0000","last_clean":"2024-09-21T18:57:35.516843+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:57:35.516843+0000","last_undegraded":"2024-09-21T18:57:35.516843+0000","last_fullsized":"2024-09-21T18:57:35.516843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":249,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517911+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:57:35.517911+0000","last_peered":"2024-09-21T18:57:35.517911+0000","last_clean":"2024-09-21T18:57:35.517911+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:57:35.517911+0000","last_undegraded":"2024-09-21T18:57:35.517911+0000","last_fullsized":"2024-09-21T18:57:35.517911+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":182,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509872+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:57:35.509872+0000","last_peered":"2024-09-21T18:57:35.509872+0000","last_clean":"2024-09-21T18:57:35.509872+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:57:35.509872+0000","last_undegraded":"2024-09-21T18:57:35.509872+0000","last_fullsized":"2024-09-21T18:57:35.509872+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.518069+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:57:35.518069+0000","last_peered":"2024-09-21T18:57:35.518069+0000","last_clean":"2024-09-21T18:57:35.518069+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:57:35.518069+0000","last_undegraded":"2024-09-21T18:57:35.518069+0000","last_fullsized":"2024-09-21T18:57:35.518069+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567713+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:57:35.567713+0000","last_peered":"2024-09-21T18:57:35.567713+0000","last_clean":"2024-09-21T18:57:35.567713+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:57:35.567713+0000","last_undegraded":"2024-09-21T18:57:35.567713+0000","last_fullsized":"2024-09-21T18:57:35.567713+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.429722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":262,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.520501+0000","last_change":"2024-09-21T18:55:51.543525+0000","last_active":"2024-09-21T18:57:35.520501+0000","last_peered":"2024-09-21T18:57:35.520501+0000","last_clean":"2024-09-21T18:57:35.520501+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:57:35.520501+0000","last_undegraded":"2024-09-21T18:57:35.520501+0000","last_fullsized":"2024-09-21T18:57:35.520501+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.924485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517726+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:57:35.517726+0000","last_peered":"2024-09-21T18:57:35.517726+0000","last_clean":"2024-09-21T18:57:35.517726+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:57:35.517726+0000","last_undegraded":"2024-09-21T18:57:35.517726+0000","last_fullsized":"2024-09-21T18:57:35.517726+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":228,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509460+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:57:35.509460+0000","last_peered":"2024-09-21T18:57:35.509460+0000","last_clean":"2024-09-21T18:57:35.509460+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:57:35.509460+0000","last_undegraded":"2024-09-21T18:57:35.509460+0000","last_fullsized":"2024-09-21T18:57:35.509460+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":186,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.509716+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:57:35.509716+0000","last_peered":"2024-09-21T18:57:35.509716+0000","last_clean":"2024-09-21T18:57:35.509716+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:57:35.509716+0000","last_undegraded":"2024-09-21T18:57:35.509716+0000","last_fullsized":"2024-09-21T18:57:35.509716+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":186,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567002+0000","last_change":"2024-09-21T18:57:25.184491+0000","last_active":"2024-09-21T18:57:35.567002+0000","last_peered":"2024-09-21T18:57:35.567002+0000","last_clean":"2024-09-21T18:57:35.567002+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:57:35.567002+0000","last_undegraded":"2024-09-21T18:57:35.567002+0000","last_fullsized":"2024-09-21T18:57:35.567002+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:25.180690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.180690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:31.159574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":189,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516455+0000","last_change":"2024-09-21T18:55:51.516509+0000","last_active":"2024-09-21T18:57:35.516455+0000","last_peered":"2024-09-21T18:57:35.516455+0000","last_clean":"2024-09-21T18:57:35.516455+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:57:35.516455+0000","last_undegraded":"2024-09-21T18:57:35.516455+0000","last_fullsized":"2024-09-21T18:57:35.516455+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":206,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567286+0000","last_change":"2024-09-21T18:55:40.664189+0000","last_active":"2024-09-21T18:57:35.567286+0000","last_peered":"2024-09-21T18:57:35.567286+0000","last_clean":"2024-09-21T18:57:35.567286+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:57:35.567286+0000","last_undegraded":"2024-09-21T18:57:35.567286+0000","last_fullsized":"2024-09-21T18:57:35.567286+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:40.664110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.664110+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568007+0000","last_change":"2024-09-21T18:57:19.082962+0000","last_active":"2024-09-21T18:57:35.568007+0000","last_peered":"2024-09-21T18:57:35.568007+0000","last_clean":"2024-09-21T18:57:35.568007+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:57:35.568007+0000","last_undegraded":"2024-09-21T18:57:35.568007+0000","last_fullsized":"2024-09-21T18:57:35.568007+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.082873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.082873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.905947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517660+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:57:35.517660+0000","last_peered":"2024-09-21T18:57:35.517660+0000","last_clean":"2024-09-21T18:57:35.517660+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:57:35.517660+0000","last_undegraded":"2024-09-21T18:57:35.517660+0000","last_fullsized":"2024-09-21T18:57:35.517660+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516289+0000","last_change":"2024-09-21T18:55:51.536412+0000","last_active":"2024-09-21T18:57:35.516289+0000","last_peered":"2024-09-21T18:57:35.516289+0000","last_clean":"2024-09-21T18:57:35.516289+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:57:35.516289+0000","last_undegraded":"2024-09-21T18:57:35.516289+0000","last_fullsized":"2024-09-21T18:57:35.516289+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":202,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516598+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:57:35.516598+0000","last_peered":"2024-09-21T18:57:35.516598+0000","last_clean":"2024-09-21T18:57:35.516598+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:57:35.516598+0000","last_undegraded":"2024-09-21T18:57:35.516598+0000","last_fullsized":"2024-09-21T18:57:35.516598+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521428+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:57:35.521428+0000","last_peered":"2024-09-21T18:57:35.521428+0000","last_clean":"2024-09-21T18:57:35.521428+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:57:35.521428+0000","last_undegraded":"2024-09-21T18:57:35.521428+0000","last_fullsized":"2024-09-21T18:57:35.521428+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.912165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":185,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567323+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:57:35.567323+0000","last_peered":"2024-09-21T18:57:35.567323+0000","last_clean":"2024-09-21T18:57:35.567323+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:57:35.567323+0000","last_undegraded":"2024-09-21T18:57:35.567323+0000","last_fullsized":"2024-09-21T18:57:35.567323+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.212178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":189,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517743+0000","last_change":"2024-09-21T18:55:51.529791+0000","last_active":"2024-09-21T18:57:35.517743+0000","last_peered":"2024-09-21T18:57:35.517743+0000","last_clean":"2024-09-21T18:57:35.517743+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:57:35.517743+0000","last_undegraded":"2024-09-21T18:57:35.517743+0000","last_fullsized":"2024-09-21T18:57:35.517743+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":202,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516480+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:57:35.516480+0000","last_peered":"2024-09-21T18:57:35.516480+0000","last_clean":"2024-09-21T18:57:35.516480+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:57:35.516480+0000","last_undegraded":"2024-09-21T18:57:35.516480+0000","last_fullsized":"2024-09-21T18:57:35.516480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516598+0000","last_change":"2024-09-21T18:55:51.529323+0000","last_active":"2024-09-21T18:57:35.516598+0000","last_peered":"2024-09-21T18:57:35.516598+0000","last_clean":"2024-09-21T18:57:35.516598+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:57:35.516598+0000","last_undegraded":"2024-09-21T18:57:35.516598+0000","last_fullsized":"2024-09-21T18:57:35.516598+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":177,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.519978+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:57:35.519978+0000","last_peered":"2024-09-21T18:57:35.519978+0000","last_clean":"2024-09-21T18:57:35.519978+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:57:35.519978+0000","last_undegraded":"2024-09-21T18:57:35.519978+0000","last_fullsized":"2024-09-21T18:57:35.519978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":173,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.567307+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:57:35.567307+0000","last_peered":"2024-09-21T18:57:35.567307+0000","last_clean":"2024-09-21T18:57:35.567307+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:57:35.567307+0000","last_undegraded":"2024-09-21T18:57:35.567307+0000","last_fullsized":"2024-09-21T18:57:35.567307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.192235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":182,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.568246+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:57:35.568246+0000","last_peered":"2024-09-21T18:57:35.568246+0000","last_clean":"2024-09-21T18:57:35.568246+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:57:35.568246+0000","last_undegraded":"2024-09-21T18:57:35.568246+0000","last_fullsized":"2024-09-21T18:57:35.568246+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":209,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.516775+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:57:35.516775+0000","last_peered":"2024-09-21T18:57:35.516775+0000","last_clean":"2024-09-21T18:57:35.516775+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:57:35.516775+0000","last_undegraded":"2024-09-21T18:57:35.516775+0000","last_fullsized":"2024-09-21T18:57:35.516775+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.521501+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:57:35.521501+0000","last_peered":"2024-09-21T18:57:35.521501+0000","last_clean":"2024-09-21T18:57:35.521501+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:57:35.521501+0000","last_undegraded":"2024-09-21T18:57:35.521501+0000","last_fullsized":"2024-09-21T18:57:35.521501+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":74,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-21T18:57:35.517652+0000","last_change":"2024-09-21T18:55:51.545494+0000","last_active":"2024-09-21T18:57:35.517652+0000","last_peered":"2024-09-21T18:57:35.517652+0000","last_clean":"2024-09-21T18:57:35.517652+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:57:35.517652+0000","last_undegraded":"2024-09-21T18:57:35.517652+0000","last_fullsized":"2024-09-21T18:57:35.517652+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3131637784,"num_objects":47786,"num_object_clones":0,"num_object_copies":143358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5555,"num_objects_unfound":0,"num_objects_dirty":47786,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230208,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31302,"num_bytes_recovered":2051342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10098126848,"data_stored":10098114632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86606,"ondisk_log_size":86606,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":258,"num_read_kb":253,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266746,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1340392,"kb_used_data":1214808,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92404760,"statfs":{"total":95995035648,"available":94622474240,"internally_reserved":0,"allocated":1243963392,"data_stored":1243550309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528383,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1297360,"kb_used_data":1167616,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92447792,"statfs":{"total":95995035648,"available":94666539008,"internally_reserved":0,"allocated":1195638784,"data_stored":1195242631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,4,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790018,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1109852,"kb_used_data":998152,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92635300,"statfs":{"total":95995035648,"available":94858547200,"internally_reserved":0,"allocated":1022107648,"data_stored":1021709886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018952,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1297976,"kb_used_data":1164060,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":92447176,"statfs":{"total":95995035648,"available":94665908224,"internally_reserved":0,"allocated":1191997440,"data_stored":1191580217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":137099718},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2169020,"kb_used_data":1917100,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91576132,"statfs":{"total":95995035648,"available":93773959168,"internally_reserved":0,"allocated":1963110400,"data_stored":1962677116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":257948106},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307676,"kb_used_data":1177924,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92437476,"statfs":{"total":95995035648,"available":94655975424,"internally_reserved":0,"allocated":1206194176,"data_stored":1205794044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328828,"kb_used_data":1162920,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92416324,"statfs":{"total":95995035648,"available":94634315776,"internally_reserved":0,"allocated":1190830080,"data_stored":1190400446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407988,"kb_used_data":1192028,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92337164,"statfs":{"total":95995035648,"available":94553255936,"internally_reserved":0,"allocated":1220636672,"data_stored":1220220182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1215889408,"data_stored":1215889408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1177948160,"data_stored":1177944088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1178533888,"data_stored":1178533888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1933971456,"data_stored":1933967384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1177157632,"data_stored":1177157632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1008730112,"data_stored":1008730112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1171329024,"data_stored":1171324952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1234567168,"data_stored":1234567168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:57:44.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T18:57:44.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:44.413 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:44.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:44.562 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:44.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:44.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:44.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:44.868 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:44 smithi081 ceph-mon[29384]: pgmap v200: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 3631/130041 objects misplaced (2.792%); 35 MiB/s, 564 objects/s recovering 2024-09-21T18:57:44.868 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:44 smithi081 ceph-mon[29384]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T18:57:44.868 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:44 smithi081 ceph-mon[39192]: pgmap v200: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 3631/130041 objects misplaced (2.792%); 35 MiB/s, 564 objects/s recovering 2024-09-21T18:57:44.868 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:44 smithi081 ceph-mon[39192]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T18:57:44.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T18:57:44.941 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:44.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:44.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:45.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:44 smithi104 ceph-mon[32438]: pgmap v200: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 3631/130041 objects misplaced (2.792%); 35 MiB/s, 564 objects/s recovering 2024-09-21T18:57:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:44 smithi104 ceph-mon[32438]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T18:57:45.042 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:45.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:45.104 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:45.104 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:45.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:45.390 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:45.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.562 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:45.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:45.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:45.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:45.783 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:45.783 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:45.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:45.790 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:45.791 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:45.830 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:45.867 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:45.867 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:45.903 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:45.903 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:45.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:45.948 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:45.948 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:45.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:45.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:45.986 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:46.022 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:46.022 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:46.078 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:46.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.244 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:46.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:46.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:46.380 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:46.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:46.508 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:46.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:46.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:46.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:46.774 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:46 smithi081 ceph-mon[29384]: pgmap v202: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 579 op/s; 3631/130041 objects misplaced (2.792%); 11 MiB/s, 178 objects/s recovering 2024-09-21T18:57:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2627234081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:46 smithi081 ceph-mon[39192]: pgmap v202: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 579 op/s; 3631/130041 objects misplaced (2.792%); 11 MiB/s, 178 objects/s recovering 2024-09-21T18:57:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2627234081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:46.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:46.928 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:46.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:47.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:46 smithi104 ceph-mon[32438]: pgmap v202: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 579 op/s; 3631/130041 objects misplaced (2.792%); 11 MiB/s, 178 objects/s recovering 2024-09-21T18:57:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2627234081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:47.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:47.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:47.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:47.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:47.399 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:57:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:57:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:57:47.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:47.463 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:47.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:47.588 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:47.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:47.735 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:47.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:47.861 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:47.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:47.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:47.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:48.041 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:48.041 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":499,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":147,"num_osds":8,"num_up_osds":8,"osd_up_since":1726944778,"num_in_osds":8,"osd_in_since":1726944950,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":145},{"state_name":"active+remapped+backfilling","count":3}],"num_pgs":148,"num_pools":7,"num_objects":38781,"data_bytes":2529040947,"bytes_used":9785147392,"bytes_avail":758175137792,"bytes_total":767960285184,"misplaced_objects":1929,"misplaced_total":116343,"misplaced_ratio":0.016580284159768959,"recovering_objects_per_sec":357,"recovering_bytes_per_sec":23401705,"recovering_keys_per_sec":0,"num_objects_recovered":2865,"num_bytes_recovered":187695128,"num_keys_recovered":0,"read_bytes_sec":1276,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":1147},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":13,"modified":"2024-09-21T18:54:37.384117+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T18:57:48.095 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:48.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:48.232 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:48.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:57:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:57:48.262 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:48.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:48.404 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:48.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:48.547 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:48.628 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:57:48.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.697 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:48.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:48.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:48.837 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:48.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:57:48.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:48.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:48.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[29384]: pgmap v203: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 1929/116343 objects misplaced (1.658%); 22 MiB/s, 357 objects/s recovering 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1633925511' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[29384]: from='client.25060 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1828148292' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[39192]: pgmap v203: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 1929/116343 objects misplaced (1.658%); 22 MiB/s, 357 objects/s recovering 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1633925511' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[39192]: from='client.25060 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:48.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1828148292' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T18:57:49.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:48 smithi104 ceph-mon[32438]: pgmap v203: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 1929/116343 objects misplaced (1.658%); 22 MiB/s, 357 objects/s recovering 2024-09-21T18:57:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1633925511' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:48 smithi104 ceph-mon[32438]: from='client.25060 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:57:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1828148292' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T18:57:49.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:49.109 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:49.127 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:57:49.127 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:241123 7.b 7:dcd16fe0:::benchmark_data_smithi081_73391_object102124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:48.476552+0000", 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0055238589999999999, 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055510000000000004, 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 241123 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.128 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476552+0000", 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476552+0000", 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476555+0000", 2024-09-21T18:57:49.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7089999999999998e-06 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476561+0000", 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2020000000000001e-06 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476577+0000", 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5724000000000001e-05 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476585+0000", 2024-09-21T18:57:49.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1550000000000007e-06 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.476755+0000", 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017002200000000001 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.480366+0000", 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036110819999999998 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:48.480489+0000", 2024-09-21T18:57:49.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000122946 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 1 2024-09-21T18:57:49.132 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:57:49.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:49.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:49.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:49.383 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:49.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:49.521 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:49.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:49.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:49.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:49.784 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:49.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:49.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:50.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:49 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:50.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:49 smithi104 ceph-mon[32438]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:57:50.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:49 smithi104 ceph-mon[32438]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T18:57:50.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:49 smithi104 ceph-mon[32438]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T18:57:50.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:50.058 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[29384]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[29384]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[29384]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[39192]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[39192]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T18:57:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:49 smithi081 ceph-mon[39192]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T18:57:50.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:50.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:50.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:50.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:50.420 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:50.442 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:50.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:50.570 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:50.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:50.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:50.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:50.833 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:50.934 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:50.957 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:50.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:50 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T18:57:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:50 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T18:57:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:50 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T18:57:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:50 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T18:57:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:50 smithi104 ceph-mon[32438]: pgmap v205: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 742 B/s rd, 0 B/s wr, 662 op/s; 1929/116343 objects misplaced (1.658%); 13 MiB/s, 207 objects/s recovering 2024-09-21T18:57:51.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:51.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[29384]: pgmap v205: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 742 B/s rd, 0 B/s wr, 662 op/s; 1929/116343 objects misplaced (1.658%); 13 MiB/s, 207 objects/s recovering 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T18:57:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:50 smithi081 ceph-mon[39192]: pgmap v205: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 742 B/s rd, 0 B/s wr, 662 op/s; 1929/116343 objects misplaced (1.658%); 13 MiB/s, 207 objects/s recovering 2024-09-21T18:57:51.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:51.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:51.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:51.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:51.429 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:51.451 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:51.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:51.573 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:51.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:51.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:51.826 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:51.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:51.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[29384]: 7.8 scrub starts 2024-09-21T18:57:51.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[29384]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T18:57:51.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:51.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:51.948 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[39192]: 7.8 scrub starts 2024-09-21T18:57:51.948 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[39192]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T18:57:51.948 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:51.948 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:51.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:51.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:51.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T18:57:52.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:51 smithi104 ceph-mon[32438]: 7.8 scrub starts 2024-09-21T18:57:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:51 smithi104 ceph-mon[32438]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T18:57:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:57:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:57:52.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:52.072 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:52.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:52.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:52.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:52.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:52.432 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:52.444 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:52.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:52.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:52.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:52.686 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:52.788 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.819 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:52.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:52.945 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:52.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:52.978 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T18:57:52.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:57:53.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:52 smithi104 ceph-mon[32438]: 7.8 scrub ok 2024-09-21T18:57:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:52 smithi104 ceph-mon[32438]: pgmap v208: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 801 op/s; 1929/115614 objects misplaced (1.668%); 15 MiB/s, 238 objects/s recovering 2024-09-21T18:57:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:52 smithi104 ceph-mon[32438]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T18:57:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:52 smithi104 ceph-mon[32438]: 3.12 scrub starts 2024-09-21T18:57:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:52 smithi104 ceph-mon[32438]: 3.12 scrub ok 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[39192]: 7.8 scrub ok 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[39192]: pgmap v208: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 801 op/s; 1929/115614 objects misplaced (1.668%); 15 MiB/s, 238 objects/s recovering 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[39192]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[39192]: 3.12 scrub starts 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[39192]: 3.12 scrub ok 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[29384]: 7.8 scrub ok 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[29384]: pgmap v208: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 801 op/s; 1929/115614 objects misplaced (1.668%); 15 MiB/s, 238 objects/s recovering 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[29384]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T18:57:53.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[29384]: 3.12 scrub starts 2024-09-21T18:57:53.016 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:52 smithi081 ceph-mon[29384]: 3.12 scrub ok 2024-09-21T18:57:53.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.096 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:53.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:53.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:53.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:53.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:53.385 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:53.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:53.613 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:53.625 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:57:53.625 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:57:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.651 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:53.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:53.667 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:57:53.667 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:57:53.699 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:57:53.699 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:57:53.734 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:57:53.734 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:57:53.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:53.768 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:57:53.768 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:57:53.795 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:53.802 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:57:53.802 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:57:53.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:57:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:57:53.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:53.922 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:53.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:53.925 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:57:53.925 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:57:53.925 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[39192]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T18:57:53.925 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[39192]: 3.17 scrub starts 2024-09-21T18:57:53.925 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[39192]: 3.17 scrub ok 2024-09-21T18:57:53.925 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2282258077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:53.926 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[29384]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T18:57:53.926 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[29384]: 3.17 scrub starts 2024-09-21T18:57:53.926 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[29384]: 3.17 scrub ok 2024-09-21T18:57:53.926 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2282258077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:54.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:53 smithi104 ceph-mon[32438]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T18:57:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:53 smithi104 ceph-mon[32438]: 3.17 scrub starts 2024-09-21T18:57:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:53 smithi104 ceph-mon[32438]: 3.17 scrub ok 2024-09-21T18:57:54.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2282258077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:57:54.024 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:54.052 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:54.138 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:57:54.138 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:57:54.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:54.192 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:54.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:54.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:54.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:57:54.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:54.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:54.601 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:54.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:54.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:54.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:54.862 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:54.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:54.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 6.12 scrub starts 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 6.12 scrub ok 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: pgmap v211: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 928 op/s; 584/105216 objects misplaced (0.555%); 19 MiB/s, 311 objects/s recovering 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 6.11 scrub starts 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 6.11 scrub ok 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T18:57:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:54 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T18:57:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.063 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:55.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 6.12 scrub starts 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 6.12 scrub ok 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: pgmap v211: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 928 op/s; 584/105216 objects misplaced (0.555%); 19 MiB/s, 311 objects/s recovering 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 6.11 scrub starts 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 6.11 scrub ok 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T18:57:55.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 6.12 scrub starts 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 6.12 scrub ok 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: pgmap v211: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 928 op/s; 584/105216 objects misplaced (0.555%); 19 MiB/s, 311 objects/s recovering 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 6.11 scrub starts 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 6.11 scrub ok 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T18:57:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:54 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T18:57:55.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:55.190 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:55.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:55.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:55.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:55.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:55.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:55.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T18:57:55.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:55.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:55.807 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:55.857 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[39192]: 6.13 scrub starts 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[39192]: 6.13 scrub ok 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[39192]: 5.15 scrub starts 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[39192]: 5.15 scrub ok 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[39192]: 5.11 scrub starts 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[39192]: 5.11 scrub ok 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[29384]: 6.13 scrub starts 2024-09-21T18:57:55.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[29384]: 6.13 scrub ok 2024-09-21T18:57:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[29384]: 5.15 scrub starts 2024-09-21T18:57:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[29384]: 5.15 scrub ok 2024-09-21T18:57:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[29384]: 5.11 scrub starts 2024-09-21T18:57:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:55 smithi081 ceph-mon[29384]: 5.11 scrub ok 2024-09-21T18:57:55.916 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:55.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:56.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.054 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:56.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:56.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.194 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:56.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:55 smithi104 ceph-mon[32438]: 6.13 scrub starts 2024-09-21T18:57:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:55 smithi104 ceph-mon[32438]: 6.13 scrub ok 2024-09-21T18:57:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:55 smithi104 ceph-mon[32438]: 5.15 scrub starts 2024-09-21T18:57:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:55 smithi104 ceph-mon[32438]: 5.15 scrub ok 2024-09-21T18:57:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:55 smithi104 ceph-mon[32438]: 5.11 scrub starts 2024-09-21T18:57:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:55 smithi104 ceph-mon[32438]: 5.11 scrub ok 2024-09-21T18:57:56.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:56.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:56.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:56.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:56.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:56.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:56.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:56.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:56.866 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:56.967 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:56.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:57:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:56.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:56.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:57.013 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:56 smithi081 ceph-mon[39192]: pgmap v212: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 857 B/s rd, 0 B/s wr, 622 op/s; 584/105216 objects misplaced (0.555%); 13 MiB/s, 209 objects/s recovering 2024-09-21T18:57:57.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:56 smithi081 ceph-mon[29384]: pgmap v212: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 857 B/s rd, 0 B/s wr, 622 op/s; 584/105216 objects misplaced (0.555%); 13 MiB/s, 209 objects/s recovering 2024-09-21T18:57:57.094 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:57.118 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:57.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:56 smithi104 ceph-mon[32438]: pgmap v212: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 857 B/s rd, 0 B/s wr, 622 op/s; 584/105216 objects misplaced (0.555%); 13 MiB/s, 209 objects/s recovering 2024-09-21T18:57:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:57.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:57.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:57:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:57:57.358 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:57.381 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:57.425 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:57.439 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:57:57.440 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:57:57.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:57.530 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:57.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:57.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.671 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:57.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:57.772 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:57.777 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:57:57.799 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:57.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:57.927 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:57.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:58.028 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:58.053 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:58.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-mon[39192]: 7.7 scrub starts 2024-09-21T18:57:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1785632119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-mon[39192]: from='client.35114 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-mon[29384]: 7.7 scrub starts 2024-09-21T18:57:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1785632119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:57 smithi081 ceph-mon[29384]: from='client.35114 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:58.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:58.191 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:58.254 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-21T18:57:55.468783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2285967923,"num_objects":35072,"num_object_clones":0,"num_object_copies":105216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":584,"num_objects_unfound":0,"num_objects_dirty":35072,"num_whiteouts":0,"num_read":1117,"num_read_kb":974,"num_write":243670,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35544,"num_bytes_recovered":2322603506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87477,"ondisk_log_size":87477,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8728024,"kb_used_data":7463660,"kb_used_omap":12,"kb_used_meta":1264179,"kb_avail":741233192,"statfs":{"total":767960285184,"available":759022788608,"internally_reserved":0,"allocated":7642787840,"data_stored":7639258841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1294519860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,6,12,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-243073024,"num_objects":-3709,"num_object_clones":0,"num_object_copies":-11127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1345,"num_objects_unfound":0,"num_objects_dirty":-3709,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3709,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.968447"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.725410+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:57:50.725410+0000","last_peered":"2024-09-21T18:57:50.725410+0000","last_clean":"2024-09-21T18:57:50.725410+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:57:50.725410+0000","last_undegraded":"2024-09-21T18:57:50.725410+0000","last_fullsized":"2024-09-21T18:57:50.725410+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":233,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714357+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:57:50.714357+0000","last_peered":"2024-09-21T18:57:50.714357+0000","last_clean":"2024-09-21T18:57:50.714357+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:57:50.714357+0000","last_undegraded":"2024-09-21T18:57:50.714357+0000","last_fullsized":"2024-09-21T18:57:50.714357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.725277+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:57:51.725277+0000","last_peered":"2024-09-21T18:57:51.725277+0000","last_clean":"2024-09-21T18:57:51.725277+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:57:51.725277+0000","last_undegraded":"2024-09-21T18:57:51.725277+0000","last_fullsized":"2024-09-21T18:57:51.725277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.239093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":184,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714477+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:57:50.714477+0000","last_peered":"2024-09-21T18:57:50.714477+0000","last_clean":"2024-09-21T18:57:50.714477+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:57:50.714477+0000","last_undegraded":"2024-09-21T18:57:50.714477+0000","last_fullsized":"2024-09-21T18:57:50.714477+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723230+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:57:51.723230+0000","last_peered":"2024-09-21T18:57:51.723230+0000","last_clean":"2024-09-21T18:57:51.723230+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:57:51.723230+0000","last_undegraded":"2024-09-21T18:57:51.723230+0000","last_fullsized":"2024-09-21T18:57:51.723230+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723254+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:57:51.723254+0000","last_peered":"2024-09-21T18:57:51.723254+0000","last_clean":"2024-09-21T18:57:51.723254+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:57:51.723254+0000","last_undegraded":"2024-09-21T18:57:51.723254+0000","last_fullsized":"2024-09-21T18:57:51.723254+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":266,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723859+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:57:51.723859+0000","last_peered":"2024-09-21T18:57:51.723859+0000","last_clean":"2024-09-21T18:57:51.723859+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:57:51.723859+0000","last_undegraded":"2024-09-21T18:57:51.723859+0000","last_fullsized":"2024-09-21T18:57:51.723859+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714273+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:57:50.714273+0000","last_peered":"2024-09-21T18:57:50.714273+0000","last_clean":"2024-09-21T18:57:50.714273+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:57:50.714273+0000","last_undegraded":"2024-09-21T18:57:50.714273+0000","last_fullsized":"2024-09-21T18:57:50.714273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706552+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:57:50.706552+0000","last_peered":"2024-09-21T18:57:50.706552+0000","last_clean":"2024-09-21T18:57:50.706552+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:57:50.706552+0000","last_undegraded":"2024-09-21T18:57:50.706552+0000","last_fullsized":"2024-09-21T18:57:50.706552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:22.885755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714889+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:57:50.714889+0000","last_peered":"2024-09-21T18:57:50.714889+0000","last_clean":"2024-09-21T18:57:50.714889+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:57:50.714889+0000","last_undegraded":"2024-09-21T18:57:50.714889+0000","last_fullsized":"2024-09-21T18:57:50.714889+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":218,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.705697+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:57:50.705697+0000","last_peered":"2024-09-21T18:57:50.705697+0000","last_clean":"2024-09-21T18:57:50.705697+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:57:50.705697+0000","last_undegraded":"2024-09-21T18:57:50.705697+0000","last_fullsized":"2024-09-21T18:57:50.705697+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.948408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.721502+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:57:50.721502+0000","last_peered":"2024-09-21T18:57:50.721502+0000","last_clean":"2024-09-21T18:57:50.721502+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:57:50.721502+0000","last_undegraded":"2024-09-21T18:57:50.721502+0000","last_fullsized":"2024-09-21T18:57:50.721502+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706279+0000","last_change":"2024-09-21T18:57:19.821897+0000","last_active":"2024-09-21T18:57:50.706279+0000","last_peered":"2024-09-21T18:57:50.706279+0000","last_clean":"2024-09-21T18:57:50.706279+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:57:50.706279+0000","last_undegraded":"2024-09-21T18:57:50.706279+0000","last_fullsized":"2024-09-21T18:57:50.706279+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.821849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.129486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":210,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714509+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:57:50.714509+0000","last_peered":"2024-09-21T18:57:50.714509+0000","last_clean":"2024-09-21T18:57:50.714509+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:57:50.714509+0000","last_undegraded":"2024-09-21T18:57:50.714509+0000","last_fullsized":"2024-09-21T18:57:50.714509+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":203,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727483+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:57:50.727483+0000","last_peered":"2024-09-21T18:57:50.727483+0000","last_clean":"2024-09-21T18:57:50.727483+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:57:50.727483+0000","last_undegraded":"2024-09-21T18:57:50.727483+0000","last_fullsized":"2024-09-21T18:57:50.727483+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:16.165478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":200,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.705625+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:57:50.705625+0000","last_peered":"2024-09-21T18:57:50.705625+0000","last_clean":"2024-09-21T18:57:50.705625+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:57:50.705625+0000","last_undegraded":"2024-09-21T18:57:50.705625+0000","last_fullsized":"2024-09-21T18:57:50.705625+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.650657+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":193,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715028+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:57:50.715028+0000","last_peered":"2024-09-21T18:57:50.715028+0000","last_clean":"2024-09-21T18:57:50.715028+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:57:50.715028+0000","last_undegraded":"2024-09-21T18:57:50.715028+0000","last_fullsized":"2024-09-21T18:57:50.715028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.724783+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:57:50.724783+0000","last_peered":"2024-09-21T18:57:50.724783+0000","last_clean":"2024-09-21T18:57:50.724783+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:57:50.724783+0000","last_undegraded":"2024-09-21T18:57:50.724783+0000","last_fullsized":"2024-09-21T18:57:50.724783+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":225,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.718668+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:57:50.718668+0000","last_peered":"2024-09-21T18:57:50.718668+0000","last_clean":"2024-09-21T18:57:50.718668+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:57:50.718668+0000","last_undegraded":"2024-09-21T18:57:50.718668+0000","last_fullsized":"2024-09-21T18:57:50.718668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.481531+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.725053+0000","last_change":"2024-09-21T18:57:20.229156+0000","last_active":"2024-09-21T18:57:51.725053+0000","last_peered":"2024-09-21T18:57:51.725053+0000","last_clean":"2024-09-21T18:57:51.725053+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:57:51.725053+0000","last_undegraded":"2024-09-21T18:57:51.725053+0000","last_fullsized":"2024-09-21T18:57:51.725053+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.229088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.229088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.406792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724200+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:57:51.724200+0000","last_peered":"2024-09-21T18:57:51.724200+0000","last_clean":"2024-09-21T18:57:51.724200+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:57:51.724200+0000","last_undegraded":"2024-09-21T18:57:51.724200+0000","last_fullsized":"2024-09-21T18:57:51.724200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":212,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715076+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:57:50.715076+0000","last_peered":"2024-09-21T18:57:50.715076+0000","last_clean":"2024-09-21T18:57:50.715076+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:57:50.715076+0000","last_undegraded":"2024-09-21T18:57:50.715076+0000","last_fullsized":"2024-09-21T18:57:50.715076+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723756+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:57:51.723756+0000","last_peered":"2024-09-21T18:57:51.723756+0000","last_clean":"2024-09-21T18:57:51.723756+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:57:51.723756+0000","last_undegraded":"2024-09-21T18:57:51.723756+0000","last_fullsized":"2024-09-21T18:57:51.723756+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715640+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:57:50.715640+0000","last_peered":"2024-09-21T18:57:50.715640+0000","last_clean":"2024-09-21T18:57:50.715640+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:57:50.715640+0000","last_undegraded":"2024-09-21T18:57:50.715640+0000","last_fullsized":"2024-09-21T18:57:50.715640+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":200,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727730+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:57:50.727730+0000","last_peered":"2024-09-21T18:57:50.727730+0000","last_clean":"2024-09-21T18:57:50.727730+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:57:50.727730+0000","last_undegraded":"2024-09-21T18:57:50.727730+0000","last_fullsized":"2024-09-21T18:57:50.727730+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":207,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.725305+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:57:51.725305+0000","last_peered":"2024-09-21T18:57:51.725305+0000","last_clean":"2024-09-21T18:57:51.725305+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:57:51.725305+0000","last_undegraded":"2024-09-21T18:57:51.725305+0000","last_fullsized":"2024-09-21T18:57:51.725305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.299356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":97,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.725679+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:57:50.725679+0000","last_peered":"2024-09-21T18:57:50.725679+0000","last_clean":"2024-09-21T18:57:50.725679+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:57:50.725679+0000","last_undegraded":"2024-09-21T18:57:50.725679+0000","last_fullsized":"2024-09-21T18:57:50.725679+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.717330+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:57:50.717330+0000","last_peered":"2024-09-21T18:57:50.717330+0000","last_clean":"2024-09-21T18:57:50.717330+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:57:50.717330+0000","last_undegraded":"2024-09-21T18:57:50.717330+0000","last_fullsized":"2024-09-21T18:57:50.717330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"150'15025","reported_seq":15180,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.012937+0000","last_change":"2024-09-21T18:54:14.451956+0000","last_active":"2024-09-21T18:57:51.012937+0000","last_peered":"2024-09-21T18:57:51.012937+0000","last_clean":"2024-09-21T18:57:51.012937+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:57:51.012937+0000","last_undegraded":"2024-09-21T18:57:51.012937+0000","last_fullsized":"2024-09-21T18:57:51.012937+0000","mapping_epoch":79,"log_start":"117'8900","ondisk_log_start":"117'8900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137166848,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15025,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":228,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723263+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:57:51.723263+0000","last_peered":"2024-09-21T18:57:51.723263+0000","last_clean":"2024-09-21T18:57:51.723263+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:57:51.723263+0000","last_undegraded":"2024-09-21T18:57:51.723263+0000","last_fullsized":"2024-09-21T18:57:51.723263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.223966+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723059+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:57:51.723059+0000","last_peered":"2024-09-21T18:57:51.723059+0000","last_clean":"2024-09-21T18:57:51.723059+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:57:51.723059+0000","last_undegraded":"2024-09-21T18:57:51.723059+0000","last_fullsized":"2024-09-21T18:57:51.723059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":284,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715591+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:57:50.715591+0000","last_peered":"2024-09-21T18:57:50.715591+0000","last_clean":"2024-09-21T18:57:50.715591+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:57:50.715591+0000","last_undegraded":"2024-09-21T18:57:50.715591+0000","last_fullsized":"2024-09-21T18:57:50.715591+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":215,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706120+0000","last_change":"2024-09-21T18:56:37.863087+0000","last_active":"2024-09-21T18:57:50.706120+0000","last_peered":"2024-09-21T18:57:50.706120+0000","last_clean":"2024-09-21T18:57:50.706120+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:57:50.706120+0000","last_undegraded":"2024-09-21T18:57:50.706120+0000","last_fullsized":"2024-09-21T18:57:50.706120+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:37.863026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:37.863026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.972736+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"151'15086","reported_seq":15243,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.842534+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:57:51.842534+0000","last_peered":"2024-09-21T18:57:51.842534+0000","last_clean":"2024-09-21T18:57:51.842534+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:57:51.842534+0000","last_undegraded":"2024-09-21T18:57:51.842534+0000","last_fullsized":"2024-09-21T18:57:51.842534+0000","mapping_epoch":79,"log_start":"123'10500","ondisk_log_start":"123'10500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138674176,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15086,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":286,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.722855+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:57:51.722855+0000","last_peered":"2024-09-21T18:57:51.722855+0000","last_clean":"2024-09-21T18:57:51.722855+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:57:51.722855+0000","last_undegraded":"2024-09-21T18:57:51.722855+0000","last_fullsized":"2024-09-21T18:57:51.722855+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":222,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724547+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:57:51.724547+0000","last_peered":"2024-09-21T18:57:51.724547+0000","last_clean":"2024-09-21T18:57:51.724547+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:57:51.724547+0000","last_undegraded":"2024-09-21T18:57:51.724547+0000","last_fullsized":"2024-09-21T18:57:51.724547+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714430+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:57:50.714430+0000","last_peered":"2024-09-21T18:57:50.714430+0000","last_clean":"2024-09-21T18:57:50.714430+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:57:50.714430+0000","last_undegraded":"2024-09-21T18:57:50.714430+0000","last_fullsized":"2024-09-21T18:57:50.714430+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":185,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714788+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:57:50.714788+0000","last_peered":"2024-09-21T18:57:50.714788+0000","last_clean":"2024-09-21T18:57:50.714788+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:57:50.714788+0000","last_undegraded":"2024-09-21T18:57:50.714788+0000","last_fullsized":"2024-09-21T18:57:50.714788+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"151'15360","reported_seq":15517,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.880156+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:57:51.880156+0000","last_peered":"2024-09-21T18:57:51.880156+0000","last_clean":"2024-09-21T18:57:51.880156+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:57:51.880156+0000","last_undegraded":"2024-09-21T18:57:51.880156+0000","last_fullsized":"2024-09-21T18:57:51.880156+0000","mapping_epoch":79,"log_start":"124'10800","ondisk_log_start":"124'10800","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15360,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":222,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714505+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:57:50.714505+0000","last_peered":"2024-09-21T18:57:50.714505+0000","last_clean":"2024-09-21T18:57:50.714505+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:57:50.714505+0000","last_undegraded":"2024-09-21T18:57:50.714505+0000","last_fullsized":"2024-09-21T18:57:50.714505+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":88,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.725987+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:57:50.725987+0000","last_peered":"2024-09-21T18:57:50.725987+0000","last_clean":"2024-09-21T18:57:50.725987+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:57:50.725987+0000","last_undegraded":"2024-09-21T18:57:50.725987+0000","last_fullsized":"2024-09-21T18:57:50.725987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":257,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.722608+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:57:51.722608+0000","last_peered":"2024-09-21T18:57:51.722608+0000","last_clean":"2024-09-21T18:57:51.722608+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:57:51.722608+0000","last_undegraded":"2024-09-21T18:57:51.722608+0000","last_fullsized":"2024-09-21T18:57:51.722608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":184,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727740+0000","last_change":"2024-09-21T18:53:13.658298+0000","last_active":"2024-09-21T18:57:50.727740+0000","last_peered":"2024-09-21T18:57:50.727740+0000","last_clean":"2024-09-21T18:57:50.727740+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:57:50.727740+0000","last_undegraded":"2024-09-21T18:57:50.727740+0000","last_fullsized":"2024-09-21T18:57:50.727740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:56.105131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"150'15112","reported_seq":15358,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.015226+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:57:51.015226+0000","last_peered":"2024-09-21T18:57:51.015226+0000","last_clean":"2024-09-21T18:57:51.015226+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:57:51.015226+0000","last_undegraded":"2024-09-21T18:57:51.015226+0000","last_fullsized":"2024-09-21T18:57:51.015226+0000","mapping_epoch":79,"log_start":"123'10400","ondisk_log_start":"123'10400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15112,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724054+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:57:51.724054+0000","last_peered":"2024-09-21T18:57:51.724054+0000","last_clean":"2024-09-21T18:57:51.724054+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:57:51.724054+0000","last_undegraded":"2024-09-21T18:57:51.724054+0000","last_fullsized":"2024-09-21T18:57:51.724054+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.091213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":213,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706624+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:57:50.706624+0000","last_peered":"2024-09-21T18:57:50.706624+0000","last_clean":"2024-09-21T18:57:50.706624+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:57:50.706624+0000","last_undegraded":"2024-09-21T18:57:50.706624+0000","last_fullsized":"2024-09-21T18:57:50.706624+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.895558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":112,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.722152+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:57:50.722152+0000","last_peered":"2024-09-21T18:57:50.722152+0000","last_clean":"2024-09-21T18:57:50.722152+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:57:50.722152+0000","last_undegraded":"2024-09-21T18:57:50.722152+0000","last_fullsized":"2024-09-21T18:57:50.722152+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":85,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.721630+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:57:50.721630+0000","last_peered":"2024-09-21T18:57:50.721630+0000","last_clean":"2024-09-21T18:57:50.721630+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:57:50.721630+0000","last_undegraded":"2024-09-21T18:57:50.721630+0000","last_fullsized":"2024-09-21T18:57:50.721630+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"150'15451","reported_seq":15756,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.019255+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:57:51.019255+0000","last_peered":"2024-09-21T18:57:51.019255+0000","last_clean":"2024-09-21T18:57:51.019255+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:57:51.019255+0000","last_undegraded":"2024-09-21T18:57:51.019255+0000","last_fullsized":"2024-09-21T18:57:51.019255+0000","mapping_epoch":79,"log_start":"119'9800","ondisk_log_start":"119'9800","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:43.658704+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144637952,"num_objects":2207,"num_object_clones":0,"num_object_copies":6621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15451,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.724871+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:57:50.724871+0000","last_peered":"2024-09-21T18:57:50.724871+0000","last_clean":"2024-09-21T18:57:50.724871+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:57:50.724871+0000","last_undegraded":"2024-09-21T18:57:50.724871+0000","last_fullsized":"2024-09-21T18:57:50.724871+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":207,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724390+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:57:51.724390+0000","last_peered":"2024-09-21T18:57:51.724390+0000","last_clean":"2024-09-21T18:57:51.724390+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:57:51.724390+0000","last_undegraded":"2024-09-21T18:57:51.724390+0000","last_fullsized":"2024-09-21T18:57:51.724390+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:47.380559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":210,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.716423+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:57:50.716423+0000","last_peered":"2024-09-21T18:57:50.716423+0000","last_clean":"2024-09-21T18:57:50.716423+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:57:50.716423+0000","last_undegraded":"2024-09-21T18:57:50.716423+0000","last_fullsized":"2024-09-21T18:57:50.716423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724888+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:57:51.724888+0000","last_peered":"2024-09-21T18:57:51.724888+0000","last_clean":"2024-09-21T18:57:51.724888+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:57:51.724888+0000","last_undegraded":"2024-09-21T18:57:51.724888+0000","last_fullsized":"2024-09-21T18:57:51.724888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.637336+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"150'15229","reported_seq":23272,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.974277+0000","last_change":"2024-09-21T18:57:34.539837+0000","last_active":"2024-09-21T18:57:50.974277+0000","last_peered":"2024-09-21T18:57:50.974277+0000","last_clean":"2024-09-21T18:57:50.974277+0000","last_became_active":"2024-09-21T18:57:34.506455+0000","last_became_peered":"2024-09-21T18:57:34.506455+0000","last_unstale":"2024-09-21T18:57:50.974277+0000","last_undegraded":"2024-09-21T18:57:50.974277+0000","last_fullsized":"2024-09-21T18:57:50.974277+0000","mapping_epoch":143,"log_start":"118'9100","ondisk_log_start":"118'9100","created":79,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":6129,"log_dups_size":0,"ondisk_log_size":6129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137560064,"num_objects":2099,"num_object_clones":0,"num_object_copies":6297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3933,"num_bytes_recovered":257753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":308,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724241+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:57:51.724241+0000","last_peered":"2024-09-21T18:57:51.724241+0000","last_clean":"2024-09-21T18:57:51.724241+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:57:51.724241+0000","last_undegraded":"2024-09-21T18:57:51.724241+0000","last_fullsized":"2024-09-21T18:57:51.724241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:41.364368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":227,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714956+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:57:50.714956+0000","last_peered":"2024-09-21T18:57:50.714956+0000","last_clean":"2024-09-21T18:57:50.714956+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:57:50.714956+0000","last_undegraded":"2024-09-21T18:57:50.714956+0000","last_fullsized":"2024-09-21T18:57:50.714956+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":223,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.705965+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:57:50.705965+0000","last_peered":"2024-09-21T18:57:50.705965+0000","last_clean":"2024-09-21T18:57:50.705965+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:57:50.705965+0000","last_undegraded":"2024-09-21T18:57:50.705965+0000","last_fullsized":"2024-09-21T18:57:50.705965+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.854902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.729632+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:57:50.729632+0000","last_peered":"2024-09-21T18:57:50.729632+0000","last_clean":"2024-09-21T18:57:50.729632+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:57:50.729632+0000","last_undegraded":"2024-09-21T18:57:50.729632+0000","last_fullsized":"2024-09-21T18:57:50.729632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723003+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:57:51.723003+0000","last_peered":"2024-09-21T18:57:51.723003+0000","last_clean":"2024-09-21T18:57:51.723003+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:57:51.723003+0000","last_undegraded":"2024-09-21T18:57:51.723003+0000","last_fullsized":"2024-09-21T18:57:51.723003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"149'15314","reported_seq":20971,"reported_epoch":149,"state":"peering","last_fresh":"2024-09-21T18:57:50.685662+0000","last_change":"2024-09-21T18:57:50.727602+0000","last_active":"2024-09-21T18:57:50.727602+0000","last_peered":"2024-09-21T18:57:50.685662+0000","last_clean":"2024-09-21T18:57:50.685662+0000","last_became_active":"2024-09-21T18:55:52.866581+0000","last_became_peered":"2024-09-21T18:55:52.866581+0000","last_unstale":"2024-09-21T18:57:50.685662+0000","last_undegraded":"2024-09-21T18:57:50.685662+0000","last_fullsized":"2024-09-21T18:57:50.685662+0000","mapping_epoch":150,"log_start":"123'10600","ondisk_log_start":"123'10600","created":79,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15314,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":179961856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":172,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.718828+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:57:50.718828+0000","last_peered":"2024-09-21T18:57:50.718828+0000","last_clean":"2024-09-21T18:57:50.718828+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:57:50.718828+0000","last_undegraded":"2024-09-21T18:57:50.718828+0000","last_fullsized":"2024-09-21T18:57:50.718828+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:44.501396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":213,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724484+0000","last_change":"2024-09-21T18:57:22.224988+0000","last_active":"2024-09-21T18:57:51.724484+0000","last_peered":"2024-09-21T18:57:51.724484+0000","last_clean":"2024-09-21T18:57:51.724484+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:57:51.724484+0000","last_undegraded":"2024-09-21T18:57:51.724484+0000","last_fullsized":"2024-09-21T18:57:51.724484+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:22.220182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:22.220182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.617863+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":244,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.719214+0000","last_change":"2024-09-21T18:57:49.456229+0000","last_active":"2024-09-21T18:57:50.719214+0000","last_peered":"2024-09-21T18:57:50.719214+0000","last_clean":"2024-09-21T18:57:50.719214+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:57:50.719214+0000","last_undegraded":"2024-09-21T18:57:50.719214+0000","last_fullsized":"2024-09-21T18:57:50.719214+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:49.456147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.456147+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:50.500491+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.719073+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:57:50.719073+0000","last_peered":"2024-09-21T18:57:50.719073+0000","last_clean":"2024-09-21T18:57:50.719073+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:57:50.719073+0000","last_undegraded":"2024-09-21T18:57:50.719073+0000","last_fullsized":"2024-09-21T18:57:50.719073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727969+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:57:50.727969+0000","last_peered":"2024-09-21T18:57:50.727969+0000","last_clean":"2024-09-21T18:57:50.727969+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:57:50.727969+0000","last_undegraded":"2024-09-21T18:57:50.727969+0000","last_fullsized":"2024-09-21T18:57:50.727969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"150'15575","reported_seq":29873,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.903348+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:57:50.903348+0000","last_peered":"2024-09-21T18:57:50.903348+0000","last_clean":"2024-09-21T18:57:50.903348+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:57:50.903348+0000","last_undegraded":"2024-09-21T18:57:50.903348+0000","last_fullsized":"2024-09-21T18:57:50.903348+0000","mapping_epoch":130,"log_start":"121'10400","ondisk_log_start":"121'10400","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.065509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147259392,"num_objects":2247,"num_object_clones":0,"num_object_copies":6741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15575,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":98,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.725813+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:57:50.725813+0000","last_peered":"2024-09-21T18:57:50.725813+0000","last_clean":"2024-09-21T18:57:50.725813+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:57:50.725813+0000","last_undegraded":"2024-09-21T18:57:50.725813+0000","last_fullsized":"2024-09-21T18:57:50.725813+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.721981+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:57:50.721981+0000","last_peered":"2024-09-21T18:57:50.721981+0000","last_clean":"2024-09-21T18:57:50.721981+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:57:50.721981+0000","last_undegraded":"2024-09-21T18:57:50.721981+0000","last_fullsized":"2024-09-21T18:57:50.721981+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":523,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727464+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:57:50.727464+0000","last_peered":"2024-09-21T18:57:50.727464+0000","last_clean":"2024-09-21T18:57:50.727464+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:57:50.727464+0000","last_undegraded":"2024-09-21T18:57:50.727464+0000","last_fullsized":"2024-09-21T18:57:50.727464+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:17.496871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":273,"num_read_kb":268,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":227,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706510+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:57:50.706510+0000","last_peered":"2024-09-21T18:57:50.706510+0000","last_clean":"2024-09-21T18:57:50.706510+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:57:50.706510+0000","last_undegraded":"2024-09-21T18:57:50.706510+0000","last_fullsized":"2024-09-21T18:57:50.706510+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.378290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723842+0000","last_change":"2024-09-21T18:57:18.176297+0000","last_active":"2024-09-21T18:57:51.723842+0000","last_peered":"2024-09-21T18:57:51.723842+0000","last_clean":"2024-09-21T18:57:51.723842+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:57:51.723842+0000","last_undegraded":"2024-09-21T18:57:51.723842+0000","last_fullsized":"2024-09-21T18:57:51.723842+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:18.176233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:18.176233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:24.204170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"150'15159","reported_seq":27194,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.958830+0000","last_change":"2024-09-21T18:57:20.837816+0000","last_active":"2024-09-21T18:57:50.958830+0000","last_peered":"2024-09-21T18:57:50.958830+0000","last_clean":"2024-09-21T18:57:50.958830+0000","last_became_active":"2024-09-21T18:57:20.750225+0000","last_became_peered":"2024-09-21T18:57:20.750225+0000","last_unstale":"2024-09-21T18:57:50.958830+0000","last_undegraded":"2024-09-21T18:57:50.958830+0000","last_fullsized":"2024-09-21T18:57:50.958830+0000","mapping_epoch":139,"log_start":"119'9602","ondisk_log_start":"119'9602","created":79,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144244736,"num_objects":2201,"num_object_clones":0,"num_object_copies":6603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15159,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":279,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.717267+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:57:50.717267+0000","last_peered":"2024-09-21T18:57:50.717267+0000","last_clean":"2024-09-21T18:57:50.717267+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:57:50.717267+0000","last_undegraded":"2024-09-21T18:57:50.717267+0000","last_fullsized":"2024-09-21T18:57:50.717267+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724401+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:57:51.724401+0000","last_peered":"2024-09-21T18:57:51.724401+0000","last_clean":"2024-09-21T18:57:51.724401+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:57:51.724401+0000","last_undegraded":"2024-09-21T18:57:51.724401+0000","last_fullsized":"2024-09-21T18:57:51.724401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":267,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727575+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:57:50.727575+0000","last_peered":"2024-09-21T18:57:50.727575+0000","last_clean":"2024-09-21T18:57:50.727575+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:57:50.727575+0000","last_undegraded":"2024-09-21T18:57:50.727575+0000","last_fullsized":"2024-09-21T18:57:50.727575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:05.121733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":188,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727847+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:57:50.727847+0000","last_peered":"2024-09-21T18:57:50.727847+0000","last_clean":"2024-09-21T18:57:50.727847+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:57:50.727847+0000","last_undegraded":"2024-09-21T18:57:50.727847+0000","last_fullsized":"2024-09-21T18:57:50.727847+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:01.123772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":193,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706132+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:57:50.706132+0000","last_peered":"2024-09-21T18:57:50.706132+0000","last_clean":"2024-09-21T18:57:50.706132+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:57:50.706132+0000","last_undegraded":"2024-09-21T18:57:50.706132+0000","last_fullsized":"2024-09-21T18:57:50.706132+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:28.894465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"150'15383","reported_seq":20681,"reported_epoch":150,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:51.488395+0000","last_change":"2024-09-21T18:57:34.219340+0000","last_active":"2024-09-21T18:57:51.488395+0000","last_peered":"2024-09-21T18:57:51.488395+0000","last_clean":"2024-09-21T18:55:50.477883+0000","last_became_active":"2024-09-21T18:55:52.879827+0000","last_became_peered":"2024-09-21T18:55:52.879827+0000","last_unstale":"2024-09-21T18:57:51.488395+0000","last_undegraded":"2024-09-21T18:57:51.488395+0000","last_fullsized":"2024-09-21T18:57:51.488395+0000","mapping_epoch":114,"log_start":"117'9200","ondisk_log_start":"117'9200","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538520,"num_objects":2237,"num_object_clones":0,"num_object_copies":6711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":151,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15383,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2520,"num_bytes_recovered":165085208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":2237}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":220,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715159+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:57:50.715159+0000","last_peered":"2024-09-21T18:57:50.715159+0000","last_clean":"2024-09-21T18:57:50.715159+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:57:50.715159+0000","last_undegraded":"2024-09-21T18:57:50.715159+0000","last_fullsized":"2024-09-21T18:57:50.715159+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714943+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:57:50.714943+0000","last_peered":"2024-09-21T18:57:50.714943+0000","last_clean":"2024-09-21T18:57:50.714943+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:57:50.714943+0000","last_undegraded":"2024-09-21T18:57:50.714943+0000","last_fullsized":"2024-09-21T18:57:50.714943+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":205,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.705983+0000","last_change":"2024-09-21T18:56:33.913317+0000","last_active":"2024-09-21T18:57:50.705983+0000","last_peered":"2024-09-21T18:57:50.705983+0000","last_clean":"2024-09-21T18:57:50.705983+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:57:50.705983+0000","last_undegraded":"2024-09-21T18:57:50.705983+0000","last_fullsized":"2024-09-21T18:57:50.705983+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:33.913250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:33.913250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:52.937683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724428+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:57:51.724428+0000","last_peered":"2024-09-21T18:57:51.724428+0000","last_clean":"2024-09-21T18:57:51.724428+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:57:51.724428+0000","last_undegraded":"2024-09-21T18:57:51.724428+0000","last_fullsized":"2024-09-21T18:57:51.724428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"150'15052","reported_seq":23531,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.006373+0000","last_change":"2024-09-21T18:57:35.557746+0000","last_active":"2024-09-21T18:57:51.006373+0000","last_peered":"2024-09-21T18:57:51.006373+0000","last_clean":"2024-09-21T18:57:51.006373+0000","last_became_active":"2024-09-21T18:57:35.521875+0000","last_became_peered":"2024-09-21T18:57:35.521875+0000","last_unstale":"2024-09-21T18:57:51.006373+0000","last_undegraded":"2024-09-21T18:57:51.006373+0000","last_fullsized":"2024-09-21T18:57:51.006373+0000","mapping_epoch":144,"log_start":"111'8300","ondisk_log_start":"111'8300","created":79,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6752,"log_dups_size":0,"ondisk_log_size":6752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136577024,"num_objects":2084,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15052,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4110,"num_bytes_recovered":269352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":229,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715413+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:57:50.715413+0000","last_peered":"2024-09-21T18:57:50.715413+0000","last_clean":"2024-09-21T18:57:50.715413+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:57:50.715413+0000","last_undegraded":"2024-09-21T18:57:50.715413+0000","last_fullsized":"2024-09-21T18:57:50.715413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.716683+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:57:50.716683+0000","last_peered":"2024-09-21T18:57:50.716683+0000","last_clean":"2024-09-21T18:57:50.716683+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:57:50.716683+0000","last_undegraded":"2024-09-21T18:57:50.716683+0000","last_fullsized":"2024-09-21T18:57:50.716683+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723923+0000","last_change":"2024-09-21T18:55:51.535384+0000","last_active":"2024-09-21T18:57:51.723923+0000","last_peered":"2024-09-21T18:57:51.723923+0000","last_clean":"2024-09-21T18:57:51.723923+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:57:51.723923+0000","last_undegraded":"2024-09-21T18:57:51.723923+0000","last_fullsized":"2024-09-21T18:57:51.723923+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.042616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":193,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.719008+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:57:50.719008+0000","last_peered":"2024-09-21T18:57:50.719008+0000","last_clean":"2024-09-21T18:57:50.719008+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:57:50.719008+0000","last_undegraded":"2024-09-21T18:57:50.719008+0000","last_fullsized":"2024-09-21T18:57:50.719008+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.540935+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"147'14919","reported_seq":20326,"reported_epoch":147,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T18:57:47.147454+0000","last_change":"2024-09-21T18:57:18.832498+0000","last_active":"2024-09-21T18:57:47.147454+0000","last_peered":"2024-09-21T18:57:47.147454+0000","last_clean":"2024-09-21T18:55:50.495661+0000","last_became_active":"2024-09-21T18:55:52.870612+0000","last_became_peered":"2024-09-21T18:55:52.870612+0000","last_unstale":"2024-09-21T18:57:47.147454+0000","last_undegraded":"2024-09-21T18:57:47.147454+0000","last_fullsized":"2024-09-21T18:57:47.147454+0000","mapping_epoch":114,"log_start":"122'10300","ondisk_log_start":"122'10300","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158531584,"num_objects":2419,"num_object_clones":0,"num_object_copies":7257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14919,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2627,"num_bytes_recovered":172163072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":2419}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":216,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715726+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:57:50.715726+0000","last_peered":"2024-09-21T18:57:50.715726+0000","last_clean":"2024-09-21T18:57:50.715726+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:57:50.715726+0000","last_undegraded":"2024-09-21T18:57:50.715726+0000","last_fullsized":"2024-09-21T18:57:50.715726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":236,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723598+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:57:51.723598+0000","last_peered":"2024-09-21T18:57:51.723598+0000","last_clean":"2024-09-21T18:57:51.723598+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:57:51.723598+0000","last_undegraded":"2024-09-21T18:57:51.723598+0000","last_fullsized":"2024-09-21T18:57:51.723598+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":262,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.718573+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:57:50.718573+0000","last_peered":"2024-09-21T18:57:50.718573+0000","last_clean":"2024-09-21T18:57:50.718573+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:57:50.718573+0000","last_undegraded":"2024-09-21T18:57:50.718573+0000","last_fullsized":"2024-09-21T18:57:50.718573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":193,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714965+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:57:50.714965+0000","last_peered":"2024-09-21T18:57:50.714965+0000","last_clean":"2024-09-21T18:57:50.714965+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:57:50.714965+0000","last_undegraded":"2024-09-21T18:57:50.714965+0000","last_fullsized":"2024-09-21T18:57:50.714965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"150'15303","reported_seq":15568,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.070236+0000","last_change":"2024-09-21T18:57:51.070236+0000","last_active":"2024-09-21T18:57:51.070236+0000","last_peered":"2024-09-21T18:57:51.070236+0000","last_clean":"2024-09-21T18:57:51.070236+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:57:51.070236+0000","last_undegraded":"2024-09-21T18:57:51.070236+0000","last_fullsized":"2024-09-21T18:57:51.070236+0000","mapping_epoch":79,"log_start":"112'8600","ondisk_log_start":"112'8600","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15303","last_scrub_stamp":"2024-09-21T18:57:51.070161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.070161+0000","objects_scrubbed":2162,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:57.001294+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":6465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15303,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":94,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.724954+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:57:50.724954+0000","last_peered":"2024-09-21T18:57:50.724954+0000","last_clean":"2024-09-21T18:57:50.724954+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:57:50.724954+0000","last_undegraded":"2024-09-21T18:57:50.724954+0000","last_fullsized":"2024-09-21T18:57:50.724954+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":213,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706426+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:57:50.706426+0000","last_peered":"2024-09-21T18:57:50.706426+0000","last_clean":"2024-09-21T18:57:50.706426+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:57:50.706426+0000","last_undegraded":"2024-09-21T18:57:50.706426+0000","last_fullsized":"2024-09-21T18:57:50.706426+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.889938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.718621+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:57:50.718621+0000","last_peered":"2024-09-21T18:57:50.718621+0000","last_clean":"2024-09-21T18:57:50.718621+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:57:50.718621+0000","last_undegraded":"2024-09-21T18:57:50.718621+0000","last_fullsized":"2024-09-21T18:57:50.718621+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.513174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.721891+0000","last_change":"2024-09-21T18:55:51.540541+0000","last_active":"2024-09-21T18:57:50.721891+0000","last_peered":"2024-09-21T18:57:50.721891+0000","last_clean":"2024-09-21T18:57:50.721891+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:57:50.721891+0000","last_undegraded":"2024-09-21T18:57:50.721891+0000","last_fullsized":"2024-09-21T18:57:50.721891+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:14.808734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:14.808734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"150'15231","reported_seq":28366,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.920424+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:57:50.920424+0000","last_peered":"2024-09-21T18:57:50.920424+0000","last_clean":"2024-09-21T18:57:50.920424+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:57:50.920424+0000","last_undegraded":"2024-09-21T18:57:50.920424+0000","last_fullsized":"2024-09-21T18:57:50.920424+0000","mapping_epoch":134,"log_start":"119'9600","ondisk_log_start":"119'9600","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144244736,"num_objects":2201,"num_object_clones":0,"num_object_copies":6603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15231,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":231,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.716511+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:57:50.716511+0000","last_peered":"2024-09-21T18:57:50.716511+0000","last_clean":"2024-09-21T18:57:50.716511+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:57:50.716511+0000","last_undegraded":"2024-09-21T18:57:50.716511+0000","last_fullsized":"2024-09-21T18:57:50.716511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.728229+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:57:50.728229+0000","last_peered":"2024-09-21T18:57:50.728229+0000","last_clean":"2024-09-21T18:57:50.728229+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:57:50.728229+0000","last_undegraded":"2024-09-21T18:57:50.728229+0000","last_fullsized":"2024-09-21T18:57:50.728229+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.169542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":211,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.718349+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:57:50.718349+0000","last_peered":"2024-09-21T18:57:50.718349+0000","last_clean":"2024-09-21T18:57:50.718349+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:57:50.718349+0000","last_undegraded":"2024-09-21T18:57:50.718349+0000","last_fullsized":"2024-09-21T18:57:50.718349+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":200,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727509+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T18:57:50.727509+0000","last_peered":"2024-09-21T18:57:50.727509+0000","last_clean":"2024-09-21T18:57:50.727509+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:57:50.727509+0000","last_undegraded":"2024-09-21T18:57:50.727509+0000","last_fullsized":"2024-09-21T18:57:50.727509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.190036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"151'14960","reported_seq":15117,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.919005+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:57:51.919005+0000","last_peered":"2024-09-21T18:57:51.919005+0000","last_clean":"2024-09-21T18:57:51.919005+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:57:51.919005+0000","last_undegraded":"2024-09-21T18:57:51.919005+0000","last_fullsized":"2024-09-21T18:57:51.919005+0000","mapping_epoch":79,"log_start":"123'10400","ondisk_log_start":"123'10400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142868480,"num_objects":2180,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14960,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":243,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714768+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:57:50.714768+0000","last_peered":"2024-09-21T18:57:50.714768+0000","last_clean":"2024-09-21T18:57:50.714768+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:57:50.714768+0000","last_undegraded":"2024-09-21T18:57:50.714768+0000","last_fullsized":"2024-09-21T18:57:50.714768+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.728467+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:57:50.728467+0000","last_peered":"2024-09-21T18:57:50.728467+0000","last_clean":"2024-09-21T18:57:50.728467+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:57:50.728467+0000","last_undegraded":"2024-09-21T18:57:50.728467+0000","last_fullsized":"2024-09-21T18:57:50.728467+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.074325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":219,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.726233+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:57:50.726233+0000","last_peered":"2024-09-21T18:57:50.726233+0000","last_clean":"2024-09-21T18:57:50.726233+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:57:50.726233+0000","last_undegraded":"2024-09-21T18:57:50.726233+0000","last_fullsized":"2024-09-21T18:57:50.726233+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723308+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:57:51.723308+0000","last_peered":"2024-09-21T18:57:51.723308+0000","last_clean":"2024-09-21T18:57:51.723308+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:57:51.723308+0000","last_undegraded":"2024-09-21T18:57:51.723308+0000","last_fullsized":"2024-09-21T18:57:51.723308+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"150'14962","reported_seq":15119,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.722862+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:57:51.722862+0000","last_peered":"2024-09-21T18:57:51.722862+0000","last_clean":"2024-09-21T18:57:51.722862+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:57:51.722862+0000","last_undegraded":"2024-09-21T18:57:51.722862+0000","last_fullsized":"2024-09-21T18:57:51.722862+0000","mapping_epoch":79,"log_start":"119'9700","ondisk_log_start":"119'9700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:29.184091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141295616,"num_objects":2156,"num_object_clones":0,"num_object_copies":6468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14962,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":234,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.705742+0000","last_change":"2024-09-21T18:56:35.846693+0000","last_active":"2024-09-21T18:57:50.705742+0000","last_peered":"2024-09-21T18:57:50.705742+0000","last_clean":"2024-09-21T18:57:50.705742+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:57:50.705742+0000","last_undegraded":"2024-09-21T18:57:50.705742+0000","last_fullsized":"2024-09-21T18:57:50.705742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:56:35.846605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:35.846605+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:51.214847+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":234,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.728492+0000","last_change":"2024-09-21T18:55:51.567353+0000","last_active":"2024-09-21T18:57:50.728492+0000","last_peered":"2024-09-21T18:57:50.728492+0000","last_clean":"2024-09-21T18:57:50.728492+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:57:50.728492+0000","last_undegraded":"2024-09-21T18:57:50.728492+0000","last_fullsized":"2024-09-21T18:57:50.728492+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:55.125124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":272,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727699+0000","last_change":"2024-09-21T18:57:23.116600+0000","last_active":"2024-09-21T18:57:50.727699+0000","last_peered":"2024-09-21T18:57:50.727699+0000","last_clean":"2024-09-21T18:57:50.727699+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:57:50.727699+0000","last_undegraded":"2024-09-21T18:57:50.727699+0000","last_fullsized":"2024-09-21T18:57:50.727699+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:57:23.116520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:23.116520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.889890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":207,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.705615+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:57:50.705615+0000","last_peered":"2024-09-21T18:57:50.705615+0000","last_clean":"2024-09-21T18:57:50.705615+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:57:50.705615+0000","last_undegraded":"2024-09-21T18:57:50.705615+0000","last_fullsized":"2024-09-21T18:57:50.705615+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:26.930637+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":218,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.716410+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:57:50.716410+0000","last_peered":"2024-09-21T18:57:50.716410+0000","last_clean":"2024-09-21T18:57:50.716410+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:57:50.716410+0000","last_undegraded":"2024-09-21T18:57:50.716410+0000","last_fullsized":"2024-09-21T18:57:50.716410+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724406+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:57:51.724406+0000","last_peered":"2024-09-21T18:57:51.724406+0000","last_clean":"2024-09-21T18:57:51.724406+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:57:51.724406+0000","last_undegraded":"2024-09-21T18:57:51.724406+0000","last_fullsized":"2024-09-21T18:57:51.724406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.709903+0000","last_change":"2024-09-21T18:56:40.848060+0000","last_active":"2024-09-21T18:57:50.709903+0000","last_peered":"2024-09-21T18:57:50.709903+0000","last_clean":"2024-09-21T18:57:50.709903+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:57:50.709903+0000","last_undegraded":"2024-09-21T18:57:50.709903+0000","last_fullsized":"2024-09-21T18:57:50.709903+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:40.847976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.847976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:57.531530+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":183,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727691+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:57:50.727691+0000","last_peered":"2024-09-21T18:57:50.727691+0000","last_clean":"2024-09-21T18:57:50.727691+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:57:50.727691+0000","last_undegraded":"2024-09-21T18:57:50.727691+0000","last_fullsized":"2024-09-21T18:57:50.727691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:00.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":94,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.725900+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:57:50.725900+0000","last_peered":"2024-09-21T18:57:50.725900+0000","last_clean":"2024-09-21T18:57:50.725900+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:57:50.725900+0000","last_undegraded":"2024-09-21T18:57:50.725900+0000","last_fullsized":"2024-09-21T18:57:50.725900+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727602+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:57:50.727602+0000","last_peered":"2024-09-21T18:57:50.727602+0000","last_clean":"2024-09-21T18:57:50.727602+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:57:50.727602+0000","last_undegraded":"2024-09-21T18:57:50.727602+0000","last_fullsized":"2024-09-21T18:57:50.727602+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:08.070910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":199,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724277+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:57:51.724277+0000","last_peered":"2024-09-21T18:57:51.724277+0000","last_clean":"2024-09-21T18:57:51.724277+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:57:51.724277+0000","last_undegraded":"2024-09-21T18:57:51.724277+0000","last_fullsized":"2024-09-21T18:57:51.724277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":192,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.718263+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:57:50.718263+0000","last_peered":"2024-09-21T18:57:50.718263+0000","last_clean":"2024-09-21T18:57:50.718263+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:57:50.718263+0000","last_undegraded":"2024-09-21T18:57:50.718263+0000","last_fullsized":"2024-09-21T18:57:50.718263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:37.565916+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":216,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706464+0000","last_change":"2024-09-21T18:57:25.890791+0000","last_active":"2024-09-21T18:57:50.706464+0000","last_peered":"2024-09-21T18:57:50.706464+0000","last_clean":"2024-09-21T18:57:50.706464+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:57:50.706464+0000","last_undegraded":"2024-09-21T18:57:50.706464+0000","last_fullsized":"2024-09-21T18:57:50.706464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:25.890710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.890710+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:51.251415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.721942+0000","last_change":"2024-09-21T18:55:51.523039+0000","last_active":"2024-09-21T18:57:50.721942+0000","last_peered":"2024-09-21T18:57:50.721942+0000","last_clean":"2024-09-21T18:57:50.721942+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:57:50.721942+0000","last_undegraded":"2024-09-21T18:57:50.721942+0000","last_fullsized":"2024-09-21T18:57:50.721942+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":265,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724572+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:57:51.724572+0000","last_peered":"2024-09-21T18:57:51.724572+0000","last_clean":"2024-09-21T18:57:51.724572+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:57:51.724572+0000","last_undegraded":"2024-09-21T18:57:51.724572+0000","last_fullsized":"2024-09-21T18:57:51.724572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":192,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715732+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:57:50.715732+0000","last_peered":"2024-09-21T18:57:50.715732+0000","last_clean":"2024-09-21T18:57:50.715732+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:57:50.715732+0000","last_undegraded":"2024-09-21T18:57:50.715732+0000","last_fullsized":"2024-09-21T18:57:50.715732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724921+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:57:51.724921+0000","last_peered":"2024-09-21T18:57:51.724921+0000","last_clean":"2024-09-21T18:57:51.724921+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:57:51.724921+0000","last_undegraded":"2024-09-21T18:57:51.724921+0000","last_fullsized":"2024-09-21T18:57:51.724921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":212,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727910+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:57:50.727910+0000","last_peered":"2024-09-21T18:57:50.727910+0000","last_clean":"2024-09-21T18:57:50.727910+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:57:50.727910+0000","last_undegraded":"2024-09-21T18:57:50.727910+0000","last_fullsized":"2024-09-21T18:57:50.727910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.429722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":275,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.706278+0000","last_change":"2024-09-21T18:55:51.543525+0000","last_active":"2024-09-21T18:57:50.706278+0000","last_peered":"2024-09-21T18:57:50.706278+0000","last_clean":"2024-09-21T18:57:50.706278+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:57:50.706278+0000","last_undegraded":"2024-09-21T18:57:50.706278+0000","last_fullsized":"2024-09-21T18:57:50.706278+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.924485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.726203+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:57:50.726203+0000","last_peered":"2024-09-21T18:57:50.726203+0000","last_clean":"2024-09-21T18:57:50.726203+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:57:50.726203+0000","last_undegraded":"2024-09-21T18:57:50.726203+0000","last_fullsized":"2024-09-21T18:57:50.726203+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":239,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.714777+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:57:50.714777+0000","last_peered":"2024-09-21T18:57:50.714777+0000","last_clean":"2024-09-21T18:57:50.714777+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:57:50.714777+0000","last_undegraded":"2024-09-21T18:57:50.714777+0000","last_fullsized":"2024-09-21T18:57:50.714777+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":197,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.715950+0000","last_change":"2024-09-21T18:53:07.408305+0000","last_active":"2024-09-21T18:57:50.715950+0000","last_peered":"2024-09-21T18:57:50.715950+0000","last_clean":"2024-09-21T18:57:50.715950+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:57:50.715950+0000","last_undegraded":"2024-09-21T18:57:50.715950+0000","last_fullsized":"2024-09-21T18:57:50.715950+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":199,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723912+0000","last_change":"2024-09-21T18:57:25.184491+0000","last_active":"2024-09-21T18:57:51.723912+0000","last_peered":"2024-09-21T18:57:51.723912+0000","last_clean":"2024-09-21T18:57:51.723912+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:57:51.723912+0000","last_undegraded":"2024-09-21T18:57:51.723912+0000","last_fullsized":"2024-09-21T18:57:51.723912+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:25.180690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.180690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:31.159574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723473+0000","last_change":"2024-09-21T18:55:51.516509+0000","last_active":"2024-09-21T18:57:51.723473+0000","last_peered":"2024-09-21T18:57:51.723473+0000","last_clean":"2024-09-21T18:57:51.723473+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:57:51.723473+0000","last_undegraded":"2024-09-21T18:57:51.723473+0000","last_fullsized":"2024-09-21T18:57:51.723473+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":227,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724241+0000","last_change":"2024-09-21T18:57:49.327777+0000","last_active":"2024-09-21T18:57:51.724241+0000","last_peered":"2024-09-21T18:57:51.724241+0000","last_clean":"2024-09-21T18:57:51.724241+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:57:51.724241+0000","last_undegraded":"2024-09-21T18:57:51.724241+0000","last_fullsized":"2024-09-21T18:57:51.724241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:49.327123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.327123+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:56.940918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":211,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.728117+0000","last_change":"2024-09-21T18:57:19.082962+0000","last_active":"2024-09-21T18:57:50.728117+0000","last_peered":"2024-09-21T18:57:50.728117+0000","last_clean":"2024-09-21T18:57:50.728117+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:57:50.728117+0000","last_undegraded":"2024-09-21T18:57:50.728117+0000","last_fullsized":"2024-09-21T18:57:50.728117+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.082873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.082873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.905947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723414+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:57:51.723414+0000","last_peered":"2024-09-21T18:57:51.723414+0000","last_clean":"2024-09-21T18:57:51.723414+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:57:51.723414+0000","last_undegraded":"2024-09-21T18:57:51.723414+0000","last_fullsized":"2024-09-21T18:57:51.723414+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.721633+0000","last_change":"2024-09-21T18:55:51.536412+0000","last_active":"2024-09-21T18:57:50.721633+0000","last_peered":"2024-09-21T18:57:50.721633+0000","last_clean":"2024-09-21T18:57:50.721633+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:57:50.721633+0000","last_undegraded":"2024-09-21T18:57:50.721633+0000","last_fullsized":"2024-09-21T18:57:50.721633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":215,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723651+0000","last_change":"2024-09-21T18:53:09.718006+0000","last_active":"2024-09-21T18:57:51.723651+0000","last_peered":"2024-09-21T18:57:51.723651+0000","last_clean":"2024-09-21T18:57:51.723651+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:57:51.723651+0000","last_undegraded":"2024-09-21T18:57:51.723651+0000","last_fullsized":"2024-09-21T18:57:51.723651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":204,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.709937+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:57:50.709937+0000","last_peered":"2024-09-21T18:57:50.709937+0000","last_clean":"2024-09-21T18:57:50.709937+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:57:50.709937+0000","last_undegraded":"2024-09-21T18:57:50.709937+0000","last_fullsized":"2024-09-21T18:57:50.709937+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.912165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":196,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727243+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:57:50.727243+0000","last_peered":"2024-09-21T18:57:50.727243+0000","last_clean":"2024-09-21T18:57:50.727243+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:57:50.727243+0000","last_undegraded":"2024-09-21T18:57:50.727243+0000","last_fullsized":"2024-09-21T18:57:50.727243+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.212178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.723519+0000","last_change":"2024-09-21T18:55:51.529791+0000","last_active":"2024-09-21T18:57:51.723519+0000","last_peered":"2024-09-21T18:57:51.723519+0000","last_clean":"2024-09-21T18:57:51.723519+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:57:51.723519+0000","last_undegraded":"2024-09-21T18:57:51.723519+0000","last_fullsized":"2024-09-21T18:57:51.723519+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":215,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.722809+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:57:51.722809+0000","last_peered":"2024-09-21T18:57:51.722809+0000","last_clean":"2024-09-21T18:57:51.722809+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:57:51.722809+0000","last_undegraded":"2024-09-21T18:57:51.722809+0000","last_fullsized":"2024-09-21T18:57:51.722809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":84,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.725296+0000","last_change":"2024-09-21T18:55:51.529323+0000","last_active":"2024-09-21T18:57:50.725296+0000","last_peered":"2024-09-21T18:57:50.725296+0000","last_clean":"2024-09-21T18:57:50.725296+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:57:50.725296+0000","last_undegraded":"2024-09-21T18:57:50.725296+0000","last_fullsized":"2024-09-21T18:57:50.725296+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":188,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.716450+0000","last_change":"2024-09-21T18:53:11.413192+0000","last_active":"2024-09-21T18:57:50.716450+0000","last_peered":"2024-09-21T18:57:50.716450+0000","last_clean":"2024-09-21T18:57:50.716450+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:57:50.716450+0000","last_undegraded":"2024-09-21T18:57:50.716450+0000","last_fullsized":"2024-09-21T18:57:50.716450+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":183,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.727408+0000","last_change":"2024-09-21T18:53:13.423010+0000","last_active":"2024-09-21T18:57:50.727408+0000","last_peered":"2024-09-21T18:57:50.727408+0000","last_clean":"2024-09-21T18:57:50.727408+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:57:50.727408+0000","last_undegraded":"2024-09-21T18:57:50.727408+0000","last_fullsized":"2024-09-21T18:57:50.727408+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:57:53.192235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.724846+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:57:51.724846+0000","last_peered":"2024-09-21T18:57:51.724846+0000","last_clean":"2024-09-21T18:57:51.724846+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:57:51.724846+0000","last_undegraded":"2024-09-21T18:57:51.724846+0000","last_fullsized":"2024-09-21T18:57:51.724846+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:48.365341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":221,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T18:57:51.722984+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:57:51.722984+0000","last_peered":"2024-09-21T18:57:51.722984+0000","last_clean":"2024-09-21T18:57:51.722984+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:57:51.722984+0000","last_undegraded":"2024-09-21T18:57:51.722984+0000","last_fullsized":"2024-09-21T18:57:51.722984+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":211,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.717371+0000","last_change":"2024-09-21T18:53:09.411220+0000","last_active":"2024-09-21T18:57:50.717371+0000","last_peered":"2024-09-21T18:57:50.717371+0000","last_clean":"2024-09-21T18:57:50.717371+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:57:50.717371+0000","last_undegraded":"2024-09-21T18:57:50.717371+0000","last_fullsized":"2024-09-21T18:57:50.717371+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":85,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-21T18:57:50.726230+0000","last_change":"2024-09-21T18:55:51.545494+0000","last_active":"2024-09-21T18:57:50.726230+0000","last_peered":"2024-09-21T18:57:50.726230+0000","last_clean":"2024-09-21T18:57:50.726230+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:57:50.726230+0000","last_undegraded":"2024-09-21T18:57:50.726230+0000","last_fullsized":"2024-09-21T18:57:50.726230+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2285371416,"num_objects":34873,"num_object_clones":0,"num_object_copies":104619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":584,"num_objects_unfound":0,"num_objects_dirty":34873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":243121,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35415,"num_bytes_recovered":2320826392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7656194048,"data_stored":7656177760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86919,"ondisk_log_size":86919,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":273,"num_read_kb":268,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266749,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039344,"kb_used_data":909588,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92705808,"statfs":{"total":95995035648,"available":94930747392,"internally_reserved":0,"allocated":931418112,"data_stored":930978700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528386,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946268,"kb_used_data":829000,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92798884,"statfs":{"total":95995035648,"available":95026057216,"internally_reserved":0,"allocated":848896000,"data_stored":848468251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":120060358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790021,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858904,"kb_used_data":751364,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92886248,"statfs":{"total":95995035648,"available":95115517952,"internally_reserved":0,"allocated":769396736,"data_stored":768972645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018955,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958196,"kb_used_data":836760,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92786956,"statfs":{"total":95995035648,"available":95013842944,"internally_reserved":0,"allocated":856842240,"data_stored":856398688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630712,"kb_used_data":1387112,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92114440,"statfs":{"total":95995035648,"available":94325186560,"internally_reserved":0,"allocated":1420402688,"data_stored":1419943075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":249428426},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946976,"kb_used_data":829708,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92798176,"statfs":{"total":95995035648,"available":95025332224,"internally_reserved":0,"allocated":849620992,"data_stored":849189264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1025656,"kb_used_data":841764,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92719496,"statfs":{"total":95995035648,"available":94944763904,"internally_reserved":0,"allocated":861966336,"data_stored":861510373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1321968,"kb_used_data":1078364,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92423184,"statfs":{"total":95995035648,"available":94641340416,"internally_reserved":0,"allocated":1104244736,"data_stored":1103797845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1137643520,"data_stored":1137639448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":861016064,"data_stored":861011992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":848691200,"data_stored":848691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1414336512,"data_stored":1414332440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":854523904,"data_stored":854523904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":762707968,"data_stored":762707968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":847384576,"data_stored":847380504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":929890304,"data_stored":929890304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:57:58.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:57:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:57:58.261 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:57 smithi104 ceph-mon[32438]: 7.7 scrub starts 2024-09-21T18:57:58.261 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1785632119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:58.261 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:57 smithi104 ceph-mon[32438]: from='client.35114 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:57:58.293 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:58.329 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:58.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:57:58.468 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:58.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:58.608 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:57:58.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:57:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:58.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:57:58.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:57:58.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:58.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:57:58.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: from='client.35108 v1:172.21.15.81:0/1231301153' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: pgmap v213: 148 pgs: 148 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s; 17 MiB/s, 267 objects/s recovering 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: from='client.35114 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:58.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: 3.14 scrub starts 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: 3.14 scrub ok 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: 7.7 scrub ok 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: 6.10 scrub starts 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[29384]: 6.10 scrub ok 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: from='client.35108 v1:172.21.15.81:0/1231301153' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: pgmap v213: 148 pgs: 148 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s; 17 MiB/s, 267 objects/s recovering 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: from='client.35114 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T18:57:58.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:58.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: 3.14 scrub starts 2024-09-21T18:57:58.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: 3.14 scrub ok 2024-09-21T18:57:58.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: 7.7 scrub ok 2024-09-21T18:57:58.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: 6.10 scrub starts 2024-09-21T18:57:58.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:58 smithi081 ceph-mon[39192]: 6.10 scrub ok 2024-09-21T18:57:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.019 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:59.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:57:59.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:59.156 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: from='client.35108 v1:172.21.15.81:0/1231301153' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: pgmap v213: 148 pgs: 148 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s; 17 MiB/s, 267 objects/s recovering 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: from='client.35114 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: 3.14 scrub starts 2024-09-21T18:57:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: 3.14 scrub ok 2024-09-21T18:57:59.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: 7.7 scrub ok 2024-09-21T18:57:59.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: 6.10 scrub starts 2024-09-21T18:57:59.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:58 smithi104 ceph-mon[32438]: 6.10 scrub ok 2024-09-21T18:57:59.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:57:59.294 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:57:59.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:57:59.422 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:57:59.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:57:59.557 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:57:59.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:59.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:59.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:57:59.818 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:57:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:57:59.915 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 3.f scrub starts 2024-09-21T18:57:59.915 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 3.f scrub ok 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 5.1e scrub starts 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 5.1e scrub ok 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: 7.c scrub starts 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[39192]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 3.f scrub starts 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 3.f scrub ok 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 5.1e scrub starts 2024-09-21T18:57:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 5.1e scrub ok 2024-09-21T18:57:59.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: 7.c scrub starts 2024-09-21T18:57:59.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:57:59 smithi081 ceph-mon[29384]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T18:57:59.920 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:57:59.957 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:57:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:57:59.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:00.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:00.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:00.104 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:00.105 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:00.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:00.141 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 3.f scrub starts 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 3.f scrub ok 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 5.1e scrub starts 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 5.1e scrub ok 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: 7.c scrub starts 2024-09-21T18:58:00.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:57:59 smithi104 ceph-mon[32438]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T18:58:00.175 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:00.175 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:00.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:00.208 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:00.208 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:00.230 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:00.243 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:00.243 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:00.277 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:00.277 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:00.310 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:00.310 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:00.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:00.346 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:00.346 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:00.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:00.496 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:00.598 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:00.637 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:00.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:00.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:00.765 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:00.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:00.892 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:58:00.892 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:58:00.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:58:00.892 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:58:00.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.892 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:99377 7.9 7:9f989195:::benchmark_data_smithi081_73391_object99376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 163.17884324799999, 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106765947, 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 99377 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707610+0000", 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.196e-06 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707638+0000", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7880000000000001e-05 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763151+0000", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055513121999999998 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763168+0000", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6152000000000001e-05 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763470+0000", 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030208399999999999 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763608+0000", 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013794999999999999 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763999+0000", 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000391691 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.771410+0000", 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074111979999999999 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.771550+0000", 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013909999999999999 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814280+0000", 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.042729984999999998 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814321+0000", 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1594999999999997e-05 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814373+0000", 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1993999999999997e-05 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101895 7.b 7:d6f8fed5:::benchmark_data_smithi081_73391_object101894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:58:00.898 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 160.9763509, 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10653628599999999, 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101895 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910101+0000", 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8109999999999999e-06 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910123+0000", 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1500000000000001e-05 2024-09-21T18:58:00.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910138+0000", 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4798e-05 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910145+0000", 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1790000000000002e-06 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910195+0000", 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9716000000000003e-05 2024-09-21T18:58:00.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910299+0000", 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104134 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910439+0000", 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001403 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.912887+0000", 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024483080000000002 2024-09-21T18:58:00.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916379+0000", 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034916880000000002 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016537+0000", 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.100158393 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016591+0000", 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3170000000000001e-05 2024-09-21T18:58:00.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016636+0000", 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5289e-05 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101904 7.b 7:d7f209bc:::benchmark_data_smithi081_73391_object101903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 160.969566507, 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10347793, 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.904 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101904 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916886+0000", 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916905+0000", 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9527000000000001e-05 2024-09-21T18:58:00.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916915+0000", 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8130000000000007e-06 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916920+0000", 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1420000000000001e-06 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916958+0000", 2024-09-21T18:58:00.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8059000000000003e-05 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.917042+0000", 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4251000000000004e-05 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.917130+0000", 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7626000000000004e-05 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.919036+0000", 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00190617 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.919594+0000", 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055802599999999999 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020228+0000", 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.100633629 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.909 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020322+0000", 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4293e-05 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020362+0000", 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9777999999999998e-05 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.910 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136125 7.b 7:df011645:::benchmark_data_smithi081_73391_object136124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 132.725470411, 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136450605, 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136125 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160983+0000", 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4490000000000002e-06 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161014+0000", 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1038000000000003e-05 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161026+0000", 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2753e-05 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161035+0000", 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0040000000000005e-06 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161146+0000", 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110518 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161292+0000", 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001461 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161445+0000", 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000153475 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170642+0000", 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0091963119999999999 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170656+0000", 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3947e-05 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297280+0000", 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12662463500000001 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297381+0000", 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101037 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297431+0000", 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9336999999999999e-05 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136126 7.b 7:d13dafa2:::benchmark_data_smithi081_73391_object136125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:58:00.916 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 132.725388552, 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139333067, 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136126 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:58:00.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.918 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161063+0000", 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3500000000000005e-07 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161084+0000", 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0755e-05 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161091+0000", 2024-09-21T18:58:00.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2169999999999997e-06 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161097+0000", 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3230000000000003e-06 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164270+0000", 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031733239999999999 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164391+0000", 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120575 2024-09-21T18:58:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164535+0000", 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014399899999999999 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170661+0000", 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061256419999999997 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170700+0000", 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9578e-05 2024-09-21T18:58:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300264+0000", 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12956358800000001 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300355+0000", 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1219999999999997e-05 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300395+0000", 2024-09-21T18:58:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0011e-05 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136128 7.b 7:df396259:::benchmark_data_smithi081_73391_object136127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 132.72374600500001, 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139089409, 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136128 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162707+0000", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162731+0000", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4485e-05 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162747+0000", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5452999999999999e-05 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162755+0000", 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9920000000000007e-06 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167113+0000", 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043581749999999997 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167237+0000", 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012398699999999999 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167386+0000", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149672 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170665+0000", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032787960000000001 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170739+0000", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4108000000000003e-05 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301654+0000", 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13091487199999999 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301749+0000", 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5187999999999997e-05 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301794+0000", 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4558000000000003e-05 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136146 7.b 7:da687baa:::benchmark_data_smithi081_73391_object136145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 132.71107629299999, 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12656247800000001, 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136146 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:58:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175376+0000", 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1040000000000001e-06 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175400+0000", 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4162e-05 2024-09-21T18:58:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175412+0000", 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1802e-05 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175420+0000", 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7149999999999998e-06 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175494+0000", 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3688000000000003e-05 2024-09-21T18:58:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175609+0000", 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115519 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175745+0000", 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013621299999999999 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177851+0000", 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021052779999999999 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177986+0000", 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013539800000000001 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301842+0000", 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.123855964 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301899+0000", 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7426000000000002e-05 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301937+0000", 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7209000000000002e-05 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136148 7.b 7:d9fea987:::benchmark_data_smithi081_73391_object136147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 132.708522276, 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12981583899999999, 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136148 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:58:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177930+0000", 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.545e-06 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177953+0000", 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.349e-05 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177962+0000", 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8890000000000008e-06 2024-09-21T18:58:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177969+0000", 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8759999999999999e-06 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178007+0000", 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8126000000000001e-05 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178088+0000", 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0684000000000002e-05 2024-09-21T18:58:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178175+0000", 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7643000000000001e-05 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.179852+0000", 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001676774 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.179997+0000", 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014443799999999999 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307615+0000", 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12761845299999999 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307693+0000", 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8103999999999996e-05 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307744+0000", 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0816999999999998e-05 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168555 7.7 e123/114 7:e5122ffb:::benchmark_data_smithi081_73391_object29556:head v 123'10492, mlcod=123'10490)", 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.113933+0000", 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.772517374000003, 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.111488108, 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113933+0000", 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113933+0000", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113934+0000", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.001e-06 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113939+0000", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1259999999999997e-06 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113946+0000", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2019999999999998e-06 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113952+0000", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4990000000000002e-06 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.117198+0000", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032465110000000001 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.117222+0000", 2024-09-21T18:58:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3991000000000001e-05 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225368+0000", 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.108145599 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:00.944 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:00.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225421+0000", 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3179e-05 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168557 7.7 e123/114 7:e8d3dc47:::benchmark_data_smithi081_73391_object29558:head v 123'10493, mlcod=123'10490)", 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.767766647000002, 2024-09-21T18:58:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10676554100000001, 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.29e-07 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118687+0000", 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3979999999999999e-06 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118692+0000", 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.233e-06 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118695+0000", 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128618+0000", 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0099226269999999998 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128640+0000", 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1719999999999999e-05 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225424+0000", 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096784673000000002 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225449+0000", 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4953999999999999e-05 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168564 7.7 e123/114 7:e16a2db8:::benchmark_data_smithi081_73391_object29565:head v 123'10494, mlcod=123'10490)", 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.122367+0000", 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.764083264000007, 2024-09-21T18:58:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.103158955, 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122367+0000", 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122367+0000", 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122368+0000", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.083e-06 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122374+0000", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6969999999999998e-06 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122383+0000", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5179999999999993e-06 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122390+0000", 2024-09-21T18:58:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.105e-06 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.135024+0000", 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012634115 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.135044+0000", 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0098e-05 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225452+0000", 2024-09-21T18:58:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.090408540999999995 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225526+0000", 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3798000000000001e-05 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169000 7.b 7:de931d3b:::benchmark_data_smithi081_73391_object30001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637210+0000", 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.249240021000006, 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106838799, 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169000 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637210+0000", 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637210+0000", 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637212+0000", 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637216+0000", 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8299999999999998e-06 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637224+0000", 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2039999999999997e-06 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637230+0000", 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0460000000000002e-06 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637377+0000", 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014751200000000001 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637486+0000", 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010838199999999999 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637585+0000", 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9575999999999997e-05 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646802+0000", 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0092170120000000001 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646840+0000", 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7957999999999998e-05 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.743910+0000", 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097069595999999994 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744010+0000", 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9980000000000002e-05 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744049+0000", 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9313000000000001e-05 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169001 7.b 7:df6f9ae3:::benchmark_data_smithi081_73391_object30002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637246+0000", 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.249204336000005, 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106883351, 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169001 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637246+0000", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637246+0000", 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637247+0000", 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3099999999999997e-07 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637249+0000", 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.74e-06 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637255+0000", 2024-09-21T18:58:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4779999999999999e-06 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637260+0000", 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5639999999999998e-06 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.640288+0000", 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030289150000000001 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.640393+0000", 2024-09-21T18:58:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010474100000000001 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.640504+0000", 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011120699999999999 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646873+0000", 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063683589999999997 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646907+0000", 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4659999999999997e-05 2024-09-21T18:58:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744056+0000", 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097148935000000006 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744095+0000", 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9070999999999999e-05 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744129+0000", 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4050000000000001e-05 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169002 7.b 7:d5eb2253:::benchmark_data_smithi081_73391_object30003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.249185505, 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10702123099999999, 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169002 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6399999999999998e-07 2024-09-21T18:58:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637268+0000", 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3939999999999999e-06 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637272+0000", 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3719999999999998e-06 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637276+0000", 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0130000000000004e-06 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646949+0000", 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009673325 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.647050+0000", 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100342 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.647154+0000", 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010430899999999999 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744164+0000", 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097010258000000002 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744175+0000", 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0299000000000001e-05 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744209+0000", 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4576000000000002e-05 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744252+0000", 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2759000000000002e-05 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744286+0000", 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4220000000000001e-05 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169003 7.9 7:9a50a309:::benchmark_data_smithi081_73391_object30004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637280+0000", 2024-09-21T18:58:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.249170034000002, 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.105220251, 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169003 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637280+0000", 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637280+0000", 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637281+0000", 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3000000000000002e-07 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637283+0000", 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.858e-06 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637287+0000", 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3250000000000001e-06 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637291+0000", 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0080000000000004e-06 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637455+0000", 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016420700000000001 2024-09-21T18:58:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637580+0000", 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000124595 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637989+0000", 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040934400000000002 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.738992+0000", 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10100292299999999 2024-09-21T18:58:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739040+0000", 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8390000000000003e-05 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739085+0000", 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4425999999999998e-05 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742244+0000", 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031589750000000001 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742372+0000", 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012819799999999999 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742501+0000", 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128672 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168997 7.4 e123/114 7:2a66f29b:::benchmark_data_smithi081_73391_object29998:head v 123'10730, mlcod=123'10727)", 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637303+0000", 2024-09-21T18:58:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.249147101000005, 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10184473600000001, 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637303+0000", 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637303+0000", 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637304+0000", 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1300000000000001e-07 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637308+0000", 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.002e-06 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637313+0000", 2024-09-21T18:58:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2290000000000002e-06 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637317+0000", 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1150000000000004e-06 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.641772+0000", 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044551060000000003 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.641794+0000", 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1693999999999999e-05 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.738772+0000", 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096977709999999995 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739148+0000", 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037646700000000001 2024-09-21T18:58:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169011 7.9 7:9db9a5ae:::benchmark_data_smithi081_73391_object30012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.645238+0000", 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 96.241212923000006, 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.108457962, 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169011 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645238+0000", 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645238+0000", 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645240+0000", 2024-09-21T18:58:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.086e-06 2024-09-21T18:58:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645244+0000", 2024-09-21T18:58:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0749999999999999e-06 2024-09-21T18:58:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645252+0000", 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4339999999999992e-06 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645258+0000", 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8329999999999999e-06 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739378+0000", 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.094119797000000005 2024-09-21T18:58:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739529+0000", 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015165299999999999 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739662+0000", 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013272699999999999 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742510+0000", 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028483240000000002 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.748769+0000", 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0062584529999999998 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.748856+0000", 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7122999999999999e-05 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.753525+0000", 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046688470000000003 2024-09-21T18:58:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.753593+0000", 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7916999999999994e-05 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.753696+0000", 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102693 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242088 7.b 7:d4630f67:::benchmark_data_smithi081_73391_object103089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512627+0000", 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.373823236, 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.165892709, 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242088 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512627+0000", 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512627+0000", 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512629+0000", 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0770000000000002e-06 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512636+0000", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.562e-06 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512649+0000", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3529e-05 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512657+0000", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5639999999999999e-06 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512723+0000", 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6489999999999995e-05 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512729+0000", 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0240000000000003e-06 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514154+0000", 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001424771 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672882+0000", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.158727757 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673023+0000", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014103799999999999 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677654+0000", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046308169999999997 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678082+0000", 2024-09-21T18:58:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000428572 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678431+0000", 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034839400000000003 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678483+0000", 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1996999999999997e-05 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678520+0000", 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7116999999999997e-05 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242087 7.d e149/114 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head v 149'15301, mlcod=148'15299)", 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512853+0000", 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.373597148, 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16432387000000001, 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512853+0000", 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512853+0000", 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512854+0000", 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.052e-06 2024-09-21T18:58:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512862+0000", 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3749999999999997e-06 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512871+0000", 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5920000000000007e-06 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512878+0000", 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6120000000000004e-06 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512943+0000", 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4738000000000003e-05 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512966+0000", 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3591000000000002e-05 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677136+0000", 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16416932200000001 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677177+0000", 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1588000000000002e-05 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242084 7.0 e149/79 7:0c19e5c0:::benchmark_data_smithi081_73391_object103085:head v 149'15371, mlcod=148'15368)", 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.372167343999999, 2024-09-21T18:58:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16410017099999999, 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5900000000000003e-07 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514287+0000", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514294+0000", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4200000000000004e-06 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674533+0000", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.160238726 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674644+0000", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111369 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674663+0000", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9338e-05 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678317+0000", 2024-09-21T18:58:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003654175 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678383+0000", 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5807000000000002e-05 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:01.002 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[29384]: pgmap v215: 148 pgs: 148 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 695 op/s; 4.5 MiB/s, 72 objects/s recovering 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[29384]: 7.c scrub ok 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[29384]: 6.9 scrub starts 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[29384]: 6.9 scrub ok 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3579315039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[39192]: pgmap v215: 148 pgs: 148 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 695 op/s; 4.5 MiB/s, 72 objects/s recovering 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[39192]: 7.c scrub ok 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[39192]: 6.9 scrub starts 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[39192]: 6.9 scrub ok 2024-09-21T18:58:01.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3579315039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:01.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:58:01.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:01.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:01.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:00 smithi104 ceph-mon[32438]: pgmap v215: 148 pgs: 148 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 695 op/s; 4.5 MiB/s, 72 objects/s recovering 2024-09-21T18:58:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:00 smithi104 ceph-mon[32438]: 7.c scrub ok 2024-09-21T18:58:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:00 smithi104 ceph-mon[32438]: 6.9 scrub starts 2024-09-21T18:58:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:00 smithi104 ceph-mon[32438]: 6.9 scrub ok 2024-09-21T18:58:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3579315039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:01.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:58:01.277 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:01.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:01.426 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:01.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:01.554 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:01.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:01.683 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:01.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:01.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:01.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:01.949 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: 6.3 scrub starts 2024-09-21T18:58:01.949 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: 6.3 scrub ok 2024-09-21T18:58:01.949 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[29384]: 7.4 scrub starts 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: 6.3 scrub starts 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: 6.3 scrub ok 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T18:58:01.950 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T18:58:01.951 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:01 smithi081 ceph-mon[39192]: 7.4 scrub starts 2024-09-21T18:58:01.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:01.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:02.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: 6.3 scrub starts 2024-09-21T18:58:02.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: 6.3 scrub ok 2024-09-21T18:58:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T18:58:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T18:58:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T18:58:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:01 smithi104 ceph-mon[32438]: 7.4 scrub starts 2024-09-21T18:58:02.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:02.109 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:02.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:02.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:02.356 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:02.388 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:02.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:02.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:02.564 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T18:58:02.596 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:02.598 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:02.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.696 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:02.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:02.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:02.854 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:02.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:02.983 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:02.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:03.050 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: pgmap v218: 148 pgs: 148 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 799 op/s; 5.8 MiB/s, 93 objects/s recovering 2024-09-21T18:58:03.050 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 7.4 scrub ok 2024-09-21T18:58:03.050 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T18:58:03.050 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T18:58:03.050 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 4.11 scrub starts 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 4.11 scrub ok 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: pgmap v218: 148 pgs: 148 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 799 op/s; 5.8 MiB/s, 93 objects/s recovering 2024-09-21T18:58:03.051 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 7.4 scrub ok 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 4.11 scrub starts 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 4.11 scrub ok 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T18:58:03.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:02 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T18:58:03.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:03.123 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:03.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:03.249 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: pgmap v218: 148 pgs: 148 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 799 op/s; 5.8 MiB/s, 93 objects/s recovering 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 7.4 scrub ok 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 4.11 scrub starts 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 4.11 scrub ok 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T18:58:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:02 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T18:58:03.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:03.384 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:03.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:03.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:03.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:03.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:03.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:03.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:03.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:03.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:03 smithi081 ceph-mon[39192]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T18:58:03.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:03 smithi081 ceph-mon[39192]: 6.1a scrub starts 2024-09-21T18:58:03.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:03 smithi081 ceph-mon[39192]: 6.1a scrub ok 2024-09-21T18:58:03.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:03 smithi081 ceph-mon[29384]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T18:58:03.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:03 smithi081 ceph-mon[29384]: 6.1a scrub starts 2024-09-21T18:58:03.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:03 smithi081 ceph-mon[29384]: 6.1a scrub ok 2024-09-21T18:58:03.997 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:58:03.999 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:58:04.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:04.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:58:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:04.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:04.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:04.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:03 smithi104 ceph-mon[32438]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T18:58:04.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:03 smithi104 ceph-mon[32438]: 6.1a scrub starts 2024-09-21T18:58:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:03 smithi104 ceph-mon[32438]: 6.1a scrub ok 2024-09-21T18:58:04.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:04.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.467 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:04.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:04.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:04.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:04.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:04.740 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:04.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:04.875 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:58:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:04.933 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:04.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:05.010 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-21T18:58:03.470940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1658526259,"num_objects":25498,"num_object_clones":0,"num_object_copies":76494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25498,"num_whiteouts":0,"num_read":1127,"num_read_kb":984,"num_write":253244,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36103,"num_bytes_recovered":2359238130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89451,"ondisk_log_size":89451,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6639060,"kb_used_data":5317616,"kb_used_omap":12,"kb_used_meta":1321075,"kb_avail":743322156,"statfs":{"total":767960285184,"available":761161887744,"internally_reserved":0,"allocated":5445238784,"data_stored":5441510601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1352781364},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,2,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-334626816,"num_objects":-5106,"num_object_clones":0,"num_object_copies":-15318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5106,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5106,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.705534"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":95,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.839260+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:58:00.839260+0000","last_peered":"2024-09-21T18:58:00.839260+0000","last_clean":"2024-09-21T18:58:00.839260+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:58:00.839260+0000","last_undegraded":"2024-09-21T18:58:00.839260+0000","last_fullsized":"2024-09-21T18:58:00.839260+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":244,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819138+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:58:00.819138+0000","last_peered":"2024-09-21T18:58:00.819138+0000","last_clean":"2024-09-21T18:58:00.819138+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:58:00.819138+0000","last_undegraded":"2024-09-21T18:58:00.819138+0000","last_fullsized":"2024-09-21T18:58:00.819138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835672+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:58:00.835672+0000","last_peered":"2024-09-21T18:58:00.835672+0000","last_clean":"2024-09-21T18:58:00.835672+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:58:00.835672+0000","last_undegraded":"2024-09-21T18:58:00.835672+0000","last_fullsized":"2024-09-21T18:58:00.835672+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.239093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":194,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819200+0000","last_change":"2024-09-21T18:53:13.428298+0000","last_active":"2024-09-21T18:58:00.819200+0000","last_peered":"2024-09-21T18:58:00.819200+0000","last_clean":"2024-09-21T18:58:00.819200+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:58:00.819200+0000","last_undegraded":"2024-09-21T18:58:00.819200+0000","last_fullsized":"2024-09-21T18:58:00.819200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835180+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:58:00.835180+0000","last_peered":"2024-09-21T18:58:00.835180+0000","last_clean":"2024-09-21T18:58:00.835180+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:58:00.835180+0000","last_undegraded":"2024-09-21T18:58:00.835180+0000","last_fullsized":"2024-09-21T18:58:00.835180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":216,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830704+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:58:00.830704+0000","last_peered":"2024-09-21T18:58:00.830704+0000","last_clean":"2024-09-21T18:58:00.830704+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:58:00.830704+0000","last_undegraded":"2024-09-21T18:58:00.830704+0000","last_fullsized":"2024-09-21T18:58:00.830704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":275,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830536+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:58:00.830536+0000","last_peered":"2024-09-21T18:58:00.830536+0000","last_clean":"2024-09-21T18:58:00.830536+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:58:00.830536+0000","last_undegraded":"2024-09-21T18:58:00.830536+0000","last_fullsized":"2024-09-21T18:58:00.830536+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.818998+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:58:00.818998+0000","last_peered":"2024-09-21T18:58:00.818998+0000","last_clean":"2024-09-21T18:58:00.818998+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:58:00.818998+0000","last_undegraded":"2024-09-21T18:58:00.818998+0000","last_fullsized":"2024-09-21T18:58:00.818998+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830236+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:58:00.830236+0000","last_peered":"2024-09-21T18:58:00.830236+0000","last_clean":"2024-09-21T18:58:00.830236+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:58:00.830236+0000","last_undegraded":"2024-09-21T18:58:00.830236+0000","last_fullsized":"2024-09-21T18:58:00.830236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:22.885755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":268,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819546+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:58:00.819546+0000","last_peered":"2024-09-21T18:58:00.819546+0000","last_clean":"2024-09-21T18:58:00.819546+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:58:00.819546+0000","last_undegraded":"2024-09-21T18:58:00.819546+0000","last_fullsized":"2024-09-21T18:58:00.819546+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":228,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.829889+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:58:00.829889+0000","last_peered":"2024-09-21T18:58:00.829889+0000","last_clean":"2024-09-21T18:58:00.829889+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:58:00.829889+0000","last_undegraded":"2024-09-21T18:58:00.829889+0000","last_fullsized":"2024-09-21T18:58:00.829889+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.948408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":94,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836431+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:58:00.836431+0000","last_peered":"2024-09-21T18:58:00.836431+0000","last_clean":"2024-09-21T18:58:00.836431+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:58:00.836431+0000","last_undegraded":"2024-09-21T18:58:00.836431+0000","last_fullsized":"2024-09-21T18:58:00.836431+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830338+0000","last_change":"2024-09-21T18:57:19.821897+0000","last_active":"2024-09-21T18:58:00.830338+0000","last_peered":"2024-09-21T18:58:00.830338+0000","last_clean":"2024-09-21T18:58:00.830338+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:58:00.830338+0000","last_undegraded":"2024-09-21T18:58:00.830338+0000","last_fullsized":"2024-09-21T18:58:00.830338+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.821849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.129486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819452+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:58:00.819452+0000","last_peered":"2024-09-21T18:58:00.819452+0000","last_clean":"2024-09-21T18:58:00.819452+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:58:00.819452+0000","last_undegraded":"2024-09-21T18:58:00.819452+0000","last_fullsized":"2024-09-21T18:58:00.819452+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":213,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827491+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:58:00.827491+0000","last_peered":"2024-09-21T18:58:00.827491+0000","last_clean":"2024-09-21T18:58:00.827491+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:58:00.827491+0000","last_undegraded":"2024-09-21T18:58:00.827491+0000","last_fullsized":"2024-09-21T18:58:00.827491+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:16.165478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.829465+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:58:00.829465+0000","last_peered":"2024-09-21T18:58:00.829465+0000","last_clean":"2024-09-21T18:58:00.829465+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:58:00.829465+0000","last_undegraded":"2024-09-21T18:58:00.829465+0000","last_fullsized":"2024-09-21T18:58:00.829465+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.650657+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.820091+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:58:00.820091+0000","last_peered":"2024-09-21T18:58:00.820091+0000","last_clean":"2024-09-21T18:58:00.820091+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:58:00.820091+0000","last_undegraded":"2024-09-21T18:58:00.820091+0000","last_fullsized":"2024-09-21T18:58:00.820091+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":94,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837464+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:58:00.837464+0000","last_peered":"2024-09-21T18:58:00.837464+0000","last_clean":"2024-09-21T18:58:00.837464+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:58:00.837464+0000","last_undegraded":"2024-09-21T18:58:00.837464+0000","last_fullsized":"2024-09-21T18:58:00.837464+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":235,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.823359+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:58:00.823359+0000","last_peered":"2024-09-21T18:58:00.823359+0000","last_clean":"2024-09-21T18:58:00.823359+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:58:00.823359+0000","last_undegraded":"2024-09-21T18:58:00.823359+0000","last_fullsized":"2024-09-21T18:58:00.823359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.481531+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.843581+0000","last_change":"2024-09-21T18:57:20.229156+0000","last_active":"2024-09-21T18:58:00.843581+0000","last_peered":"2024-09-21T18:58:00.843581+0000","last_clean":"2024-09-21T18:58:00.843581+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:58:00.843581+0000","last_undegraded":"2024-09-21T18:58:00.843581+0000","last_fullsized":"2024-09-21T18:58:00.843581+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.229088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.229088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.406792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":196,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835365+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:58:00.835365+0000","last_peered":"2024-09-21T18:58:00.835365+0000","last_clean":"2024-09-21T18:58:00.835365+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:58:00.835365+0000","last_undegraded":"2024-09-21T18:58:00.835365+0000","last_fullsized":"2024-09-21T18:58:00.835365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819554+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:58:00.819554+0000","last_peered":"2024-09-21T18:58:00.819554+0000","last_clean":"2024-09-21T18:58:00.819554+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:58:00.819554+0000","last_undegraded":"2024-09-21T18:58:00.819554+0000","last_fullsized":"2024-09-21T18:58:00.819554+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":234,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830293+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:58:00.830293+0000","last_peered":"2024-09-21T18:58:00.830293+0000","last_clean":"2024-09-21T18:58:00.830293+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:58:00.830293+0000","last_undegraded":"2024-09-21T18:58:00.830293+0000","last_fullsized":"2024-09-21T18:58:00.830293+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819698+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:58:00.819698+0000","last_peered":"2024-09-21T18:58:00.819698+0000","last_clean":"2024-09-21T18:58:00.819698+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:58:00.819698+0000","last_undegraded":"2024-09-21T18:58:00.819698+0000","last_fullsized":"2024-09-21T18:58:00.819698+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831036+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:58:00.831036+0000","last_peered":"2024-09-21T18:58:00.831036+0000","last_clean":"2024-09-21T18:58:00.831036+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:58:00.831036+0000","last_undegraded":"2024-09-21T18:58:00.831036+0000","last_fullsized":"2024-09-21T18:58:00.831036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":216,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835755+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:58:00.835755+0000","last_peered":"2024-09-21T18:58:00.835755+0000","last_clean":"2024-09-21T18:58:00.835755+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:58:00.835755+0000","last_undegraded":"2024-09-21T18:58:00.835755+0000","last_fullsized":"2024-09-21T18:58:00.835755+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.299356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":107,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837780+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:58:00.837780+0000","last_peered":"2024-09-21T18:58:00.837780+0000","last_clean":"2024-09-21T18:58:00.837780+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:58:00.837780+0000","last_undegraded":"2024-09-21T18:58:00.837780+0000","last_fullsized":"2024-09-21T18:58:00.837780+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":276,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.833972+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:58:00.833972+0000","last_peered":"2024-09-21T18:58:00.833972+0000","last_clean":"2024-09-21T18:58:00.833972+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:58:00.833972+0000","last_undegraded":"2024-09-21T18:58:00.833972+0000","last_fullsized":"2024-09-21T18:58:00.833972+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"155'15581","reported_seq":15772,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.446766+0000","last_change":"2024-09-21T18:57:59.815796+0000","last_active":"2024-09-21T18:58:01.446766+0000","last_peered":"2024-09-21T18:58:01.446766+0000","last_clean":"2024-09-21T18:58:01.446766+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:58:01.446766+0000","last_undegraded":"2024-09-21T18:58:01.446766+0000","last_fullsized":"2024-09-21T18:58:01.446766+0000","mapping_epoch":79,"log_start":"119'9300","ondisk_log_start":"119'9300","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'15478","last_scrub_stamp":"2024-09-21T18:57:59.815673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.815673+0000","objects_scrubbed":1655,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.208391+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100728832,"num_objects":1537,"num_object_clones":0,"num_object_copies":4611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15581,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":236,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835976+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:58:00.835976+0000","last_peered":"2024-09-21T18:58:00.835976+0000","last_clean":"2024-09-21T18:58:00.835976+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:58:00.835976+0000","last_undegraded":"2024-09-21T18:58:00.835976+0000","last_fullsized":"2024-09-21T18:58:00.835976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.223966+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":216,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830869+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:58:00.830869+0000","last_peered":"2024-09-21T18:58:00.830869+0000","last_clean":"2024-09-21T18:58:00.830869+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:58:00.830869+0000","last_undegraded":"2024-09-21T18:58:00.830869+0000","last_fullsized":"2024-09-21T18:58:00.830869+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":296,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.820100+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:58:00.820100+0000","last_peered":"2024-09-21T18:58:00.820100+0000","last_clean":"2024-09-21T18:58:00.820100+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:58:00.820100+0000","last_undegraded":"2024-09-21T18:58:00.820100+0000","last_fullsized":"2024-09-21T18:58:00.820100+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":233,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830012+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T18:58:00.830012+0000","last_peered":"2024-09-21T18:58:00.830012+0000","last_clean":"2024-09-21T18:58:00.830012+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:58:00.830012+0000","last_undegraded":"2024-09-21T18:58:00.830012+0000","last_fullsized":"2024-09-21T18:58:00.830012+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.949065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"155'15666","reported_seq":15831,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:02.162652+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:58:02.162652+0000","last_peered":"2024-09-21T18:58:02.162652+0000","last_clean":"2024-09-21T18:58:02.162652+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:58:02.162652+0000","last_undegraded":"2024-09-21T18:58:02.162652+0000","last_fullsized":"2024-09-21T18:58:02.162652+0000","mapping_epoch":79,"log_start":"125'11000","ondisk_log_start":"125'11000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":1536,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15666,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":294,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830611+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:58:00.830611+0000","last_peered":"2024-09-21T18:58:00.830611+0000","last_clean":"2024-09-21T18:58:00.830611+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:58:00.830611+0000","last_undegraded":"2024-09-21T18:58:00.830611+0000","last_fullsized":"2024-09-21T18:58:00.830611+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835715+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:58:00.835715+0000","last_peered":"2024-09-21T18:58:00.835715+0000","last_clean":"2024-09-21T18:58:00.835715+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:58:00.835715+0000","last_undegraded":"2024-09-21T18:58:00.835715+0000","last_fullsized":"2024-09-21T18:58:00.835715+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819116+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:58:00.819116+0000","last_peered":"2024-09-21T18:58:00.819116+0000","last_clean":"2024-09-21T18:58:00.819116+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:58:00.819116+0000","last_undegraded":"2024-09-21T18:58:00.819116+0000","last_fullsized":"2024-09-21T18:58:00.819116+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":195,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819377+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:58:00.819377+0000","last_peered":"2024-09-21T18:58:00.819377+0000","last_clean":"2024-09-21T18:58:00.819377+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:58:00.819377+0000","last_undegraded":"2024-09-21T18:58:00.819377+0000","last_fullsized":"2024-09-21T18:58:00.819377+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"155'15928","reported_seq":16093,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:02.125540+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:58:02.125540+0000","last_peered":"2024-09-21T18:58:02.125540+0000","last_clean":"2024-09-21T18:58:02.125540+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:58:02.125540+0000","last_undegraded":"2024-09-21T18:58:02.125540+0000","last_fullsized":"2024-09-21T18:58:02.125540+0000","mapping_epoch":79,"log_start":"125'11300","ondisk_log_start":"125'11300","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":232,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819235+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:58:00.819235+0000","last_peered":"2024-09-21T18:58:00.819235+0000","last_clean":"2024-09-21T18:58:00.819235+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:58:00.819235+0000","last_undegraded":"2024-09-21T18:58:00.819235+0000","last_fullsized":"2024-09-21T18:58:00.819235+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":99,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.838158+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:58:00.838158+0000","last_peered":"2024-09-21T18:58:00.838158+0000","last_clean":"2024-09-21T18:58:00.838158+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:58:00.838158+0000","last_undegraded":"2024-09-21T18:58:00.838158+0000","last_fullsized":"2024-09-21T18:58:00.838158+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":267,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830361+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:58:00.830361+0000","last_peered":"2024-09-21T18:58:00.830361+0000","last_clean":"2024-09-21T18:58:00.830361+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:58:00.830361+0000","last_undegraded":"2024-09-21T18:58:00.830361+0000","last_fullsized":"2024-09-21T18:58:00.830361+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827745+0000","last_change":"2024-09-21T18:58:00.240553+0000","last_active":"2024-09-21T18:58:00.827745+0000","last_peered":"2024-09-21T18:58:00.827745+0000","last_clean":"2024-09-21T18:58:00.827745+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:58:00.827745+0000","last_undegraded":"2024-09-21T18:58:00.827745+0000","last_fullsized":"2024-09-21T18:58:00.827745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.240476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.240476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:25.073873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"155'15721","reported_seq":15977,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.461134+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:58:01.461134+0000","last_peered":"2024-09-21T18:58:01.461134+0000","last_clean":"2024-09-21T18:58:01.461134+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:58:01.461134+0000","last_undegraded":"2024-09-21T18:58:01.461134+0000","last_fullsized":"2024-09-21T18:58:01.461134+0000","mapping_epoch":79,"log_start":"125'10900","ondisk_log_start":"125'10900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100990976,"num_objects":1541,"num_object_clones":0,"num_object_copies":4623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15721,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":218,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836269+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:58:00.836269+0000","last_peered":"2024-09-21T18:58:00.836269+0000","last_clean":"2024-09-21T18:58:00.836269+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:58:00.836269+0000","last_undegraded":"2024-09-21T18:58:00.836269+0000","last_fullsized":"2024-09-21T18:58:00.836269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.091213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":223,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830341+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:58:00.830341+0000","last_peered":"2024-09-21T18:58:00.830341+0000","last_clean":"2024-09-21T18:58:00.830341+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:58:00.830341+0000","last_undegraded":"2024-09-21T18:58:00.830341+0000","last_fullsized":"2024-09-21T18:58:00.830341+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.895558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":125,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837356+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:58:00.837356+0000","last_peered":"2024-09-21T18:58:00.837356+0000","last_clean":"2024-09-21T18:58:00.837356+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:58:00.837356+0000","last_undegraded":"2024-09-21T18:58:00.837356+0000","last_fullsized":"2024-09-21T18:58:00.837356+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":95,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836533+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:58:00.836533+0000","last_peered":"2024-09-21T18:58:00.836533+0000","last_clean":"2024-09-21T18:58:00.836533+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:58:00.836533+0000","last_undegraded":"2024-09-21T18:58:00.836533+0000","last_fullsized":"2024-09-21T18:58:00.836533+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"155'16038","reported_seq":16353,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.465214+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:58:01.465214+0000","last_peered":"2024-09-21T18:58:01.465214+0000","last_clean":"2024-09-21T18:58:01.465214+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:58:01.465214+0000","last_undegraded":"2024-09-21T18:58:01.465214+0000","last_fullsized":"2024-09-21T18:58:01.465214+0000","mapping_epoch":79,"log_start":"122'10400","ondisk_log_start":"122'10400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:43.658704+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":113,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837559+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:58:00.837559+0000","last_peered":"2024-09-21T18:58:00.837559+0000","last_clean":"2024-09-21T18:58:00.837559+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:58:00.837559+0000","last_undegraded":"2024-09-21T18:58:00.837559+0000","last_fullsized":"2024-09-21T18:58:00.837559+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":215,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837558+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:58:00.837558+0000","last_peered":"2024-09-21T18:58:00.837558+0000","last_clean":"2024-09-21T18:58:00.837558+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:58:00.837558+0000","last_undegraded":"2024-09-21T18:58:00.837558+0000","last_fullsized":"2024-09-21T18:58:00.837558+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:47.380559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.833127+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:58:00.833127+0000","last_peered":"2024-09-21T18:58:00.833127+0000","last_clean":"2024-09-21T18:58:00.833127+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:58:00.833127+0000","last_undegraded":"2024-09-21T18:58:00.833127+0000","last_fullsized":"2024-09-21T18:58:00.833127+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835556+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:58:00.835556+0000","last_peered":"2024-09-21T18:58:00.835556+0000","last_clean":"2024-09-21T18:58:00.835556+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:58:00.835556+0000","last_undegraded":"2024-09-21T18:58:00.835556+0000","last_fullsized":"2024-09-21T18:58:00.835556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.637336+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"155'15785","reported_seq":23864,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.475060+0000","last_change":"2024-09-21T18:57:57.958393+0000","last_active":"2024-09-21T18:58:01.475060+0000","last_peered":"2024-09-21T18:58:01.475060+0000","last_clean":"2024-09-21T18:58:01.475060+0000","last_became_active":"2024-09-21T18:57:34.506455+0000","last_became_peered":"2024-09-21T18:57:34.506455+0000","last_unstale":"2024-09-21T18:58:01.475060+0000","last_undegraded":"2024-09-21T18:58:01.475060+0000","last_fullsized":"2024-09-21T18:58:01.475060+0000","mapping_epoch":143,"log_start":"119'9500","ondisk_log_start":"119'9500","created":79,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'15561","last_scrub_stamp":"2024-09-21T18:57:57.958164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:57.958164+0000","objects_scrubbed":1789,"log_size":6285,"log_dups_size":0,"ondisk_log_size":6285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:26.531439+0000","scrub_duration":626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15785,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3933,"num_bytes_recovered":257753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":318,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837484+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:58:00.837484+0000","last_peered":"2024-09-21T18:58:00.837484+0000","last_clean":"2024-09-21T18:58:00.837484+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:58:00.837484+0000","last_undegraded":"2024-09-21T18:58:00.837484+0000","last_fullsized":"2024-09-21T18:58:00.837484+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:41.364368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":237,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819750+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:58:00.819750+0000","last_peered":"2024-09-21T18:58:00.819750+0000","last_clean":"2024-09-21T18:58:00.819750+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:58:00.819750+0000","last_undegraded":"2024-09-21T18:58:00.819750+0000","last_fullsized":"2024-09-21T18:58:00.819750+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":233,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831162+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:58:00.831162+0000","last_peered":"2024-09-21T18:58:00.831162+0000","last_clean":"2024-09-21T18:58:00.831162+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:58:00.831162+0000","last_undegraded":"2024-09-21T18:58:00.831162+0000","last_fullsized":"2024-09-21T18:58:00.831162+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.854902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":95,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836984+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:58:00.836984+0000","last_peered":"2024-09-21T18:58:00.836984+0000","last_clean":"2024-09-21T18:58:00.836984+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:58:00.836984+0000","last_undegraded":"2024-09-21T18:58:00.836984+0000","last_fullsized":"2024-09-21T18:58:00.836984+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830377+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:58:00.830377+0000","last_peered":"2024-09-21T18:58:00.830377+0000","last_clean":"2024-09-21T18:58:00.830377+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:58:00.830377+0000","last_undegraded":"2024-09-21T18:58:00.830377+0000","last_fullsized":"2024-09-21T18:58:00.830377+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"155'15889","reported_seq":21582,"reported_epoch":155,"state":"active+clean+scrubbing","last_fresh":"2024-09-21T18:58:01.562536+0000","last_change":"2024-09-21T18:58:01.024563+0000","last_active":"2024-09-21T18:58:01.562536+0000","last_peered":"2024-09-21T18:58:01.562536+0000","last_clean":"2024-09-21T18:58:01.562536+0000","last_became_active":"2024-09-21T18:57:51.729319+0000","last_became_peered":"2024-09-21T18:57:51.729319+0000","last_unstale":"2024-09-21T18:58:01.562536+0000","last_undegraded":"2024-09-21T18:58:01.562536+0000","last_fullsized":"2024-09-21T18:58:01.562536+0000","mapping_epoch":150,"log_start":"123'10600","ondisk_log_start":"123'10600","created":79,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":1399,"log_size":5289,"log_dups_size":0,"ondisk_log_size":5289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":4863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15889,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":179961856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":182,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.823086+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:58:00.823086+0000","last_peered":"2024-09-21T18:58:00.823086+0000","last_clean":"2024-09-21T18:58:00.823086+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:58:00.823086+0000","last_undegraded":"2024-09-21T18:58:00.823086+0000","last_fullsized":"2024-09-21T18:58:00.823086+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:44.501396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837863+0000","last_change":"2024-09-21T18:57:22.224988+0000","last_active":"2024-09-21T18:58:00.837863+0000","last_peered":"2024-09-21T18:58:00.837863+0000","last_clean":"2024-09-21T18:58:00.837863+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:58:00.837863+0000","last_undegraded":"2024-09-21T18:58:00.837863+0000","last_fullsized":"2024-09-21T18:58:00.837863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:22.220182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:22.220182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.617863+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":254,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.822775+0000","last_change":"2024-09-21T18:57:49.456229+0000","last_active":"2024-09-21T18:58:00.822775+0000","last_peered":"2024-09-21T18:58:00.822775+0000","last_clean":"2024-09-21T18:58:00.822775+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:58:00.822775+0000","last_undegraded":"2024-09-21T18:58:00.822775+0000","last_fullsized":"2024-09-21T18:58:00.822775+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:49.456147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.456147+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:50.500491+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831241+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:58:00.831241+0000","last_peered":"2024-09-21T18:58:00.831241+0000","last_clean":"2024-09-21T18:58:00.831241+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:58:00.831241+0000","last_undegraded":"2024-09-21T18:58:00.831241+0000","last_fullsized":"2024-09-21T18:58:00.831241+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.828220+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:58:00.828220+0000","last_peered":"2024-09-21T18:58:00.828220+0000","last_clean":"2024-09-21T18:58:00.828220+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:58:00.828220+0000","last_undegraded":"2024-09-21T18:58:00.828220+0000","last_fullsized":"2024-09-21T18:58:00.828220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"155'16184","reported_seq":30492,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.459805+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:58:01.459805+0000","last_peered":"2024-09-21T18:58:01.459805+0000","last_clean":"2024-09-21T18:58:01.459805+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:58:01.459805+0000","last_undegraded":"2024-09-21T18:58:01.459805+0000","last_fullsized":"2024-09-21T18:58:01.459805+0000","mapping_epoch":130,"log_start":"124'11000","ondisk_log_start":"124'11000","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:07.065509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16184,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":109,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837945+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:58:00.837945+0000","last_peered":"2024-09-21T18:58:00.837945+0000","last_clean":"2024-09-21T18:58:00.837945+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:58:00.837945+0000","last_undegraded":"2024-09-21T18:58:00.837945+0000","last_fullsized":"2024-09-21T18:58:00.837945+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":95,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836934+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:58:00.836934+0000","last_peered":"2024-09-21T18:58:00.836934+0000","last_clean":"2024-09-21T18:58:00.836934+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:58:00.836934+0000","last_undegraded":"2024-09-21T18:58:00.836934+0000","last_fullsized":"2024-09-21T18:58:00.836934+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":543,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827627+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:58:00.827627+0000","last_peered":"2024-09-21T18:58:00.827627+0000","last_clean":"2024-09-21T18:58:00.827627+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:58:00.827627+0000","last_undegraded":"2024-09-21T18:58:00.827627+0000","last_fullsized":"2024-09-21T18:58:00.827627+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:17.496871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":283,"num_read_kb":278,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":237,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831005+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:58:00.831005+0000","last_peered":"2024-09-21T18:58:00.831005+0000","last_clean":"2024-09-21T18:58:00.831005+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:58:00.831005+0000","last_undegraded":"2024-09-21T18:58:00.831005+0000","last_fullsized":"2024-09-21T18:58:00.831005+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.378290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837268+0000","last_change":"2024-09-21T18:57:18.176297+0000","last_active":"2024-09-21T18:58:00.837268+0000","last_peered":"2024-09-21T18:58:00.837268+0000","last_clean":"2024-09-21T18:58:00.837268+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:58:00.837268+0000","last_undegraded":"2024-09-21T18:58:00.837268+0000","last_fullsized":"2024-09-21T18:58:00.837268+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:18.176233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:18.176233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:24.204170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"155'15748","reported_seq":27793,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.563835+0000","last_change":"2024-09-21T18:57:20.837816+0000","last_active":"2024-09-21T18:58:01.563835+0000","last_peered":"2024-09-21T18:58:01.563835+0000","last_clean":"2024-09-21T18:58:01.563835+0000","last_became_active":"2024-09-21T18:57:20.750225+0000","last_became_peered":"2024-09-21T18:57:20.750225+0000","last_unstale":"2024-09-21T18:58:01.563835+0000","last_undegraded":"2024-09-21T18:58:01.563835+0000","last_fullsized":"2024-09-21T18:58:01.563835+0000","mapping_epoch":139,"log_start":"121'10102","ondisk_log_start":"121'10102","created":79,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":291,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.833907+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:58:00.833907+0000","last_peered":"2024-09-21T18:58:00.833907+0000","last_clean":"2024-09-21T18:58:00.833907+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:58:00.833907+0000","last_undegraded":"2024-09-21T18:58:00.833907+0000","last_fullsized":"2024-09-21T18:58:00.833907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":216,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831647+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:58:00.831647+0000","last_peered":"2024-09-21T18:58:00.831647+0000","last_clean":"2024-09-21T18:58:00.831647+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:58:00.831647+0000","last_undegraded":"2024-09-21T18:58:00.831647+0000","last_fullsized":"2024-09-21T18:58:00.831647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":277,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827497+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:58:00.827497+0000","last_peered":"2024-09-21T18:58:00.827497+0000","last_clean":"2024-09-21T18:58:00.827497+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:58:00.827497+0000","last_undegraded":"2024-09-21T18:58:00.827497+0000","last_fullsized":"2024-09-21T18:58:00.827497+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:05.121733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831135+0000","last_change":"2024-09-21T18:53:11.414589+0000","last_active":"2024-09-21T18:58:00.831135+0000","last_peered":"2024-09-21T18:58:00.831135+0000","last_clean":"2024-09-21T18:58:00.831135+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:58:00.831135+0000","last_undegraded":"2024-09-21T18:58:00.831135+0000","last_fullsized":"2024-09-21T18:58:00.831135+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:01.123772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830892+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:58:00.830892+0000","last_peered":"2024-09-21T18:58:00.830892+0000","last_clean":"2024-09-21T18:58:00.830892+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:58:00.830892+0000","last_undegraded":"2024-09-21T18:58:00.830892+0000","last_fullsized":"2024-09-21T18:58:00.830892+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:28.894465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"155'15929","reported_seq":21548,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.409413+0000","last_change":"2024-09-21T18:57:52.759473+0000","last_active":"2024-09-21T18:58:01.409413+0000","last_peered":"2024-09-21T18:58:01.409413+0000","last_clean":"2024-09-21T18:58:01.409413+0000","last_became_active":"2024-09-21T18:57:52.731182+0000","last_became_peered":"2024-09-21T18:57:52.731182+0000","last_unstale":"2024-09-21T18:58:01.409413+0000","last_undegraded":"2024-09-21T18:58:01.409413+0000","last_fullsized":"2024-09-21T18:58:01.409413+0000","mapping_epoch":151,"log_start":"119'9600","ondisk_log_start":"119'9600","created":79,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110755864,"num_objects":1691,"num_object_clones":0,"num_object_copies":5073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15929,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2670,"num_bytes_recovered":174915608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.820183+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:58:00.820183+0000","last_peered":"2024-09-21T18:58:00.820183+0000","last_clean":"2024-09-21T18:58:00.820183+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:58:00.820183+0000","last_undegraded":"2024-09-21T18:58:00.820183+0000","last_fullsized":"2024-09-21T18:58:00.820183+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819404+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:58:00.819404+0000","last_peered":"2024-09-21T18:58:00.819404+0000","last_clean":"2024-09-21T18:58:00.819404+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:58:00.819404+0000","last_undegraded":"2024-09-21T18:58:00.819404+0000","last_fullsized":"2024-09-21T18:58:00.819404+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":223,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830798+0000","last_change":"2024-09-21T18:57:58.653399+0000","last_active":"2024-09-21T18:58:00.830798+0000","last_peered":"2024-09-21T18:58:00.830798+0000","last_clean":"2024-09-21T18:58:00.830798+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:58:00.830798+0000","last_undegraded":"2024-09-21T18:58:00.830798+0000","last_fullsized":"2024-09-21T18:58:00.830798+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.653339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.653339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:20.676781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":196,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831103+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:58:00.831103+0000","last_peered":"2024-09-21T18:58:00.831103+0000","last_clean":"2024-09-21T18:58:00.831103+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:58:00.831103+0000","last_undegraded":"2024-09-21T18:58:00.831103+0000","last_fullsized":"2024-09-21T18:58:00.831103+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"155'15628","reported_seq":24117,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.573981+0000","last_change":"2024-09-21T18:57:35.557746+0000","last_active":"2024-09-21T18:58:01.573981+0000","last_peered":"2024-09-21T18:58:01.573981+0000","last_clean":"2024-09-21T18:58:01.573981+0000","last_became_active":"2024-09-21T18:57:35.521875+0000","last_became_peered":"2024-09-21T18:57:35.521875+0000","last_unstale":"2024-09-21T18:58:01.573981+0000","last_undegraded":"2024-09-21T18:58:01.573981+0000","last_fullsized":"2024-09-21T18:58:01.573981+0000","mapping_epoch":144,"log_start":"117'8900","ondisk_log_start":"117'8900","created":79,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6728,"log_dups_size":0,"ondisk_log_size":6728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98828288,"num_objects":1508,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15628,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4110,"num_bytes_recovered":269352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":239,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819993+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:58:00.819993+0000","last_peered":"2024-09-21T18:58:00.819993+0000","last_clean":"2024-09-21T18:58:00.819993+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:58:00.819993+0000","last_undegraded":"2024-09-21T18:58:00.819993+0000","last_fullsized":"2024-09-21T18:58:00.819993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.833327+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:58:00.833327+0000","last_peered":"2024-09-21T18:58:00.833327+0000","last_clean":"2024-09-21T18:58:00.833327+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:58:00.833327+0000","last_undegraded":"2024-09-21T18:58:00.833327+0000","last_fullsized":"2024-09-21T18:58:00.833327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":213,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837331+0000","last_change":"2024-09-21T18:55:51.535384+0000","last_active":"2024-09-21T18:58:00.837331+0000","last_peered":"2024-09-21T18:58:00.837331+0000","last_clean":"2024-09-21T18:58:00.837331+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:58:00.837331+0000","last_undegraded":"2024-09-21T18:58:00.837331+0000","last_fullsized":"2024-09-21T18:58:00.837331+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.042616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.823247+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:58:00.823247+0000","last_peered":"2024-09-21T18:58:00.823247+0000","last_clean":"2024-09-21T18:58:00.823247+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:58:00.823247+0000","last_undegraded":"2024-09-21T18:58:00.823247+0000","last_fullsized":"2024-09-21T18:58:00.823247+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.540935+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"155'15727","reported_seq":21974,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.553290+0000","last_change":"2024-09-21T18:57:52.776315+0000","last_active":"2024-09-21T18:58:01.553290+0000","last_peered":"2024-09-21T18:58:01.553290+0000","last_clean":"2024-09-21T18:58:01.553290+0000","last_became_active":"2024-09-21T18:57:52.730444+0000","last_became_peered":"2024-09-21T18:57:52.730444+0000","last_unstale":"2024-09-21T18:58:01.553290+0000","last_undegraded":"2024-09-21T18:58:01.553290+0000","last_fullsized":"2024-09-21T18:58:01.553290+0000","mapping_epoch":151,"log_start":"123'10500","ondisk_log_start":"123'10500","created":79,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105578496,"num_objects":1611,"num_object_clones":0,"num_object_copies":4833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15727,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3036,"num_bytes_recovered":198967296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":226,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819751+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:58:00.819751+0000","last_peered":"2024-09-21T18:58:00.819751+0000","last_clean":"2024-09-21T18:58:00.819751+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:58:00.819751+0000","last_undegraded":"2024-09-21T18:58:00.819751+0000","last_fullsized":"2024-09-21T18:58:00.819751+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":244,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835825+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:58:00.835825+0000","last_peered":"2024-09-21T18:58:00.835825+0000","last_clean":"2024-09-21T18:58:00.835825+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:58:00.835825+0000","last_undegraded":"2024-09-21T18:58:00.835825+0000","last_fullsized":"2024-09-21T18:58:00.835825+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":274,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.823500+0000","last_change":"2024-09-21T18:55:22.245681+0000","last_active":"2024-09-21T18:58:00.823500+0000","last_peered":"2024-09-21T18:58:00.823500+0000","last_clean":"2024-09-21T18:58:00.823500+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:58:00.823500+0000","last_undegraded":"2024-09-21T18:58:00.823500+0000","last_fullsized":"2024-09-21T18:58:00.823500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:22.245585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.245585+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819997+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:58:00.819997+0000","last_peered":"2024-09-21T18:58:00.819997+0000","last_clean":"2024-09-21T18:58:00.819997+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:58:00.819997+0000","last_undegraded":"2024-09-21T18:58:00.819997+0000","last_fullsized":"2024-09-21T18:58:00.819997+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"155'15887","reported_seq":16162,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.546062+0000","last_change":"2024-09-21T18:57:51.070236+0000","last_active":"2024-09-21T18:58:01.546062+0000","last_peered":"2024-09-21T18:58:01.546062+0000","last_clean":"2024-09-21T18:58:01.546062+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:58:01.546062+0000","last_undegraded":"2024-09-21T18:58:01.546062+0000","last_fullsized":"2024-09-21T18:58:01.546062+0000","mapping_epoch":79,"log_start":"118'9200","ondisk_log_start":"118'9200","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15303","last_scrub_stamp":"2024-09-21T18:57:51.070161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.070161+0000","objects_scrubbed":2162,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:57.001294+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":104,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.837659+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:58:00.837659+0000","last_peered":"2024-09-21T18:58:00.837659+0000","last_clean":"2024-09-21T18:58:00.837659+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:58:00.837659+0000","last_undegraded":"2024-09-21T18:58:00.837659+0000","last_fullsized":"2024-09-21T18:58:00.837659+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":223,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830453+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:58:00.830453+0000","last_peered":"2024-09-21T18:58:00.830453+0000","last_clean":"2024-09-21T18:58:00.830453+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:58:00.830453+0000","last_undegraded":"2024-09-21T18:58:00.830453+0000","last_fullsized":"2024-09-21T18:58:00.830453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:04.889938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.822818+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:58:00.822818+0000","last_peered":"2024-09-21T18:58:00.822818+0000","last_clean":"2024-09-21T18:58:00.822818+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:58:00.822818+0000","last_undegraded":"2024-09-21T18:58:00.822818+0000","last_fullsized":"2024-09-21T18:58:00.822818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.513174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":103,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836873+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T18:58:00.836873+0000","last_peered":"2024-09-21T18:58:00.836873+0000","last_clean":"2024-09-21T18:58:00.836873+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:58:00.836873+0000","last_undegraded":"2024-09-21T18:58:00.836873+0000","last_fullsized":"2024-09-21T18:58:00.836873+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.098728+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"155'15859","reported_seq":29005,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.528206+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:58:01.528206+0000","last_peered":"2024-09-21T18:58:01.528206+0000","last_clean":"2024-09-21T18:58:01.528206+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:58:01.528206+0000","last_undegraded":"2024-09-21T18:58:01.528206+0000","last_fullsized":"2024-09-21T18:58:01.528206+0000","mapping_epoch":134,"log_start":"122'10300","ondisk_log_start":"122'10300","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":4719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15859,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":241,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.833229+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:58:00.833229+0000","last_peered":"2024-09-21T18:58:00.833229+0000","last_clean":"2024-09-21T18:58:00.833229+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:58:00.833229+0000","last_undegraded":"2024-09-21T18:58:00.833229+0000","last_fullsized":"2024-09-21T18:58:00.833229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.828583+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:58:00.828583+0000","last_peered":"2024-09-21T18:58:00.828583+0000","last_clean":"2024-09-21T18:58:00.828583+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:58:00.828583+0000","last_undegraded":"2024-09-21T18:58:00.828583+0000","last_fullsized":"2024-09-21T18:58:00.828583+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.169542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.824145+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:58:00.824145+0000","last_peered":"2024-09-21T18:58:00.824145+0000","last_clean":"2024-09-21T18:58:00.824145+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:58:00.824145+0000","last_undegraded":"2024-09-21T18:58:00.824145+0000","last_fullsized":"2024-09-21T18:58:00.824145+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827400+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T18:58:00.827400+0000","last_peered":"2024-09-21T18:58:00.827400+0000","last_clean":"2024-09-21T18:58:00.827400+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:58:00.827400+0000","last_undegraded":"2024-09-21T18:58:00.827400+0000","last_fullsized":"2024-09-21T18:58:00.827400+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.190036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"155'15585","reported_seq":15750,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:02.161858+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:58:02.161858+0000","last_peered":"2024-09-21T18:58:02.161858+0000","last_clean":"2024-09-21T18:58:02.161858+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:58:02.161858+0000","last_undegraded":"2024-09-21T18:58:02.161858+0000","last_fullsized":"2024-09-21T18:58:02.161858+0000","mapping_epoch":79,"log_start":"125'10900","ondisk_log_start":"125'10900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101908480,"num_objects":1555,"num_object_clones":0,"num_object_copies":4665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15585,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":253,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819306+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:58:00.819306+0000","last_peered":"2024-09-21T18:58:00.819306+0000","last_clean":"2024-09-21T18:58:00.819306+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:58:00.819306+0000","last_undegraded":"2024-09-21T18:58:00.819306+0000","last_fullsized":"2024-09-21T18:58:00.819306+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.828611+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:58:00.828611+0000","last_peered":"2024-09-21T18:58:00.828611+0000","last_clean":"2024-09-21T18:58:00.828611+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:58:00.828611+0000","last_undegraded":"2024-09-21T18:58:00.828611+0000","last_fullsized":"2024-09-21T18:58:00.828611+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:06.074325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":229,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.822926+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:58:00.822926+0000","last_peered":"2024-09-21T18:58:00.822926+0000","last_clean":"2024-09-21T18:58:00.822926+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:58:00.822926+0000","last_undegraded":"2024-09-21T18:58:00.822926+0000","last_fullsized":"2024-09-21T18:58:00.822926+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":196,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835340+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:58:00.835340+0000","last_peered":"2024-09-21T18:58:00.835340+0000","last_clean":"2024-09-21T18:58:00.835340+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:58:00.835340+0000","last_undegraded":"2024-09-21T18:58:00.835340+0000","last_fullsized":"2024-09-21T18:58:00.835340+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"155'15540","reported_seq":15705,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.736934+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:58:01.736934+0000","last_peered":"2024-09-21T18:58:01.736934+0000","last_clean":"2024-09-21T18:58:01.736934+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:58:01.736934+0000","last_undegraded":"2024-09-21T18:58:01.736934+0000","last_fullsized":"2024-09-21T18:58:01.736934+0000","mapping_epoch":79,"log_start":"123'10300","ondisk_log_start":"123'10300","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:29.184091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103415808,"num_objects":1578,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15540,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":252,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.829506+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T18:58:00.829506+0000","last_peered":"2024-09-21T18:58:00.829506+0000","last_clean":"2024-09-21T18:58:00.829506+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:58:00.829506+0000","last_undegraded":"2024-09-21T18:58:00.829506+0000","last_fullsized":"2024-09-21T18:58:00.829506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.873039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":252,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.828922+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T18:58:00.828922+0000","last_peered":"2024-09-21T18:58:00.828922+0000","last_clean":"2024-09-21T18:58:00.828922+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:58:00.828922+0000","last_undegraded":"2024-09-21T18:58:00.828922+0000","last_fullsized":"2024-09-21T18:58:00.828922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:09.842647+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":284,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827006+0000","last_change":"2024-09-21T18:57:23.116600+0000","last_active":"2024-09-21T18:58:00.827006+0000","last_peered":"2024-09-21T18:58:00.827006+0000","last_clean":"2024-09-21T18:58:00.827006+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:58:00.827006+0000","last_undegraded":"2024-09-21T18:58:00.827006+0000","last_fullsized":"2024-09-21T18:58:00.827006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:57:23.116520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:23.116520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.889890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":217,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.829665+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:58:00.829665+0000","last_peered":"2024-09-21T18:58:00.829665+0000","last_clean":"2024-09-21T18:58:00.829665+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:58:00.829665+0000","last_undegraded":"2024-09-21T18:58:00.829665+0000","last_fullsized":"2024-09-21T18:58:00.829665+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:26.930637+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":228,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.832622+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:58:00.832622+0000","last_peered":"2024-09-21T18:58:00.832622+0000","last_clean":"2024-09-21T18:58:00.832622+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:58:00.832622+0000","last_undegraded":"2024-09-21T18:58:00.832622+0000","last_fullsized":"2024-09-21T18:58:00.832622+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":213,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835624+0000","last_change":"2024-09-21T18:53:09.717993+0000","last_active":"2024-09-21T18:58:00.835624+0000","last_peered":"2024-09-21T18:58:00.835624+0000","last_clean":"2024-09-21T18:58:00.835624+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:58:00.835624+0000","last_undegraded":"2024-09-21T18:58:00.835624+0000","last_fullsized":"2024-09-21T18:58:00.835624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830725+0000","last_change":"2024-09-21T18:56:40.848060+0000","last_active":"2024-09-21T18:58:00.830725+0000","last_peered":"2024-09-21T18:58:00.830725+0000","last_clean":"2024-09-21T18:58:00.830725+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:58:00.830725+0000","last_undegraded":"2024-09-21T18:58:00.830725+0000","last_fullsized":"2024-09-21T18:58:00.830725+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:40.847976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.847976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":194,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.828159+0000","last_change":"2024-09-21T18:53:13.657369+0000","last_active":"2024-09-21T18:58:00.828159+0000","last_peered":"2024-09-21T18:58:00.828159+0000","last_clean":"2024-09-21T18:58:00.828159+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:58:00.828159+0000","last_undegraded":"2024-09-21T18:58:00.828159+0000","last_fullsized":"2024-09-21T18:58:00.828159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":104,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.838014+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:58:00.838014+0000","last_peered":"2024-09-21T18:58:00.838014+0000","last_clean":"2024-09-21T18:58:00.838014+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:58:00.838014+0000","last_undegraded":"2024-09-21T18:58:00.838014+0000","last_fullsized":"2024-09-21T18:58:00.838014+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":214,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827290+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:58:00.827290+0000","last_peered":"2024-09-21T18:58:00.827290+0000","last_clean":"2024-09-21T18:58:00.827290+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:58:00.827290+0000","last_undegraded":"2024-09-21T18:58:00.827290+0000","last_fullsized":"2024-09-21T18:58:00.827290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:08.070910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835489+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:58:00.835489+0000","last_peered":"2024-09-21T18:58:00.835489+0000","last_clean":"2024-09-21T18:58:00.835489+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:58:00.835489+0000","last_undegraded":"2024-09-21T18:58:00.835489+0000","last_fullsized":"2024-09-21T18:58:00.835489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.823013+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:58:00.823013+0000","last_peered":"2024-09-21T18:58:00.823013+0000","last_clean":"2024-09-21T18:58:00.823013+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:58:00.823013+0000","last_undegraded":"2024-09-21T18:58:00.823013+0000","last_fullsized":"2024-09-21T18:58:00.823013+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:37.565916+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":227,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830235+0000","last_change":"2024-09-21T18:57:25.890791+0000","last_active":"2024-09-21T18:58:00.830235+0000","last_peered":"2024-09-21T18:58:00.830235+0000","last_clean":"2024-09-21T18:58:00.830235+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:58:00.830235+0000","last_undegraded":"2024-09-21T18:58:00.830235+0000","last_fullsized":"2024-09-21T18:58:00.830235+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:25.890710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.890710+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:51.251415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":102,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836897+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T18:58:00.836897+0000","last_peered":"2024-09-21T18:58:00.836897+0000","last_clean":"2024-09-21T18:58:00.836897+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:58:00.836897+0000","last_undegraded":"2024-09-21T18:58:00.836897+0000","last_fullsized":"2024-09-21T18:58:00.836897+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.445017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":275,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.831205+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:58:00.831205+0000","last_peered":"2024-09-21T18:58:00.831205+0000","last_clean":"2024-09-21T18:58:00.831205+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:58:00.831205+0000","last_undegraded":"2024-09-21T18:58:00.831205+0000","last_fullsized":"2024-09-21T18:58:00.831205+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.820150+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:58:00.820150+0000","last_peered":"2024-09-21T18:58:00.820150+0000","last_clean":"2024-09-21T18:58:00.820150+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:58:00.820150+0000","last_undegraded":"2024-09-21T18:58:00.820150+0000","last_fullsized":"2024-09-21T18:58:00.820150+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":219,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.832119+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:58:00.832119+0000","last_peered":"2024-09-21T18:58:00.832119+0000","last_clean":"2024-09-21T18:58:00.832119+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:58:00.832119+0000","last_undegraded":"2024-09-21T18:58:00.832119+0000","last_fullsized":"2024-09-21T18:58:00.832119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.827914+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:58:00.827914+0000","last_peered":"2024-09-21T18:58:00.827914+0000","last_clean":"2024-09-21T18:58:00.827914+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:58:00.827914+0000","last_undegraded":"2024-09-21T18:58:00.827914+0000","last_fullsized":"2024-09-21T18:58:00.827914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.429722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":296,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830178+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T18:58:00.830178+0000","last_peered":"2024-09-21T18:58:00.830178+0000","last_clean":"2024-09-21T18:58:00.830178+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:58:00.830178+0000","last_undegraded":"2024-09-21T18:58:00.830178+0000","last_fullsized":"2024-09-21T18:58:00.830178+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:13.388257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":94,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.838399+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:58:00.838399+0000","last_peered":"2024-09-21T18:58:00.838399+0000","last_clean":"2024-09-21T18:58:00.838399+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:58:00.838399+0000","last_undegraded":"2024-09-21T18:58:00.838399+0000","last_fullsized":"2024-09-21T18:58:00.838399+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":249,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819495+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:58:00.819495+0000","last_peered":"2024-09-21T18:58:00.819495+0000","last_clean":"2024-09-21T18:58:00.819495+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:58:00.819495+0000","last_undegraded":"2024-09-21T18:58:00.819495+0000","last_fullsized":"2024-09-21T18:58:00.819495+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":215,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.819852+0000","last_change":"2024-09-21T18:57:57.837867+0000","last_active":"2024-09-21T18:58:00.819852+0000","last_peered":"2024-09-21T18:58:00.819852+0000","last_clean":"2024-09-21T18:58:00.819852+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:58:00.819852+0000","last_undegraded":"2024-09-21T18:58:00.819852+0000","last_fullsized":"2024-09-21T18:58:00.819852+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:57.837826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:57.837826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:58.681714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836180+0000","last_change":"2024-09-21T18:57:25.184491+0000","last_active":"2024-09-21T18:58:00.836180+0000","last_peered":"2024-09-21T18:58:00.836180+0000","last_clean":"2024-09-21T18:58:00.836180+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:58:00.836180+0000","last_undegraded":"2024-09-21T18:58:00.836180+0000","last_fullsized":"2024-09-21T18:58:00.836180+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:25.180690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.180690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:31.159574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":218,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830024+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T18:58:00.830024+0000","last_peered":"2024-09-21T18:58:00.830024+0000","last_clean":"2024-09-21T18:58:00.830024+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:58:00.830024+0000","last_undegraded":"2024-09-21T18:58:00.830024+0000","last_fullsized":"2024-09-21T18:58:00.830024+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.201536+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":235,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836339+0000","last_change":"2024-09-21T18:57:49.327777+0000","last_active":"2024-09-21T18:58:00.836339+0000","last_peered":"2024-09-21T18:58:00.836339+0000","last_clean":"2024-09-21T18:58:00.836339+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:58:00.836339+0000","last_undegraded":"2024-09-21T18:58:00.836339+0000","last_fullsized":"2024-09-21T18:58:00.836339+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:49.327123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.327123+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:56.940918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.828439+0000","last_change":"2024-09-21T18:57:19.082962+0000","last_active":"2024-09-21T18:58:00.828439+0000","last_peered":"2024-09-21T18:58:00.828439+0000","last_clean":"2024-09-21T18:58:00.828439+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:58:00.828439+0000","last_undegraded":"2024-09-21T18:58:00.828439+0000","last_fullsized":"2024-09-21T18:58:00.828439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.082873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.082873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.905947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":222,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835424+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:58:00.835424+0000","last_peered":"2024-09-21T18:58:00.835424+0000","last_clean":"2024-09-21T18:58:00.835424+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:58:00.835424+0000","last_undegraded":"2024-09-21T18:58:00.835424+0000","last_fullsized":"2024-09-21T18:58:00.835424+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":102,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.836567+0000","last_change":"2024-09-21T18:57:58.019346+0000","last_active":"2024-09-21T18:58:00.836567+0000","last_peered":"2024-09-21T18:58:00.836567+0000","last_clean":"2024-09-21T18:58:00.836567+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:58:00.836567+0000","last_undegraded":"2024-09-21T18:58:00.836567+0000","last_fullsized":"2024-09-21T18:58:00.836567+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.019278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.019278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.581215+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":231,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:01.818014+0000","last_change":"2024-09-21T18:58:01.818014+0000","last_active":"2024-09-21T18:58:01.818014+0000","last_peered":"2024-09-21T18:58:01.818014+0000","last_clean":"2024-09-21T18:58:01.818014+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:58:01.818014+0000","last_undegraded":"2024-09-21T18:58:01.818014+0000","last_fullsized":"2024-09-21T18:58:01.818014+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:58:01.817792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.817792+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:30.058894+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":215,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830848+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:58:00.830848+0000","last_peered":"2024-09-21T18:58:00.830848+0000","last_clean":"2024-09-21T18:58:00.830848+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:58:00.830848+0000","last_undegraded":"2024-09-21T18:58:00.830848+0000","last_fullsized":"2024-09-21T18:58:00.830848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:03.912165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.826972+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:58:00.826972+0000","last_peered":"2024-09-21T18:58:00.826972+0000","last_clean":"2024-09-21T18:58:00.826972+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:58:00.826972+0000","last_undegraded":"2024-09-21T18:58:00.826972+0000","last_fullsized":"2024-09-21T18:58:00.826972+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.212178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":218,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.835726+0000","last_change":"2024-09-21T18:57:54.750968+0000","last_active":"2024-09-21T18:58:00.835726+0000","last_peered":"2024-09-21T18:58:00.835726+0000","last_clean":"2024-09-21T18:58:00.835726+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:58:00.835726+0000","last_undegraded":"2024-09-21T18:58:00.835726+0000","last_fullsized":"2024-09-21T18:58:00.835726+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:54.750896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.750896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.812996+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":223,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830522+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:58:00.830522+0000","last_peered":"2024-09-21T18:58:00.830522+0000","last_clean":"2024-09-21T18:58:00.830522+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:58:00.830522+0000","last_undegraded":"2024-09-21T18:58:00.830522+0000","last_fullsized":"2024-09-21T18:58:00.830522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":102,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.839091+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T18:58:00.839091+0000","last_peered":"2024-09-21T18:58:00.839091+0000","last_clean":"2024-09-21T18:58:00.839091+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:58:00.839091+0000","last_undegraded":"2024-09-21T18:58:00.839091+0000","last_fullsized":"2024-09-21T18:58:00.839091+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.894402+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.833000+0000","last_change":"2024-09-21T18:57:55.356422+0000","last_active":"2024-09-21T18:58:00.833000+0000","last_peered":"2024-09-21T18:58:00.833000+0000","last_clean":"2024-09-21T18:58:00.833000+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:58:00.833000+0000","last_undegraded":"2024-09-21T18:58:00.833000+0000","last_fullsized":"2024-09-21T18:58:00.833000+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:55.356360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:55.356360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:04.334971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.826734+0000","last_change":"2024-09-21T18:57:53.263833+0000","last_active":"2024-09-21T18:58:00.826734+0000","last_peered":"2024-09-21T18:58:00.826734+0000","last_clean":"2024-09-21T18:58:00.826734+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:58:00.826734+0000","last_undegraded":"2024-09-21T18:58:00.826734+0000","last_fullsized":"2024-09-21T18:58:00.826734+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.263783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.263783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.950038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.843143+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:58:00.843143+0000","last_peered":"2024-09-21T18:58:00.843143+0000","last_clean":"2024-09-21T18:58:00.843143+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:58:00.843143+0000","last_undegraded":"2024-09-21T18:58:00.843143+0000","last_fullsized":"2024-09-21T18:58:00.843143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:48.365341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.830800+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:58:00.830800+0000","last_peered":"2024-09-21T18:58:00.830800+0000","last_clean":"2024-09-21T18:58:00.830800+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:58:00.830800+0000","last_undegraded":"2024-09-21T18:58:00.830800+0000","last_fullsized":"2024-09-21T18:58:00.830800+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":229,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.834062+0000","last_change":"2024-09-21T18:57:54.359262+0000","last_active":"2024-09-21T18:58:00.834062+0000","last_peered":"2024-09-21T18:58:00.834062+0000","last_clean":"2024-09-21T18:58:00.834062+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:58:00.834062+0000","last_undegraded":"2024-09-21T18:58:00.834062+0000","last_fullsized":"2024-09-21T18:58:00.834062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:57:54.359175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.359175+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.117172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":103,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T18:58:00.838322+0000","last_change":"2024-09-21T18:57:58.985012+0000","last_active":"2024-09-21T18:58:00.838322+0000","last_peered":"2024-09-21T18:58:00.838322+0000","last_clean":"2024-09-21T18:58:00.838322+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:58:00.838322+0000","last_undegraded":"2024-09-21T18:58:00.838322+0000","last_fullsized":"2024-09-21T18:58:00.838322+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.984945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.984945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:03.530436+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1657929752,"num_objects":25299,"num_object_clones":0,"num_object_copies":75897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252695,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35974,"num_bytes_recovered":2357461016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5342441472,"data_stored":5342429256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88893,"ondisk_log_size":88893,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":283,"num_read_kb":278,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829412,"kb_used_data":707984,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92915740,"statfs":{"total":95995035648,"available":95145717760,"internally_reserved":0,"allocated":724975616,"data_stored":724509466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528388,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640220,"kb_used_data":510468,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93104932,"statfs":{"total":95995035648,"available":95339450368,"internally_reserved":0,"allocated":522719232,"data_stored":522264745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710344,"kb_used_data":593076,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93034808,"statfs":{"total":95995035648,"available":95267643392,"internally_reserved":0,"allocated":607309824,"data_stored":606864319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018957,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648104,"kb_used_data":518356,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93097048,"statfs":{"total":95995035648,"available":95331377152,"internally_reserved":0,"allocated":530796544,"data_stored":530326254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163380,"kb_used_data":901728,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92581772,"statfs":{"total":95995035648,"available":94803734528,"internally_reserved":0,"allocated":923369472,"data_stored":922887193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":267909578},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758348,"kb_used_data":628476,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92986804,"statfs":{"total":95995035648,"available":95218487296,"internally_reserved":0,"allocated":643559424,"data_stored":643106282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":806580,"kb_used_data":622688,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92938572,"statfs":{"total":95995035648,"available":95169097728,"internally_reserved":0,"allocated":637632512,"data_stored":637149811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1082672,"kb_used_data":834840,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92662480,"statfs":{"total":95995035648,"available":94886379520,"internally_reserved":0,"allocated":854876160,"data_stored":854402531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":837226496,"data_stored":837222424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":629280768,"data_stored":629276696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":621215744,"data_stored":621215744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":907935744,"data_stored":907935744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":517799936,"data_stored":517799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":592576512,"data_stored":592576512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":517607424,"data_stored":517603352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":718798848,"data_stored":718798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:58:05.017 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T18:58:05.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:58:05.017 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T18:58:05.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T18:58:05.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:05.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:05.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: 6.15 scrub starts 2024-09-21T18:58:05.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: 6.15 scrub ok 2024-09-21T18:58:05.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: pgmap v220: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 895 op/s 2024-09-21T18:58:05.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2774895806' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: from='client.25096 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: from='client.15258 v1:172.21.15.81:0/688936975' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: 6.15 scrub starts 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: 6.15 scrub ok 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: pgmap v220: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 895 op/s 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2774895806' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:05.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: from='client.25096 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:05.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: from='client.15258 v1:172.21.15.81:0/688936975' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:05.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:04 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:58:05.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.216 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:05.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:05.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: 6.15 scrub starts 2024-09-21T18:58:05.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: 6.15 scrub ok 2024-09-21T18:58:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: pgmap v220: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 895 op/s 2024-09-21T18:58:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2774895806' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: from='client.25096 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: from='client.15258 v1:172.21.15.81:0/688936975' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:04 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:58:05.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:05.364 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:05.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:05.492 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:05.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:05.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:05.781 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:05.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:05.952 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:05.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: 5.4 scrub starts 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: 5.4 scrub ok 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: 5.c scrub starts 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: 5.c scrub ok 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: 4.17 scrub starts 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: 4.17 scrub ok 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: from='client.25096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:58:05.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[29384]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: 5.4 scrub starts 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: 5.4 scrub ok 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: 5.c scrub starts 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: 5.c scrub ok 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: 4.17 scrub starts 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: 4.17 scrub ok 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: from='client.25096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:58:05.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:05 smithi081 ceph-mon[39192]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T18:58:05.991 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:256989 7.a e157/79 7:5c12b9d6:::benchmark_data_smithi081_73391_object117990:head v 157'15827, mlcod=157'15824)", 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:05.982775+0000", 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0038975419999999999, 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039254149999999998, 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:05.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982775+0000", 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982775+0000", 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982777+0000", 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3909999999999999e-06 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:05.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982784+0000", 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8079999999999999e-06 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982794+0000", 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0451e-05 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982801+0000", 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6429999999999998e-06 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.982964+0000", 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016291499999999999 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.983006+0000", 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2218999999999997e-05 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:256996 7.3 e157/79 7:cc39b06d:::benchmark_data_smithi081_73391_object117997:head v 157'15882, mlcod=157'15880)", 2024-09-21T18:58:05.995 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:05.984152+0000", 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.002521316, 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026028959999999999, 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984152+0000", 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984152+0000", 2024-09-21T18:58:05.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984153+0000", 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0130000000000001e-06 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984159+0000", 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2670000000000003e-06 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984170+0000", 2024-09-21T18:58:05.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1338000000000001e-05 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984176+0000", 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2020000000000001e-06 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984239+0000", 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2236999999999994e-05 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:05.984255+0000", 2024-09-21T18:58:05.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6674e-05 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 2 2024-09-21T18:58:05.999 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:58:06.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:06.094 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:06.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:06.228 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:06.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: 5.4 scrub starts 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: 5.4 scrub ok 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: 5.c scrub starts 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: 5.c scrub ok 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: 4.17 scrub starts 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: 4.17 scrub ok 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: from='client.25096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:58:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:05 smithi104 ceph-mon[32438]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T18:58:06.330 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:06.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:06.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:06.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:06.618 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:06.720 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:06.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:06.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:06.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:06.997 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:06.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:07.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[39192]: pgmap v222: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 858 B/s rd, 0 B/s wr, 856 op/s 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[29384]: pgmap v222: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 858 B/s rd, 0 B/s wr, 856 op/s 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:07.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:07.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:06 smithi104 ceph-mon[32438]: pgmap v222: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 858 B/s rd, 0 B/s wr, 856 op/s 2024-09-21T18:58:07.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:58:07.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:58:07.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:07.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:07.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:07.376 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:07.404 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:07.469 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:58:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:58:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:58:07.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T18:58:07.494 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:07.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:07.519 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:07.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:07.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:07.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:58:07.695 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:07.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:07.855 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:07.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:07.981 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:07.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:08.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:08.108 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:08.203 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:08.210 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:08.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:08.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:58:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:58:08.335 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:08.360 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:08.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:08.486 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 158, 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T18:58:07.869676+0000", 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T18:52:58.333331+0000", 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T18:55:50.490072+0000", 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5801984, 2024-09-21T18:58:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "noscrub", 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 18, 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 7, 2024-09-21T18:58:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 1, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 1, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 1, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 1, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 1, 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "21", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:58:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 7.8899998664855957, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 7.8899998664855957, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T18:58:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 3, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 3, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T18:58:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.545 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.546 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6500000953674316, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6500000953674316, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T18:58:08.547 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:58:08.548 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.549 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.550 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.551 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.552 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.5, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.5, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.553 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.554 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:58:08.555 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.556 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.557 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.558 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.559 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.25, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.25, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.25, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.25, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T18:58:08.560 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.561 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.562 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.563 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.564 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.565 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.25, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.25, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.25, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.25, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T18:58:08.566 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:58:08.567 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.568 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.569 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.570 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.571 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.75, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.75, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.75, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.75, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 7, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_0", 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:54:12.574080+0000", 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:58:08.572 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 16, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 16, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 16, 2024-09-21T18:58:08.573 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 16, 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 16, 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "111", 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:58:08.574 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:58:08.575 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:58:08.576 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T18:58:08.577 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.578 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 151, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.580 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T18:58:08.581 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.582 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 133, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.585 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 144, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.587 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T18:58:08.588 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.589 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.590 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 150, 2024-09-21T18:58:08.591 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.593 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.594 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 149, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.595 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T18:58:08.596 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.597 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 37, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 113, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:58:08.600 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.601 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/2569144231", 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/2569144231", 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/2569144231", 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/2569144231", 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.602 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 45, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 151, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.603 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/185975065", 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/185975065", 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/185975065", 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/185975065", 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:58:08.606 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 130, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T18:58:08.607 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.608 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T18:58:08.609 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:08.610 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:01.882583+0000", 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.611 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:57:26.753959+0000", 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:57:07.195001+0000", 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:57:18.940069+0000", 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.613 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:00.418311+0000", 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:07.454216+0000", 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T18:58:08.614 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:57:25.357666+0000", 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:57:42.010989+0000", 2024-09-21T18:58:08.615 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.7", 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.616 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.10", 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T18:58:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.13", 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.18", 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [], 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:58:08.619 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T18:58:08.620 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T18:58:08.621 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:08.622 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:58:08.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:08.662 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:08.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:08.798 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:08.899 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:08.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:08.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:08.985 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:08.985 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:09.026 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:09.026 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:09.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:09.070 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:09.070 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:09.108 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:09.109 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:09.143 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[39192]: pgmap v223: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[39192]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/182994992' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T18:58:09.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[29384]: pgmap v223: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T18:58:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[29384]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T18:58:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/182994992' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T18:58:09.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:09.172 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:09.201 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:09.205 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:09.205 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:09.242 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:09.242 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:08 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:08 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:08 smithi104 ceph-mon[32438]: pgmap v223: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T18:58:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:08 smithi104 ceph-mon[32438]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T18:58:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/182994992' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T18:58:09.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:09.336 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:09.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:09.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:09.569 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:09.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:09.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:09.751 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:09.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:09.881 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:09.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:09.973 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2781539640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:09.973 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:09 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:09.973 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:09 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:09.973 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2781539640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:09.974 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:09 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:09.974 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:09 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:09.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:10.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:10.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:10.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2781539640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:09 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:09 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:10.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:10.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:10.526 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:10.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:10.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:10.690 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:10.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:10.795 INFO:tasks.thrashosds.thrasher:j is {'epoch': 158, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T18:58:07.869676+0000', 'last_up_change': '2024-09-21T18:52:58.333331+0000', 'last_in_change': '2024-09-21T18:55:50.490072+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T18:54:12.574080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 151, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 113, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 2569144231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 2569144231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 2569144231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 2569144231}]}, 'public_addr': '172.21.15.104:6804/2569144231', 'cluster_addr': '172.21.15.104:6805/2569144231', 'heartbeat_back_addr': '172.21.15.104:6807/2569144231', 'heartbeat_front_addr': '172.21.15.104:6806/2569144231', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 151, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 185975065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 185975065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 185975065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 185975065}]}, 'public_addr': '172.21.15.104:6808/185975065', 'cluster_addr': '172.21.15.104:6809/185975065', 'heartbeat_back_addr': '172.21.15.104:6811/185975065', 'heartbeat_front_addr': '172.21.15.104:6810/185975065', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:01.882583+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:57:26.753959+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:57:07.195001+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:57:18.940069+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:00.418311+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:07.454216+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:57:25.357666+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:57:42.010989+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T18:58:10.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:58:10.817 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:10.863 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:10.863 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:10.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:10.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:10.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:11.057 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:11.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:10 smithi081 ceph-mon[29384]: pgmap v225: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 771 B/s rd, 0 B/s wr, 755 op/s 2024-09-21T18:58:11.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:10 smithi081 ceph-mon[29384]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T18:58:11.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:10 smithi081 ceph-mon[39192]: pgmap v225: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 771 B/s rd, 0 B/s wr, 755 op/s 2024-09-21T18:58:11.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:10 smithi081 ceph-mon[39192]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T18:58:11.084 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:11.145 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:11.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:11.237 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:10 smithi104 ceph-mon[32438]: pgmap v225: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 771 B/s rd, 0 B/s wr, 755 op/s 2024-09-21T18:58:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:10 smithi104 ceph-mon[32438]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T18:58:11.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.375 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:11.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:11.476 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:11.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:11.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:11.630 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:11.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:11.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:11.860 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:11.895 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:11.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:11.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:12.024 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:12.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:58:12.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:12.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T18:58:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.201 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:12.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:12.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:12.327 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:12.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:12.471 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:12.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:12.599 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:12.701 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:12.727 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:12.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:12.850 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:12.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:12.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:12.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:13.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:13.127 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:13.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:13.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:12 smithi104 ceph-mon[32438]: pgmap v227: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 758 op/s 2024-09-21T18:58:13.264 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:13.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:12 smithi081 ceph-mon[39192]: pgmap v227: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 758 op/s 2024-09-21T18:58:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:12 smithi081 ceph-mon[29384]: pgmap v227: 148 pgs: 148 active+clean; 1.2 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 758 op/s 2024-09-21T18:58:13.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:13.391 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:13.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.527 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:13.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:13.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:13.656 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:13.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:13.784 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:13.862 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:13.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:13.908 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:13.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:14.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:14.036 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:14.138 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:14.166 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:14.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.312 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:14.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:14.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:14.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:14.513 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:58:14.553 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:14.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:14.580 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:14.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:14.726 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:14.789 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:14.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:14.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:14.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:14.941 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:58:14.943 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:58:14.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:15.105 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:15.129 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:15.231 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:14 smithi104 ceph-mon[32438]: pgmap v228: 148 pgs: 148 active+clean; 938 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T18:58:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3343954324' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:14 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3343954324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:14 smithi104 ceph-mon[32438]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T18:58:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[39192]: pgmap v228: 148 pgs: 148 active+clean; 938 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T18:58:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3343954324' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3343954324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[39192]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T18:58:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[29384]: pgmap v228: 148 pgs: 148 active+clean; 938 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T18:58:15.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3343954324' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:15.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:15.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3343954324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:15.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:14 smithi081 ceph-mon[29384]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T18:58:15.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:15.380 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:15.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:15.618 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:15.643 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:15.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:15.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:15.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:16.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:16.038 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:16.139 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:16.202 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:15 smithi081 ceph-mon[39192]: from='client.25111 v1:172.21.15.81:0/1639289077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:16.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:15 smithi081 ceph-mon[29384]: from='client.25111 v1:172.21.15.81:0/1639289077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-21T18:58:14.509774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":983964211,"num_objects":15205,"num_object_clones":0,"num_object_copies":45615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15205,"num_whiteouts":0,"num_read":1137,"num_read_kb":994,"num_write":263537,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36103,"num_bytes_recovered":2359238130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90244,"ondisk_log_size":90244,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4680548,"kb_used_data":3304904,"kb_used_omap":12,"kb_used_meta":1375219,"kb_avail":745280668,"statfs":{"total":767960285184,"available":763167404032,"internally_reserved":0,"allocated":3384221696,"data_stored":3380312097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1408224820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-346292224,"num_objects":-5284,"num_object_clones":0,"num_object_copies":-15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5284,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5284,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.038009"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900840+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:58:09.900840+0000","last_peered":"2024-09-21T18:58:09.900840+0000","last_clean":"2024-09-21T18:58:09.900840+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:58:09.900840+0000","last_undegraded":"2024-09-21T18:58:09.900840+0000","last_fullsized":"2024-09-21T18:58:09.900840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":252,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.894919+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:58:09.894919+0000","last_peered":"2024-09-21T18:58:09.894919+0000","last_clean":"2024-09-21T18:58:09.894919+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:58:09.894919+0000","last_undegraded":"2024-09-21T18:58:09.894919+0000","last_fullsized":"2024-09-21T18:58:09.894919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.917836+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:58:09.917836+0000","last_peered":"2024-09-21T18:58:09.917836+0000","last_clean":"2024-09-21T18:58:09.917836+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:58:09.917836+0000","last_undegraded":"2024-09-21T18:58:09.917836+0000","last_fullsized":"2024-09-21T18:58:09.917836+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.239093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895048+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T18:58:09.895048+0000","last_peered":"2024-09-21T18:58:09.895048+0000","last_clean":"2024-09-21T18:58:09.895048+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:58:09.895048+0000","last_undegraded":"2024-09-21T18:58:09.895048+0000","last_fullsized":"2024-09-21T18:58:09.895048+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.087878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907887+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:58:09.907887+0000","last_peered":"2024-09-21T18:58:09.907887+0000","last_clean":"2024-09-21T18:58:09.907887+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:58:09.907887+0000","last_undegraded":"2024-09-21T18:58:09.907887+0000","last_fullsized":"2024-09-21T18:58:09.907887+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":224,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908928+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:58:09.908928+0000","last_peered":"2024-09-21T18:58:09.908928+0000","last_clean":"2024-09-21T18:58:09.908928+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:58:09.908928+0000","last_undegraded":"2024-09-21T18:58:09.908928+0000","last_fullsized":"2024-09-21T18:58:09.908928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":283,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908037+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:58:09.908037+0000","last_peered":"2024-09-21T18:58:09.908037+0000","last_clean":"2024-09-21T18:58:09.908037+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:58:09.908037+0000","last_undegraded":"2024-09-21T18:58:09.908037+0000","last_fullsized":"2024-09-21T18:58:09.908037+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.894829+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:58:09.894829+0000","last_peered":"2024-09-21T18:58:09.894829+0000","last_clean":"2024-09-21T18:58:09.894829+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:58:09.894829+0000","last_undegraded":"2024-09-21T18:58:09.894829+0000","last_fullsized":"2024-09-21T18:58:09.894829+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":223,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909359+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:58:09.909359+0000","last_peered":"2024-09-21T18:58:09.909359+0000","last_clean":"2024-09-21T18:58:09.909359+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:58:09.909359+0000","last_undegraded":"2024-09-21T18:58:09.909359+0000","last_fullsized":"2024-09-21T18:58:09.909359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:22.885755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":277,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895290+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:58:09.895290+0000","last_peered":"2024-09-21T18:58:09.895290+0000","last_clean":"2024-09-21T18:58:09.895290+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:58:09.895290+0000","last_undegraded":"2024-09-21T18:58:09.895290+0000","last_fullsized":"2024-09-21T18:58:09.895290+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":236,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908911+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:58:09.908911+0000","last_peered":"2024-09-21T18:58:09.908911+0000","last_clean":"2024-09-21T18:58:09.908911+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:58:09.908911+0000","last_undegraded":"2024-09-21T18:58:09.908911+0000","last_fullsized":"2024-09-21T18:58:09.908911+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":102,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.897796+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:58:09.897796+0000","last_peered":"2024-09-21T18:58:09.897796+0000","last_clean":"2024-09-21T18:58:09.897796+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:58:09.897796+0000","last_undegraded":"2024-09-21T18:58:09.897796+0000","last_fullsized":"2024-09-21T18:58:09.897796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":223,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909069+0000","last_change":"2024-09-21T18:57:19.821897+0000","last_active":"2024-09-21T18:58:09.909069+0000","last_peered":"2024-09-21T18:58:09.909069+0000","last_clean":"2024-09-21T18:58:09.909069+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:58:09.909069+0000","last_undegraded":"2024-09-21T18:58:09.909069+0000","last_fullsized":"2024-09-21T18:58:09.909069+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.821849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.129486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895187+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:58:09.895187+0000","last_peered":"2024-09-21T18:58:09.895187+0000","last_clean":"2024-09-21T18:58:09.895187+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:58:09.895187+0000","last_undegraded":"2024-09-21T18:58:09.895187+0000","last_fullsized":"2024-09-21T18:58:09.895187+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900010+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:58:09.900010+0000","last_peered":"2024-09-21T18:58:09.900010+0000","last_clean":"2024-09-21T18:58:09.900010+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:58:09.900010+0000","last_undegraded":"2024-09-21T18:58:09.900010+0000","last_fullsized":"2024-09-21T18:58:09.900010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:16.165478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":219,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908678+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:58:09.908678+0000","last_peered":"2024-09-21T18:58:09.908678+0000","last_clean":"2024-09-21T18:58:09.908678+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:58:09.908678+0000","last_undegraded":"2024-09-21T18:58:09.908678+0000","last_fullsized":"2024-09-21T18:58:09.908678+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.650657+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895714+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:58:09.895714+0000","last_peered":"2024-09-21T18:58:09.895714+0000","last_clean":"2024-09-21T18:58:09.895714+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:58:09.895714+0000","last_undegraded":"2024-09-21T18:58:09.895714+0000","last_fullsized":"2024-09-21T18:58:09.895714+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":102,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900402+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:58:09.900402+0000","last_peered":"2024-09-21T18:58:09.900402+0000","last_clean":"2024-09-21T18:58:09.900402+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:58:09.900402+0000","last_undegraded":"2024-09-21T18:58:09.900402+0000","last_fullsized":"2024-09-21T18:58:09.900402+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":244,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.901625+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:58:09.901625+0000","last_peered":"2024-09-21T18:58:09.901625+0000","last_clean":"2024-09-21T18:58:09.901625+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:58:09.901625+0000","last_undegraded":"2024-09-21T18:58:09.901625+0000","last_fullsized":"2024-09-21T18:58:09.901625+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.481531+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.914186+0000","last_change":"2024-09-21T18:57:20.229156+0000","last_active":"2024-09-21T18:58:09.914186+0000","last_peered":"2024-09-21T18:58:09.914186+0000","last_clean":"2024-09-21T18:58:09.914186+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:58:09.914186+0000","last_undegraded":"2024-09-21T18:58:09.914186+0000","last_fullsized":"2024-09-21T18:58:09.914186+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.229088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.229088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.406792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909637+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:58:09.909637+0000","last_peered":"2024-09-21T18:58:09.909637+0000","last_clean":"2024-09-21T18:58:09.909637+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:58:09.909637+0000","last_undegraded":"2024-09-21T18:58:09.909637+0000","last_fullsized":"2024-09-21T18:58:09.909637+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.745019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":231,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895451+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:58:09.895451+0000","last_peered":"2024-09-21T18:58:09.895451+0000","last_clean":"2024-09-21T18:58:09.895451+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:58:09.895451+0000","last_undegraded":"2024-09-21T18:58:09.895451+0000","last_fullsized":"2024-09-21T18:58:09.895451+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":243,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.906291+0000","last_change":"2024-09-21T18:54:49.116601+0000","last_active":"2024-09-21T18:58:09.906291+0000","last_peered":"2024-09-21T18:58:09.906291+0000","last_clean":"2024-09-21T18:58:09.906291+0000","last_became_active":"2024-09-21T18:53:09.406600+0000","last_became_peered":"2024-09-21T18:53:09.406600+0000","last_unstale":"2024-09-21T18:58:09.906291+0000","last_undegraded":"2024-09-21T18:58:09.906291+0000","last_fullsized":"2024-09-21T18:58:09.906291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895269+0000","last_change":"2024-09-21T18:55:24.111167+0000","last_active":"2024-09-21T18:58:09.895269+0000","last_peered":"2024-09-21T18:58:09.895269+0000","last_clean":"2024-09-21T18:58:09.895269+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:58:09.895269+0000","last_undegraded":"2024-09-21T18:58:09.895269+0000","last_fullsized":"2024-09-21T18:58:09.895269+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.110965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.110965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900293+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:58:09.900293+0000","last_peered":"2024-09-21T18:58:09.900293+0000","last_clean":"2024-09-21T18:58:09.900293+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:58:09.900293+0000","last_undegraded":"2024-09-21T18:58:09.900293+0000","last_fullsized":"2024-09-21T18:58:09.900293+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":224,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.918020+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:58:09.918020+0000","last_peered":"2024-09-21T18:58:09.918020+0000","last_clean":"2024-09-21T18:58:09.918020+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:58:09.918020+0000","last_undegraded":"2024-09-21T18:58:09.918020+0000","last_fullsized":"2024-09-21T18:58:09.918020+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.299356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":115,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904173+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:58:09.904173+0000","last_peered":"2024-09-21T18:58:09.904173+0000","last_clean":"2024-09-21T18:58:09.904173+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:58:09.904173+0000","last_undegraded":"2024-09-21T18:58:09.904173+0000","last_fullsized":"2024-09-21T18:58:09.904173+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.058858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":286,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.916448+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:58:09.916448+0000","last_peered":"2024-09-21T18:58:09.916448+0000","last_clean":"2024-09-21T18:58:09.916448+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:58:09.916448+0000","last_undegraded":"2024-09-21T18:58:09.916448+0000","last_fullsized":"2024-09-21T18:58:09.916448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:08.276721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"159'16215","reported_seq":16414,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.448093+0000","last_change":"2024-09-21T18:57:59.815796+0000","last_active":"2024-09-21T18:58:11.448093+0000","last_peered":"2024-09-21T18:58:11.448093+0000","last_clean":"2024-09-21T18:58:11.448093+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:58:11.448093+0000","last_undegraded":"2024-09-21T18:58:11.448093+0000","last_fullsized":"2024-09-21T18:58:11.448093+0000","mapping_epoch":79,"log_start":"121'9900","ondisk_log_start":"121'9900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'15478","last_scrub_stamp":"2024-09-21T18:57:59.815673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.815673+0000","objects_scrubbed":1655,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.208391+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":2709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16215,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":244,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.914292+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:58:09.914292+0000","last_peered":"2024-09-21T18:58:09.914292+0000","last_clean":"2024-09-21T18:58:09.914292+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:58:09.914292+0000","last_undegraded":"2024-09-21T18:58:09.914292+0000","last_fullsized":"2024-09-21T18:58:09.914292+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.223966+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":224,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.920282+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:58:09.920282+0000","last_peered":"2024-09-21T18:58:09.920282+0000","last_clean":"2024-09-21T18:58:09.920282+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:58:09.920282+0000","last_undegraded":"2024-09-21T18:58:09.920282+0000","last_fullsized":"2024-09-21T18:58:09.920282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":306,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895794+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:58:09.895794+0000","last_peered":"2024-09-21T18:58:09.895794+0000","last_clean":"2024-09-21T18:58:09.895794+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:58:09.895794+0000","last_undegraded":"2024-09-21T18:58:09.895794+0000","last_fullsized":"2024-09-21T18:58:09.895794+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":241,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909124+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T18:58:09.909124+0000","last_peered":"2024-09-21T18:58:09.909124+0000","last_clean":"2024-09-21T18:58:09.909124+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:58:09.909124+0000","last_undegraded":"2024-09-21T18:58:09.909124+0000","last_fullsized":"2024-09-21T18:58:09.909124+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.949065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"159'16300","reported_seq":16473,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:12.123447+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:58:12.123447+0000","last_peered":"2024-09-21T18:58:12.123447+0000","last_clean":"2024-09-21T18:58:12.123447+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:58:12.123447+0000","last_undegraded":"2024-09-21T18:58:12.123447+0000","last_fullsized":"2024-09-21T18:58:12.123447+0000","mapping_epoch":79,"log_start":"128'11600","ondisk_log_start":"128'11600","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59113472,"num_objects":902,"num_object_clones":0,"num_object_copies":2706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16300,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":302,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.920043+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:58:09.920043+0000","last_peered":"2024-09-21T18:58:09.920043+0000","last_clean":"2024-09-21T18:58:09.920043+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:58:09.920043+0000","last_undegraded":"2024-09-21T18:58:09.920043+0000","last_fullsized":"2024-09-21T18:58:09.920043+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":238,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909994+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:58:09.909994+0000","last_peered":"2024-09-21T18:58:09.909994+0000","last_clean":"2024-09-21T18:58:09.909994+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:58:09.909994+0000","last_undegraded":"2024-09-21T18:58:09.909994+0000","last_fullsized":"2024-09-21T18:58:09.909994+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895039+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:58:09.895039+0000","last_peered":"2024-09-21T18:58:09.895039+0000","last_clean":"2024-09-21T18:58:09.895039+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:58:09.895039+0000","last_undegraded":"2024-09-21T18:58:09.895039+0000","last_fullsized":"2024-09-21T18:58:09.895039+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":203,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895198+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:58:09.895198+0000","last_peered":"2024-09-21T18:58:09.895198+0000","last_clean":"2024-09-21T18:58:09.895198+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:58:09.895198+0000","last_undegraded":"2024-09-21T18:58:09.895198+0000","last_fullsized":"2024-09-21T18:58:09.895198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.900161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"159'16571","reported_seq":16744,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:12.138820+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:58:12.138820+0000","last_peered":"2024-09-21T18:58:12.138820+0000","last_clean":"2024-09-21T18:58:12.138820+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:58:12.138820+0000","last_undegraded":"2024-09-21T18:58:12.138820+0000","last_fullsized":"2024-09-21T18:58:12.138820+0000","mapping_epoch":79,"log_start":"129'11900","ondisk_log_start":"129'11900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16571,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895091+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:58:09.895091+0000","last_peered":"2024-09-21T18:58:09.895091+0000","last_clean":"2024-09-21T18:58:09.895091+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:58:09.895091+0000","last_undegraded":"2024-09-21T18:58:09.895091+0000","last_fullsized":"2024-09-21T18:58:09.895091+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":107,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904429+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:58:09.904429+0000","last_peered":"2024-09-21T18:58:09.904429+0000","last_clean":"2024-09-21T18:58:09.904429+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:58:09.904429+0000","last_undegraded":"2024-09-21T18:58:09.904429+0000","last_fullsized":"2024-09-21T18:58:09.904429+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":277,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.911020+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:58:09.911020+0000","last_peered":"2024-09-21T18:58:09.911020+0000","last_clean":"2024-09-21T18:58:09.911020+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:58:09.911020+0000","last_undegraded":"2024-09-21T18:58:09.911020+0000","last_fullsized":"2024-09-21T18:58:09.911020+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900165+0000","last_change":"2024-09-21T18:58:00.240553+0000","last_active":"2024-09-21T18:58:09.900165+0000","last_peered":"2024-09-21T18:58:09.900165+0000","last_clean":"2024-09-21T18:58:09.900165+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:58:09.900165+0000","last_undegraded":"2024-09-21T18:58:09.900165+0000","last_fullsized":"2024-09-21T18:58:09.900165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.240476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.240476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:25.073873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"159'16359","reported_seq":16623,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.466573+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:58:11.466573+0000","last_peered":"2024-09-21T18:58:11.466573+0000","last_clean":"2024-09-21T18:58:11.466573+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:58:11.466573+0000","last_undegraded":"2024-09-21T18:58:11.466573+0000","last_fullsized":"2024-09-21T18:58:11.466573+0000","mapping_epoch":79,"log_start":"127'11500","ondisk_log_start":"127'11500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":2709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16359,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":226,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.914926+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:58:09.914926+0000","last_peered":"2024-09-21T18:58:09.914926+0000","last_clean":"2024-09-21T18:58:09.914926+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:58:09.914926+0000","last_undegraded":"2024-09-21T18:58:09.914926+0000","last_fullsized":"2024-09-21T18:58:09.914926+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":231,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909371+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:58:09.909371+0000","last_peered":"2024-09-21T18:58:09.909371+0000","last_clean":"2024-09-21T18:58:09.909371+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:58:09.909371+0000","last_undegraded":"2024-09-21T18:58:09.909371+0000","last_fullsized":"2024-09-21T18:58:09.909371+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900648+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:58:09.900648+0000","last_peered":"2024-09-21T18:58:09.900648+0000","last_clean":"2024-09-21T18:58:09.900648+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:58:09.900648+0000","last_undegraded":"2024-09-21T18:58:09.900648+0000","last_fullsized":"2024-09-21T18:58:09.900648+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.897928+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:58:09.897928+0000","last_peered":"2024-09-21T18:58:09.897928+0000","last_clean":"2024-09-21T18:58:09.897928+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:58:09.897928+0000","last_undegraded":"2024-09-21T18:58:09.897928+0000","last_fullsized":"2024-09-21T18:58:09.897928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"159'16725","reported_seq":17048,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.472939+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:58:11.472939+0000","last_peered":"2024-09-21T18:58:11.472939+0000","last_clean":"2024-09-21T18:58:11.472939+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:58:11.472939+0000","last_undegraded":"2024-09-21T18:58:11.472939+0000","last_fullsized":"2024-09-21T18:58:11.472939+0000","mapping_epoch":79,"log_start":"124'11100","ondisk_log_start":"124'11100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:43.658704+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16725,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900488+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:58:09.900488+0000","last_peered":"2024-09-21T18:58:09.900488+0000","last_clean":"2024-09-21T18:58:09.900488+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:58:09.900488+0000","last_undegraded":"2024-09-21T18:58:09.900488+0000","last_fullsized":"2024-09-21T18:58:09.900488+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":223,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.918592+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:58:09.918592+0000","last_peered":"2024-09-21T18:58:09.918592+0000","last_clean":"2024-09-21T18:58:09.918592+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:58:09.918592+0000","last_undegraded":"2024-09-21T18:58:09.918592+0000","last_fullsized":"2024-09-21T18:58:09.918592+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:47.380559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":228,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.915833+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:58:09.915833+0000","last_peered":"2024-09-21T18:58:09.915833+0000","last_clean":"2024-09-21T18:58:09.915833+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:58:09.915833+0000","last_undegraded":"2024-09-21T18:58:09.915833+0000","last_fullsized":"2024-09-21T18:58:09.915833+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.917277+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:58:09.917277+0000","last_peered":"2024-09-21T18:58:09.917277+0000","last_clean":"2024-09-21T18:58:09.917277+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:58:09.917277+0000","last_undegraded":"2024-09-21T18:58:09.917277+0000","last_fullsized":"2024-09-21T18:58:09.917277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:15.637336+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"159'16423","reported_seq":24509,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.438972+0000","last_change":"2024-09-21T18:57:57.958393+0000","last_active":"2024-09-21T18:58:11.438972+0000","last_peered":"2024-09-21T18:58:11.438972+0000","last_clean":"2024-09-21T18:58:11.438972+0000","last_became_active":"2024-09-21T18:57:34.506455+0000","last_became_peered":"2024-09-21T18:57:34.506455+0000","last_unstale":"2024-09-21T18:58:11.438972+0000","last_undegraded":"2024-09-21T18:58:11.438972+0000","last_fullsized":"2024-09-21T18:58:11.438972+0000","mapping_epoch":143,"log_start":"121'10100","ondisk_log_start":"121'10100","created":79,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'15561","last_scrub_stamp":"2024-09-21T18:57:57.958164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:57.958164+0000","objects_scrubbed":1789,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:26.531439+0000","scrub_duration":626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16423,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3933,"num_bytes_recovered":257753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":328,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.918334+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:58:09.918334+0000","last_peered":"2024-09-21T18:58:09.918334+0000","last_clean":"2024-09-21T18:58:09.918334+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:58:09.918334+0000","last_undegraded":"2024-09-21T18:58:09.918334+0000","last_fullsized":"2024-09-21T18:58:09.918334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:41.364368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":245,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895580+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:58:09.895580+0000","last_peered":"2024-09-21T18:58:09.895580+0000","last_clean":"2024-09-21T18:58:09.895580+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:58:09.895580+0000","last_undegraded":"2024-09-21T18:58:09.895580+0000","last_fullsized":"2024-09-21T18:58:09.895580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":241,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909754+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:58:09.909754+0000","last_peered":"2024-09-21T18:58:09.909754+0000","last_clean":"2024-09-21T18:58:09.909754+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:58:09.909754+0000","last_undegraded":"2024-09-21T18:58:09.909754+0000","last_fullsized":"2024-09-21T18:58:09.909754+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.902375+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:58:09.902375+0000","last_peered":"2024-09-21T18:58:09.902375+0000","last_clean":"2024-09-21T18:58:09.902375+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:58:09.902375+0000","last_undegraded":"2024-09-21T18:58:09.902375+0000","last_fullsized":"2024-09-21T18:58:09.902375+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908629+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:58:09.908629+0000","last_peered":"2024-09-21T18:58:09.908629+0000","last_clean":"2024-09-21T18:58:09.908629+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:58:09.908629+0000","last_undegraded":"2024-09-21T18:58:09.908629+0000","last_fullsized":"2024-09-21T18:58:09.908629+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"159'16544","reported_seq":22253,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.543663+0000","last_change":"2024-09-21T18:58:01.679316+0000","last_active":"2024-09-21T18:58:11.543663+0000","last_peered":"2024-09-21T18:58:11.543663+0000","last_clean":"2024-09-21T18:58:11.543663+0000","last_became_active":"2024-09-21T18:57:51.729319+0000","last_became_peered":"2024-09-21T18:57:51.729319+0000","last_unstale":"2024-09-21T18:58:11.543663+0000","last_undegraded":"2024-09-21T18:58:11.543663+0000","last_fullsized":"2024-09-21T18:58:11.543663+0000","mapping_epoch":150,"log_start":"124'10900","ondisk_log_start":"124'10900","created":79,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'15900","last_scrub_stamp":"2024-09-21T18:58:01.679101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.679101+0000","objects_scrubbed":1623,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.524249+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16544,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":179961856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":190,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.901758+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:58:09.901758+0000","last_peered":"2024-09-21T18:58:09.901758+0000","last_clean":"2024-09-21T18:58:09.901758+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:58:09.901758+0000","last_undegraded":"2024-09-21T18:58:09.901758+0000","last_fullsized":"2024-09-21T18:58:09.901758+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:44.501396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.911689+0000","last_change":"2024-09-21T18:57:22.224988+0000","last_active":"2024-09-21T18:58:09.911689+0000","last_peered":"2024-09-21T18:58:09.911689+0000","last_clean":"2024-09-21T18:58:09.911689+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:58:09.911689+0000","last_undegraded":"2024-09-21T18:58:09.911689+0000","last_fullsized":"2024-09-21T18:58:09.911689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:22.220182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:22.220182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.617863+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":262,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.901963+0000","last_change":"2024-09-21T18:57:49.456229+0000","last_active":"2024-09-21T18:58:09.901963+0000","last_peered":"2024-09-21T18:58:09.901963+0000","last_clean":"2024-09-21T18:58:09.901963+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:58:09.901963+0000","last_undegraded":"2024-09-21T18:58:09.901963+0000","last_fullsized":"2024-09-21T18:58:09.901963+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:49.456147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.456147+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:50.500491+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.902668+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:58:09.902668+0000","last_peered":"2024-09-21T18:58:09.902668+0000","last_clean":"2024-09-21T18:58:09.902668+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:58:09.902668+0000","last_undegraded":"2024-09-21T18:58:09.902668+0000","last_fullsized":"2024-09-21T18:58:09.902668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900292+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:58:09.900292+0000","last_peered":"2024-09-21T18:58:09.900292+0000","last_clean":"2024-09-21T18:58:09.900292+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:58:09.900292+0000","last_undegraded":"2024-09-21T18:58:09.900292+0000","last_fullsized":"2024-09-21T18:58:09.900292+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:10.461376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"159'16852","reported_seq":31168,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.454298+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:58:11.454298+0000","last_peered":"2024-09-21T18:58:11.454298+0000","last_clean":"2024-09-21T18:58:11.454298+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:58:11.454298+0000","last_undegraded":"2024-09-21T18:58:11.454298+0000","last_fullsized":"2024-09-21T18:58:11.454298+0000","mapping_epoch":130,"log_start":"127'11700","ondisk_log_start":"127'11700","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.080321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16852,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":117,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904256+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:58:09.904256+0000","last_peered":"2024-09-21T18:58:09.904256+0000","last_clean":"2024-09-21T18:58:09.904256+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:58:09.904256+0000","last_undegraded":"2024-09-21T18:58:09.904256+0000","last_fullsized":"2024-09-21T18:58:09.904256+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.899526+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:58:09.899526+0000","last_peered":"2024-09-21T18:58:09.899526+0000","last_clean":"2024-09-21T18:58:09.899526+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:58:09.899526+0000","last_undegraded":"2024-09-21T18:58:09.899526+0000","last_fullsized":"2024-09-21T18:58:09.899526+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":561,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900281+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:58:09.900281+0000","last_peered":"2024-09-21T18:58:09.900281+0000","last_clean":"2024-09-21T18:58:09.900281+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:58:09.900281+0000","last_undegraded":"2024-09-21T18:58:09.900281+0000","last_fullsized":"2024-09-21T18:58:09.900281+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:17.496871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":293,"num_read_kb":288,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":245,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909686+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:58:09.909686+0000","last_peered":"2024-09-21T18:58:09.909686+0000","last_clean":"2024-09-21T18:58:09.909686+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:58:09.909686+0000","last_undegraded":"2024-09-21T18:58:09.909686+0000","last_fullsized":"2024-09-21T18:58:09.909686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.378290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.911751+0000","last_change":"2024-09-21T18:57:18.176297+0000","last_active":"2024-09-21T18:58:09.911751+0000","last_peered":"2024-09-21T18:58:09.911751+0000","last_clean":"2024-09-21T18:58:09.911751+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:58:09.911751+0000","last_undegraded":"2024-09-21T18:58:09.911751+0000","last_fullsized":"2024-09-21T18:58:09.911751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:18.176233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:18.176233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:24.204170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"159'16386","reported_seq":28439,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.570351+0000","last_change":"2024-09-21T18:57:20.837816+0000","last_active":"2024-09-21T18:58:11.570351+0000","last_peered":"2024-09-21T18:58:11.570351+0000","last_clean":"2024-09-21T18:58:11.570351+0000","last_became_active":"2024-09-21T18:57:20.750225+0000","last_became_peered":"2024-09-21T18:57:20.750225+0000","last_unstale":"2024-09-21T18:58:11.570351+0000","last_undegraded":"2024-09-21T18:58:11.570351+0000","last_fullsized":"2024-09-21T18:58:11.570351+0000","mapping_epoch":139,"log_start":"124'10802","ondisk_log_start":"124'10802","created":79,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.016370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63832064,"num_objects":974,"num_object_clones":0,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16386,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":301,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.916265+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:58:09.916265+0000","last_peered":"2024-09-21T18:58:09.916265+0000","last_clean":"2024-09-21T18:58:09.916265+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:58:09.916265+0000","last_undegraded":"2024-09-21T18:58:09.916265+0000","last_fullsized":"2024-09-21T18:58:09.916265+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.234034+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":224,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.910711+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:58:09.910711+0000","last_peered":"2024-09-21T18:58:09.910711+0000","last_clean":"2024-09-21T18:58:09.910711+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:58:09.910711+0000","last_undegraded":"2024-09-21T18:58:09.910711+0000","last_fullsized":"2024-09-21T18:58:09.910711+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":285,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900207+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:58:09.900207+0000","last_peered":"2024-09-21T18:58:09.900207+0000","last_clean":"2024-09-21T18:58:09.900207+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:58:09.900207+0000","last_undegraded":"2024-09-21T18:58:09.900207+0000","last_fullsized":"2024-09-21T18:58:09.900207+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:08.083396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900392+0000","last_change":"2024-09-21T18:58:04.113670+0000","last_active":"2024-09-21T18:58:09.900392+0000","last_peered":"2024-09-21T18:58:09.900392+0000","last_clean":"2024-09-21T18:58:09.900392+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:58:09.900392+0000","last_undegraded":"2024-09-21T18:58:09.900392+0000","last_fullsized":"2024-09-21T18:58:09.900392+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.113596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.113596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:24.182248+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909632+0000","last_change":"2024-09-21T18:56:12.095400+0000","last_active":"2024-09-21T18:58:09.909632+0000","last_peered":"2024-09-21T18:58:09.909632+0000","last_clean":"2024-09-21T18:58:09.909632+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:58:09.909632+0000","last_undegraded":"2024-09-21T18:58:09.909632+0000","last_fullsized":"2024-09-21T18:58:09.909632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:12.095341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:12.095341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:28.894465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"159'16600","reported_seq":22227,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.491343+0000","last_change":"2024-09-21T18:57:52.759473+0000","last_active":"2024-09-21T18:58:11.491343+0000","last_peered":"2024-09-21T18:58:11.491343+0000","last_clean":"2024-09-21T18:58:11.491343+0000","last_became_active":"2024-09-21T18:57:52.731182+0000","last_became_peered":"2024-09-21T18:57:52.731182+0000","last_unstale":"2024-09-21T18:58:11.491343+0000","last_undegraded":"2024-09-21T18:58:11.491343+0000","last_fullsized":"2024-09-21T18:58:11.491343+0000","mapping_epoch":151,"log_start":"121'10300","ondisk_log_start":"121'10300","created":79,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6300,"log_dups_size":0,"ondisk_log_size":6300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66781208,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16600,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2670,"num_bytes_recovered":174915608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":238,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895877+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:58:09.895877+0000","last_peered":"2024-09-21T18:58:09.895877+0000","last_clean":"2024-09-21T18:58:09.895877+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:58:09.895877+0000","last_undegraded":"2024-09-21T18:58:09.895877+0000","last_fullsized":"2024-09-21T18:58:09.895877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895250+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:58:09.895250+0000","last_peered":"2024-09-21T18:58:09.895250+0000","last_clean":"2024-09-21T18:58:09.895250+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:58:09.895250+0000","last_undegraded":"2024-09-21T18:58:09.895250+0000","last_fullsized":"2024-09-21T18:58:09.895250+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.857138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":231,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909582+0000","last_change":"2024-09-21T18:57:58.653399+0000","last_active":"2024-09-21T18:58:09.909582+0000","last_peered":"2024-09-21T18:58:09.909582+0000","last_clean":"2024-09-21T18:58:09.909582+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:58:09.909582+0000","last_undegraded":"2024-09-21T18:58:09.909582+0000","last_fullsized":"2024-09-21T18:58:09.909582+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.653339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.653339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:20.676781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907063+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:58:09.907063+0000","last_peered":"2024-09-21T18:58:09.907063+0000","last_clean":"2024-09-21T18:58:09.907063+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:58:09.907063+0000","last_undegraded":"2024-09-21T18:58:09.907063+0000","last_fullsized":"2024-09-21T18:58:09.907063+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"159'16249","reported_seq":24746,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.574992+0000","last_change":"2024-09-21T18:57:35.557746+0000","last_active":"2024-09-21T18:58:11.574992+0000","last_peered":"2024-09-21T18:58:11.574992+0000","last_clean":"2024-09-21T18:58:11.574992+0000","last_became_active":"2024-09-21T18:57:35.521875+0000","last_became_peered":"2024-09-21T18:57:35.521875+0000","last_unstale":"2024-09-21T18:58:11.574992+0000","last_undegraded":"2024-09-21T18:58:11.574992+0000","last_fullsized":"2024-09-21T18:58:11.574992+0000","mapping_epoch":144,"log_start":"119'9500","ondisk_log_start":"119'9500","created":79,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6749,"log_dups_size":0,"ondisk_log_size":6749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:10.592711+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58130432,"num_objects":887,"num_object_clones":0,"num_object_copies":2661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16249,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4110,"num_bytes_recovered":269352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":247,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895501+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:58:09.895501+0000","last_peered":"2024-09-21T18:58:09.895501+0000","last_clean":"2024-09-21T18:58:09.895501+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:58:09.895501+0000","last_undegraded":"2024-09-21T18:58:09.895501+0000","last_fullsized":"2024-09-21T18:58:09.895501+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.916089+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:58:09.916089+0000","last_peered":"2024-09-21T18:58:09.916089+0000","last_clean":"2024-09-21T18:58:09.916089+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:58:09.916089+0000","last_undegraded":"2024-09-21T18:58:09.916089+0000","last_fullsized":"2024-09-21T18:58:09.916089+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.223741+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.911835+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T18:58:09.911835+0000","last_peered":"2024-09-21T18:58:09.911835+0000","last_clean":"2024-09-21T18:58:09.911835+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:58:09.911835+0000","last_undegraded":"2024-09-21T18:58:09.911835+0000","last_fullsized":"2024-09-21T18:58:09.911835+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:23.332129+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.902119+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:58:09.902119+0000","last_peered":"2024-09-21T18:58:09.902119+0000","last_clean":"2024-09-21T18:58:09.902119+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:58:09.902119+0000","last_undegraded":"2024-09-21T18:58:09.902119+0000","last_fullsized":"2024-09-21T18:58:09.902119+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.540935+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"159'16368","reported_seq":22625,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.573444+0000","last_change":"2024-09-21T18:57:52.776315+0000","last_active":"2024-09-21T18:58:11.573444+0000","last_peered":"2024-09-21T18:58:11.573444+0000","last_clean":"2024-09-21T18:58:11.573444+0000","last_became_active":"2024-09-21T18:57:52.730444+0000","last_became_peered":"2024-09-21T18:57:52.730444+0000","last_unstale":"2024-09-21T18:58:11.573444+0000","last_undegraded":"2024-09-21T18:58:11.573444+0000","last_fullsized":"2024-09-21T18:58:11.573444+0000","mapping_epoch":151,"log_start":"124'10800","ondisk_log_start":"124'10800","created":79,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:09.988393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16368,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3036,"num_bytes_recovered":198967296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":234,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895355+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:58:09.895355+0000","last_peered":"2024-09-21T18:58:09.895355+0000","last_clean":"2024-09-21T18:58:09.895355+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:58:09.895355+0000","last_undegraded":"2024-09-21T18:58:09.895355+0000","last_fullsized":"2024-09-21T18:58:09.895355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":252,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908246+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:58:09.908246+0000","last_peered":"2024-09-21T18:58:09.908246+0000","last_clean":"2024-09-21T18:58:09.908246+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:58:09.908246+0000","last_undegraded":"2024-09-21T18:58:09.908246+0000","last_fullsized":"2024-09-21T18:58:09.908246+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":292,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.902550+0000","last_change":"2024-09-21T18:58:02.631905+0000","last_active":"2024-09-21T18:58:09.902550+0000","last_peered":"2024-09-21T18:58:09.902550+0000","last_clean":"2024-09-21T18:58:09.902550+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:58:09.902550+0000","last_undegraded":"2024-09-21T18:58:09.902550+0000","last_fullsized":"2024-09-21T18:58:09.902550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:58:02.629537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.629537+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:04.420331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895631+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:58:09.895631+0000","last_peered":"2024-09-21T18:58:09.895631+0000","last_clean":"2024-09-21T18:58:09.895631+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:58:09.895631+0000","last_undegraded":"2024-09-21T18:58:09.895631+0000","last_fullsized":"2024-09-21T18:58:09.895631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"159'16537","reported_seq":16820,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.581350+0000","last_change":"2024-09-21T18:57:51.070236+0000","last_active":"2024-09-21T18:58:11.581350+0000","last_peered":"2024-09-21T18:58:11.581350+0000","last_clean":"2024-09-21T18:58:11.581350+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:58:11.581350+0000","last_undegraded":"2024-09-21T18:58:11.581350+0000","last_fullsized":"2024-09-21T18:58:11.581350+0000","mapping_epoch":79,"log_start":"119'9800","ondisk_log_start":"119'9800","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15303","last_scrub_stamp":"2024-09-21T18:57:51.070161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.070161+0000","objects_scrubbed":2162,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:57.001294+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16537,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":113,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900568+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:58:09.900568+0000","last_peered":"2024-09-21T18:58:09.900568+0000","last_clean":"2024-09-21T18:58:09.900568+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:58:09.900568+0000","last_undegraded":"2024-09-21T18:58:09.900568+0000","last_fullsized":"2024-09-21T18:58:09.900568+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":231,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909094+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:58:09.909094+0000","last_peered":"2024-09-21T18:58:09.909094+0000","last_clean":"2024-09-21T18:58:09.909094+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:58:09.909094+0000","last_undegraded":"2024-09-21T18:58:09.909094+0000","last_fullsized":"2024-09-21T18:58:09.909094+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.565897+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.901347+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:58:09.901347+0000","last_peered":"2024-09-21T18:58:09.901347+0000","last_clean":"2024-09-21T18:58:09.901347+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:58:09.901347+0000","last_undegraded":"2024-09-21T18:58:09.901347+0000","last_fullsized":"2024-09-21T18:58:09.901347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.513174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.899440+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T18:58:09.899440+0000","last_peered":"2024-09-21T18:58:09.899440+0000","last_clean":"2024-09-21T18:58:09.899440+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:58:09.899440+0000","last_undegraded":"2024-09-21T18:58:09.899440+0000","last_fullsized":"2024-09-21T18:58:09.899440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.098728+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"159'16460","reported_seq":29614,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.518744+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:58:11.518744+0000","last_peered":"2024-09-21T18:58:11.518744+0000","last_clean":"2024-09-21T18:58:11.518744+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:58:11.518744+0000","last_undegraded":"2024-09-21T18:58:11.518744+0000","last_fullsized":"2024-09-21T18:58:11.518744+0000","mapping_epoch":134,"log_start":"124'10900","ondisk_log_start":"124'10900","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.024781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63700992,"num_objects":972,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16460,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":249,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.915929+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:58:09.915929+0000","last_peered":"2024-09-21T18:58:09.915929+0000","last_clean":"2024-09-21T18:58:09.915929+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:58:09.915929+0000","last_undegraded":"2024-09-21T18:58:09.915929+0000","last_fullsized":"2024-09-21T18:58:09.915929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900682+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:58:09.900682+0000","last_peered":"2024-09-21T18:58:09.900682+0000","last_clean":"2024-09-21T18:58:09.900682+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:58:09.900682+0000","last_undegraded":"2024-09-21T18:58:09.900682+0000","last_fullsized":"2024-09-21T18:58:09.900682+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.169542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.902411+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:58:09.902411+0000","last_peered":"2024-09-21T18:58:09.902411+0000","last_clean":"2024-09-21T18:58:09.902411+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:58:09.902411+0000","last_undegraded":"2024-09-21T18:58:09.902411+0000","last_fullsized":"2024-09-21T18:58:09.902411+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900102+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T18:58:09.900102+0000","last_peered":"2024-09-21T18:58:09.900102+0000","last_clean":"2024-09-21T18:58:09.900102+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:58:09.900102+0000","last_undegraded":"2024-09-21T18:58:09.900102+0000","last_fullsized":"2024-09-21T18:58:09.900102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.190036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"159'16209","reported_seq":16382,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:12.164574+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:58:12.164574+0000","last_peered":"2024-09-21T18:58:12.164574+0000","last_clean":"2024-09-21T18:58:12.164574+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:58:12.164574+0000","last_undegraded":"2024-09-21T18:58:12.164574+0000","last_fullsized":"2024-09-21T18:58:12.164574+0000","mapping_epoch":79,"log_start":"127'11500","ondisk_log_start":"127'11500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61014016,"num_objects":931,"num_object_clones":0,"num_object_copies":2793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16209,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":261,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895174+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:58:09.895174+0000","last_peered":"2024-09-21T18:58:09.895174+0000","last_clean":"2024-09-21T18:58:09.895174+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:58:09.895174+0000","last_undegraded":"2024-09-21T18:58:09.895174+0000","last_fullsized":"2024-09-21T18:58:09.895174+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.902407+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:58:09.902407+0000","last_peered":"2024-09-21T18:58:09.902407+0000","last_clean":"2024-09-21T18:58:09.902407+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:58:09.902407+0000","last_undegraded":"2024-09-21T18:58:09.902407+0000","last_fullsized":"2024-09-21T18:58:09.902407+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":237,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904073+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:58:09.904073+0000","last_peered":"2024-09-21T18:58:09.904073+0000","last_clean":"2024-09-21T18:58:09.904073+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:58:09.904073+0000","last_undegraded":"2024-09-21T18:58:09.904073+0000","last_fullsized":"2024-09-21T18:58:09.904073+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907970+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:58:09.907970+0000","last_peered":"2024-09-21T18:58:09.907970+0000","last_clean":"2024-09-21T18:58:09.907970+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:58:09.907970+0000","last_undegraded":"2024-09-21T18:58:09.907970+0000","last_fullsized":"2024-09-21T18:58:09.907970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"159'16190","reported_seq":16363,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:11.729299+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:58:11.729299+0000","last_peered":"2024-09-21T18:58:11.729299+0000","last_clean":"2024-09-21T18:58:11.729299+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:58:11.729299+0000","last_undegraded":"2024-09-21T18:58:11.729299+0000","last_fullsized":"2024-09-21T18:58:11.729299+0000","mapping_epoch":79,"log_start":"125'11000","ondisk_log_start":"125'11000","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5190,"log_dups_size":0,"ondisk_log_size":5190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:29.184091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817408,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16190,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":260,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908590+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T18:58:09.908590+0000","last_peered":"2024-09-21T18:58:09.908590+0000","last_clean":"2024-09-21T18:58:09.908590+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:58:09.908590+0000","last_undegraded":"2024-09-21T18:58:09.908590+0000","last_fullsized":"2024-09-21T18:58:09.908590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.873039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":260,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900503+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T18:58:09.900503+0000","last_peered":"2024-09-21T18:58:09.900503+0000","last_clean":"2024-09-21T18:58:09.900503+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:58:09.900503+0000","last_undegraded":"2024-09-21T18:58:09.900503+0000","last_fullsized":"2024-09-21T18:58:09.900503+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:09.842647+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":294,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.899963+0000","last_change":"2024-09-21T18:57:23.116600+0000","last_active":"2024-09-21T18:58:09.899963+0000","last_peered":"2024-09-21T18:58:09.899963+0000","last_clean":"2024-09-21T18:58:09.899963+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:58:09.899963+0000","last_undegraded":"2024-09-21T18:58:09.899963+0000","last_fullsized":"2024-09-21T18:58:09.899963+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:57:23.116520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:23.116520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.889890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":225,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908738+0000","last_change":"2024-09-21T18:55:51.525074+0000","last_active":"2024-09-21T18:58:09.908738+0000","last_peered":"2024-09-21T18:58:09.908738+0000","last_clean":"2024-09-21T18:58:09.908738+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:58:09.908738+0000","last_undegraded":"2024-09-21T18:58:09.908738+0000","last_fullsized":"2024-09-21T18:58:09.908738+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.051292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.051292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:26.930637+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":236,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.915861+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:58:09.915861+0000","last_peered":"2024-09-21T18:58:09.915861+0000","last_clean":"2024-09-21T18:58:09.915861+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:58:09.915861+0000","last_undegraded":"2024-09-21T18:58:09.915861+0000","last_fullsized":"2024-09-21T18:58:09.915861+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909861+0000","last_change":"2024-09-21T18:58:04.758385+0000","last_active":"2024-09-21T18:58:09.909861+0000","last_peered":"2024-09-21T18:58:09.909861+0000","last_clean":"2024-09-21T18:58:09.909861+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:58:09.909861+0000","last_undegraded":"2024-09-21T18:58:09.909861+0000","last_fullsized":"2024-09-21T18:58:09.909861+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T18:58:04.757944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.757944+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:23.873942+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":231,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909566+0000","last_change":"2024-09-21T18:58:01.738746+0000","last_active":"2024-09-21T18:58:09.909566+0000","last_peered":"2024-09-21T18:58:09.909566+0000","last_clean":"2024-09-21T18:58:09.909566+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:58:09.909566+0000","last_undegraded":"2024-09-21T18:58:09.909566+0000","last_fullsized":"2024-09-21T18:58:09.909566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:01.738627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.738627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.225962+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900204+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T18:58:09.900204+0000","last_peered":"2024-09-21T18:58:09.900204+0000","last_clean":"2024-09-21T18:58:09.900204+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:58:09.900204+0000","last_undegraded":"2024-09-21T18:58:09.900204+0000","last_fullsized":"2024-09-21T18:58:09.900204+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.935335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904337+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:58:09.904337+0000","last_peered":"2024-09-21T18:58:09.904337+0000","last_clean":"2024-09-21T18:58:09.904337+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:58:09.904337+0000","last_undegraded":"2024-09-21T18:58:09.904337+0000","last_fullsized":"2024-09-21T18:58:09.904337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900090+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:58:09.900090+0000","last_peered":"2024-09-21T18:58:09.900090+0000","last_clean":"2024-09-21T18:58:09.900090+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:58:09.900090+0000","last_undegraded":"2024-09-21T18:58:09.900090+0000","last_fullsized":"2024-09-21T18:58:09.900090+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":216,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909723+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:58:09.909723+0000","last_peered":"2024-09-21T18:58:09.909723+0000","last_clean":"2024-09-21T18:58:09.909723+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:58:09.909723+0000","last_undegraded":"2024-09-21T18:58:09.909723+0000","last_fullsized":"2024-09-21T18:58:09.909723+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.901182+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:58:09.901182+0000","last_peered":"2024-09-21T18:58:09.901182+0000","last_clean":"2024-09-21T18:58:09.901182+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:58:09.901182+0000","last_undegraded":"2024-09-21T18:58:09.901182+0000","last_fullsized":"2024-09-21T18:58:09.901182+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:37.565916+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":235,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909289+0000","last_change":"2024-09-21T18:57:25.890791+0000","last_active":"2024-09-21T18:58:09.909289+0000","last_peered":"2024-09-21T18:58:09.909289+0000","last_clean":"2024-09-21T18:58:09.909289+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:58:09.909289+0000","last_undegraded":"2024-09-21T18:58:09.909289+0000","last_fullsized":"2024-09-21T18:58:09.909289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:25.890710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.890710+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:51.251415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900061+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T18:58:09.900061+0000","last_peered":"2024-09-21T18:58:09.900061+0000","last_clean":"2024-09-21T18:58:09.900061+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:58:09.900061+0000","last_undegraded":"2024-09-21T18:58:09.900061+0000","last_fullsized":"2024-09-21T18:58:09.900061+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.445017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":285,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907175+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:58:09.907175+0000","last_peered":"2024-09-21T18:58:09.907175+0000","last_clean":"2024-09-21T18:58:09.907175+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:58:09.907175+0000","last_undegraded":"2024-09-21T18:58:09.907175+0000","last_fullsized":"2024-09-21T18:58:09.907175+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":211,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895723+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:58:09.895723+0000","last_peered":"2024-09-21T18:58:09.895723+0000","last_clean":"2024-09-21T18:58:09.895723+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:58:09.895723+0000","last_undegraded":"2024-09-21T18:58:09.895723+0000","last_fullsized":"2024-09-21T18:58:09.895723+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":227,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907455+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:58:09.907455+0000","last_peered":"2024-09-21T18:58:09.907455+0000","last_clean":"2024-09-21T18:58:09.907455+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:58:09.907455+0000","last_undegraded":"2024-09-21T18:58:09.907455+0000","last_fullsized":"2024-09-21T18:58:09.907455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900284+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:58:09.900284+0000","last_peered":"2024-09-21T18:58:09.900284+0000","last_clean":"2024-09-21T18:58:09.900284+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:58:09.900284+0000","last_undegraded":"2024-09-21T18:58:09.900284+0000","last_fullsized":"2024-09-21T18:58:09.900284+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:19.429722+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":306,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909204+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T18:58:09.909204+0000","last_peered":"2024-09-21T18:58:09.909204+0000","last_clean":"2024-09-21T18:58:09.909204+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:58:09.909204+0000","last_undegraded":"2024-09-21T18:58:09.909204+0000","last_fullsized":"2024-09-21T18:58:09.909204+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:13.388257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":102,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904688+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:58:09.904688+0000","last_peered":"2024-09-21T18:58:09.904688+0000","last_clean":"2024-09-21T18:58:09.904688+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:58:09.904688+0000","last_undegraded":"2024-09-21T18:58:09.904688+0000","last_fullsized":"2024-09-21T18:58:09.904688+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":257,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895294+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:58:09.895294+0000","last_peered":"2024-09-21T18:58:09.895294+0000","last_clean":"2024-09-21T18:58:09.895294+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:58:09.895294+0000","last_undegraded":"2024-09-21T18:58:09.895294+0000","last_fullsized":"2024-09-21T18:58:09.895294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":223,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.895478+0000","last_change":"2024-09-21T18:57:57.837867+0000","last_active":"2024-09-21T18:58:09.895478+0000","last_peered":"2024-09-21T18:58:09.895478+0000","last_clean":"2024-09-21T18:58:09.895478+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:58:09.895478+0000","last_undegraded":"2024-09-21T18:58:09.895478+0000","last_fullsized":"2024-09-21T18:58:09.895478+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:57.837826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:57.837826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:58.681714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":215,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.914842+0000","last_change":"2024-09-21T18:57:25.184491+0000","last_active":"2024-09-21T18:58:09.914842+0000","last_peered":"2024-09-21T18:58:09.914842+0000","last_clean":"2024-09-21T18:58:09.914842+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:58:09.914842+0000","last_undegraded":"2024-09-21T18:58:09.914842+0000","last_fullsized":"2024-09-21T18:58:09.914842+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:25.180690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.180690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:31.159574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":226,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907445+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T18:58:09.907445+0000","last_peered":"2024-09-21T18:58:09.907445+0000","last_clean":"2024-09-21T18:58:09.907445+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:58:09.907445+0000","last_undegraded":"2024-09-21T18:58:09.907445+0000","last_fullsized":"2024-09-21T18:58:09.907445+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.201536+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":243,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.915136+0000","last_change":"2024-09-21T18:57:49.327777+0000","last_active":"2024-09-21T18:58:09.915136+0000","last_peered":"2024-09-21T18:58:09.915136+0000","last_clean":"2024-09-21T18:58:09.915136+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:58:09.915136+0000","last_undegraded":"2024-09-21T18:58:09.915136+0000","last_fullsized":"2024-09-21T18:58:09.915136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:49.327123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.327123+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:56.940918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900624+0000","last_change":"2024-09-21T18:57:19.082962+0000","last_active":"2024-09-21T18:58:09.900624+0000","last_peered":"2024-09-21T18:58:09.900624+0000","last_clean":"2024-09-21T18:58:09.900624+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:58:09.900624+0000","last_undegraded":"2024-09-21T18:58:09.900624+0000","last_fullsized":"2024-09-21T18:58:09.900624+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.082873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.082873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:21.905947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908045+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:58:09.908045+0000","last_peered":"2024-09-21T18:58:09.908045+0000","last_clean":"2024-09-21T18:58:09.908045+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:58:09.908045+0000","last_undegraded":"2024-09-21T18:58:09.908045+0000","last_fullsized":"2024-09-21T18:58:09.908045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.898032+0000","last_change":"2024-09-21T18:57:58.019346+0000","last_active":"2024-09-21T18:58:09.898032+0000","last_peered":"2024-09-21T18:58:09.898032+0000","last_clean":"2024-09-21T18:58:09.898032+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:58:09.898032+0000","last_undegraded":"2024-09-21T18:58:09.898032+0000","last_fullsized":"2024-09-21T18:58:09.898032+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.019278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.019278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.581215+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":239,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.907775+0000","last_change":"2024-09-21T18:58:01.818014+0000","last_active":"2024-09-21T18:58:09.907775+0000","last_peered":"2024-09-21T18:58:09.907775+0000","last_clean":"2024-09-21T18:58:09.907775+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:58:09.907775+0000","last_undegraded":"2024-09-21T18:58:09.907775+0000","last_fullsized":"2024-09-21T18:58:09.907775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:58:01.817792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.817792+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:30.058894+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":223,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.909598+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:58:09.909598+0000","last_peered":"2024-09-21T18:58:09.909598+0000","last_clean":"2024-09-21T18:58:09.909598+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:58:09.909598+0000","last_undegraded":"2024-09-21T18:58:09.909598+0000","last_fullsized":"2024-09-21T18:58:09.909598+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.605024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.899740+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:58:09.899740+0000","last_peered":"2024-09-21T18:58:09.899740+0000","last_clean":"2024-09-21T18:58:09.899740+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:58:09.899740+0000","last_undegraded":"2024-09-21T18:58:09.899740+0000","last_fullsized":"2024-09-21T18:58:09.899740+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.212178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":226,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.908172+0000","last_change":"2024-09-21T18:57:54.750968+0000","last_active":"2024-09-21T18:58:09.908172+0000","last_peered":"2024-09-21T18:58:09.908172+0000","last_clean":"2024-09-21T18:58:09.908172+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:58:09.908172+0000","last_undegraded":"2024-09-21T18:58:09.908172+0000","last_fullsized":"2024-09-21T18:58:09.908172+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:54.750896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.750896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.812996+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":231,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.919939+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:58:09.919939+0000","last_peered":"2024-09-21T18:58:09.919939+0000","last_clean":"2024-09-21T18:58:09.919939+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:58:09.919939+0000","last_undegraded":"2024-09-21T18:58:09.919939+0000","last_fullsized":"2024-09-21T18:58:09.919939+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":110,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.900705+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T18:58:09.900705+0000","last_peered":"2024-09-21T18:58:09.900705+0000","last_clean":"2024-09-21T18:58:09.900705+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:58:09.900705+0000","last_undegraded":"2024-09-21T18:58:09.900705+0000","last_fullsized":"2024-09-21T18:58:09.900705+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.894402+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.915807+0000","last_change":"2024-09-21T18:57:55.356422+0000","last_active":"2024-09-21T18:58:09.915807+0000","last_peered":"2024-09-21T18:58:09.915807+0000","last_clean":"2024-09-21T18:58:09.915807+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:58:09.915807+0000","last_undegraded":"2024-09-21T18:58:09.915807+0000","last_fullsized":"2024-09-21T18:58:09.915807+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:55.356360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:55.356360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:04.334971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.899875+0000","last_change":"2024-09-21T18:57:53.263833+0000","last_active":"2024-09-21T18:58:09.899875+0000","last_peered":"2024-09-21T18:58:09.899875+0000","last_clean":"2024-09-21T18:58:09.899875+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:58:09.899875+0000","last_undegraded":"2024-09-21T18:58:09.899875+0000","last_fullsized":"2024-09-21T18:58:09.899875+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.263783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.263783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.950038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.914355+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:58:09.914355+0000","last_peered":"2024-09-21T18:58:09.914355+0000","last_clean":"2024-09-21T18:58:09.914355+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:58:09.914355+0000","last_undegraded":"2024-09-21T18:58:09.914355+0000","last_fullsized":"2024-09-21T18:58:09.914355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:48.365341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":238,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.920198+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:58:09.920198+0000","last_peered":"2024-09-21T18:58:09.920198+0000","last_clean":"2024-09-21T18:58:09.920198+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:58:09.920198+0000","last_undegraded":"2024-09-21T18:58:09.920198+0000","last_fullsized":"2024-09-21T18:58:09.920198+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":237,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.916473+0000","last_change":"2024-09-21T18:57:54.359262+0000","last_active":"2024-09-21T18:58:09.916473+0000","last_peered":"2024-09-21T18:58:09.916473+0000","last_clean":"2024-09-21T18:58:09.916473+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:58:09.916473+0000","last_undegraded":"2024-09-21T18:58:09.916473+0000","last_fullsized":"2024-09-21T18:58:09.916473+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:57:54.359175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.359175+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.117172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-21T18:58:09.904600+0000","last_change":"2024-09-21T18:57:58.985012+0000","last_active":"2024-09-21T18:58:09.904600+0000","last_peered":"2024-09-21T18:58:09.904600+0000","last_clean":"2024-09-21T18:58:09.904600+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:58:09.904600+0000","last_undegraded":"2024-09-21T18:58:09.904600+0000","last_fullsized":"2024-09-21T18:58:09.904600+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.984945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.984945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:03.530436+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":983367704,"num_objects":15006,"num_object_clones":0,"num_object_copies":45018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262988,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35974,"num_bytes_recovered":2357461016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3282644992,"data_stored":3282632776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89686,"ondisk_log_size":89686,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":293,"num_read_kb":288,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266753,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625812,"kb_used_data":486336,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93119340,"statfs":{"total":95995035648,"available":95354204160,"internally_reserved":0,"allocated":498008064,"data_stored":497520568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528390,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":447684,"kb_used_data":304052,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93297468,"statfs":{"total":95995035648,"available":95536607232,"internally_reserved":0,"allocated":311349248,"data_stored":310873415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543236,"kb_used_data":413488,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93201916,"statfs":{"total":95995035648,"available":95438761984,"internally_reserved":0,"allocated":423411712,"data_stored":422939537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018959,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461080,"kb_used_data":317444,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93284072,"statfs":{"total":95995035648,"available":95522889728,"internally_reserved":0,"allocated":325062656,"data_stored":324571020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799332,"kb_used_data":533328,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92945820,"statfs":{"total":95995035648,"available":95176519680,"internally_reserved":0,"allocated":546127872,"data_stored":545624247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":272169418},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":486988,"kb_used_data":361400,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93258164,"statfs":{"total":95995035648,"available":95496359936,"internally_reserved":0,"allocated":370073600,"data_stored":369593788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":558244,"kb_used_data":378512,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93186908,"statfs":{"total":95995035648,"available":95423393792,"internally_reserved":0,"allocated":387596288,"data_stored":387092241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758172,"kb_used_data":510344,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92986980,"statfs":{"total":95995035648,"available":95218667520,"internally_reserved":0,"allocated":522592256,"data_stored":522097281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":500633600,"data_stored":500629528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":373886976,"data_stored":373882904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368836608,"data_stored":368836608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":530907136,"data_stored":530907136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":309919744,"data_stored":309919744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":403701760,"data_stored":403701760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":306778112,"data_stored":306774040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":487981056,"data_stored":487981056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:58:16.254 INFO:tasks.thrashosds.thrasher:Setting 4.1d to [0, 1, 4, 6, 7, 3] 2024-09-21T18:58:16.255 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.1d', '0', '1', '4', '6', '7', '3'] 2024-09-21T18:58:16.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 4.1d 0 1 4 6 7 3 2024-09-21T18:58:16.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:15 smithi104 ceph-mon[32438]: from='client.25111 v1:172.21.15.81:0/1639289077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:16.266 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:16.395 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:16.523 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:16.576 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:16.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:16.704 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:16.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:16.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:16.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:16.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:16.950 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:16.950 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:17.009 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:17.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:17.137 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:16 smithi081 ceph-mon[29384]: pgmap v230: 148 pgs: 148 active+clean; 938 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 674 B/s rd, 0 B/s wr, 696 op/s 2024-09-21T18:58:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:16 smithi081 ceph-mon[39192]: pgmap v230: 148 pgs: 148 active+clean; 938 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 674 B/s rd, 0 B/s wr, 696 op/s 2024-09-21T18:58:17.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:17.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:16 smithi104 ceph-mon[32438]: pgmap v230: 148 pgs: 148 active+clean; 938 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 674 B/s rd, 0 B/s wr, 696 op/s 2024-09-21T18:58:17.264 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:17.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:17.398 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:17.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:17.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:58:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:58:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:58:17.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:17.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:17.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T18:58:17.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.839 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:17.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:17.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:17.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:17.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:18.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:18.092 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:18.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:18.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:18.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:58:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:58:18.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:18.377 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:18.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:18.506 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:18.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.638 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:18.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:18.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:58:18.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:18.764 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:18.866 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:18.893 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:18.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:18.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:19.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:18 smithi081 ceph-mon[39192]: pgmap v231: 148 pgs: 148 active+clean; 623 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T18:58:19.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:19.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:18 smithi081 ceph-mon[29384]: pgmap v231: 148 pgs: 148 active+clean; 623 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T18:58:19.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:19.017 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:19.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:19.144 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:19.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:18 smithi104 ceph-mon[32438]: pgmap v231: 148 pgs: 148 active+clean; 623 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T18:58:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:19.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:19.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:19.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:19.546 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:19.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:19.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:19.799 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:19.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:19.905 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:19.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:20.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:20.062 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:20.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:20.172 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:20.240 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/599343988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:20.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/599343988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:20.289 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/599343988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:20.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:20.368 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:20.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:20.502 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:20.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:20.627 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:20.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:20.754 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:20.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:20.881 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:20.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:20.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:21.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:21.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:21.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:21.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:20 smithi104 ceph-mon[32438]: pgmap v232: 148 pgs: 148 active+clean; 623 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-21T18:58:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:21.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:20 smithi081 ceph-mon[39192]: pgmap v232: 148 pgs: 148 active+clean; 623 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-21T18:58:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:20 smithi081 ceph-mon[29384]: pgmap v232: 148 pgs: 148 active+clean; 623 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-21T18:58:21.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:21.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:21.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:21.545 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:21.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:21.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:21.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:21.958 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:21.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:22.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:22.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:22.060 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:22.085 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:22.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:22.210 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:22.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:22.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:22.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:22.461 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:22.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:22.590 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:22.612 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:58:22.612 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:274135 7.7 e160/143 7:ef92f1d0:::benchmark_data_smithi081_73391_object135136:head v 160'17120, mlcod=160'17118)", 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.435188+0000", 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0027429590000000001, 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002771187, 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435188+0000", 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435188+0000", 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435189+0000", 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5409999999999999e-06 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435196+0000", 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4509999999999997e-06 2024-09-21T18:58:22.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435210+0000", 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4107000000000001e-05 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435217+0000", 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8079999999999999e-06 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435294+0000", 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6922000000000007e-05 2024-09-21T18:58:22.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.435312+0000", 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8289000000000001e-05 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:274127 7.f e160/151)", 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.436332+0000", 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0015991309999999999, 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016843299999999999, 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.616 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436332+0000", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436332+0000", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436333+0000", 2024-09-21T18:58:22.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.221e-06 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436339+0000", 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5200000000000003e-06 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436350+0000", 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0198e-05 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436357+0000", 2024-09-21T18:58:22.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2309999999999999e-06 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274109 7.2 7:4a3fa668:::benchmark_data_smithi081_73391_object135110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.402019+0000", 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.035911480000000003, 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.036046745999999998, 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274109 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.402019+0000", 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.402019+0000", 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.402020+0000", 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8500000000000001e-07 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.402024+0000", 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4529999999999998e-06 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.402031+0000", 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4220000000000003e-06 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.402037+0000", 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1970000000000001e-06 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.424742+0000", 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022704449000000002 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.424843+0000", 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101771 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:22.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.424944+0000", 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100597 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.437872+0000", 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01292832 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:274128 7.f e160/151)", 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.436406+0000", 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001524879, 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001730868, 2024-09-21T18:58:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436406+0000", 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436406+0000", 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436407+0000", 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1600000000000001e-07 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436411+0000", 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436418+0000", 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6259999999999997e-06 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436423+0000", 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9590000000000003e-06 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:274134 7.f e160/151)", 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.436429+0000", 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001501704, 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017493980000000001, 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.626 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436429+0000", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436429+0000", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436429+0000", 2024-09-21T18:58:22.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4400000000000001e-07 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436436+0000", 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9399999999999996e-06 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436441+0000", 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.25e-06 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436444+0000", 2024-09-21T18:58:22.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6490000000000001e-06 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274111 7.2 7:49d0ed8b:::benchmark_data_smithi081_73391_object135112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.405468+0000", 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.032463095999999997, 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.032758071, 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274111 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.405468+0000", 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.405468+0000", 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.405470+0000", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.781e-06 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.405475+0000", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5210000000000002e-06 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.405490+0000", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.456e-05 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.405499+0000", 2024-09-21T18:58:22.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9989999999999997e-06 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.427506+0000", 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022007478 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.427609+0000", 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103164 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.427718+0000", 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108379 2024-09-21T18:58:22.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.437877+0000", 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010159039 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274138 7.3 7:c709c667:::benchmark_data_smithi081_73391_object135139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.436591+0000", 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0013403460000000001, 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001709143, 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.633 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274138 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436591+0000", 2024-09-21T18:58:22.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436591+0000", 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436592+0000", 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.778e-06 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436596+0000", 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1819999999999999e-06 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436606+0000", 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436612+0000", 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0449999999999997e-06 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436663+0000", 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.147e-05 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436767+0000", 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103483 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.436870+0000", 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103027 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274120 7.3 7:cd76569b:::benchmark_data_smithi081_73391_object135121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.416841+0000", 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.021090344, 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021528037, 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274120 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416841+0000", 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416841+0000", 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416843+0000", 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2390000000000001e-06 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416849+0000", 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6000000000000003e-06 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416862+0000", 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2522e-05 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416870+0000", 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5310000000000001e-06 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.417045+0000", 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000174611 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.417172+0000", 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012699 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.417294+0000", 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012234 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.419911+0000", 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026163229999999998 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.424672+0000", 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047609649999999998 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274121 7.2 7:411e38f1:::benchmark_data_smithi081_73391_object135122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.416892+0000", 2024-09-21T18:58:22.642 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.021039354, 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021554015999999999, 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274121 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416892+0000", 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416892+0000", 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416892+0000", 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9999999999999997e-07 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416896+0000", 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4410000000000002e-06 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416903+0000", 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.328e-06 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.416909+0000", 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1469999999999998e-06 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.433577+0000", 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016668127000000001 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.433759+0000", 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000181578 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.433865+0000", 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106177 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274131 7.3 7:c6f20c61:::benchmark_data_smithi081_73391_object135132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.646 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.421599+0000", 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.016331928999999999, 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016922959000000001, 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274131 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421599+0000", 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421599+0000", 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421601+0000", 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.703e-06 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421605+0000", 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8199999999999996e-06 2024-09-21T18:58:22.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421616+0000", 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0376e-05 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421623+0000", 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5129999999999999e-06 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421732+0000", 2024-09-21T18:58:22.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108646 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421838+0000", 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106179 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.421946+0000", 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010740599999999999 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.425061+0000", 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031153750000000001 2024-09-21T18:58:22.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.432688+0000", 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0076267130000000002 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:274105 7.a 7:5d84baa5:::benchmark_data_smithi081_73391_object135106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:58:22.398894+0000", 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.039036666999999997, 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.039713789999999999, 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:22.651 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 274105 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.398894+0000", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.398894+0000", 2024-09-21T18:58:22.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.398895+0000", 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.077e-06 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.398900+0000", 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.352e-06 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.398907+0000", 2024-09-21T18:58:22.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0949999999999998e-06 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.398914+0000", 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2359999999999998e-06 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.417511+0000", 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018597497000000001 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.417606+0000", 2024-09-21T18:58:22.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4505000000000004e-05 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.417706+0000", 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9592999999999993e-05 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.425032+0000", 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0073266160000000002 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:58:22.437867+0000", 2024-09-21T18:58:22.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012835113 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 11 2024-09-21T18:58:22.656 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:58:22.659 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:58:22.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:22.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:22.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:22.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:22.970 INFO:teuthology.orchestra.run.smithi081.stderr:set 4.1d pg_upmap_items mapping to [0->1,4->6,7->3] 2024-09-21T18:58:23.016 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:23.043 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:23.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:23.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:23.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:22 smithi104 ceph-mon[32438]: pgmap v233: 148 pgs: 148 active+clean; 603 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-21T18:58:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4090629667' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]} : dispatch 2024-09-21T18:58:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:22 smithi104 ceph-mon[32438]: from='client.25120 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]} : dispatch 2024-09-21T18:58:23.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:23.315 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:22 smithi081 ceph-mon[29384]: pgmap v233: 148 pgs: 148 active+clean; 603 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-21T18:58:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4090629667' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]} : dispatch 2024-09-21T18:58:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:22 smithi081 ceph-mon[29384]: from='client.25120 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]} : dispatch 2024-09-21T18:58:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:22 smithi081 ceph-mon[39192]: pgmap v233: 148 pgs: 148 active+clean; 603 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-21T18:58:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4090629667' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]} : dispatch 2024-09-21T18:58:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:22 smithi081 ceph-mon[39192]: from='client.25120 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]} : dispatch 2024-09-21T18:58:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:23.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.444 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:23.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:23.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:23.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T18:58:23.591 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:23.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:23.753 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:23.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:23.918 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:23.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:24.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:24.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:24.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.192 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:24.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:24.254 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:23 smithi081 ceph-mon[29384]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]}]': finished 2024-09-21T18:58:24.254 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:23 smithi081 ceph-mon[29384]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T18:58:24.254 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:23 smithi081 ceph-mon[39192]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]}]': finished 2024-09-21T18:58:24.255 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:23 smithi081 ceph-mon[39192]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T18:58:24.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:23 smithi104 ceph-mon[32438]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1d", "id": [0, 1, 4, 6, 7, 3]}]': finished 2024-09-21T18:58:24.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:23 smithi104 ceph-mon[32438]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T18:58:24.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:24.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:24.335 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:24.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:24.457 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:24.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:24.598 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:24.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:24.741 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:24.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:24.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:24.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.009 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:25.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:24 smithi081 ceph-mon[29384]: pgmap v235: 148 pgs: 148 active+clean; 290 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T18:58:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:24 smithi081 ceph-mon[29384]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T18:58:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:24 smithi081 ceph-mon[39192]: pgmap v235: 148 pgs: 148 active+clean; 290 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T18:58:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:24 smithi081 ceph-mon[39192]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T18:58:25.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:25.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:25.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:25.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:24 smithi104 ceph-mon[32438]: pgmap v235: 148 pgs: 148 active+clean; 290 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T18:58:25.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:24 smithi104 ceph-mon[32438]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T18:58:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:25.299 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:25.339 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:25.339 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:25.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:25.380 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:25.381 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:25.388 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:25.421 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:25.421 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:25.462 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:25.462 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:25.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:25.508 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:25.508 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:25.514 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:25.555 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:25.555 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:25.565 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:25.595 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:25.596 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:25.616 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:25.641 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:25.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:25.766 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:25.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:25.894 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:25.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:25.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:25.997 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:58:26.042 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:26.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:26.145 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Removed 138996 objects 2024-09-21T18:58:26.146 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Clean up completed and total clean up time :152.964 2024-09-21T18:58:26.164 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:26.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2595923761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:26.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3452051728' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:26.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.287 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:26.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:26.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2595923761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:26.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3452051728' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:26.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2595923761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:26.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3452051728' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:26.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:26.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:26.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:26.657 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:26.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:26.780 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:26.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:26.904 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:26.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:27.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:27.028 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:27.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[39192]: pgmap v237: 148 pgs: 148 active+clean; 290 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 666 op/s 2024-09-21T18:58:27.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:27.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3452051728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:58:27.033 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[39192]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T18:58:27.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[29384]: pgmap v237: 148 pgs: 148 active+clean; 290 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 666 op/s 2024-09-21T18:58:27.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:27.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3452051728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:58:27.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:26 smithi081 ceph-mon[29384]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T18:58:27.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:27.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:27.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:26 smithi104 ceph-mon[32438]: pgmap v237: 148 pgs: 148 active+clean; 290 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 666 op/s 2024-09-21T18:58:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:26 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3452051728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:58:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:26 smithi104 ceph-mon[32438]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T18:58:27.274 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:27.301 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:58:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:58:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:58:27.376 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:27.398 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:27.500 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:27.523 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:27.595 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:27.595 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:27.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:27.645 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:27.747 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:27.768 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:27.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:27.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:27.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:28.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:28.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:28.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:28.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:28.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:27 smithi104 ceph-mon[32438]: 3.6 scrub starts 2024-09-21T18:58:28.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:27 smithi104 ceph-mon[32438]: 3.6 scrub ok 2024-09-21T18:58:28.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:58:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:58:28.274 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:27 smithi081 ceph-mon[39192]: 3.6 scrub starts 2024-09-21T18:58:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:27 smithi081 ceph-mon[39192]: 3.6 scrub ok 2024-09-21T18:58:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:27 smithi081 ceph-mon[29384]: 3.6 scrub starts 2024-09-21T18:58:28.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:27 smithi081 ceph-mon[29384]: 3.6 scrub ok 2024-09-21T18:58:28.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:28.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:58:28.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:28.523 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:28.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:28.648 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:28.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T18:58:28.716 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:28.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:28.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:28.897 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:28.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:28.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.022 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:29.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:29.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:29.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:29.227 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:29.227 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:29.245 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:29.246 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:29.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[29384]: pgmap v239: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.61k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T18:58:29.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:29.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[29384]: 3.18 scrub starts 2024-09-21T18:58:29.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[29384]: 3.18 scrub ok 2024-09-21T18:58:29.247 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[39192]: pgmap v239: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.61k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T18:58:29.247 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:29.247 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[39192]: 3.18 scrub starts 2024-09-21T18:58:29.247 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:28 smithi081 ceph-mon[39192]: 3.18 scrub ok 2024-09-21T18:58:29.248 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:29.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:28 smithi104 ceph-mon[32438]: pgmap v239: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.61k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T18:58:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:28 smithi104 ceph-mon[32438]: 3.18 scrub starts 2024-09-21T18:58:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:28 smithi104 ceph-mon[32438]: 3.18 scrub ok 2024-09-21T18:58:29.263 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:29.280 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:29.280 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:29.297 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:29.297 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:29.299 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:58:29.299 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:58:29.299 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T18:58:29.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T18:58:29.314 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:29.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:58:29.331 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:29.332 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:29.349 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:29.349 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:29.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:29.397 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:29.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:29.522 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:29.624 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:29.647 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:29.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:29.772 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:29.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:29.896 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:29.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:29.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:30.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:30.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:30.143 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:30.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:29 smithi104 ceph-mon[32438]: 5.1c scrub starts 2024-09-21T18:58:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:29 smithi104 ceph-mon[32438]: 5.1c scrub ok 2024-09-21T18:58:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:29 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T18:58:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:29 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T18:58:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2431967877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:29 smithi104 ceph-mon[32438]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T18:58:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[29384]: 5.1c scrub starts 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[29384]: 5.1c scrub ok 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2431967877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[29384]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[39192]: 5.1c scrub starts 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[39192]: 5.1c scrub ok 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T18:58:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T18:58:30.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2431967877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:30.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:29 smithi081 ceph-mon[39192]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T18:58:30.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:30.390 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145372 7.f 7:f3db53f7:::benchmark_data_smithi081_73391_object6373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:58:30.487 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.333907+0000", 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 151.141492137, 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.132201821, 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145372 2024-09-21T18:58:30.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333907+0000", 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333907+0000", 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333908+0000", 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1289999999999999e-06 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333912+0000", 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7720000000000001e-06 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333918+0000", 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6059999999999999e-06 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333923+0000", 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1220000000000003e-06 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431117+0000", 2024-09-21T18:58:30.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097193494000000005 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431239+0000", 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012223400000000001 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431352+0000", 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011341900000000001 2024-09-21T18:58:30.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462090+0000", 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030737417 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462135+0000", 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5228999999999997e-05 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462174+0000", 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9218000000000003e-05 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.465927+0000", 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037523449999999998 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.465992+0000", 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5376999999999994e-05 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.466108+0000", 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116459 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:211482 7.e e138/114 7:7d19c424:::benchmark_data_smithi081_73391_object72483:head v 138'13046, mlcod=138'13043)", 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.060608+0000", 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 76.414790503999996, 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13969134899999999, 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060608+0000", 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060608+0000", 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060609+0000", 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6200000000000006e-07 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060615+0000", 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4049999999999999e-06 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060622+0000", 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.712e-06 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060628+0000", 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0120000000000002e-06 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.175684+0000", 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.115055169 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.175703+0000", 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9955000000000001e-05 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.199415+0000", 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.023711368 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200300+0000", 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00088476600000000005 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.500 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:211485 7.f 7:f98a38d8:::benchmark_data_smithi081_73391_object72486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.060874+0000", 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 76.414524338000007, 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.143099278, 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.501 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 211485 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060874+0000", 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060874+0000", 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060876+0000", 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2270000000000001e-06 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060880+0000", 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6120000000000001e-06 2024-09-21T18:58:30.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.173490+0000", 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.1126096 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.173500+0000", 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0583e-05 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.179620+0000", 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006119691 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.179760+0000", 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013987099999999999 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.179898+0000", 2024-09-21T18:58:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013791699999999999 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.184545+0000", 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046469789999999999 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.197397+0000", 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012852542999999999 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200323+0000", 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002925108 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.203537+0000", 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032148229999999999 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.203599+0000", 2024-09-21T18:58:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1186000000000007e-05 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.203974+0000", 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037513800000000001 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242073 7.3 7:c6fda3ec:::benchmark_data_smithi081_73391_object103074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.501575+0000", 2024-09-21T18:58:30.508 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.973823707000001, 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17268383800000001, 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242073 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501575+0000", 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501575+0000", 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501577+0000", 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1289999999999999e-06 2024-09-21T18:58:30.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501583+0000", 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4500000000000003e-06 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501595+0000", 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2077e-05 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501603+0000", 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3399999999999998e-06 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501722+0000", 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118658 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501830+0000", 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010876 2024-09-21T18:58:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501939+0000", 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108435 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511744+0000", 2024-09-21T18:58:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0098048200000000005 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511797+0000", 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3733999999999997e-05 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674170+0000", 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16237236399999999 2024-09-21T18:58:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674221+0000", 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1059e-05 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674259+0000", 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8012000000000001e-05 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242076 7.a 7:5697a0ec:::benchmark_data_smithi081_73391_object103077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.501662+0000", 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.973736768999999, 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.173717652, 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242076 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501662+0000", 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501662+0000", 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501662+0000", 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8999999999999997e-07 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501665+0000", 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.424e-06 2024-09-21T18:58:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501670+0000", 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5949999999999998e-06 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501681+0000", 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0934000000000001e-05 2024-09-21T18:58:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502591+0000", 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000909408 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502695+0000", 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104381 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502808+0000", 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011254799999999999 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511271+0000", 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084630829999999997 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512006+0000", 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00073563299999999997 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674954+0000", 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16294760899999999 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.675342+0000", 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038748999999999999 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.675380+0000", 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8056999999999998e-05 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242074 7.0 e148/79 7:061b77dd:::benchmark_data_smithi081_73391_object103075:head v 148'15370, mlcod=148'15368)", 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.502435+0000", 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.972964036, 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17104414200000001, 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502435+0000", 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502435+0000", 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502435+0000", 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8100000000000002e-07 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502441+0000", 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.31e-06 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502446+0000", 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5219999999999998e-06 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502450+0000", 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2490000000000003e-06 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509857+0000", 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074064150000000004 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509877+0000", 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0219000000000001e-05 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673418+0000", 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16354082 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673479+0000", 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0925999999999999e-05 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242079 7.3 7:cff90720:::benchmark_data_smithi081_73391_object103080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.503647+0000", 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.971751709999999, 2024-09-21T18:58:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.170710009, 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242079 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503647+0000", 2024-09-21T18:58:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503647+0000", 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503649+0000", 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8729999999999999e-06 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503654+0000", 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.541e-06 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503666+0000", 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1511e-05 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503674+0000", 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2670000000000006e-06 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.504624+0000", 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00095014000000000003 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.504730+0000", 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105939 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.504835+0000", 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010462 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511778+0000", 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0069427839999999996 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512126+0000", 2024-09-21T18:58:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034867699999999998 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674292+0000", 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16216533999999999 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674329+0000", 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6977999999999997e-05 2024-09-21T18:58:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674357+0000", 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8339000000000001e-05 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242080 7.1 e149/79 7:892cd020:::benchmark_data_smithi081_73391_object103081:head v 149'15044, mlcod=148'15042)", 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.506111+0000", 2024-09-21T18:58:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.969287545, 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.166020997, 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506111+0000", 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506111+0000", 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506112+0000", 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0559999999999999e-06 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506118+0000", 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5999999999999997e-06 2024-09-21T18:58:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506127+0000", 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.465e-06 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506134+0000", 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7360000000000004e-06 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.507428+0000", 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012939819999999999 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.507447+0000", 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9091999999999999e-05 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672081+0000", 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.164633475 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672132+0000", 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1591000000000001e-05 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242081 7.7 e149/143 7:eb350bf9:::benchmark_data_smithi081_73391_object103082:head v 149'15160, mlcod=148'15158)", 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.509291+0000", 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.966107974000003, 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16315574999999999, 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509291+0000", 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509291+0000", 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509291+0000", 2024-09-21T18:58:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7800000000000001e-07 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509294+0000", 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.258e-06 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509299+0000", 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.198e-06 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509304+0000", 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5120000000000002e-06 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509344+0000", 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0198999999999999e-05 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509360+0000", 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5909e-05 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672405+0000", 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.163044511 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672447+0000", 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1984999999999999e-05 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242082 7.1 e149/79 7:83ef7caf:::benchmark_data_smithi081_73391_object103083:head v 149'15045, mlcod=148'15042)", 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.510875+0000", 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.964524216000001, 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16130624199999999, 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510875+0000", 2024-09-21T18:58:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510875+0000", 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510875+0000", 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4900000000000005e-07 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510881+0000", 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8490000000000002e-06 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510888+0000", 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1060000000000002e-06 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510893+0000", 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9400000000000001e-06 2024-09-21T18:58:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511398+0000", 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00050453499999999997 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511417+0000", 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9273e-05 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672136+0000", 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16071918600000001 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672181+0000", 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4604000000000002e-05 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242087 7.d e149/114 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head v 149'15301, mlcod=148'15299)", 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.513200+0000", 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.962198622999999, 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20102450699999999, 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513200+0000", 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513200+0000", 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513202+0000", 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.403e-06 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513208+0000", 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5769999999999999e-06 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672834+0000", 2024-09-21T18:58:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.15962562899999999 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672843+0000", 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3409999999999992e-06 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702934+0000", 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030091353000000001 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702958+0000", 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4054e-05 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.714194+0000", 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011235121000000001 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.714225+0000", 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1028999999999997e-05 2024-09-21T18:58:30.557 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242089 7.2 7:47cc2c63:::benchmark_data_smithi081_73391_object103090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.513246+0000", 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.962152867, 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19053052500000001, 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.558 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242089 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513246+0000", 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513246+0000", 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513248+0000", 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.759e-06 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513252+0000", 2024-09-21T18:58:30.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673414+0000", 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16016180499999999 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673424+0000", 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.893e-06 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673481+0000", 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6764000000000001e-05 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673610+0000", 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012902599999999999 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673721+0000", 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110931 2024-09-21T18:58:30.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.699219+0000", 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025498257 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702051+0000", 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028315860000000001 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702478+0000", 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00042716299999999999 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.703734+0000", 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012565180000000001 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.703776+0000", 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1999999999999998e-05 2024-09-21T18:58:30.564 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242083 7.1 e149/79 7:8f0bc0f0:::benchmark_data_smithi081_73391_object103084:head v 149'15046, mlcod=148'15042)", 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.514317+0000", 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.961081870000001, 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.18463781100000001, 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514317+0000", 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514317+0000", 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514318+0000", 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2699999999999999e-06 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514325+0000", 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8820000000000003e-06 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.671916+0000", 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.15759063000000001 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.671926+0000", 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0431000000000001e-05 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.695786+0000", 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.023859405 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.695808+0000", 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2393000000000001e-05 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.698910+0000", 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031024149999999999 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.698955+0000", 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4385000000000003e-05 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.569 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242810 7.a 7:50694f8c:::benchmark_data_smithi081_73391_object103811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.341722+0000", 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.133676596999997, 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136267994, 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.570 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242810 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341722+0000", 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341722+0000", 2024-09-21T18:58:30.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341724+0000", 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.609e-06 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341729+0000", 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3870000000000003e-06 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341739+0000", 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0122000000000001e-05 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341746+0000", 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.55e-06 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.343863+0000", 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021173540000000001 2024-09-21T18:58:30.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.343965+0000", 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101864 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.344070+0000", 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105202 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.347901+0000", 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038303790000000001 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348980+0000", 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010796 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477914+0000", 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.128933728 2024-09-21T18:58:30.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477948+0000", 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3618000000000002e-05 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477990+0000", 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2580999999999998e-05 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.576 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242814 7.2 7:4a8d833a:::benchmark_data_smithi081_73391_object103815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.348817+0000", 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.126582173999999, 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13874188600000001, 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.577 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242814 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348817+0000", 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348817+0000", 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348818+0000", 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3400000000000001e-06 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348822+0000", 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8419999999999998e-06 2024-09-21T18:58:30.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348828+0000", 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6109999999999999e-06 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348834+0000", 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4940000000000003e-06 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.580 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349009+0000", 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017474600000000001 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349111+0000", 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102105 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349211+0000", 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010063799999999999 2024-09-21T18:58:30.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351733+0000", 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025221409999999999 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477886+0000", 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12615274400000001 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487304+0000", 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0094181649999999992 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487511+0000", 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020704799999999999 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487558+0000", 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7012000000000003e-05 2024-09-21T18:58:30.583 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242817 7.3 7:c6315629:::benchmark_data_smithi081_73391_object103818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.348843+0000", 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.126555351999997, 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13292791100000001, 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.584 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242817 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348843+0000", 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348843+0000", 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348844+0000", 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8699999999999995e-07 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348846+0000", 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.181e-06 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348851+0000", 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6129999999999997e-06 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348855+0000", 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3529999999999997e-06 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351592+0000", 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027369949999999999 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351697+0000", 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010456800000000001 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351802+0000", 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010498800000000001 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354541+0000", 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027390589999999999 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.473798+0000", 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.119257029 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481514+0000", 2024-09-21T18:58:30.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0077163730000000003 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481725+0000", 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000210762 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481771+0000", 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6502999999999998e-05 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.590 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242821 7.a 7:5657abd2:::benchmark_data_smithi081_73391_object103822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.348867+0000", 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.126531788000001, 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.138406739, 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242821 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348867+0000", 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348867+0000", 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348867+0000", 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0999999999999999e-07 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348870+0000", 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1859999999999999e-06 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348874+0000", 2024-09-21T18:58:30.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6469999999999997e-06 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348878+0000", 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1869999999999999e-06 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350332+0000", 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001453338 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350437+0000", 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010491499999999999 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350555+0000", 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118657 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353037+0000", 2024-09-21T18:58:30.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024819899999999999 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475426+0000", 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12238887900000001 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487175+0000", 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011749177 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487232+0000", 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6196999999999997e-05 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487274+0000", 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2156000000000001e-05 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.597 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242822 7.2 7:4babd305:::benchmark_data_smithi081_73391_object103823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.353427+0000", 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.121971852000001, 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13814759800000001, 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242822 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353427+0000", 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353427+0000", 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353429+0000", 2024-09-21T18:58:30.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8250000000000001e-06 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353434+0000", 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5609999999999998e-06 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353445+0000", 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0652000000000001e-05 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353452+0000", 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2969999999999998e-06 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354086+0000", 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063416299999999998 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354193+0000", 2024-09-21T18:58:30.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106693 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354303+0000", 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010956800000000001 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477935+0000", 2024-09-21T18:58:30.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12363234200000001 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477947+0000", 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2431e-05 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491475+0000", 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013527656000000001 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491527+0000", 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1622999999999997e-05 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491575+0000", 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7787000000000002e-05 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242823 7.2 7:4c0cbab2:::benchmark_data_smithi081_73391_object103824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.353478+0000", 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.121920375999998, 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.138776869, 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242823 2024-09-21T18:58:30.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353478+0000", 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353478+0000", 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353479+0000", 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6199999999999997e-07 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353482+0000", 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8140000000000001e-06 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353487+0000", 2024-09-21T18:58:30.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6019999999999999e-06 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353493+0000", 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1730000000000003e-06 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356641+0000", 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031481109999999999 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356732+0000", 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1620999999999996e-05 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356830+0000", 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7289999999999999e-05 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477952+0000", 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12112197299999999 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487050+0000", 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009098531 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491616+0000", 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045655089999999997 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491894+0000", 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027829200000000003 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492255+0000", 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036129100000000002 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242827 7.2 7:42b56fec:::benchmark_data_smithi081_73391_object103828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.354737+0000", 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 40.120661312000003, 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.137678105, 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:30.612 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242827 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354737+0000", 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354737+0000", 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354738+0000", 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.006e-06 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354743+0000", 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3970000000000004e-06 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:58:30.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354752+0000", 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8899999999999996e-06 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354758+0000", 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6769999999999998e-06 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475439+0000", 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12068037199999999 2024-09-21T18:58:30.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475564+0000", 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012529800000000001 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475681+0000", 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011681200000000001 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:58:30.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.482300+0000", 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066191899999999996 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487092+0000", 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047915659999999997 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492297+0000", 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005205093 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492382+0000", 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5304999999999997e-05 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492416+0000", 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3498999999999999e-05 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:58:30.619 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:58:30.619 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:58:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:30.718 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:30.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:30.767 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:30.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:30.897 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:30.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:30.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:31.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:31.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:31.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:31.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:30 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T18:58:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:31 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T18:58:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:31 smithi104 ceph-mon[32438]: pgmap v240: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 709 op/s; 33 B/s, 0 objects/s recovering 2024-09-21T18:58:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:31 smithi104 ceph-mon[32438]: 3.b scrub starts 2024-09-21T18:58:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:31 smithi104 ceph-mon[32438]: 3.b scrub ok 2024-09-21T18:58:31.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[29384]: pgmap v240: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 709 op/s; 33 B/s, 0 objects/s recovering 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[29384]: 3.b scrub starts 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[29384]: 3.b scrub ok 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[39192]: pgmap v240: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 709 op/s; 33 B/s, 0 objects/s recovering 2024-09-21T18:58:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[39192]: 3.b scrub starts 2024-09-21T18:58:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:30 smithi081 ceph-mon[39192]: 3.b scrub ok 2024-09-21T18:58:31.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:31.399 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:31.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:31.501 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:31.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T18:58:31.525 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:31.563 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:31.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:31.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:31.770 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:31.787 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:31.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:31.893 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:31.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:31.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.018 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:32.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:32.095 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:58:32.116 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[29384]: 7.b scrub starts 2024-09-21T18:58:32.116 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[29384]: 7.b scrub ok 2024-09-21T18:58:32.116 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[29384]: 6.7 scrub starts 2024-09-21T18:58:32.116 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[29384]: 6.7 scrub ok 2024-09-21T18:58:32.116 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[29384]: 3.1f scrub starts 2024-09-21T18:58:32.116 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[29384]: 3.1f scrub ok 2024-09-21T18:58:32.117 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[39192]: 7.b scrub starts 2024-09-21T18:58:32.117 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[39192]: 7.b scrub ok 2024-09-21T18:58:32.117 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[39192]: 6.7 scrub starts 2024-09-21T18:58:32.117 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[39192]: 6.7 scrub ok 2024-09-21T18:58:32.117 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[39192]: 3.1f scrub starts 2024-09-21T18:58:32.117 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:32 smithi081 ceph-mon[39192]: 3.1f scrub ok 2024-09-21T18:58:32.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:32.140 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:32.167 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:32 smithi104 ceph-mon[32438]: 7.b scrub starts 2024-09-21T18:58:32.167 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:32 smithi104 ceph-mon[32438]: 7.b scrub ok 2024-09-21T18:58:32.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:32 smithi104 ceph-mon[32438]: 6.7 scrub starts 2024-09-21T18:58:32.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:32 smithi104 ceph-mon[32438]: 6.7 scrub ok 2024-09-21T18:58:32.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:32 smithi104 ceph-mon[32438]: 3.1f scrub starts 2024-09-21T18:58:32.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:32 smithi104 ceph-mon[32438]: 3.1f scrub ok 2024-09-21T18:58:32.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:32.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:32.271 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:32.271 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:58:32.271 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T18:58:32.271 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:32.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:32.389 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:32.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.514 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:32.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:32.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:32.636 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:32.672 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:58:32.673 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T18:58:32.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:32.759 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:32.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:32.881 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:32.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:32.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:33.005 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:33.018 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:58:33.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:33.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:33.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:33.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: pgmap v242: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 616 op/s; 29 B/s, 0 objects/s recovering 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: 7.a scrub starts 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: 7.a scrub ok 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3368729172' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:33 smithi104 ceph-mon[32438]: from='client.25144 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: pgmap v242: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 616 op/s; 29 B/s, 0 objects/s recovering 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: 7.a scrub starts 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: 7.a scrub ok 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3368729172' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[29384]: from='client.25144 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: pgmap v242: 148 pgs: 148 active+clean; 583 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 616 op/s; 29 B/s, 0 objects/s recovering 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: 7.a scrub starts 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: 7.a scrub ok 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3368729172' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:33 smithi081 ceph-mon[39192]: from='client.25144 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:33.326 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T18:58:33.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:33.389 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:33.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:33.513 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:33.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:33.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:33.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.759 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-21T18:58:31.481845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1152,"num_read_kb":1009,"num_write":278543,"num_write_kb":8897418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36108,"num_bytes_recovered":2359238350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89766,"ondisk_log_size":89766,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1744940,"kb_used_data":281932,"kb_used_omap":12,"kb_used_meta":1462643,"kb_avail":748216276,"statfs":{"total":767960285184,"available":766173466624,"internally_reserved":0,"allocated":288698368,"data_stored":284620592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":1497746996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-302972952,"num_objects":-4624,"num_object_clones":0,"num_object_copies":-13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4624,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.506922"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.017014+0000","last_change":"2024-09-21T18:55:51.525761+0000","last_active":"2024-09-21T18:58:26.017014+0000","last_peered":"2024-09-21T18:58:26.017014+0000","last_clean":"2024-09-21T18:58:26.017014+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T18:58:26.017014+0000","last_undegraded":"2024-09-21T18:58:26.017014+0000","last_fullsized":"2024-09-21T18:58:26.017014+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:25.183310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.183310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":262,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513905+0000","last_change":"2024-09-21T18:55:39.951935+0000","last_active":"2024-09-21T18:58:29.513905+0000","last_peered":"2024-09-21T18:58:29.513905+0000","last_clean":"2024-09-21T18:58:29.513905+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T18:58:29.513905+0000","last_undegraded":"2024-09-21T18:58:29.513905+0000","last_fullsized":"2024-09-21T18:58:29.513905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:55:39.951860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.951860+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006295+0000","last_change":"2024-09-21T18:54:13.159900+0000","last_active":"2024-09-21T18:58:26.006295+0000","last_peered":"2024-09-21T18:58:26.006295+0000","last_clean":"2024-09-21T18:58:26.006295+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T18:58:26.006295+0000","last_undegraded":"2024-09-21T18:58:26.006295+0000","last_fullsized":"2024-09-21T18:58:26.006295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.159851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.159851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.239093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":221,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513783+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T18:58:29.513783+0000","last_peered":"2024-09-21T18:58:29.513783+0000","last_clean":"2024-09-21T18:58:29.513783+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T18:58:29.513783+0000","last_undegraded":"2024-09-21T18:58:29.513783+0000","last_fullsized":"2024-09-21T18:58:29.513783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.087878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005139+0000","last_change":"2024-09-21T18:54:54.069966+0000","last_active":"2024-09-21T18:58:26.005139+0000","last_peered":"2024-09-21T18:58:26.005139+0000","last_clean":"2024-09-21T18:58:26.005139+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T18:58:26.005139+0000","last_undegraded":"2024-09-21T18:58:26.005139+0000","last_fullsized":"2024-09-21T18:58:26.005139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:54.069766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:54.069766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005691+0000","last_change":"2024-09-21T18:54:50.090079+0000","last_active":"2024-09-21T18:58:26.005691+0000","last_peered":"2024-09-21T18:58:26.005691+0000","last_clean":"2024-09-21T18:58:26.005691+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T18:58:26.005691+0000","last_undegraded":"2024-09-21T18:58:26.005691+0000","last_fullsized":"2024-09-21T18:58:26.005691+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:50.089335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:50.089335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":291,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005813+0000","last_change":"2024-09-21T18:55:51.560419+0000","last_active":"2024-09-21T18:58:26.005813+0000","last_peered":"2024-09-21T18:58:26.005813+0000","last_clean":"2024-09-21T18:58:26.005813+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T18:58:26.005813+0000","last_undegraded":"2024-09-21T18:58:26.005813+0000","last_fullsized":"2024-09-21T18:58:26.005813+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:13.817777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":225,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513678+0000","last_change":"2024-09-21T18:55:28.046258+0000","last_active":"2024-09-21T18:58:29.513678+0000","last_peered":"2024-09-21T18:58:29.513678+0000","last_clean":"2024-09-21T18:58:29.513678+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T18:58:29.513678+0000","last_undegraded":"2024-09-21T18:58:29.513678+0000","last_fullsized":"2024-09-21T18:58:29.513678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:28.046199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:28.046199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":233,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253853+0000","last_change":"2024-09-21T18:57:15.761415+0000","last_active":"2024-09-21T18:58:30.253853+0000","last_peered":"2024-09-21T18:58:30.253853+0000","last_clean":"2024-09-21T18:58:30.253853+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T18:58:30.253853+0000","last_undegraded":"2024-09-21T18:58:30.253853+0000","last_fullsized":"2024-09-21T18:58:30.253853+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:15.761351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.761351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:24.451897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":287,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514305+0000","last_change":"2024-09-21T18:56:02.969282+0000","last_active":"2024-09-21T18:58:29.514305+0000","last_peered":"2024-09-21T18:58:29.514305+0000","last_clean":"2024-09-21T18:58:29.514305+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T18:58:29.514305+0000","last_undegraded":"2024-09-21T18:58:29.514305+0000","last_fullsized":"2024-09-21T18:58:29.514305+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:56:02.969228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:02.969228+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":247,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.257001+0000","last_change":"2024-09-21T18:56:34.885282+0000","last_active":"2024-09-21T18:58:30.257001+0000","last_peered":"2024-09-21T18:58:30.257001+0000","last_clean":"2024-09-21T18:58:30.257001+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T18:58:30.257001+0000","last_undegraded":"2024-09-21T18:58:30.257001+0000","last_fullsized":"2024-09-21T18:58:30.257001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:34.885209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:34.885209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.553709+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.015749+0000","last_change":"2024-09-21T18:55:51.540347+0000","last_active":"2024-09-21T18:58:26.015749+0000","last_peered":"2024-09-21T18:58:26.015749+0000","last_clean":"2024-09-21T18:58:26.015749+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T18:58:26.015749+0000","last_undegraded":"2024-09-21T18:58:26.015749+0000","last_fullsized":"2024-09-21T18:58:26.015749+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.662687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.662687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":233,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253419+0000","last_change":"2024-09-21T18:57:19.821897+0000","last_active":"2024-09-21T18:58:30.253419+0000","last_peered":"2024-09-21T18:58:30.253419+0000","last_clean":"2024-09-21T18:58:30.253419+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T18:58:30.253419+0000","last_undegraded":"2024-09-21T18:58:30.253419+0000","last_fullsized":"2024-09-21T18:58:30.253419+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.821849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.129486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":239,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514119+0000","last_change":"2024-09-21T18:54:11.001752+0000","last_active":"2024-09-21T18:58:29.514119+0000","last_peered":"2024-09-21T18:58:29.514119+0000","last_clean":"2024-09-21T18:58:29.514119+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T18:58:29.514119+0000","last_undegraded":"2024-09-21T18:58:29.514119+0000","last_fullsized":"2024-09-21T18:58:29.514119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:54:11.001686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.001686+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:13.865820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009420+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T18:58:26.009420+0000","last_peered":"2024-09-21T18:58:26.009420+0000","last_clean":"2024-09-21T18:58:26.009420+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T18:58:26.009420+0000","last_undegraded":"2024-09-21T18:58:26.009420+0000","last_fullsized":"2024-09-21T18:58:26.009420+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:17.989380+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":229,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.252976+0000","last_change":"2024-09-21T18:56:41.896412+0000","last_active":"2024-09-21T18:58:30.252976+0000","last_peered":"2024-09-21T18:58:30.252976+0000","last_clean":"2024-09-21T18:58:30.252976+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T18:58:30.252976+0000","last_undegraded":"2024-09-21T18:58:30.252976+0000","last_fullsized":"2024-09-21T18:58:30.252976+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:41.896351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:41.896351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:46.650657+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":221,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514632+0000","last_change":"2024-09-21T18:55:32.077916+0000","last_active":"2024-09-21T18:58:29.514632+0000","last_peered":"2024-09-21T18:58:29.514632+0000","last_clean":"2024-09-21T18:58:29.514632+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T18:58:29.514632+0000","last_undegraded":"2024-09-21T18:58:29.514632+0000","last_fullsized":"2024-09-21T18:58:29.514632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.077846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.077846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.016544+0000","last_change":"2024-09-21T18:55:51.520744+0000","last_active":"2024-09-21T18:58:26.016544+0000","last_peered":"2024-09-21T18:58:26.016544+0000","last_clean":"2024-09-21T18:58:26.016544+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T18:58:26.016544+0000","last_undegraded":"2024-09-21T18:58:26.016544+0000","last_fullsized":"2024-09-21T18:58:26.016544+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:29.982501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.982501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":252,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008332+0000","last_change":"2024-09-21T18:54:43.084572+0000","last_active":"2024-09-21T18:58:26.008332+0000","last_peered":"2024-09-21T18:58:26.008332+0000","last_clean":"2024-09-21T18:58:26.008332+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T18:58:26.008332+0000","last_undegraded":"2024-09-21T18:58:26.008332+0000","last_fullsized":"2024-09-21T18:58:26.008332+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:54:43.084397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.084397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.481531+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008241+0000","last_change":"2024-09-21T18:57:20.229156+0000","last_active":"2024-09-21T18:58:26.008241+0000","last_peered":"2024-09-21T18:58:26.008241+0000","last_clean":"2024-09-21T18:58:26.008241+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T18:58:26.008241+0000","last_undegraded":"2024-09-21T18:58:26.008241+0000","last_fullsized":"2024-09-21T18:58:26.008241+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.229088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.229088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.406792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008084+0000","last_change":"2024-09-21T18:53:13.657671+0000","last_active":"2024-09-21T18:58:26.008084+0000","last_peered":"2024-09-21T18:58:26.008084+0000","last_clean":"2024-09-21T18:58:26.008084+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T18:58:26.008084+0000","last_undegraded":"2024-09-21T18:58:26.008084+0000","last_fullsized":"2024-09-21T18:58:26.008084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.745019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":241,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514440+0000","last_change":"2024-09-21T18:56:07.998284+0000","last_active":"2024-09-21T18:58:29.514440+0000","last_peered":"2024-09-21T18:58:29.514440+0000","last_clean":"2024-09-21T18:58:29.514440+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T18:58:29.514440+0000","last_undegraded":"2024-09-21T18:58:29.514440+0000","last_fullsized":"2024-09-21T18:58:29.514440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:07.998220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:07.998220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":270,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005160+0000","last_change":"2024-09-21T18:58:24.011703+0000","last_active":"2024-09-21T18:58:26.005160+0000","last_peered":"2024-09-21T18:58:26.005160+0000","last_clean":"2024-09-21T18:58:26.005160+0000","last_became_active":"2024-09-21T18:58:23.993245+0000","last_became_peered":"2024-09-21T18:58:23.993245+0000","last_unstale":"2024-09-21T18:58:26.005160+0000","last_undegraded":"2024-09-21T18:58:26.005160+0000","last_fullsized":"2024-09-21T18:58:26.005160+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:49.116507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.116507+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":233,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514254+0000","last_change":"2024-09-21T18:58:28.912619+0000","last_active":"2024-09-21T18:58:29.514254+0000","last_peered":"2024-09-21T18:58:29.514254+0000","last_clean":"2024-09-21T18:58:29.514254+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T18:58:29.514254+0000","last_undegraded":"2024-09-21T18:58:29.514254+0000","last_fullsized":"2024-09-21T18:58:29.514254+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:28.912554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:28.912554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:30.902185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":227,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009353+0000","last_change":"2024-09-21T18:57:14.035912+0000","last_active":"2024-09-21T18:58:26.009353+0000","last_peered":"2024-09-21T18:58:26.009353+0000","last_clean":"2024-09-21T18:58:26.009353+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T18:58:26.009353+0000","last_undegraded":"2024-09-21T18:58:26.009353+0000","last_fullsized":"2024-09-21T18:58:26.009353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:57:14.035837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:14.035837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:27.846135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006381+0000","last_change":"2024-09-21T18:54:15.240674+0000","last_active":"2024-09-21T18:58:26.006381+0000","last_peered":"2024-09-21T18:58:26.006381+0000","last_clean":"2024-09-21T18:58:26.006381+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T18:58:26.006381+0000","last_undegraded":"2024-09-21T18:58:26.006381+0000","last_fullsized":"2024-09-21T18:58:26.006381+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:15.240604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.240604+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.299356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":123,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.021637+0000","last_change":"2024-09-21T18:55:51.572566+0000","last_active":"2024-09-21T18:58:26.021637+0000","last_peered":"2024-09-21T18:58:26.021637+0000","last_clean":"2024-09-21T18:58:26.021637+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T18:58:26.021637+0000","last_undegraded":"2024-09-21T18:58:26.021637+0000","last_fullsized":"2024-09-21T18:58:26.021637+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:54:11.898593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:11.898593+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.058858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":297,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.998933+0000","last_change":"2024-09-21T18:55:21.493043+0000","last_active":"2024-09-21T18:58:25.998933+0000","last_peered":"2024-09-21T18:58:25.998933+0000","last_clean":"2024-09-21T18:58:25.998933+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T18:58:25.998933+0000","last_undegraded":"2024-09-21T18:58:25.998933+0000","last_fullsized":"2024-09-21T18:58:25.998933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:21.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.492989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:08.276721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"163'17118","reported_seq":17325,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.115104+0000","last_change":"2024-09-21T18:57:59.815796+0000","last_active":"2024-09-21T18:58:26.115104+0000","last_peered":"2024-09-21T18:58:26.115104+0000","last_clean":"2024-09-21T18:58:26.115104+0000","last_became_active":"2024-09-21T18:54:14.451675+0000","last_became_peered":"2024-09-21T18:54:14.451675+0000","last_unstale":"2024-09-21T18:58:26.115104+0000","last_undegraded":"2024-09-21T18:58:26.115104+0000","last_fullsized":"2024-09-21T18:58:26.115104+0000","mapping_epoch":79,"log_start":"125'10965","ondisk_log_start":"125'10965","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'15478","last_scrub_stamp":"2024-09-21T18:57:59.815673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.815673+0000","objects_scrubbed":1655,"log_size":6153,"log_dups_size":0,"ondisk_log_size":6153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.208391+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17118,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":252,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006696+0000","last_change":"2024-09-21T18:57:15.119607+0000","last_active":"2024-09-21T18:58:26.006696+0000","last_peered":"2024-09-21T18:58:26.006696+0000","last_clean":"2024-09-21T18:58:26.006696+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T18:58:26.006696+0000","last_undegraded":"2024-09-21T18:58:26.006696+0000","last_fullsized":"2024-09-21T18:58:26.006696+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:15.119448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.119448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:21.282475+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.004214+0000","last_change":"2024-09-21T18:54:53.084595+0000","last_active":"2024-09-21T18:58:26.004214+0000","last_peered":"2024-09-21T18:58:26.004214+0000","last_clean":"2024-09-21T18:58:26.004214+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T18:58:26.004214+0000","last_undegraded":"2024-09-21T18:58:26.004214+0000","last_fullsized":"2024-09-21T18:58:26.004214+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:53.084525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:53.084525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":320,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514742+0000","last_change":"2024-09-21T18:55:53.018341+0000","last_active":"2024-09-21T18:58:29.514742+0000","last_peered":"2024-09-21T18:58:29.514742+0000","last_clean":"2024-09-21T18:58:29.514742+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T18:58:29.514742+0000","last_undegraded":"2024-09-21T18:58:29.514742+0000","last_fullsized":"2024-09-21T18:58:29.514742+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:55:53.018260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:53.018260+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":251,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253442+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T18:58:30.253442+0000","last_peered":"2024-09-21T18:58:30.253442+0000","last_clean":"2024-09-21T18:58:30.253442+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T18:58:30.253442+0000","last_undegraded":"2024-09-21T18:58:30.253442+0000","last_fullsized":"2024-09-21T18:58:30.253442+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.949065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"163'17202","reported_seq":17384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.107328+0000","last_change":"2024-09-21T18:54:14.453085+0000","last_active":"2024-09-21T18:58:26.107328+0000","last_peered":"2024-09-21T18:58:26.107328+0000","last_clean":"2024-09-21T18:58:26.107328+0000","last_became_active":"2024-09-21T18:54:14.452690+0000","last_became_peered":"2024-09-21T18:54:14.452690+0000","last_unstale":"2024-09-21T18:58:26.107328+0000","last_undegraded":"2024-09-21T18:58:26.107328+0000","last_fullsized":"2024-09-21T18:58:26.107328+0000","mapping_epoch":79,"log_start":"135'12500","ondisk_log_start":"135'12500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:21.933805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17202,"num_write_kb":550464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":310,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.003997+0000","last_change":"2024-09-21T18:55:51.566454+0000","last_active":"2024-09-21T18:58:26.003997+0000","last_peered":"2024-09-21T18:58:26.003997+0000","last_clean":"2024-09-21T18:58:26.003997+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T18:58:26.003997+0000","last_undegraded":"2024-09-21T18:58:26.003997+0000","last_fullsized":"2024-09-21T18:58:26.003997+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:55:20.959737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:20.959737+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":246,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008423+0000","last_change":"2024-09-21T18:55:51.532374+0000","last_active":"2024-09-21T18:58:26.008423+0000","last_peered":"2024-09-21T18:58:26.008423+0000","last_clean":"2024-09-21T18:58:26.008423+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T18:58:26.008423+0000","last_undegraded":"2024-09-21T18:58:26.008423+0000","last_fullsized":"2024-09-21T18:58:26.008423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.017608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.017608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":225,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513818+0000","last_change":"2024-09-21T18:56:05.987982+0000","last_active":"2024-09-21T18:58:29.513818+0000","last_peered":"2024-09-21T18:58:29.513818+0000","last_clean":"2024-09-21T18:58:29.513818+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T18:58:29.513818+0000","last_undegraded":"2024-09-21T18:58:29.513818+0000","last_fullsized":"2024-09-21T18:58:29.513818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:05.987945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:56:05.987945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":213,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514070+0000","last_change":"2024-09-21T18:53:13.422226+0000","last_active":"2024-09-21T18:58:29.514070+0000","last_peered":"2024-09-21T18:58:29.514070+0000","last_clean":"2024-09-21T18:58:29.514070+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T18:58:29.514070+0000","last_undegraded":"2024-09-21T18:58:29.514070+0000","last_fullsized":"2024-09-21T18:58:29.514070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.900161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"163'17492","reported_seq":17674,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.140545+0000","last_change":"2024-09-21T18:54:14.455349+0000","last_active":"2024-09-21T18:58:26.140545+0000","last_peered":"2024-09-21T18:58:26.140545+0000","last_clean":"2024-09-21T18:58:26.140545+0000","last_became_active":"2024-09-21T18:54:14.454990+0000","last_became_peered":"2024-09-21T18:54:14.454990+0000","last_unstale":"2024-09-21T18:58:26.140545+0000","last_undegraded":"2024-09-21T18:58:26.140545+0000","last_fullsized":"2024-09-21T18:58:26.140545+0000","mapping_epoch":79,"log_start":"135'12800","ondisk_log_start":"135'12800","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:25.942950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17492,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":250,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513915+0000","last_change":"2024-09-21T18:54:49.074923+0000","last_active":"2024-09-21T18:58:29.513915+0000","last_peered":"2024-09-21T18:58:29.513915+0000","last_clean":"2024-09-21T18:58:29.513915+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T18:58:29.513915+0000","last_undegraded":"2024-09-21T18:58:29.513915+0000","last_fullsized":"2024-09-21T18:58:29.513915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:54:49.074859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:49.074859+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.997123+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":115,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.021038+0000","last_change":"2024-09-21T18:55:51.564530+0000","last_active":"2024-09-21T18:58:26.021038+0000","last_peered":"2024-09-21T18:58:26.021038+0000","last_clean":"2024-09-21T18:58:26.021038+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T18:58:26.021038+0000","last_undegraded":"2024-09-21T18:58:26.021038+0000","last_fullsized":"2024-09-21T18:58:26.021038+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:55:27.735823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:27.735823+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":288,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.003757+0000","last_change":"2024-09-21T18:53:11.411491+0000","last_active":"2024-09-21T18:58:26.003757+0000","last_peered":"2024-09-21T18:58:26.003757+0000","last_clean":"2024-09-21T18:58:26.003757+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T18:58:26.003757+0000","last_undegraded":"2024-09-21T18:58:26.003757+0000","last_fullsized":"2024-09-21T18:58:26.003757+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:53:10.392064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:10.815359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009331+0000","last_change":"2024-09-21T18:58:00.240553+0000","last_active":"2024-09-21T18:58:26.009331+0000","last_peered":"2024-09-21T18:58:26.009331+0000","last_clean":"2024-09-21T18:58:26.009331+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T18:58:26.009331+0000","last_undegraded":"2024-09-21T18:58:26.009331+0000","last_fullsized":"2024-09-21T18:58:26.009331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.240476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.240476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:25.073873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"163'17262","reported_seq":17537,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513679+0000","last_change":"2024-09-21T18:56:01.273900+0000","last_active":"2024-09-21T18:58:29.513679+0000","last_peered":"2024-09-21T18:58:29.513679+0000","last_clean":"2024-09-21T18:58:29.513679+0000","last_became_active":"2024-09-21T18:54:14.452613+0000","last_became_peered":"2024-09-21T18:54:14.452613+0000","last_unstale":"2024-09-21T18:58:29.513679+0000","last_undegraded":"2024-09-21T18:58:29.513679+0000","last_fullsized":"2024-09-21T18:58:29.513679+0000","mapping_epoch":79,"log_start":"133'12400","ondisk_log_start":"133'12400","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9119","last_scrub_stamp":"2024-09-21T18:56:01.273375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:01.273375+0000","objects_scrubbed":8203,"log_size":4862,"log_dups_size":0,"ondisk_log_size":4862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17262,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006954+0000","last_change":"2024-09-21T18:53:09.413967+0000","last_active":"2024-09-21T18:58:26.006954+0000","last_peered":"2024-09-21T18:58:26.006954+0000","last_clean":"2024-09-21T18:58:26.006954+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T18:58:26.006954+0000","last_undegraded":"2024-09-21T18:58:26.006954+0000","last_fullsized":"2024-09-21T18:58:26.006954+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:14.276901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":241,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253932+0000","last_change":"2024-09-21T18:55:36.016602+0000","last_active":"2024-09-21T18:58:30.253932+0000","last_peered":"2024-09-21T18:58:30.253932+0000","last_clean":"2024-09-21T18:58:30.253932+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T18:58:30.253932+0000","last_undegraded":"2024-09-21T18:58:30.253932+0000","last_fullsized":"2024-09-21T18:58:30.253932+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.016546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.016546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:12.525467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":146,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.016441+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T18:58:26.016441+0000","last_peered":"2024-09-21T18:58:26.016441+0000","last_clean":"2024-09-21T18:58:26.016441+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T18:58:26.016441+0000","last_undegraded":"2024-09-21T18:58:26.016441+0000","last_fullsized":"2024-09-21T18:58:26.016441+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.910894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.015895+0000","last_change":"2024-09-21T18:55:51.529085+0000","last_active":"2024-09-21T18:58:26.015895+0000","last_peered":"2024-09-21T18:58:26.015895+0000","last_clean":"2024-09-21T18:58:26.015895+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T18:58:26.015895+0000","last_undegraded":"2024-09-21T18:58:26.015895+0000","last_fullsized":"2024-09-21T18:58:26.015895+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:32.030893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:32.030893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"163'17658","reported_seq":17991,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.252767+0000","last_change":"2024-09-21T18:56:40.641073+0000","last_active":"2024-09-21T18:58:30.252767+0000","last_peered":"2024-09-21T18:58:30.252767+0000","last_clean":"2024-09-21T18:58:30.252767+0000","last_became_active":"2024-09-21T18:54:14.451634+0000","last_became_peered":"2024-09-21T18:54:14.451634+0000","last_unstale":"2024-09-21T18:58:30.252767+0000","last_undegraded":"2024-09-21T18:58:30.252767+0000","last_fullsized":"2024-09-21T18:58:30.252767+0000","mapping_epoch":79,"log_start":"129'12100","ondisk_log_start":"129'12100","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11678","last_scrub_stamp":"2024-09-21T18:56:40.640883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:56:40.640883+0000","objects_scrubbed":6037,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:43.658704+0000","scrub_duration":1848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17658,"num_write_kb":565056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":129,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.016616+0000","last_change":"2024-09-21T18:55:51.566017+0000","last_active":"2024-09-21T18:58:26.016616+0000","last_peered":"2024-09-21T18:58:26.016616+0000","last_clean":"2024-09-21T18:58:26.016616+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T18:58:26.016616+0000","last_undegraded":"2024-09-21T18:58:26.016616+0000","last_fullsized":"2024-09-21T18:58:26.016616+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:21.240843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:21.240843+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006919+0000","last_change":"2024-09-21T18:54:43.377875+0000","last_active":"2024-09-21T18:58:26.006919+0000","last_peered":"2024-09-21T18:58:26.006919+0000","last_clean":"2024-09-21T18:58:26.006919+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T18:58:26.006919+0000","last_undegraded":"2024-09-21T18:58:26.006919+0000","last_fullsized":"2024-09-21T18:58:26.006919+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:43.377639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:43.377639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:47.380559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":236,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.998119+0000","last_change":"2024-09-21T18:55:51.514243+0000","last_active":"2024-09-21T18:58:25.998119+0000","last_peered":"2024-09-21T18:58:25.998119+0000","last_clean":"2024-09-21T18:58:25.998119+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T18:58:25.998119+0000","last_undegraded":"2024-09-21T18:58:25.998119+0000","last_fullsized":"2024-09-21T18:58:25.998119+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.672486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.672486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.316949+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006208+0000","last_change":"2024-09-21T18:57:13.102974+0000","last_active":"2024-09-21T18:58:26.006208+0000","last_peered":"2024-09-21T18:58:26.006208+0000","last_clean":"2024-09-21T18:58:26.006208+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T18:58:26.006208+0000","last_undegraded":"2024-09-21T18:58:26.006208+0000","last_fullsized":"2024-09-21T18:58:26.006208+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.102919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.102919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:24.198745+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"163'17328","reported_seq":25423,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.138694+0000","last_change":"2024-09-21T18:57:57.958393+0000","last_active":"2024-09-21T18:58:26.138694+0000","last_peered":"2024-09-21T18:58:26.138694+0000","last_clean":"2024-09-21T18:58:26.138694+0000","last_became_active":"2024-09-21T18:57:34.506455+0000","last_became_peered":"2024-09-21T18:57:34.506455+0000","last_unstale":"2024-09-21T18:58:26.138694+0000","last_undegraded":"2024-09-21T18:58:26.138694+0000","last_fullsized":"2024-09-21T18:58:26.138694+0000","mapping_epoch":143,"log_start":"125'11181","ondisk_log_start":"125'11181","created":79,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'15561","last_scrub_stamp":"2024-09-21T18:57:57.958164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:57.958164+0000","objects_scrubbed":1789,"log_size":6147,"log_dups_size":0,"ondisk_log_size":6147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:26.531439+0000","scrub_duration":626,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17328,"num_write_kb":554496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3933,"num_bytes_recovered":257753088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":339,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006835+0000","last_change":"2024-09-21T18:54:46.394029+0000","last_active":"2024-09-21T18:58:26.006835+0000","last_peered":"2024-09-21T18:58:26.006835+0000","last_clean":"2024-09-21T18:58:26.006835+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T18:58:26.006835+0000","last_undegraded":"2024-09-21T18:58:26.006835+0000","last_fullsized":"2024-09-21T18:58:26.006835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:54:46.393937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.393937+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:41.364368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":255,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514413+0000","last_change":"2024-09-21T18:56:00.925383+0000","last_active":"2024-09-21T18:58:29.514413+0000","last_peered":"2024-09-21T18:58:29.514413+0000","last_clean":"2024-09-21T18:58:29.514413+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T18:58:29.514413+0000","last_undegraded":"2024-09-21T18:58:29.514413+0000","last_fullsized":"2024-09-21T18:58:29.514413+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:56:00.925315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:56:00.925315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":251,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253399+0000","last_change":"2024-09-21T18:55:55.096856+0000","last_active":"2024-09-21T18:58:30.253399+0000","last_peered":"2024-09-21T18:58:30.253399+0000","last_clean":"2024-09-21T18:58:30.253399+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T18:58:30.253399+0000","last_undegraded":"2024-09-21T18:58:30.253399+0000","last_fullsized":"2024-09-21T18:58:30.253399+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:55:55.096776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:55:55.096776+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:15.593168+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.018870+0000","last_change":"2024-09-21T18:55:51.539737+0000","last_active":"2024-09-21T18:58:26.018870+0000","last_peered":"2024-09-21T18:58:26.018870+0000","last_clean":"2024-09-21T18:58:26.018870+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T18:58:26.018870+0000","last_undegraded":"2024-09-21T18:58:26.018870+0000","last_fullsized":"2024-09-21T18:58:26.018870+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:41.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.369577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:12.965106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005487+0000","last_change":"2024-09-21T18:55:26.996852+0000","last_active":"2024-09-21T18:58:26.005487+0000","last_peered":"2024-09-21T18:58:26.005487+0000","last_clean":"2024-09-21T18:58:26.005487+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T18:58:26.005487+0000","last_undegraded":"2024-09-21T18:58:26.005487+0000","last_fullsized":"2024-09-21T18:58:26.005487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.996607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.996607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"163'17510","reported_seq":23227,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.128415+0000","last_change":"2024-09-21T18:58:01.679316+0000","last_active":"2024-09-21T18:58:26.128415+0000","last_peered":"2024-09-21T18:58:26.128415+0000","last_clean":"2024-09-21T18:58:26.128415+0000","last_became_active":"2024-09-21T18:57:51.729319+0000","last_became_peered":"2024-09-21T18:57:51.729319+0000","last_unstale":"2024-09-21T18:58:26.128415+0000","last_undegraded":"2024-09-21T18:58:26.128415+0000","last_fullsized":"2024-09-21T18:58:26.128415+0000","mapping_epoch":150,"log_start":"128'11900","ondisk_log_start":"128'11900","created":79,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'15900","last_scrub_stamp":"2024-09-21T18:58:01.679101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.679101+0000","objects_scrubbed":1623,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.524249+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17510,"num_write_kb":560320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":179961856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":198,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008880+0000","last_change":"2024-09-21T18:55:51.553119+0000","last_active":"2024-09-21T18:58:26.008880+0000","last_peered":"2024-09-21T18:58:26.008880+0000","last_clean":"2024-09-21T18:58:26.008880+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T18:58:26.008880+0000","last_undegraded":"2024-09-21T18:58:26.008880+0000","last_fullsized":"2024-09-21T18:58:26.008880+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:54:40.059523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.059523+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:44.501396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006103+0000","last_change":"2024-09-21T18:57:22.224988+0000","last_active":"2024-09-21T18:58:26.006103+0000","last_peered":"2024-09-21T18:58:26.006103+0000","last_clean":"2024-09-21T18:58:26.006103+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T18:58:26.006103+0000","last_undegraded":"2024-09-21T18:58:26.006103+0000","last_fullsized":"2024-09-21T18:58:26.006103+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:22.220182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:22.220182+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:45.617863+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":270,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.004241+0000","last_change":"2024-09-21T18:57:49.456229+0000","last_active":"2024-09-21T18:58:26.004241+0000","last_peered":"2024-09-21T18:58:26.004241+0000","last_clean":"2024-09-21T18:58:26.004241+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T18:58:26.004241+0000","last_undegraded":"2024-09-21T18:58:26.004241+0000","last_fullsized":"2024-09-21T18:58:26.004241+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:57:49.456147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.456147+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:50.500491+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.013818+0000","last_change":"2024-09-21T18:55:36.310521+0000","last_active":"2024-09-21T18:58:26.013818+0000","last_peered":"2024-09-21T18:58:26.013818+0000","last_clean":"2024-09-21T18:58:26.013818+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T18:58:26.013818+0000","last_undegraded":"2024-09-21T18:58:26.013818+0000","last_fullsized":"2024-09-21T18:58:26.013818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.304348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.304348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:13.667140+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009895+0000","last_change":"2024-09-21T18:56:52.176652+0000","last_active":"2024-09-21T18:58:26.009895+0000","last_peered":"2024-09-21T18:58:26.009895+0000","last_clean":"2024-09-21T18:58:26.009895+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T18:58:26.009895+0000","last_undegraded":"2024-09-21T18:58:26.009895+0000","last_fullsized":"2024-09-21T18:58:26.009895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:56:52.176581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:56:52.176581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.999868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"163'17822","reported_seq":32147,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.124467+0000","last_change":"2024-09-21T18:56:54.046315+0000","last_active":"2024-09-21T18:58:26.124467+0000","last_peered":"2024-09-21T18:58:26.124467+0000","last_clean":"2024-09-21T18:58:26.124467+0000","last_became_active":"2024-09-21T18:56:54.029307+0000","last_became_peered":"2024-09-21T18:56:54.029307+0000","last_unstale":"2024-09-21T18:58:26.124467+0000","last_undegraded":"2024-09-21T18:58:26.124467+0000","last_fullsized":"2024-09-21T18:58:26.124467+0000","mapping_epoch":130,"log_start":"131'12700","ondisk_log_start":"131'12700","created":79,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.080321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17822,"num_write_kb":570304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7059,"num_bytes_recovered":462618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":125,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.020775+0000","last_change":"2024-09-21T18:55:51.563690+0000","last_active":"2024-09-21T18:58:26.020775+0000","last_peered":"2024-09-21T18:58:26.020775+0000","last_clean":"2024-09-21T18:58:26.020775+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T18:58:26.020775+0000","last_undegraded":"2024-09-21T18:58:26.020775+0000","last_fullsized":"2024-09-21T18:58:26.020775+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:55:23.675118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:23.675118+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.017829+0000","last_change":"2024-09-21T18:55:51.538400+0000","last_active":"2024-09-21T18:58:26.017829+0000","last_peered":"2024-09-21T18:58:26.017829+0000","last_clean":"2024-09-21T18:58:26.017829+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T18:58:26.017829+0000","last_undegraded":"2024-09-21T18:58:26.017829+0000","last_fullsized":"2024-09-21T18:58:26.017829+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:22.082541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:22.082541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":584,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.012475+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T18:58:26.012475+0000","last_peered":"2024-09-21T18:58:26.012475+0000","last_clean":"2024-09-21T18:58:26.012475+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T18:58:26.012475+0000","last_undegraded":"2024-09-21T18:58:26.012475+0000","last_fullsized":"2024-09-21T18:58:26.012475+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.029905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":308,"num_read_kb":303,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":255,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.254079+0000","last_change":"2024-09-21T18:57:12.834964+0000","last_active":"2024-09-21T18:58:30.254079+0000","last_peered":"2024-09-21T18:58:30.254079+0000","last_clean":"2024-09-21T18:58:30.254079+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T18:58:30.254079+0000","last_undegraded":"2024-09-21T18:58:30.254079+0000","last_fullsized":"2024-09-21T18:58:30.254079+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:12.834895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:12.834895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.520233+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006514+0000","last_change":"2024-09-21T18:57:18.176297+0000","last_active":"2024-09-21T18:58:26.006514+0000","last_peered":"2024-09-21T18:58:26.006514+0000","last_clean":"2024-09-21T18:58:26.006514+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T18:58:26.006514+0000","last_undegraded":"2024-09-21T18:58:26.006514+0000","last_fullsized":"2024-09-21T18:58:26.006514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:18.176233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:18.176233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:25.244424+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"163'17360","reported_seq":29422,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.125107+0000","last_change":"2024-09-21T18:57:20.837816+0000","last_active":"2024-09-21T18:58:26.125107+0000","last_peered":"2024-09-21T18:58:26.125107+0000","last_clean":"2024-09-21T18:58:26.125107+0000","last_became_active":"2024-09-21T18:57:20.750225+0000","last_became_peered":"2024-09-21T18:57:20.750225+0000","last_unstale":"2024-09-21T18:58:26.125107+0000","last_undegraded":"2024-09-21T18:58:26.125107+0000","last_fullsized":"2024-09-21T18:58:26.125107+0000","mapping_epoch":139,"log_start":"129'11802","ondisk_log_start":"129'11802","created":79,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.016370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17360,"num_write_kb":555520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388825088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":312,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.999220+0000","last_change":"2024-09-21T18:55:30.556536+0000","last_active":"2024-09-21T18:58:25.999220+0000","last_peered":"2024-09-21T18:58:25.999220+0000","last_clean":"2024-09-21T18:58:25.999220+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T18:58:25.999220+0000","last_undegraded":"2024-09-21T18:58:25.999220+0000","last_fullsized":"2024-09-21T18:58:25.999220+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:55:30.556453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:30.556453+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.234034+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.004935+0000","last_change":"2024-09-21T18:55:31.069114+0000","last_active":"2024-09-21T18:58:26.004935+0000","last_peered":"2024-09-21T18:58:26.004935+0000","last_clean":"2024-09-21T18:58:26.004935+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T18:58:26.004935+0000","last_undegraded":"2024-09-21T18:58:26.004935+0000","last_fullsized":"2024-09-21T18:58:26.004935+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.069057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.069057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":293,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.012561+0000","last_change":"2024-09-21T18:55:51.568828+0000","last_active":"2024-09-21T18:58:26.012561+0000","last_peered":"2024-09-21T18:58:26.012561+0000","last_clean":"2024-09-21T18:58:26.012561+0000","last_became_active":"2024-09-21T18:55:51.535713+0000","last_became_peered":"2024-09-21T18:55:51.535713+0000","last_unstale":"2024-09-21T18:58:26.012561+0000","last_undegraded":"2024-09-21T18:58:26.012561+0000","last_fullsized":"2024-09-21T18:58:26.012561+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T18:53:29.766231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:53:29.766231+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:08.083396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009666+0000","last_change":"2024-09-21T18:58:04.113670+0000","last_active":"2024-09-21T18:58:26.009666+0000","last_peered":"2024-09-21T18:58:26.009666+0000","last_clean":"2024-09-21T18:58:26.009666+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T18:58:26.009666+0000","last_undegraded":"2024-09-21T18:58:26.009666+0000","last_fullsized":"2024-09-21T18:58:26.009666+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.113596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.113596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:24.182248+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":229,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.445025+0000","last_change":"2024-09-21T18:58:30.445025+0000","last_active":"2024-09-21T18:58:30.445025+0000","last_peered":"2024-09-21T18:58:30.445025+0000","last_clean":"2024-09-21T18:58:30.445025+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T18:58:30.445025+0000","last_undegraded":"2024-09-21T18:58:30.445025+0000","last_fullsized":"2024-09-21T18:58:30.445025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:30.444954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:30.444954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:36.189409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"163'17620","reported_seq":23255,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.130052+0000","last_change":"2024-09-21T18:57:52.759473+0000","last_active":"2024-09-21T18:58:26.130052+0000","last_peered":"2024-09-21T18:58:26.130052+0000","last_clean":"2024-09-21T18:58:26.130052+0000","last_became_active":"2024-09-21T18:57:52.731182+0000","last_became_peered":"2024-09-21T18:57:52.731182+0000","last_unstale":"2024-09-21T18:58:26.130052+0000","last_undegraded":"2024-09-21T18:58:26.130052+0000","last_fullsized":"2024-09-21T18:58:26.130052+0000","mapping_epoch":151,"log_start":"125'11438","ondisk_log_start":"125'11438","created":79,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'7928","last_scrub_stamp":"2024-09-21T18:55:40.234365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:40.234365+0000","objects_scrubbed":7820,"log_size":6182,"log_dups_size":0,"ondisk_log_size":6182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:24.384491+0000","scrub_duration":2600,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17620,"num_write_kb":563777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2670,"num_bytes_recovered":174915608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":248,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514745+0000","last_change":"2024-09-21T18:55:35.022891+0000","last_active":"2024-09-21T18:58:29.514745+0000","last_peered":"2024-09-21T18:58:29.514745+0000","last_clean":"2024-09-21T18:58:29.514745+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T18:58:29.514745+0000","last_undegraded":"2024-09-21T18:58:29.514745+0000","last_fullsized":"2024-09-21T18:58:29.514745+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:55:35.022812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.022812+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":225,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514044+0000","last_change":"2024-09-21T18:53:07.400275+0000","last_active":"2024-09-21T18:58:29.514044+0000","last_peered":"2024-09-21T18:58:29.514044+0000","last_clean":"2024-09-21T18:58:29.514044+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T18:58:29.514044+0000","last_undegraded":"2024-09-21T18:58:29.514044+0000","last_fullsized":"2024-09-21T18:58:29.514044+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:53:06.375817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.857138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":241,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253788+0000","last_change":"2024-09-21T18:57:58.653399+0000","last_active":"2024-09-21T18:58:30.253788+0000","last_peered":"2024-09-21T18:58:30.253788+0000","last_clean":"2024-09-21T18:58:30.253788+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T18:58:30.253788+0000","last_undegraded":"2024-09-21T18:58:30.253788+0000","last_fullsized":"2024-09-21T18:58:30.253788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.653339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.653339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:20.676781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006153+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T18:58:26.006153+0000","last_peered":"2024-09-21T18:58:26.006153+0000","last_clean":"2024-09-21T18:58:26.006153+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T18:58:26.006153+0000","last_undegraded":"2024-09-21T18:58:26.006153+0000","last_fullsized":"2024-09-21T18:58:26.006153+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.894642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"163'17136","reported_seq":25641,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.120029+0000","last_change":"2024-09-21T18:57:35.557746+0000","last_active":"2024-09-21T18:58:26.120029+0000","last_peered":"2024-09-21T18:58:26.120029+0000","last_clean":"2024-09-21T18:58:26.120029+0000","last_became_active":"2024-09-21T18:57:35.521875+0000","last_became_peered":"2024-09-21T18:57:35.521875+0000","last_unstale":"2024-09-21T18:58:26.120029+0000","last_undegraded":"2024-09-21T18:58:26.120029+0000","last_fullsized":"2024-09-21T18:58:26.120029+0000","mapping_epoch":144,"log_start":"123'10400","ondisk_log_start":"123'10400","created":79,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'7040","last_scrub_stamp":"2024-09-21T18:55:29.650201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:55:29.650201+0000","objects_scrubbed":6970,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:10.592711+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17136,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4110,"num_bytes_recovered":269352960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":257,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514431+0000","last_change":"2024-09-21T18:55:57.951276+0000","last_active":"2024-09-21T18:58:29.514431+0000","last_peered":"2024-09-21T18:58:29.514431+0000","last_clean":"2024-09-21T18:58:29.514431+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T18:58:29.514431+0000","last_undegraded":"2024-09-21T18:58:29.514431+0000","last_fullsized":"2024-09-21T18:58:29.514431+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:55:57.951206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:57.951206+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.998286+0000","last_change":"2024-09-21T18:55:34.599493+0000","last_active":"2024-09-21T18:58:25.998286+0000","last_peered":"2024-09-21T18:58:25.998286+0000","last_clean":"2024-09-21T18:58:25.998286+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T18:58:25.998286+0000","last_undegraded":"2024-09-21T18:58:25.998286+0000","last_fullsized":"2024-09-21T18:58:25.998286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.599450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.599450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.223741+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006603+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T18:58:26.006603+0000","last_peered":"2024-09-21T18:58:26.006603+0000","last_clean":"2024-09-21T18:58:26.006603+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T18:58:26.006603+0000","last_undegraded":"2024-09-21T18:58:26.006603+0000","last_fullsized":"2024-09-21T18:58:26.006603+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:23.332129+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009017+0000","last_change":"2024-09-21T18:54:47.041077+0000","last_active":"2024-09-21T18:58:26.009017+0000","last_peered":"2024-09-21T18:58:26.009017+0000","last_clean":"2024-09-21T18:58:26.009017+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T18:58:26.009017+0000","last_undegraded":"2024-09-21T18:58:26.009017+0000","last_fullsized":"2024-09-21T18:58:26.009017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:47.041005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:47.041005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:36.540935+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"163'17338","reported_seq":23603,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.136729+0000","last_change":"2024-09-21T18:57:52.776315+0000","last_active":"2024-09-21T18:58:26.136729+0000","last_peered":"2024-09-21T18:58:26.136729+0000","last_clean":"2024-09-21T18:58:26.136729+0000","last_became_active":"2024-09-21T18:57:52.730444+0000","last_became_peered":"2024-09-21T18:57:52.730444+0000","last_unstale":"2024-09-21T18:58:26.136729+0000","last_undegraded":"2024-09-21T18:58:26.136729+0000","last_fullsized":"2024-09-21T18:58:26.136729+0000","mapping_epoch":151,"log_start":"128'11700","ondisk_log_start":"128'11700","created":79,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:09.988393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17338,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3036,"num_bytes_recovered":198967296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":244,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514280+0000","last_change":"2024-09-21T18:55:59.954625+0000","last_active":"2024-09-21T18:58:29.514280+0000","last_peered":"2024-09-21T18:58:29.514280+0000","last_clean":"2024-09-21T18:58:29.514280+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T18:58:29.514280+0000","last_undegraded":"2024-09-21T18:58:29.514280+0000","last_fullsized":"2024-09-21T18:58:29.514280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:55:59.954404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:59.954404+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":260,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005552+0000","last_change":"2024-09-21T18:55:51.548329+0000","last_active":"2024-09-21T18:58:26.005552+0000","last_peered":"2024-09-21T18:58:26.005552+0000","last_clean":"2024-09-21T18:58:26.005552+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T18:58:26.005552+0000","last_undegraded":"2024-09-21T18:58:26.005552+0000","last_fullsized":"2024-09-21T18:58:26.005552+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:54:07.936976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:07.936976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":303,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008026+0000","last_change":"2024-09-21T18:58:02.631905+0000","last_active":"2024-09-21T18:58:26.008026+0000","last_peered":"2024-09-21T18:58:26.008026+0000","last_clean":"2024-09-21T18:58:26.008026+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T18:58:26.008026+0000","last_undegraded":"2024-09-21T18:58:26.008026+0000","last_fullsized":"2024-09-21T18:58:26.008026+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:58:02.629537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.629537+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:04.420331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":221,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514630+0000","last_change":"2024-09-21T18:54:56.044953+0000","last_active":"2024-09-21T18:58:29.514630+0000","last_peered":"2024-09-21T18:58:29.514630+0000","last_clean":"2024-09-21T18:58:29.514630+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T18:58:29.514630+0000","last_undegraded":"2024-09-21T18:58:29.514630+0000","last_fullsized":"2024-09-21T18:58:29.514630+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:56.044886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:56.044886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:17.940183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"163'17458","reported_seq":17749,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.141618+0000","last_change":"2024-09-21T18:57:51.070236+0000","last_active":"2024-09-21T18:58:26.141618+0000","last_peered":"2024-09-21T18:58:26.141618+0000","last_clean":"2024-09-21T18:58:26.141618+0000","last_became_active":"2024-09-21T18:54:14.454019+0000","last_became_peered":"2024-09-21T18:54:14.454019+0000","last_unstale":"2024-09-21T18:58:26.141618+0000","last_undegraded":"2024-09-21T18:58:26.141618+0000","last_fullsized":"2024-09-21T18:58:26.141618+0000","mapping_epoch":79,"log_start":"124'10700","ondisk_log_start":"124'10700","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15303","last_scrub_stamp":"2024-09-21T18:57:51.070161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.070161+0000","objects_scrubbed":2162,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:57.001294+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17458,"num_write_kb":558656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.016692+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T18:58:26.016692+0000","last_peered":"2024-09-21T18:58:26.016692+0000","last_clean":"2024-09-21T18:58:26.016692+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T18:58:26.016692+0000","last_undegraded":"2024-09-21T18:58:26.016692+0000","last_fullsized":"2024-09-21T18:58:26.016692+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:17.896636+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":241,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253504+0000","last_change":"2024-09-21T18:55:37.064551+0000","last_active":"2024-09-21T18:58:30.253504+0000","last_peered":"2024-09-21T18:58:30.253504+0000","last_clean":"2024-09-21T18:58:30.253504+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T18:58:30.253504+0000","last_undegraded":"2024-09-21T18:58:30.253504+0000","last_fullsized":"2024-09-21T18:58:30.253504+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.064474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.064474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.565897+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.007657+0000","last_change":"2024-09-21T18:54:46.041146+0000","last_active":"2024-09-21T18:58:26.007657+0000","last_peered":"2024-09-21T18:58:26.007657+0000","last_clean":"2024-09-21T18:58:26.007657+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T18:58:26.007657+0000","last_undegraded":"2024-09-21T18:58:26.007657+0000","last_fullsized":"2024-09-21T18:58:26.007657+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:46.040844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:46.040844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.513174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":119,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.017755+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T18:58:26.017755+0000","last_peered":"2024-09-21T18:58:26.017755+0000","last_clean":"2024-09-21T18:58:26.017755+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T18:58:26.017755+0000","last_undegraded":"2024-09-21T18:58:26.017755+0000","last_fullsized":"2024-09-21T18:58:26.017755+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.098728+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"163'17432","reported_seq":30594,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.108511+0000","last_change":"2024-09-21T18:57:03.607699+0000","last_active":"2024-09-21T18:58:26.108511+0000","last_peered":"2024-09-21T18:58:26.108511+0000","last_clean":"2024-09-21T18:58:26.108511+0000","last_became_active":"2024-09-21T18:57:03.567321+0000","last_became_peered":"2024-09-21T18:57:03.567321+0000","last_unstale":"2024-09-21T18:58:26.108511+0000","last_undegraded":"2024-09-21T18:58:26.108511+0000","last_fullsized":"2024-09-21T18:58:26.108511+0000","mapping_epoch":134,"log_start":"128'11800","ondisk_log_start":"128'11800","created":79,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.024781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17432,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6486,"num_bytes_recovered":425066496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":257,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.998161+0000","last_change":"2024-09-21T18:55:39.610642+0000","last_active":"2024-09-21T18:58:25.998161+0000","last_peered":"2024-09-21T18:58:25.998161+0000","last_clean":"2024-09-21T18:58:25.998161+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T18:58:25.998161+0000","last_undegraded":"2024-09-21T18:58:25.998161+0000","last_fullsized":"2024-09-21T18:58:25.998161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:55:39.610566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.610566+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:21.295998+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.010177+0000","last_change":"2024-09-21T18:55:26.999629+0000","last_active":"2024-09-21T18:58:26.010177+0000","last_peered":"2024-09-21T18:58:26.010177+0000","last_clean":"2024-09-21T18:58:26.010177+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T18:58:26.010177+0000","last_undegraded":"2024-09-21T18:58:26.010177+0000","last_fullsized":"2024-09-21T18:58:26.010177+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:26.999482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:26.999482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:35.169542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008144+0000","last_change":"2024-09-21T18:55:51.525363+0000","last_active":"2024-09-21T18:58:26.008144+0000","last_peered":"2024-09-21T18:58:26.008144+0000","last_clean":"2024-09-21T18:58:26.008144+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T18:58:26.008144+0000","last_undegraded":"2024-09-21T18:58:26.008144+0000","last_fullsized":"2024-09-21T18:58:26.008144+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.316926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.316926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:15.691524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.012554+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T18:58:26.012554+0000","last_peered":"2024-09-21T18:58:26.012554+0000","last_clean":"2024-09-21T18:58:26.012554+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T18:58:26.012554+0000","last_undegraded":"2024-09-21T18:58:26.012554+0000","last_fullsized":"2024-09-21T18:58:26.012554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:25.100406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"163'17140","reported_seq":17321,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.114344+0000","last_change":"2024-09-21T18:54:14.455120+0000","last_active":"2024-09-21T18:58:26.114344+0000","last_peered":"2024-09-21T18:58:26.114344+0000","last_clean":"2024-09-21T18:58:26.114344+0000","last_became_active":"2024-09-21T18:54:14.454851+0000","last_became_peered":"2024-09-21T18:54:14.454851+0000","last_unstale":"2024-09-21T18:58:26.114344+0000","last_undegraded":"2024-09-21T18:58:26.114344+0000","last_fullsized":"2024-09-21T18:58:26.114344+0000","mapping_epoch":79,"log_start":"135'12500","ondisk_log_start":"135'12500","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17140,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":271,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.513948+0000","last_change":"2024-09-21T18:54:40.066221+0000","last_active":"2024-09-21T18:58:29.513948+0000","last_peered":"2024-09-21T18:58:29.513948+0000","last_clean":"2024-09-21T18:58:29.513948+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T18:58:29.513948+0000","last_undegraded":"2024-09-21T18:58:29.513948+0000","last_fullsized":"2024-09-21T18:58:29.513948+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:54:40.066144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:40.066144+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:14.887522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009862+0000","last_change":"2024-09-21T18:54:08.915601+0000","last_active":"2024-09-21T18:58:26.009862+0000","last_peered":"2024-09-21T18:58:26.009862+0000","last_clean":"2024-09-21T18:58:26.009862+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T18:58:26.009862+0000","last_undegraded":"2024-09-21T18:58:26.009862+0000","last_fullsized":"2024-09-21T18:58:26.009862+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:08.915555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:08.915555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:10.062214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.014417+0000","last_change":"2024-09-21T18:55:51.525737+0000","last_active":"2024-09-21T18:58:26.014417+0000","last_peered":"2024-09-21T18:58:26.014417+0000","last_clean":"2024-09-21T18:58:26.014417+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T18:58:26.014417+0000","last_undegraded":"2024-09-21T18:58:26.014417+0000","last_fullsized":"2024-09-21T18:58:26.014417+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:37.339735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:37.339735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.610568+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005213+0000","last_change":"2024-09-21T18:53:13.429832+0000","last_active":"2024-09-21T18:58:26.005213+0000","last_peered":"2024-09-21T18:58:26.005213+0000","last_clean":"2024-09-21T18:58:26.005213+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T18:58:26.005213+0000","last_undegraded":"2024-09-21T18:58:26.005213+0000","last_fullsized":"2024-09-21T18:58:26.005213+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:15.837150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"163'17118","reported_seq":17299,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.113579+0000","last_change":"2024-09-21T18:54:14.456186+0000","last_active":"2024-09-21T18:58:26.113579+0000","last_peered":"2024-09-21T18:58:26.113579+0000","last_clean":"2024-09-21T18:58:26.113579+0000","last_became_active":"2024-09-21T18:54:14.455918+0000","last_became_peered":"2024-09-21T18:54:14.455918+0000","last_unstale":"2024-09-21T18:58:26.113579+0000","last_undegraded":"2024-09-21T18:58:26.113579+0000","last_fullsized":"2024-09-21T18:58:26.113579+0000","mapping_epoch":79,"log_start":"130'11900","ondisk_log_start":"130'11900","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:54:13.457909+0000","last_clean_scrub_stamp":"2024-09-21T18:54:13.457909+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:29.184091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17118,"num_write_kb":547776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":270,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.252849+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T18:58:30.252849+0000","last_peered":"2024-09-21T18:58:30.252849+0000","last_clean":"2024-09-21T18:58:30.252849+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T18:58:30.252849+0000","last_undegraded":"2024-09-21T18:58:30.252849+0000","last_fullsized":"2024-09-21T18:58:30.252849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.873039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":268,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009957+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T18:58:26.009957+0000","last_peered":"2024-09-21T18:58:26.009957+0000","last_clean":"2024-09-21T18:58:26.009957+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T18:58:26.009957+0000","last_undegraded":"2024-09-21T18:58:26.009957+0000","last_fullsized":"2024-09-21T18:58:26.009957+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:09.842647+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":305,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008825+0000","last_change":"2024-09-21T18:57:23.116600+0000","last_active":"2024-09-21T18:58:26.008825+0000","last_peered":"2024-09-21T18:58:26.008825+0000","last_clean":"2024-09-21T18:58:26.008825+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T18:58:26.008825+0000","last_undegraded":"2024-09-21T18:58:26.008825+0000","last_fullsized":"2024-09-21T18:58:26.008825+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:57:23.116520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:23.116520+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:38.889890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":243,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253102+0000","last_change":"2024-09-21T18:58:29.399508+0000","last_active":"2024-09-21T18:58:30.253102+0000","last_peered":"2024-09-21T18:58:30.253102+0000","last_clean":"2024-09-21T18:58:30.253102+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T18:58:30.253102+0000","last_undegraded":"2024-09-21T18:58:30.253102+0000","last_fullsized":"2024-09-21T18:58:30.253102+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:29.399458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:29.399458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:33.460368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.998328+0000","last_change":"2024-09-21T18:55:51.514267+0000","last_active":"2024-09-21T18:58:25.998328+0000","last_peered":"2024-09-21T18:58:25.998328+0000","last_clean":"2024-09-21T18:58:25.998328+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T18:58:25.998328+0000","last_undegraded":"2024-09-21T18:58:25.998328+0000","last_fullsized":"2024-09-21T18:58:25.998328+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:24.758299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:24.758299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.374962+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008297+0000","last_change":"2024-09-21T18:58:04.758385+0000","last_active":"2024-09-21T18:58:26.008297+0000","last_peered":"2024-09-21T18:58:26.008297+0000","last_clean":"2024-09-21T18:58:26.008297+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T18:58:26.008297+0000","last_undegraded":"2024-09-21T18:58:26.008297+0000","last_fullsized":"2024-09-21T18:58:26.008297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T18:58:04.757944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.757944+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:23.873942+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":241,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253947+0000","last_change":"2024-09-21T18:58:01.738746+0000","last_active":"2024-09-21T18:58:30.253947+0000","last_peered":"2024-09-21T18:58:30.253947+0000","last_clean":"2024-09-21T18:58:30.253947+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T18:58:30.253947+0000","last_undegraded":"2024-09-21T18:58:30.253947+0000","last_fullsized":"2024-09-21T18:58:30.253947+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:01.738627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.738627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.225962+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009748+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T18:58:26.009748+0000","last_peered":"2024-09-21T18:58:26.009748+0000","last_clean":"2024-09-21T18:58:26.009748+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T18:58:26.009748+0000","last_undegraded":"2024-09-21T18:58:26.009748+0000","last_fullsized":"2024-09-21T18:58:26.009748+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.935335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.020952+0000","last_change":"2024-09-21T18:55:51.571525+0000","last_active":"2024-09-21T18:58:26.020952+0000","last_peered":"2024-09-21T18:58:26.020952+0000","last_clean":"2024-09-21T18:58:26.020952+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T18:58:26.020952+0000","last_undegraded":"2024-09-21T18:58:26.020952+0000","last_fullsized":"2024-09-21T18:58:26.020952+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:51.117167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.117167+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:15.948606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008992+0000","last_change":"2024-09-21T18:54:15.008099+0000","last_active":"2024-09-21T18:58:26.008992+0000","last_peered":"2024-09-21T18:58:26.008992+0000","last_clean":"2024-09-21T18:58:26.008992+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T18:58:26.008992+0000","last_undegraded":"2024-09-21T18:58:26.008992+0000","last_fullsized":"2024-09-21T18:58:26.008992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:15.008055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:15.008055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:16.029972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":224,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008158+0000","last_change":"2024-09-21T18:55:34.132690+0000","last_active":"2024-09-21T18:58:26.008158+0000","last_peered":"2024-09-21T18:58:26.008158+0000","last_clean":"2024-09-21T18:58:26.008158+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T18:58:26.008158+0000","last_undegraded":"2024-09-21T18:58:26.008158+0000","last_fullsized":"2024-09-21T18:58:26.008158+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:34.132246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:34.132246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.007966+0000","last_change":"2024-09-21T18:54:48.048647+0000","last_active":"2024-09-21T18:58:26.007966+0000","last_peered":"2024-09-21T18:58:26.007966+0000","last_clean":"2024-09-21T18:58:26.007966+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T18:58:26.007966+0000","last_undegraded":"2024-09-21T18:58:26.007966+0000","last_fullsized":"2024-09-21T18:58:26.007966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:48.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:48.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:37.565916+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":245,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.252662+0000","last_change":"2024-09-21T18:57:25.890791+0000","last_active":"2024-09-21T18:58:30.252662+0000","last_peered":"2024-09-21T18:58:30.252662+0000","last_clean":"2024-09-21T18:58:30.252662+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T18:58:30.252662+0000","last_undegraded":"2024-09-21T18:58:30.252662+0000","last_fullsized":"2024-09-21T18:58:30.252662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:25.890710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.890710+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:51.251415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":119,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.015754+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T18:58:26.015754+0000","last_peered":"2024-09-21T18:58:26.015754+0000","last_clean":"2024-09-21T18:58:26.015754+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T18:58:26.015754+0000","last_undegraded":"2024-09-21T18:58:26.015754+0000","last_fullsized":"2024-09-21T18:58:26.015754+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:19.445017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":296,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006247+0000","last_change":"2024-09-21T18:55:25.994495+0000","last_active":"2024-09-21T18:58:26.006247+0000","last_peered":"2024-09-21T18:58:26.006247+0000","last_clean":"2024-09-21T18:58:26.006247+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T18:58:26.006247+0000","last_undegraded":"2024-09-21T18:58:26.006247+0000","last_fullsized":"2024-09-21T18:58:26.006247+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:55:25.994420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:25.994420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":221,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514739+0000","last_change":"2024-09-21T18:54:51.062813+0000","last_active":"2024-09-21T18:58:29.514739+0000","last_peered":"2024-09-21T18:58:29.514739+0000","last_clean":"2024-09-21T18:58:29.514739+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T18:58:29.514739+0000","last_undegraded":"2024-09-21T18:58:29.514739+0000","last_fullsized":"2024-09-21T18:58:29.514739+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:51.062769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:54:51.062769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:21.994576+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":235,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.007217+0000","last_change":"2024-09-21T18:53:09.717986+0000","last_active":"2024-09-21T18:58:26.007217+0000","last_peered":"2024-09-21T18:58:26.007217+0000","last_clean":"2024-09-21T18:58:26.007217+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T18:58:26.007217+0000","last_undegraded":"2024-09-21T18:58:26.007217+0000","last_fullsized":"2024-09-21T18:58:26.007217+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.800647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.009729+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T18:58:26.009729+0000","last_peered":"2024-09-21T18:58:26.009729+0000","last_clean":"2024-09-21T18:58:26.009729+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T18:58:26.009729+0000","last_undegraded":"2024-09-21T18:58:26.009729+0000","last_fullsized":"2024-09-21T18:58:26.009729+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.088928+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":320,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.253623+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T18:58:30.253623+0000","last_peered":"2024-09-21T18:58:30.253623+0000","last_clean":"2024-09-21T18:58:30.253623+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T18:58:30.253623+0000","last_undegraded":"2024-09-21T18:58:30.253623+0000","last_fullsized":"2024-09-21T18:58:30.253623+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:13.388257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.021268+0000","last_change":"2024-09-21T18:55:51.546532+0000","last_active":"2024-09-21T18:58:26.021268+0000","last_peered":"2024-09-21T18:58:26.021268+0000","last_clean":"2024-09-21T18:58:26.021268+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T18:58:26.021268+0000","last_undegraded":"2024-09-21T18:58:26.021268+0000","last_fullsized":"2024-09-21T18:58:26.021268+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:39.046790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:39.046790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":267,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514145+0000","last_change":"2024-09-21T18:56:09.001446+0000","last_active":"2024-09-21T18:58:29.514145+0000","last_peered":"2024-09-21T18:58:29.514145+0000","last_clean":"2024-09-21T18:58:29.514145+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T18:58:29.514145+0000","last_undegraded":"2024-09-21T18:58:29.514145+0000","last_fullsized":"2024-09-21T18:58:29.514145+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:56:09.001366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:56:09.001366+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":233,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:29.514552+0000","last_change":"2024-09-21T18:57:57.837867+0000","last_active":"2024-09-21T18:58:29.514552+0000","last_peered":"2024-09-21T18:58:29.514552+0000","last_clean":"2024-09-21T18:58:29.514552+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T18:58:29.514552+0000","last_undegraded":"2024-09-21T18:58:29.514552+0000","last_fullsized":"2024-09-21T18:58:29.514552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:57.837826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:57.837826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:58.681714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.006800+0000","last_change":"2024-09-21T18:57:25.184491+0000","last_active":"2024-09-21T18:58:26.006800+0000","last_peered":"2024-09-21T18:58:26.006800+0000","last_clean":"2024-09-21T18:58:26.006800+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T18:58:26.006800+0000","last_undegraded":"2024-09-21T18:58:26.006800+0000","last_fullsized":"2024-09-21T18:58:26.006800+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:25.180690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:25.180690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:31.159574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.004697+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T18:58:26.004697+0000","last_peered":"2024-09-21T18:58:26.004697+0000","last_clean":"2024-09-21T18:58:26.004697+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T18:58:26.004697+0000","last_undegraded":"2024-09-21T18:58:26.004697+0000","last_fullsized":"2024-09-21T18:58:26.004697+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:07.201536+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":251,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.007090+0000","last_change":"2024-09-21T18:57:49.327777+0000","last_active":"2024-09-21T18:58:26.007090+0000","last_peered":"2024-09-21T18:58:26.007090+0000","last_clean":"2024-09-21T18:58:26.007090+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T18:58:26.007090+0000","last_undegraded":"2024-09-21T18:58:26.007090+0000","last_fullsized":"2024-09-21T18:58:26.007090+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:57:49.327123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:49.327123+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:56.940918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.010118+0000","last_change":"2024-09-21T18:57:19.082962+0000","last_active":"2024-09-21T18:58:26.010118+0000","last_peered":"2024-09-21T18:58:26.010118+0000","last_clean":"2024-09-21T18:58:26.010118+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T18:58:26.010118+0000","last_undegraded":"2024-09-21T18:58:26.010118+0000","last_fullsized":"2024-09-21T18:58:26.010118+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:19.082873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:19.082873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005306+0000","last_change":"2024-09-21T18:55:51.516013+0000","last_active":"2024-09-21T18:58:26.005306+0000","last_peered":"2024-09-21T18:58:26.005306+0000","last_clean":"2024-09-21T18:58:26.005306+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T18:58:26.005306+0000","last_undegraded":"2024-09-21T18:58:26.005306+0000","last_fullsized":"2024-09-21T18:58:26.005306+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:33.072080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:33.072080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":119,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.015993+0000","last_change":"2024-09-21T18:57:58.019346+0000","last_active":"2024-09-21T18:58:26.015993+0000","last_peered":"2024-09-21T18:58:26.015993+0000","last_clean":"2024-09-21T18:58:26.015993+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T18:58:26.015993+0000","last_undegraded":"2024-09-21T18:58:26.015993+0000","last_fullsized":"2024-09-21T18:58:26.015993+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.019278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.019278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.581215+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":247,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.004837+0000","last_change":"2024-09-21T18:58:01.818014+0000","last_active":"2024-09-21T18:58:26.004837+0000","last_peered":"2024-09-21T18:58:26.004837+0000","last_clean":"2024-09-21T18:58:26.004837+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T18:58:26.004837+0000","last_undegraded":"2024-09-21T18:58:26.004837+0000","last_fullsized":"2024-09-21T18:58:26.004837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:58:01.817792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:01.817792+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:30.058894+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":233,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-21T18:58:30.254040+0000","last_change":"2024-09-21T18:55:35.017076+0000","last_active":"2024-09-21T18:58:30.254040+0000","last_peered":"2024-09-21T18:58:30.254040+0000","last_clean":"2024-09-21T18:58:30.254040+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T18:58:30.254040+0000","last_undegraded":"2024-09-21T18:58:30.254040+0000","last_fullsized":"2024-09-21T18:58:30.254040+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:35.017000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:55:35.017000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:05.605024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008488+0000","last_change":"2024-09-21T18:55:31.026854+0000","last_active":"2024-09-21T18:58:26.008488+0000","last_peered":"2024-09-21T18:58:26.008488+0000","last_clean":"2024-09-21T18:58:26.008488+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T18:58:26.008488+0000","last_undegraded":"2024-09-21T18:58:26.008488+0000","last_fullsized":"2024-09-21T18:58:26.008488+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:31.026663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:55:31.026663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:34.212178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.005453+0000","last_change":"2024-09-21T18:57:54.750968+0000","last_active":"2024-09-21T18:58:26.005453+0000","last_peered":"2024-09-21T18:58:26.005453+0000","last_clean":"2024-09-21T18:58:26.005453+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T18:58:26.005453+0000","last_undegraded":"2024-09-21T18:58:26.005453+0000","last_fullsized":"2024-09-21T18:58:26.005453+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:54.750896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.750896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.812996+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":239,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.003912+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T18:58:26.003912+0000","last_peered":"2024-09-21T18:58:26.003912+0000","last_clean":"2024-09-21T18:58:26.003912+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T18:58:26.003912+0000","last_undegraded":"2024-09-21T18:58:26.003912+0000","last_fullsized":"2024-09-21T18:58:26.003912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:18.909641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":119,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.016894+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T18:58:26.016894+0000","last_peered":"2024-09-21T18:58:26.016894+0000","last_clean":"2024-09-21T18:58:26.016894+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T18:58:26.016894+0000","last_undegraded":"2024-09-21T18:58:26.016894+0000","last_fullsized":"2024-09-21T18:58:26.016894+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:11.894402+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.998338+0000","last_change":"2024-09-21T18:57:55.356422+0000","last_active":"2024-09-21T18:58:25.998338+0000","last_peered":"2024-09-21T18:58:25.998338+0000","last_clean":"2024-09-21T18:58:25.998338+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T18:58:25.998338+0000","last_undegraded":"2024-09-21T18:58:25.998338+0000","last_fullsized":"2024-09-21T18:58:25.998338+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:55.356360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:55.356360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:04.334971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008594+0000","last_change":"2024-09-21T18:57:53.263833+0000","last_active":"2024-09-21T18:58:26.008594+0000","last_peered":"2024-09-21T18:58:26.008594+0000","last_clean":"2024-09-21T18:58:26.008594+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T18:58:26.008594+0000","last_undegraded":"2024-09-21T18:58:26.008594+0000","last_fullsized":"2024-09-21T18:58:26.008594+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.263783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.263783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:22.950038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.008119+0000","last_change":"2024-09-21T18:55:36.693594+0000","last_active":"2024-09-21T18:58:26.008119+0000","last_peered":"2024-09-21T18:58:26.008119+0000","last_clean":"2024-09-21T18:58:26.008119+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T18:58:26.008119+0000","last_undegraded":"2024-09-21T18:58:26.008119+0000","last_fullsized":"2024-09-21T18:58:26.008119+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:55:36.693528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:55:36.693528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:58:48.365341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":246,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.004161+0000","last_change":"2024-09-21T18:55:51.516552+0000","last_active":"2024-09-21T18:58:26.004161+0000","last_peered":"2024-09-21T18:58:26.004161+0000","last_clean":"2024-09-21T18:58:26.004161+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T18:58:26.004161+0000","last_undegraded":"2024-09-21T18:58:26.004161+0000","last_fullsized":"2024-09-21T18:58:26.004161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:54:12.878793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:54:12.878793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:25.999013+0000","last_change":"2024-09-21T18:57:54.359262+0000","last_active":"2024-09-21T18:58:25.999013+0000","last_peered":"2024-09-21T18:58:25.999013+0000","last_clean":"2024-09-21T18:58:25.999013+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T18:58:25.999013+0000","last_undegraded":"2024-09-21T18:58:25.999013+0000","last_fullsized":"2024-09-21T18:58:25.999013+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:57:54.359175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.359175+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:06.117172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":119,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-21T18:58:26.021198+0000","last_change":"2024-09-21T18:57:58.985012+0000","last_active":"2024-09-21T18:58:26.021198+0000","last_peered":"2024-09-21T18:58:26.021198+0000","last_clean":"2024-09-21T18:58:26.021198+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T18:58:26.021198+0000","last_undegraded":"2024-09-21T18:58:26.021198+0000","last_fullsized":"2024-09-21T18:58:26.021198+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:58.984945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.984945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:59:03.530436+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277994,"num_write_kb":8895745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35974,"num_bytes_recovered":2357461016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264175616,"data_stored":264175616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89208,"ondisk_log_size":89208,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":308,"num_read_kb":303,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266756,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286148,"kb_used_data":146672,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93459004,"statfs":{"total":95995035648,"available":95702020096,"internally_reserved":0,"allocated":150192128,"data_stored":149683260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528393,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149436,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93595716,"statfs":{"total":95995035648,"available":95842013184,"internally_reserved":0,"allocated":1679360,"data_stored":1178255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790029,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":247744,"kb_used_data":117996,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93497408,"statfs":{"total":95995035648,"available":95741345792,"internally_reserved":0,"allocated":120827904,"data_stored":120329012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018963,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148876,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596276,"statfs":{"total":95995035648,"available":95842586624,"internally_reserved":0,"allocated":1105920,"data_stored":595539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":275220,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93469932,"statfs":{"total":95995035648,"available":95713210368,"internally_reserved":0,"allocated":1114112,"data_stored":589115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":280689098},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148860,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596292,"statfs":{"total":95995035648,"available":95842603008,"internally_reserved":0,"allocated":1089536,"data_stored":588352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184976,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93560176,"statfs":{"total":95995035648,"available":95805620224,"internally_reserved":0,"allocated":1110016,"data_stored":588669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303680,"kb_used_data":11308,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93441472,"statfs":{"total":95995035648,"available":95684067328,"internally_reserved":0,"allocated":11579392,"data_stored":11068390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":117768192,"data_stored":117768192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":146407424,"data_stored":146407424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:58:33.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:33.765 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T18:58:33.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T18:58:33.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:33.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:33.888 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:33.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:33.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:34.025 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:34.025 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:34.040 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:34.041 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:34.061 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:34.062 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:34.080 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:34.080 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:34.096 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:34.097 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:34.113 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:34.113 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:34.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:34.135 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:34.135 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:34.135 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:34.153 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:34.153 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:34.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.259 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:34.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:34.268 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: from='client.15324 v1:172.21.15.81:0/3833785840' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:34.268 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T18:58:34.268 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T18:58:34.268 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:58:34.269 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:58:34.269 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: from='client.25144 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:34.269 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T18:58:34.269 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/725239262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:34.283 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:34.283 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: from='client.15324 v1:172.21.15.81:0/3833785840' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: from='client.25144 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T18:58:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/725239262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: from='client.15324 v1:172.21.15.81:0/3833785840' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: from='client.25144 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T18:58:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/725239262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:34.361 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:34.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:34.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:34.611 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:34.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.758 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:34.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:34.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:34.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:34.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:34.889 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:34.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T18:58:34.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:35.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:35.105 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:35.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:35.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:35.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:35 smithi104 ceph-mon[32438]: pgmap v244: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 618 op/s; 29 B/s, 0 objects/s recovering 2024-09-21T18:58:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:35 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T18:58:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:35 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T18:58:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:35 smithi104 ceph-mon[32438]: 3.8 scrub starts 2024-09-21T18:58:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:35 smithi104 ceph-mon[32438]: 3.8 scrub ok 2024-09-21T18:58:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:35 smithi104 ceph-mon[32438]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[39192]: pgmap v244: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 618 op/s; 29 B/s, 0 objects/s recovering 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[39192]: 3.8 scrub starts 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[39192]: 3.8 scrub ok 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[39192]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[29384]: pgmap v244: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 618 op/s; 29 B/s, 0 objects/s recovering 2024-09-21T18:58:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T18:58:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T18:58:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[29384]: 3.8 scrub starts 2024-09-21T18:58:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[29384]: 3.8 scrub ok 2024-09-21T18:58:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:35 smithi081 ceph-mon[29384]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T18:58:35.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:35.374 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:35.476 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:35.496 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:35.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:35.684 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:35.684 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:35.720 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:35.741 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:35.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:35.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:35.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:35.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:35.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:36.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:36.112 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:36.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:36.246 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:36 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T18:58:36.246 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:36 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T18:58:36.246 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:36 smithi104 ceph-mon[32438]: 6.1b scrub starts 2024-09-21T18:58:36.247 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:36 smithi104 ceph-mon[32438]: 6.1b scrub ok 2024-09-21T18:58:36.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[29384]: 6.1b scrub starts 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[29384]: 6.1b scrub ok 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[39192]: 6.1b scrub starts 2024-09-21T18:58:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:36 smithi081 ceph-mon[39192]: 6.1b scrub ok 2024-09-21T18:58:36.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.358 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:36.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:36.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:36.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:36.518 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:36.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:36.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:36.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:58:36.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:36.726 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:36.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:36.852 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:36.915 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:36.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:36.965 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:36.966 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:36.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 1 pg_num to 11 2024-09-21T18:58:37.078 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:37.099 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:37.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: pgmap v246: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T18:58:37.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2123703165' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: from='client.35195 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:37.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:37.221 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:58:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: pgmap v246: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2123703165' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T18:58:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: from='client.35195 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: pgmap v246: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T18:58:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T18:58:37.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2123703165' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T18:58:37.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: from='client.35195 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T18:58:37.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T18:58:37.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T18:58:37.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:37.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:37.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:37.343 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:37.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:58:37.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:37.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T18:58:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:37.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:37.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:37.589 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:37.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:37.712 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:37.715 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:37.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:37.835 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:37.883 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T18:58:37.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T18:58:37.936 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:37.960 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:37.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:38.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.082 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:38.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:38.115 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:58:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: 3.1e scrub starts 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: 3.1e scrub ok 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: from='client.35195 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: pgmap v248: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:38.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:38.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.204 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:38.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:38.213 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:38.213 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: 3.1e scrub starts 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: 3.1e scrub ok 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: from='client.35195 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: pgmap v248: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:38.218 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: 3.1e scrub starts 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: 3.1e scrub ok 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: from='client.35195 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: pgmap v248: 148 pgs: 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:38.283 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:38.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:38.327 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:38.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:38.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:38.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:38.577 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:38.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.701 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:38.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:38.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:38.823 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:38.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T18:58:38.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:38.953 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:38.953 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:38.961 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:38.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:38.969 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:38.969 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:38.985 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:38.985 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:39.001 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:39.001 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:39.017 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:39.018 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:39.034 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:39.034 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:39.048 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:58:39.058 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:39.059 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:39.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:39.079 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:39.079 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:39.084 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:39.155 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:39.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:39.207 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: 3.d scrub starts 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: 3.d scrub ok 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: 5.14 scrub starts 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: 5.14 scrub ok 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1222068866' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2117744145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:39.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: 3.d scrub starts 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: 3.d scrub ok 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: 5.14 scrub starts 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: 5.14 scrub ok 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1222068866' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2117744145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: 3.d scrub starts 2024-09-21T18:58:39.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: 3.d scrub ok 2024-09-21T18:58:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: 5.14 scrub starts 2024-09-21T18:58:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: 5.14 scrub ok 2024-09-21T18:58:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T18:58:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T18:58:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1222068866' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:58:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2117744145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:39.330 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:39.431 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:39.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:39.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.595 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:39.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99372 7.0 7:018c5cb9:::benchmark_data_smithi081_73391_object99371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 201.96164928100001, 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100414906, 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.667 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99372 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701742+0000", 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701744+0000", 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3889999999999999e-06 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701794+0000", 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0031000000000003e-05 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701809+0000", 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4639e-05 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701817+0000", 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0090000000000007e-06 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.701879+0000", 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1820000000000001e-05 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.702011+0000", 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013177800000000001 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.702160+0000", 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014952100000000001 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.704287+0000", 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021267270000000001 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.704646+0000", 2024-09-21T18:58:39.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035917799999999998 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802063+0000", 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097417227999999995 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802117+0000", 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3488e-05 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802157+0000", 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0098000000000001e-05 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99378 7.0 7:0717396a:::benchmark_data_smithi081_73391_object99377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 201.95563728299999, 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094525928999999995, 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99378 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707754+0000", 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707756+0000", 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1110000000000002e-06 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707812+0000", 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6208000000000001e-05 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707824+0000", 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1838000000000001e-05 2024-09-21T18:58:39.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707832+0000", 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5560000000000002e-06 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707876+0000", 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4354000000000003e-05 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.707962+0000", 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5986e-05 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.708079+0000", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117478 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710054+0000", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019749530000000002 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710919+0000", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00086474499999999999 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802199+0000", 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.091279896999999999 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802247+0000", 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7550999999999999e-05 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802280+0000", 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3251999999999999e-05 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.679 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99386 7.0 7:06617387:::benchmark_data_smithi081_73391_object99385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:58:39.679 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:58:39.679 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 201.94955347699999, 2024-09-21T18:58:39.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088596121999999999, 2024-09-21T18:58:39.679 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.679 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99386 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713838+0000", 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713840+0000", 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7990000000000001e-06 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713885+0000", 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5139999999999998e-05 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713895+0000", 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0193999999999999e-05 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713901+0000", 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7049999999999996e-06 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713952+0000", 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1641999999999998e-05 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.714080+0000", 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012742999999999999 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.714184+0000", 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010401899999999999 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.718889+0000", 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047058029999999997 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.718919+0000", 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9291000000000001e-05 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802318+0000", 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083398876999999996 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802398+0000", 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0391999999999999e-05 2024-09-21T18:58:39.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802434+0000", 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5830000000000001e-05 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99389 7.0 7:02d2e699:::benchmark_data_smithi081_73391_object99388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 201.948236837, 2024-09-21T18:58:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087440860999999995, 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99389 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715154+0000", 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715156+0000", 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6980000000000001e-06 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715227+0000", 2024-09-21T18:58:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0919000000000003e-05 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715243+0000", 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5559000000000001e-05 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.715250+0000", 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6340000000000001e-06 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716563+0000", 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013130399999999999 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716688+0000", 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012472500000000001 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.716876+0000", 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000187586 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.719345+0000", 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024699539999999999 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.720446+0000", 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011003219999999999 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802480+0000", 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082033990000000001 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802566+0000", 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6119999999999995e-05 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.802595+0000", 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9314000000000001e-05 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114537 7.0 7:04929e7d:::benchmark_data_smithi081_73391_object114536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:58:39.691 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 188.93999330299999, 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105603195, 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114537 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723400+0000", 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.119e-06 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723433+0000", 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3302000000000001e-05 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723444+0000", 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0523000000000001e-05 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723452+0000", 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8199999999999997e-06 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727483+0000", 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040313839999999998 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727571+0000", 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7770999999999997e-05 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727655+0000", 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4322999999999999e-05 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730033+0000", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002377948 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730045+0000", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2091e-05 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828840+0000", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098794333999999998 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828919+0000", 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9752000000000005e-05 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.829001+0000", 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1828000000000003e-05 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114544 7.0 7:016dd1d4:::benchmark_data_smithi081_73391_object114543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 188.93246950700001, 2024-09-21T18:58:39.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101948043, 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114544 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730924+0000", 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1280000000000002e-06 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730973+0000", 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8687000000000003e-05 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730987+0000", 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.448e-05 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730996+0000", 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9719999999999998e-06 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731081+0000", 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5292999999999998e-05 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731162+0000", 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0603000000000001e-05 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731260+0000", 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8259999999999998e-05 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.733069+0000", 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018090840000000001 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.780503+0000", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.047434142999999998 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832639+0000", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052135212 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832711+0000", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2327000000000001e-05 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832870+0000", 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015885399999999999 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145381 7.0 7:07a4e617:::benchmark_data_smithi081_73391_object6382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.342438+0000", 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 160.32095368500001, 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11277905000000001, 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.703 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145381 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342438+0000", 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342438+0000", 2024-09-21T18:58:39.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342439+0000", 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7179999999999999e-06 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342444+0000", 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342456+0000", 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2302e-05 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342464+0000", 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.447e-06 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342510+0000", 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5977999999999999e-05 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342606+0000", 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6277e-05 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342712+0000", 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106092 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.348117+0000", 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005405371 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.349145+0000", 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010274399999999999 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455138+0000", 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105993714 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455174+0000", 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5976999999999998e-05 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455217+0000", 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2103e-05 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145387 7.0 7:0b525978:::benchmark_data_smithi081_73391_object6388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.343230+0000", 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 160.32016090299999, 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112080033, 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.709 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145387 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343230+0000", 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343230+0000", 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343232+0000", 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.421e-06 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343237+0000", 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3650000000000003e-06 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343252+0000", 2024-09-21T18:58:39.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4705000000000001e-05 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343260+0000", 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3629999999999994e-06 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345358+0000", 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020982319999999998 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345479+0000", 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012078699999999999 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345612+0000", 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013247700000000001 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.348281+0000", 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002669186 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.349239+0000", 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095836999999999995 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455251+0000", 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10601202899999999 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455284+0000", 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3090000000000003e-05 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455310+0000", 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6007999999999999e-05 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.714 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:147845 7.0 7:0886cefa:::benchmark_data_smithi081_73391_object8846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:02.071897+0000", 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 157.59149428699999, 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083496556, 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 147845 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.715 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071897+0000", 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071897+0000", 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071899+0000", 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.244e-06 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071906+0000", 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8669999999999996e-06 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071920+0000", 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4124999999999999e-05 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071929+0000", 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6260000000000001e-06 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.071990+0000", 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1334000000000006e-05 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.072102+0000", 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111965 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.072174+0000", 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1947999999999996e-05 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.074271+0000", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002097444 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.076397+0000", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021253969999999998 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.155291+0000", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.078894174999999997 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.155349+0000", 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7510999999999999e-05 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:02.155393+0000", 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4919999999999997e-05 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:168583 7.0 7:0243c025:::benchmark_data_smithi081_73391_object29584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.139820+0000", 2024-09-21T18:58:39.720 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 135.52357163100001, 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085977963000000004, 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 168583 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139820+0000", 2024-09-21T18:58:39.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139820+0000", 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139822+0000", 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.323e-06 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139829+0000", 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.725e-06 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139841+0000", 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1979e-05 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139849+0000", 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9359999999999999e-06 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.139915+0000", 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6311999999999998e-05 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.140049+0000", 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013385200000000001 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.140134+0000", 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5255999999999998e-05 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.142437+0000", 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023029779999999998 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.199006+0000", 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056569341000000002 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.225678+0000", 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026671836000000001 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.225745+0000", 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6773000000000006e-05 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.225798+0000", 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2651999999999997e-05 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.726 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:169006 7.0 7:0bdc217b:::benchmark_data_smithi081_73391_object30007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.638054+0000", 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 135.02533710899999, 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101027187, 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 169006 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638054+0000", 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638054+0000", 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638056+0000", 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.057e-06 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638062+0000", 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1940000000000003e-06 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638079+0000", 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6223999999999999e-05 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638086+0000", 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6199999999999999e-06 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638134+0000", 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7886000000000002e-05 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638238+0000", 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104114 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638337+0000", 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8924000000000004e-05 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.641212+0000", 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028744550000000002 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.646975+0000", 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057632689999999997 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:39.732 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_0' removed 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.738982+0000", 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092007007000000002 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739038+0000", 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6113999999999998e-05 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739081+0000", 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3322999999999998e-05 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:169009 7.0 7:01cb78f3:::benchmark_data_smithi081_73391_object30010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.644717+0000", 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 135.018674678, 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106867806, 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.734 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 169009 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644717+0000", 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644717+0000", 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644718+0000", 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7910000000000001e-06 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644726+0000", 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6960000000000005e-06 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644736+0000", 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0375e-05 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644743+0000", 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3679999999999998e-06 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644795+0000", 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2070999999999997e-05 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644903+0000", 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108062 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644986+0000", 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3425999999999997e-05 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.647116+0000", 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021293950000000001 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739125+0000", 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092009274000000002 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751496+0000", 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012371312000000001 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751539+0000", 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2358999999999999e-05 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751584+0000", 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5677000000000003e-05 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211274 7.0 7:0d096775:::benchmark_data_smithi081_73391_object72275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.770245+0000", 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 85.893146383000001, 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.111625435, 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.740 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211274 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770245+0000", 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770245+0000", 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770246+0000", 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770248+0000", 2024-09-21T18:58:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770254+0000", 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1460000000000004e-06 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770258+0000", 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8149999999999996e-06 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776110+0000", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005852037 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776217+0000", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106234 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776328+0000", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011097 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782087+0000", 2024-09-21T18:58:39.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057596749999999997 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.786947+0000", 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048601130000000001 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881796+0000", 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094848779999999994 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881834+0000", 2024-09-21T18:58:39.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7861999999999998e-05 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881870+0000", 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6266999999999997e-05 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211279 7.0 7:05526e4b:::benchmark_data_smithi081_73391_object72280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.776231+0000", 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 85.887159826000001, 2024-09-21T18:58:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105721686, 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211279 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776231+0000", 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776231+0000", 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776234+0000", 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2989999999999999e-06 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776239+0000", 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5840000000000002e-06 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776250+0000", 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1207e-05 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776258+0000", 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5469999999999999e-06 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.779375+0000", 2024-09-21T18:58:39.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031171200000000001 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.779473+0000", 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8119e-05 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.779587+0000", 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011357 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782093+0000", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025064290000000001 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787025+0000", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049313739999999997 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881902+0000", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094877192999999999 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881930+0000", 2024-09-21T18:58:39.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8305000000000001e-05 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881953+0000", 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2938999999999998e-05 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211292 7.0 7:0d9ec215:::benchmark_data_smithi081_73391_object72293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:39.752 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.787634+0000", 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 85.875757707000005, 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107733332, 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211292 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787634+0000", 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787634+0000", 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787635+0000", 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9190000000000002e-06 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787641+0000", 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8139999999999997e-06 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787654+0000", 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2692e-05 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787662+0000", 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7989999999999994e-06 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787713+0000", 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1008999999999999e-05 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787831+0000", 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117778 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787907+0000", 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6227e-05 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.790256+0000", 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023488369999999999 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791100+0000", 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00084454799999999998 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895304+0000", 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104203948 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895338+0000", 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4042999999999999e-05 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895367+0000", 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8717999999999998e-05 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.758 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:212960 7.0 7:00a22706:::benchmark_data_smithi081_73391_object73961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.850426+0000", 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.812965676000005, 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083215539000000005, 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 212960 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850426+0000", 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850426+0000", 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850427+0000", 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850432+0000", 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8770000000000001e-06 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850442+0000", 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5510000000000005e-06 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850449+0000", 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3379999999999999e-06 2024-09-21T18:58:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850502+0000", 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3371000000000001e-05 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850604+0000", 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102198 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.850686+0000", 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1489000000000005e-05 2024-09-21T18:58:39.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.852832+0000", 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002146462 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.857731+0000", 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048987950000000001 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.933535+0000", 2024-09-21T18:58:39.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.075804060000000006 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.933592+0000", 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6993000000000001e-05 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.933641+0000", 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8785000000000001e-05 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:212967 7.0 7:06b671d4:::benchmark_data_smithi081_73391_object73968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:15.853961+0000", 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.809429964000003, 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079818790000000001, 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 212967 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.853961+0000", 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.853961+0000", 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.853963+0000", 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9099999999999999e-06 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.853969+0000", 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1129999999999997e-06 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.853982+0000", 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3115000000000001e-05 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.853991+0000", 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3999999999999992e-06 2024-09-21T18:58:39.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.854042+0000", 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1347999999999997e-05 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.854151+0000", 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108923 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.854276+0000", 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012464600000000001 2024-09-21T18:58:39.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.858082+0000", 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038060730000000001 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.862868+0000", 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047860580000000002 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.933693+0000", 2024-09-21T18:58:39.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.070825186999999998 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.933742+0000", 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9391999999999997e-05 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:15.933780+0000", 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7625e-05 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242074 7.0 7:061b77dd:::benchmark_data_smithi081_73391_object103075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.497997+0000", 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 50.165394558000003, 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.175851656, 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242074 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.497997+0000", 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.497997+0000", 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.497999+0000", 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0600000000000002e-06 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498005+0000", 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.908e-06 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498020+0000", 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5089000000000001e-05 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498029+0000", 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1970000000000002e-06 2024-09-21T18:58:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498086+0000", 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6614999999999999e-05 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498200+0000", 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011442099999999999 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498280+0000", 2024-09-21T18:58:39.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0018999999999994e-05 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.501049+0000", 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027687020000000001 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511643+0000", 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010594643000000001 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.673781+0000", 2024-09-21T18:58:39.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.162137751 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.673817+0000", 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5805e-05 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.673848+0000", 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1445999999999997e-05 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242084 7.0 7:0c19e5c0:::benchmark_data_smithi081_73391_object103085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.512045+0000", 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 50.151345880999997, 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19629996199999999, 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242084 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512045+0000", 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512045+0000", 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512047+0000", 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.804e-06 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512052+0000", 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9400000000000001e-06 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512061+0000", 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.251e-06 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512067+0000", 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512105+0000", 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7907000000000001e-05 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512108+0000", 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8970000000000001e-06 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.513850+0000", 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001741289 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.513953+0000", 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103705 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.514153+0000", 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019997999999999999 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516849+0000", 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026958249999999998 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.678567+0000", 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16171744499999999 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.708270+0000", 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029703475999999999 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.708314+0000", 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3520999999999999e-05 2024-09-21T18:58:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.708345+0000", 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1631999999999998e-05 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242824 7.0 7:0ae306ee:::benchmark_data_smithi081_73391_object103825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:50.350874+0000", 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 49.312516819999999, 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12322082400000001, 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242824 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350874+0000", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350874+0000", 2024-09-21T18:58:39.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350877+0000", 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.446e-06 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350884+0000", 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4039999999999998e-06 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:39.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350901+0000", 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6529e-05 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350909+0000", 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-06 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350971+0000", 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1586000000000003e-05 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.351093+0000", 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012243900000000001 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.351207+0000", 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113706 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.354027+0000", 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028203239999999999 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.359333+0000", 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00530541 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.474025+0000", 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.114692632 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.474063+0000", 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7687e-05 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:39.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.474095+0000", 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2030999999999999e-05 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:39.789 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:39.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:39.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:39.913 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:39.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:40.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:40.036 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:40.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.159 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:40.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:40.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:40.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T18:58:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T18:58:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T18:58:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T18:58:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1222068866' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/369597576' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: pgmap v251: 158 pgs: 10 unknown, 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/369597576' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[39192]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T18:58:40.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1222068866' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/369597576' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: pgmap v251: 158 pgs: 10 unknown, 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/369597576' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T18:58:40.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:40 smithi081 ceph-mon[29384]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T18:58:40.383 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:40.403 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:40.430 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1222068866' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/369597576' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: pgmap v251: 158 pgs: 10 unknown, 148 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:40.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T18:58:40.434 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/369597576' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T18:58:40.434 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T18:58:40.434 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:40 smithi104 ceph-mon[32438]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T18:58:40.452 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T18:58:40.452 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T18:58:40.452 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T18:58:40.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T18:58:40.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T18:58:40.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:40.557 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:40.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:40.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:40.748 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:40.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:40.802 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:40.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:40.926 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:40.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:41.028 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:41.048 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:41.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:41 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T18:58:41.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:41 smithi104 ceph-mon[32438]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T18:58:41.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:41.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:41.224 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:58:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T18:58:41.225 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:41.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:41.294 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:41.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:41.417 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:41.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:41.539 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:41 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T18:58:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:41 smithi081 ceph-mon[29384]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T18:58:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:41 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T18:58:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:41 smithi081 ceph-mon[39192]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T18:58:41.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:41.661 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:41.763 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:41.784 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:41.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T18:58:41.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:41.908 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:41.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:42.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.031 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:42.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:42.051 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:42.132 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:42.153 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:42.255 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:42.277 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:42.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:42 smithi081 ceph-mon[29384]: pgmap v254: 142 pgs: 6 unknown, 136 active+clean; 792 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 11 op/s 2024-09-21T18:58:42.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:42 smithi081 ceph-mon[39192]: pgmap v254: 142 pgs: 6 unknown, 136 active+clean; 792 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 11 op/s 2024-09-21T18:58:42.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:42.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:42 smithi104 ceph-mon[32438]: pgmap v254: 142 pgs: 6 unknown, 136 active+clean; 792 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 11 op/s 2024-09-21T18:58:42.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:42.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:42.522 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:42.606 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:42.606 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:58:42.606 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T18:58:42.606 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T18:58:42.607 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:42.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.645 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:42.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:42.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:42.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:42.884 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:58:42.884 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:58:42.884 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-21T18:58:42.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-21T18:58:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:42.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:42.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:42.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:43.014 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_1' created 2024-09-21T18:58:43.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:58:43.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:43.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:43 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3775149050' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T18:58:43.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:43 smithi104 ceph-mon[32438]: from='client.35219 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T18:58:43.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T18:58:43.239 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:43.260 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:43 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3775149050' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T18:58:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:43 smithi081 ceph-mon[29384]: from='client.35219 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T18:58:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:43 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3775149050' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T18:58:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:43 smithi081 ceph-mon[39192]: from='client.35219 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T18:58:43.361 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.383 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:43.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:43.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:43.489 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:43.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:43.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:43.629 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:43.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:58:43.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:43.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:43.854 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:43.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:43.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:43.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102355 7.4 e105/82 7:2a73ca6f:::benchmark_data_smithi081_73391_object102354:head v 105'6448, mlcod=105'6446)", 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.327165+0000", 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.64545890700001, 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16628193299999999, 2024-09-21T18:58:43.976 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327165+0000", 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327165+0000", 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327166+0000", 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5990000000000001e-06 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327197+0000", 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1355e-05 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327209+0000", 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1109e-05 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327217+0000", 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1960000000000008e-06 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327305+0000", 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7937999999999997e-05 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327323+0000", 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8034999999999999e-05 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493402+0000", 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.166079438 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493446+0000", 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4263e-05 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:43.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102359 7.c 7:3c684ab9:::benchmark_data_smithi081_73391_object102358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.329664+0000", 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.64295929299999, 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16395372499999999, 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102359 2024-09-21T18:58:43.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329664+0000", 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329664+0000", 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:43.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329666+0000", 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.477e-06 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329703+0000", 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7696e-05 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329715+0000", 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1430000000000001e-05 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:43.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329722+0000", 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4100000000000002e-06 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329782+0000", 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9558000000000001e-05 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329907+0000", 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554600000000001 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330061+0000", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015371300000000001 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337366+0000", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073049380000000004 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493489+0000", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15612330799999999 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493524+0000", 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4567999999999998e-05 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493575+0000", 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1141999999999999e-05 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493618+0000", 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2938999999999997e-05 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:43.986 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102362 7.d 7:b4fc79ed:::benchmark_data_smithi081_73391_object102361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.330143+0000", 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.64248062499999, 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16063384, 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102362 2024-09-21T18:58:43.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330143+0000", 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330143+0000", 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330144+0000", 2024-09-21T18:58:43.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.361e-06 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330179+0000", 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4944000000000003e-05 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:43.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330189+0000", 2024-09-21T18:58:43.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:43.990 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9639999999999998e-06 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330196+0000", 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5259999999999999e-06 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333790+0000", 2024-09-21T18:58:43.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035948120000000002 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333921+0000", 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000131051 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.334318+0000", 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000396463 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:43.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337942+0000", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003624132 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489748+0000", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15180602900000001 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490678+0000", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093001499999999996 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490730+0000", 2024-09-21T18:58:43.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1604000000000003e-05 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490777+0000", 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6938999999999999e-05 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102363 7.c 7:396ad605:::benchmark_data_smithi081_73391_object102362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.64241587000001, 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.165470859, 2024-09-21T18:58:43.994 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102363 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3300000000000002e-07 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330233+0000", 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5241000000000001e-05 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330239+0000", 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9950000000000002e-06 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330244+0000", 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8119999999999998e-06 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333287+0000", 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030428870000000002 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333401+0000", 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113414 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333571+0000", 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017085599999999999 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.492806+0000", 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15923499299999999 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493628+0000", 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082127600000000002 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495595+0000", 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001967445 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495642+0000", 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7024000000000002e-05 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:43.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495678+0000", 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6282999999999998e-05 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102361 7.4 e105/82 7:2b82beac:::benchmark_data_smithi081_73391_object102360:head v 105'6449, mlcod=105'6446)", 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333737+0000", 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.638886216, 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.162036767, 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333737+0000", 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333737+0000", 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333739+0000", 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333760+0000", 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1755000000000001e-05 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333770+0000", 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9699999999999994e-06 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333777+0000", 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3740000000000003e-06 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335178+0000", 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014012409999999999 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335197+0000", 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8695e-05 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495739+0000", 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16054214 2024-09-21T18:58:44.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495774+0000", 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5228000000000002e-05 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102367 7.4 e105/82 7:2bc3d420:::benchmark_data_smithi081_73391_object102366:head v 105'6450, mlcod=105'6446)", 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333834+0000", 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.63878913900001, 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16196540800000001, 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.004 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333834+0000", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333834+0000", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333835+0000", 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6399999999999998e-07 2024-09-21T18:58:44.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333854+0000", 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9420999999999999e-05 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333859+0000", 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1490000000000002e-06 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333863+0000", 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.16e-06 2024-09-21T18:58:44.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337397+0000", 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00353382 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337414+0000", 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7051e-05 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495777+0000", 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.158363108 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495800+0000", 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2334999999999999e-05 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102368 7.4 e105/82 7:29e74c0a:::benchmark_data_smithi081_73391_object102367:head v 105'6451, mlcod=105'6446)", 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.336323+0000", 2024-09-21T18:58:44.008 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.636300804, 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15951227800000001, 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336323+0000", 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336323+0000", 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336324+0000", 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3260000000000001e-06 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336346+0000", 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1977e-05 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336355+0000", 2024-09-21T18:58:44.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3090000000000001e-06 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336361+0000", 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1689999999999997e-06 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.491302+0000", 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.154940193 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.491320+0000", 2024-09-21T18:58:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8844000000000001e-05 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495810+0000", 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004489537 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495835+0000", 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4922999999999998e-05 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102371 7.7 7:e4277e62:::benchmark_data_smithi081_73391_object102370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.337854+0000", 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.63476949899999, 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15779875900000001, 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102371 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337854+0000", 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337854+0000", 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337855+0000", 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5239999999999999e-06 2024-09-21T18:58:44.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337887+0000", 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1814999999999999e-05 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337895+0000", 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6639999999999998e-06 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337900+0000", 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.48e-06 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338644+0000", 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074344799999999996 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338755+0000", 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011141099999999999 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489524+0000", 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15076826500000001 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495470+0000", 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059459880000000001 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495536+0000", 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6666000000000002e-05 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495551+0000", 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5211000000000001e-05 2024-09-21T18:58:44.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495608+0000", 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6668e-05 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495653+0000", 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4619000000000001e-05 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.018 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102375 7.c 7:3f5265de:::benchmark_data_smithi081_73391_object102374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.338287+0000", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.634336182, 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15767785000000001, 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102375 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338287+0000", 2024-09-21T18:58:44.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338287+0000", 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338288+0000", 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.029e-06 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338354+0000", 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5264000000000004e-05 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338362+0000", 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.524e-06 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338369+0000", 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.421e-06 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489653+0000", 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15128423099999999 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489791+0000", 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013800800000000001 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489995+0000", 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000203773 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495713+0000", 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057185710000000004 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495809+0000", 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5754000000000005e-05 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495889+0000", 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9679999999999996e-05 2024-09-21T18:58:44.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495934+0000", 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5609000000000003e-05 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495965+0000", 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0985999999999998e-05 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop_reply(client.34742.0:102362 7.d e105/82 ondisk, result = 0)", 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.338914+0000", 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.63370932199999, 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15084166600000001, 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338914+0000", 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338914+0000", 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338915+0000", 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2810000000000001e-06 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338920+0000", 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3749999999999996e-06 2024-09-21T18:58:44.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489604+0000", 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150684605 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489671+0000", 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6698000000000004e-05 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489735+0000", 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4158999999999993e-05 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489742+0000", 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2150000000000004e-06 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489756+0000", 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3332999999999999e-05 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102370 7.4 e105/82 7:261d1db0:::benchmark_data_smithi081_73391_object102369:head v 105'6452, mlcod=105'6446)", 2024-09-21T18:58:44.028 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.339637+0000", 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 203.632986899, 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15622318700000001, 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339637+0000", 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339637+0000", 2024-09-21T18:58:44.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339638+0000", 2024-09-21T18:58:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1790000000000001e-06 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339663+0000", 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5381999999999999e-05 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489911+0000", 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15024772 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489917+0000", 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1090000000000003e-06 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493494+0000", 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035770260000000001 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493514+0000", 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9647999999999999e-05 2024-09-21T18:58:44.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495837+0000", 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002323215 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495860+0000", 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2908000000000002e-05 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136121 7.c 7:3d398460:::benchmark_data_smithi081_73391_object136120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.155866+0000", 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 175.81675719099999, 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13614505299999999, 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136121 2024-09-21T18:58:44.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155866+0000", 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155866+0000", 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155868+0000", 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9180000000000001e-06 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155929+0000", 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0381999999999999e-05 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155939+0000", 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0815999999999999e-05 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155946+0000", 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9920000000000001e-06 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155991+0000", 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4379999999999999e-05 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.156091+0000", 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010033 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.156182+0000", 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1057999999999995e-05 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.158936+0000", 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027542299999999999 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159467+0000", 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053071599999999998 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291914+0000", 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13244737400000001 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291965+0000", 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0878999999999999e-05 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292011+0000", 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5977999999999999e-05 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242081 7.7 7:eb350bf9:::benchmark_data_smithi081_73391_object103082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.508138+0000", 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 54.464485904, 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.194433776, 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.041 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242081 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508138+0000", 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508138+0000", 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508140+0000", 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0360000000000001e-06 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508146+0000", 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9549999999999997e-06 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508155+0000", 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6719999999999999e-06 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508162+0000", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7309999999999996e-06 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508367+0000", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000205353 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508471+0000", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103912 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,3", 2024-09-21T18:58:44.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508686+0000", 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215054 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511410+0000", 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027232530000000001 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699220+0000", 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18781029699999999 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.702479+0000", 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032595760000000001 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.702539+0000", 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9150000000000001e-05 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.702571+0000", 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2787000000000003e-05 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.046 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242087 7.d 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.512359+0000", 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 54.460264815999999, 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20359369999999999, 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.047 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242087 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512359+0000", 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512359+0000", 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512360+0000", 2024-09-21T18:58:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.519e-06 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512365+0000", 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7809999999999996e-06 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512377+0000", 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2211e-05 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512384+0000", 2024-09-21T18:58:44.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2699999999999999e-06 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512443+0000", 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8335000000000003e-05 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512575+0000", 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013186299999999999 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512712+0000", 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013723899999999999 2024-09-21T18:58:44.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516039+0000", 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033274899999999998 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.677347+0000", 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16130797199999999 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.678572+0000", 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00122426 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.715855+0000", 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.037282933999999997 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.715891+0000", 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6803e-05 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.715952+0000", 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1023000000000002e-05 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.052 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243316 7.d 7:bf0cd40c:::benchmark_data_smithi081_73391_object104317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.765579+0000", 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.207044443999997, 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99853839899999997, 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243316 2024-09-21T18:58:44.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765579+0000", 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765579+0000", 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765581+0000", 2024-09-21T18:58:44.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9300000000000002e-06 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765586+0000", 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6840000000000004e-06 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765595+0000", 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.798e-06 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765602+0000", 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9380000000000003e-06 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765644+0000", 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1282000000000002e-05 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765649+0000", 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.857e-06 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.756878+0000", 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99122877300000001 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.757004+0000", 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012556700000000001 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.757126+0000", 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012181 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.763931+0000", 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068057559999999996 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.763974+0000", 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2883999999999998e-05 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764026+0000", 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1981000000000003e-05 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764078+0000", 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2176000000000003e-05 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764117+0000", 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8962999999999999e-05 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243350 7.d 7:bfd5d960:::benchmark_data_smithi081_73391_object104351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.796299+0000", 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.176324497000003, 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97577692699999996, 2024-09-21T18:58:44.059 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243350 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796299+0000", 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.060 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796299+0000", 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796300+0000", 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1540000000000001e-06 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796304+0000", 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1960000000000001e-06 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796313+0000", 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4079999999999998e-06 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796319+0000", 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4400000000000002e-06 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796397+0000", 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7651999999999994e-05 2024-09-21T18:58:44.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.797134+0000", 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073711200000000005 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.759680+0000", 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96254598499999999 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.759744+0000", 2024-09-21T18:58:44.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3629999999999999e-05 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.759811+0000", 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7885999999999993e-05 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764195+0000", 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043838039999999998 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764234+0000", 2024-09-21T18:58:44.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9187999999999998e-05 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.771939+0000", 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0077048699999999999 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772013+0000", 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4122999999999995e-05 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772076+0000", 2024-09-21T18:58:44.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2478999999999996e-05 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243371 7.d 7:b3855816:::benchmark_data_smithi081_73391_object104372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.814706+0000", 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.157917384999998, 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95757182200000002, 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.066 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243371 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814706+0000", 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814706+0000", 2024-09-21T18:58:44.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814708+0000", 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8759999999999999e-06 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814714+0000", 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1199999999999999e-06 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814725+0000", 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1362000000000001e-05 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814733+0000", 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6079999999999998e-06 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814787+0000", 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4459999999999997e-05 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814796+0000", 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6519999999999995e-06 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.761141+0000", 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94634492999999997 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.761197+0000", 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5989000000000002e-05 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.761256+0000", 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8699000000000002e-05 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764201+0000", 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029451989999999999 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767108+0000", 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029069629999999998 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772190+0000", 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050818169999999998 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772240+0000", 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0204999999999998e-05 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772278+0000", 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7941999999999997e-05 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243372 7.d 7:b59e4c53:::benchmark_data_smithi081_73391_object104373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.815401+0000", 2024-09-21T18:58:44.072 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.157222746999999, 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.956966714, 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243372 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815401+0000", 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815401+0000", 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815402+0000", 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.443e-06 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815407+0000", 2024-09-21T18:58:44.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4789999999999998e-06 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815421+0000", 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3917000000000001e-05 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815428+0000", 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1300000000000003e-06 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815476+0000", 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7919e-05 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815483+0000", 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9519999999999997e-06 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.762409+0000", 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94692642900000001 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.762512+0000", 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102755 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.762629+0000", 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117614 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765635+0000", 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030059599999999998 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767146+0000", 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015110869999999999 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772321+0000", 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005174583 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772349+0000", 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7605000000000001e-05 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772367+0000", 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8841000000000001e-05 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243449 7.d 7:b7cc7edf:::benchmark_data_smithi081_73391_object104450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.919192+0000", 2024-09-21T18:58:44.079 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.053431666999998, 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.85324788200000001, 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243449 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919192+0000", 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919192+0000", 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919194+0000", 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.075e-06 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919199+0000", 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.485e-06 2024-09-21T18:58:44.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919212+0000", 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2564e-05 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919221+0000", 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1589999999999999e-06 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919289+0000", 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8225999999999994e-05 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919296+0000", 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6039999999999998e-06 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764271+0000", 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.84497510200000003 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764379+0000", 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010764099999999999 2024-09-21T18:58:44.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764479+0000", 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100608 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767124+0000", 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002644286 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.769391+0000", 2024-09-21T18:58:44.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022677740000000002 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772399+0000", 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030081679999999999 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772422+0000", 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2461000000000001e-05 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772440+0000", 2024-09-21T18:58:44.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7728999999999999e-05 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243457 7.d 7:b5e36520:::benchmark_data_smithi081_73391_object104458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.942755+0000", 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 52.029868778000001, 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82976072599999995, 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:44.086 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243457 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942755+0000", 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942755+0000", 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942758+0000", 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4800000000000001e-06 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942769+0000", 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0363999999999999e-05 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942795+0000", 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6043000000000001e-05 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942815+0000", 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0806e-05 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942928+0000", 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000112965 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942934+0000", 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2440000000000001e-06 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765669+0000", 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822735624 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765735+0000", 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6048000000000002e-05 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765796+0000", 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0699000000000003e-05 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767208+0000", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412127 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.769429+0000", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022209420000000001 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772472+0000", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030429020000000001 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:44.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772497+0000", 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.533e-05 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772515+0000", 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8151999999999998e-05 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:44.092 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:44.093 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:44.093 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:44.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:44.215 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[39192]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[39192]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[39192]: pgmap v256: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[29384]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T18:58:44.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[29384]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T18:58:44.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[29384]: pgmap v256: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-09-21T18:58:44.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:44.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:44 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:44 smithi104 ceph-mon[32438]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T18:58:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:44 smithi104 ceph-mon[32438]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T18:58:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:44 smithi104 ceph-mon[32438]: pgmap v256: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 461 B/s rd, 0 op/s 2024-09-21T18:58:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:44 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:44 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:44.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:44.338 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:44.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:44.460 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:44.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:44.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:44.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:44.706 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:44.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:44.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T18:58:44.828 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:44.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:44.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:44.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:45.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:45.059 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:45.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:45.197 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:45.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:45.321 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:45.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:45 smithi104 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:58:45.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:45 smithi104 ceph-mon[32438]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T18:58:45.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:45 smithi104 ceph-mon[32438]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T18:58:45.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:45.444 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:45.493 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:45.493 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:58:45.493 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T18:58:45.493 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:45.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:45.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:45 smithi081 ceph-mon[29384]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:58:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:45 smithi081 ceph-mon[29384]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T18:58:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:45 smithi081 ceph-mon[29384]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T18:58:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:45 smithi081 ceph-mon[39192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T18:58:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:45 smithi081 ceph-mon[39192]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T18:58:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:45 smithi081 ceph-mon[39192]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T18:58:45.569 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:45.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:45.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:45.817 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:45.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:45.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:45.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:46.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:46.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:46.121 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:46 smithi104 ceph-mon[32438]: pgmap v259: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 618 B/s rd, 0 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T18:58:46.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T18:58:46.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:46.189 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:46.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.313 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:46.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:46 smithi081 ceph-mon[29384]: pgmap v259: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 618 B/s rd, 0 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T18:58:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:46 smithi081 ceph-mon[39192]: pgmap v259: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 618 B/s rd, 0 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T18:58:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:46.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:46.435 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:46.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-21T18:58:46.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:46.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:46.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:46.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:46.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:46.805 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:46.811 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:46.824 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:46.824 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T18:58:46.824 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T18:58:46.824 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T18:58:46.825 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:46.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:46.928 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:46.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:47.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:47.052 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:47.098 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:58:47.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:47.175 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:47.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:47.401 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:47.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:47.470 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1350432390' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:47.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T18:58:47.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:47.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1350432390' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:47.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1350432390' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:58:47.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:58:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:58:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:58:47.649 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.671 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:47.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:47.745 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T18:58:47.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:47.793 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:47.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:47.915 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:47.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:48.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:48.037 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:48.107 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:58:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:58:48.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:48 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:48.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1350432390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:48.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:48 smithi104 ceph-mon[32438]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T18:58:48.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:48 smithi104 ceph-mon[32438]: pgmap v261: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T18:58:48.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:48.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:48.160 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54921 7.5 7:a314aa6c:::benchmark_data_smithi081_73391_object54920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 241.74773818200001, 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.074546457999999996, 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.208 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54921 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6599999999999996e-07 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457166+0000", 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5052000000000003e-05 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457172+0000", 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457178+0000", 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9259999999999997e-06 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459247+0000", 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020685429999999999 2024-09-21T18:58:48.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459371+0000", 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012465100000000001 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459488+0000", 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011696000000000001 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461191+0000", 2024-09-21T18:58:48.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001702924 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.502032+0000", 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.040840333999999999 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531576+0000", 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029544002999999999 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531634+0000", 2024-09-21T18:58:48.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8527999999999999e-05 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531676+0000", 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2054000000000001e-05 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54929 7.5 7:af33521c:::benchmark_data_smithi081_73391_object54928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.461656+0000", 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 241.74321169699999, 2024-09-21T18:58:48.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076295636, 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54929 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461656+0000", 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461656+0000", 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461657+0000", 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.094e-06 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461720+0000", 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2421000000000004e-05 2024-09-21T18:58:48.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461729+0000", 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9320000000000002e-06 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461735+0000", 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4479999999999999e-06 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461793+0000", 2024-09-21T18:58:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7732000000000002e-05 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461920+0000", 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000126957 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.462069+0000", 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014942500000000001 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.463573+0000", 2024-09-21T18:58:48.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015035960000000001 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.502383+0000", 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.038809896000000003 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537843+0000", 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.035460301 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537910+0000", 2024-09-21T18:58:48.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537952+0000", 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1569000000000001e-05 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99380 7.5 7:a5c9fcf3:::benchmark_data_smithi081_73391_object99379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.710751+0000", 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.49411726, 2024-09-21T18:58:48.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079490342000000005, 2024-09-21T18:58:48.221 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.221 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.221 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.221 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99380 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710751+0000", 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710751+0000", 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710752+0000", 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7129999999999999e-06 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710804+0000", 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1554999999999997e-05 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710813+0000", 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3589999999999997e-06 2024-09-21T18:58:48.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710820+0000", 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7950000000000002e-06 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710870+0000", 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0315000000000002e-05 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710979+0000", 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010852 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.711073+0000", 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4493000000000005e-05 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.712997+0000", 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019239000000000001 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713798+0000", 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080073700000000004 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790157+0000", 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076358910000000002 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790215+0000", 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8378999999999998e-05 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790241+0000", 2024-09-21T18:58:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5666000000000001e-05 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101889 7.f e105/82 7:f2f30f0c:::benchmark_data_smithi081_73391_object101888:head v 105'6311, mlcod=105'6308)", 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.908994+0000", 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 208.29587380999999, 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087156066000000004, 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908995+0000", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9800000000000003e-07 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909027+0000", 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2353e-05 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909033+0000", 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1859999999999997e-06 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909038+0000", 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7249999999999997e-06 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909131+0000", 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2363999999999999e-05 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909149+0000", 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8450000000000001e-05 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996090+0000", 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086940559000000001 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996150+0000", 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0730999999999998e-05 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101894 7.8 e105/79 7:1138a4d6:::benchmark_data_smithi081_73391_object101893:head v 105'6424, mlcod=105'6422)", 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910208+0000", 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 208.29466009000001, 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086428049000000007, 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910209+0000", 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.24e-06 2024-09-21T18:58:48.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910259+0000", 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9756000000000003e-05 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910269+0000", 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0035e-05 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910276+0000", 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8040000000000004e-06 2024-09-21T18:58:48.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914457+0000", 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041814679999999998 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914478+0000", 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0528999999999999e-05 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996573+0000", 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082095020000000005 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996636+0000", 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3196999999999998e-05 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101892 7.f e105/82 7:f5e7532a:::benchmark_data_smithi081_73391_object101891:head v 105'6312, mlcod=105'6308)", 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910906+0000", 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 208.29396222899999, 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085280135000000007, 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.235 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910907+0000", 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910962+0000", 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5427000000000003e-05 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910972+0000", 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2520000000000005e-06 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910978+0000", 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.117e-06 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914620+0000", 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036426869999999999 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914639+0000", 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8848999999999999e-05 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996155+0000", 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.081515729999999995 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996186+0000", 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0874000000000003e-05 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101891 7.c e105/79 7:3b3bc22b:::benchmark_data_smithi081_73391_object101890:head v 105'6296, mlcod=105'6294)", 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.912575+0000", 2024-09-21T18:58:48.239 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 208.29229264, 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084158549999999999, 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912575+0000", 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912575+0000", 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912577+0000", 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.316e-06 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912603+0000", 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5993999999999999e-05 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912614+0000", 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1082e-05 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912621+0000", 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.063e-06 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.913518+0000", 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089733900000000004 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.913546+0000", 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7746000000000002e-05 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916419+0000", 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028730499999999998 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996734+0000", 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080314960000000005 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101901 7.5 7:a8e9101c:::benchmark_data_smithi081_73391_object101900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:48.243 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916321+0000", 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 208.28854708399999, 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087318653999999996, 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101901 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T18:58:48.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916329+0000", 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3990000000000004e-06 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916364+0000", 2024-09-21T18:58:48.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4808999999999998e-05 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916374+0000", 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0298e-05 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916382+0000", 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996259+0000", 2024-09-21T18:58:48.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079877760000000006 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996412+0000", 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015201099999999999 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996586+0000", 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017437 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003419+0000", 2024-09-21T18:58:48.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068332979999999998 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003455+0000", 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5438000000000002e-05 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003478+0000", 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3048e-05 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003584+0000", 2024-09-21T18:58:48.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106331 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003640+0000", 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5583e-05 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101902 7.5 7:a3b6d405:::benchmark_data_smithi081_73391_object101901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916400+0000", 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 208.288468075, 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087823038000000006, 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.249 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101902 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916401+0000", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5899999999999996e-07 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916430+0000", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9076000000000001e-05 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996083+0000", 2024-09-21T18:58:48.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079653529000000001 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996103+0000", 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0015999999999999e-05 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998560+0000", 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002457126 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998676+0000", 2024-09-21T18:58:48.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115328 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998809+0000", 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133767 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003647+0000", 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048375040000000003 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003682+0000", 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004144+0000", 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046231000000000001 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004190+0000", 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5958999999999998e-05 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004223+0000", 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2840999999999999e-05 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.254 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114519 7.6 7:6b5e3486:::benchmark_data_smithi081_73391_object114518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709106+0000", 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 197.49576225800001, 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12712907900000001, 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114519 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709107+0000", 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.66e-06 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709157+0000", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9911999999999999e-05 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709171+0000", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3947e-05 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709179+0000", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3529999999999997e-06 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716464+0000", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072858719999999997 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716590+0000", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554999999999999 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716718+0000", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012810499999999999 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723181+0000", 2024-09-21T18:58:48.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006462944 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723217+0000", 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5593000000000003e-05 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836081+0000", 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11286423700000001 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836188+0000", 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107315 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836235+0000", 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6591000000000002e-05 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114521 7.6 7:6fe397a7:::benchmark_data_smithi081_73391_object114520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709300+0000", 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 197.495567709, 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12709793799999999, 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.260 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114521 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709301+0000", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6199999999999996e-07 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709354+0000", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2936999999999998e-05 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709361+0000", 2024-09-21T18:58:48.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.097e-06 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709367+0000", 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6550000000000001e-06 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719193+0000", 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098261459999999991 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719453+0000", 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026041699999999998 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.720410+0000", 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095699200000000004 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723227+0000", 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002816888 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723832+0000", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060487799999999999 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836280+0000", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11244789099999999 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836362+0000", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1967999999999999e-05 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836398+0000", 2024-09-21T18:58:48.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6207000000000001e-05 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114538 7.6 7:657d97a1:::benchmark_data_smithi081_73391_object114537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723638+0000", 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 197.48123003699999, 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11594671600000001, 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.266 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114538 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723640+0000", 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.801e-06 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723678+0000", 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7873000000000001e-05 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723688+0000", 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0071000000000001e-05 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723695+0000", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3429999999999998e-06 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723868+0000", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017280900000000001 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724003+0000", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013505 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:58:48.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724166+0000", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162764 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729028+0000", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048623590000000001 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729096+0000", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7625000000000004e-05 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836438+0000", 2024-09-21T18:58:48.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107342302 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839527+0000", 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030888589999999998 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839585+0000", 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7859999999999998e-05 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114539 7.6 7:6de988ff:::benchmark_data_smithi081_73391_object114538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723706+0000", 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 197.48116158600001, 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11604534699999999, 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114539 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723707+0000", 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4399999999999999e-07 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723743+0000", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5395999999999998e-05 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723749+0000", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1850000000000001e-06 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723754+0000", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6219999999999997e-06 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726769+0000", 2024-09-21T18:58:48.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030147289999999998 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726889+0000", 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012032999999999999 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727041+0000", 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015136699999999999 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729293+0000", 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002252085 2024-09-21T18:58:48.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730090+0000", 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00079701999999999996 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839629+0000", 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10953908599999999 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839710+0000", 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0630000000000006e-05 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839752+0000", 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2153000000000001e-05 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121044 7.6 7:62f149b0:::benchmark_data_smithi081_73391_object121043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.421001+0000", 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 191.783867407, 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084735675999999996, 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.278 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121044 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T18:58:48.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421002+0000", 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.336e-06 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421062+0000", 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0475e-05 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421072+0000", 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9389999999999992e-06 2024-09-21T18:58:48.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421078+0000", 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2870000000000004e-06 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421128+0000", 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9938000000000002e-05 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421252+0000", 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012434499999999999 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421447+0000", 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019541299999999999 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425265+0000", 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038176640000000001 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428690+0000", 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034252710000000001 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505655+0000", 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076964640000000001 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505691+0000", 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5887999999999999e-05 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505736+0000", 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5479999999999998e-05 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121051 7.6 7:619bd2cd:::benchmark_data_smithi081_73391_object121050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.425560+0000", 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 191.77930818600001, 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088495110000000002, 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121051 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.284 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425562+0000", 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.154e-06 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425607+0000", 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5229999999999999e-05 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425620+0000", 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2346e-05 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425627+0000", 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425678+0000", 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1053000000000001e-05 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425785+0000", 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107099 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425916+0000", 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013072999999999999 2024-09-21T18:58:48.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428156+0000", 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022408649999999999 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428832+0000", 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067540999999999999 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.513961+0000", 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085129596000000002 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514015+0000", 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3313000000000003e-05 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514055+0000", 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0156e-05 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145390 7.5 7:a9dbf476:::benchmark_data_smithi081_73391_object6391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.350105+0000", 2024-09-21T18:58:48.289 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 168.85476346600001, 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080355056999999994, 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145390 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350105+0000", 2024-09-21T18:58:48.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350105+0000", 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350106+0000", 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5820000000000001e-06 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350111+0000", 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1220000000000003e-06 2024-09-21T18:58:48.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350120+0000", 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6680000000000007e-06 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350125+0000", 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3329999999999996e-06 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350166+0000", 2024-09-21T18:58:48.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0370000000000001e-05 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350248+0000", 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2588000000000003e-05 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,2", 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350334+0000", 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5597000000000001e-05 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.352607+0000", 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022733499999999999 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.353408+0000", 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080050800000000001 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.353423+0000", 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4961e-05 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.430264+0000", 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076841719000000003 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.430327+0000", 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3082000000000004e-05 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.430460+0000", 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013217700000000001 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211656 7.5 7:a3c1ba51:::benchmark_data_smithi081_73391_object72657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.348024+0000", 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 93.856843624999996, 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107860995, 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211656 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348024+0000", 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348024+0000", 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348027+0000", 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6709999999999999e-06 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348033+0000", 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4779999999999999e-06 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348046+0000", 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348053+0000", 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2980000000000003e-06 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348113+0000", 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9852999999999998e-05 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348226+0000", 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011289 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348343+0000", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116623 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.350450+0000", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021071620000000001 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.357782+0000", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073323420000000004 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455803+0000", 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098020501999999995 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455840+0000", 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7206000000000003e-05 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455885+0000", 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5330999999999997e-05 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211679 7.5 7:a0aaf166:::benchmark_data_smithi081_73391_object72680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.371480+0000", 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 93.833387974999994, 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084602176000000001, 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211679 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371480+0000", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371480+0000", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371482+0000", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371486+0000", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5820000000000003e-06 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371497+0000", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0801e-05 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371503+0000", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9000000000000003e-06 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371564+0000", 2024-09-21T18:58:48.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0491000000000001e-05 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371687+0000", 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012302999999999999 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371815+0000", 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864899999999999 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.373861+0000", 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020455870000000002 2024-09-21T18:58:48.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.375558+0000", 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016976879999999999 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455985+0000", 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080426912000000003 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.456037+0000", 2024-09-21T18:58:48.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1109999999999997e-05 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.456082+0000", 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5686000000000003e-05 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:241829 7.5 7:a585a8ba:::benchmark_data_smithi081_73391_object102830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.216357+0000", 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 58.988511045999999, 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073916367999999996, 2024-09-21T18:58:48.307 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 241829 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216357+0000", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216357+0000", 2024-09-21T18:58:48.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216359+0000", 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216366+0000", 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.669e-06 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216381+0000", 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5705999999999998e-05 2024-09-21T18:58:48.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216390+0000", 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5499999999999995e-06 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216448+0000", 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7556000000000003e-05 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216597+0000", 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014921199999999999 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216707+0000", 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110716 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.219439+0000", 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027312679999999998 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.287801+0000", 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.068362676999999997 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.290143+0000", 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023415229999999999 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.290216+0000", 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3202999999999997e-05 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.290273+0000", 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7166999999999997e-05 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.312 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242836 7.5 7:a22dd50b:::benchmark_data_smithi081_73391_object103837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:50.363052+0000", 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 57.841816489999999, 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.121274044, 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242836 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T18:58:48.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363052+0000", 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363052+0000", 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363054+0000", 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1160000000000002e-06 2024-09-21T18:58:48.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363059+0000", 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6729999999999997e-06 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363072+0000", 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2897e-05 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363080+0000", 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4229999999999999e-06 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363131+0000", 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1759999999999999e-05 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363205+0000", 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3712000000000002e-05 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363271+0000", 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5877999999999995e-05 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.365405+0000", 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021336670000000001 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.366435+0000", 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010303689999999999 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484220+0000", 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11778470100000001 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484278+0000", 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8464000000000001e-05 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484326+0000", 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7383999999999999e-05 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T18:58:48.318 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T18:58:48.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:48.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1350432390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:48.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[29384]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T18:58:48.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[29384]: pgmap v261: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T18:58:48.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:48.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:48.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1350432390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:58:48.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[39192]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T18:58:48.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[39192]: pgmap v261: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T18:58:48.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:48.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:48.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:48.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:48.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:48.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:48.705 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:48.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:48.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:48.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:48.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:48.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:49.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T18:58:49.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:49.071 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:49.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.195 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:49.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:49.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:49.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:49.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-21T18:58:49.442 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:49.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:49.566 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:49.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:49.674 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:49.675 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.691 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:49.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:49.693 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:49.693 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:49.709 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:49.710 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:49.725 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:49.725 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:49.742 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:49.742 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:49.758 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:49.758 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:49.774 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:49.774 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:49.790 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:49.790 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:49.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:49.813 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:49.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:49.936 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:49.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:50.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:50.059 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:50.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.184 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:50.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:50.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:50.307 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:50.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:50.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:50.452 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:50.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:50.548 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T18:58:50.556 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:50.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:50.680 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[29384]: pgmap v262: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[29384]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2302877334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1676723562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[39192]: pgmap v262: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[39192]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2302877334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:50.691 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1676723562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:58:50.782 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:50.803 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:50.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:50.925 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:50.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:50.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:58:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:50 smithi104 ceph-mon[32438]: pgmap v262: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:50 smithi104 ceph-mon[32438]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T18:58:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2302877334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1676723562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T18:58:51.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:51.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:51.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:51.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:51.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:51.313 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:51.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:51.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:51.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1676723562' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[29384]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1676723562' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[39192]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:51.567 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:51.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:51.691 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:51.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:51.819 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:51.920 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:51.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:51.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:51.957 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:58:51.957 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:58:52.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:51 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1676723562' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T18:58:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:51 smithi104 ceph-mon[32438]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T18:58:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:58:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:58:52.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:52.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:52.188 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:52.205 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:52.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.312 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:52.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:52.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:52.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:58:52.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:52.563 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:52.639 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:52 smithi081 ceph-mon[39192]: pgmap v265: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:52.639 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:52 smithi081 ceph-mon[39192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:58:52.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:52 smithi081 ceph-mon[29384]: pgmap v265: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:52.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:52 smithi081 ceph-mon[29384]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:58:52.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:52.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T18:58:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.689 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:52.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:52.717 DEBUG:teuthology.orchestra.run.smithi081:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T18:58:52.732 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T18:58:52.732 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T18:58:52.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:hints = 1 2024-09-21T18:58:52.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T18:58:52.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object prefix: benchmark_data_smithi081_105538 2024-09-21T18:58:52.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:58:52.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 0 0 0 0 0 0 - 0 2024-09-21T18:58:52.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.839 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:52.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:52.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:52.965 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:52.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:53.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:52 smithi104 ceph-mon[32438]: pgmap v265: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:53.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:52 smithi104 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T18:58:53.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:53.097 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:53.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:53.228 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:53.318 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:53.330 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:53.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:53.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:53.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:53.630 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:53.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:53.763 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:53.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 1 16 3435 3419 213.676 213.688 0.0049985 0.00466387 2024-09-21T18:58:53.864 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:53.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:53.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:53.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:54.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:54.042 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:54.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:54.175 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:54.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:54.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:54.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:54.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:54.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.573 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:54.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:54.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.723 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:54.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[29384]: pgmap v266: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 801 B/s rd, 0 op/s 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[29384]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[39192]: pgmap v266: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 801 B/s rd, 0 op/s 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:58:54.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:58:54.732 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:54 smithi081 ceph-mon[39192]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T18:58:54.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 2 15 6471 6456 201.736 189.812 0.0111968 0.00494246 2024-09-21T18:58:54.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:54.854 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:54.856 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:58:54.856 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:58:54.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:58:54.880 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:58:54.915 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:58:54.915 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:58:54.953 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:58:54.953 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:58:54.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:54.980 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:54.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:54.985 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:58:54.985 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:58:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:54 smithi104 ceph-mon[32438]: pgmap v266: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 801 B/s rd, 0 op/s 2024-09-21T18:58:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:54 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:58:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:54 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:58:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:54 smithi104 ceph-mon[32438]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T18:58:55.021 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:58:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:58:55.056 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:58:55.057 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:58:55.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:55.092 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:58:55.092 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:58:55.120 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:55.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:55.253 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:55.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:55.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:55.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:55.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-21T18:58:55.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:55.584 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:58:55.632 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:55.658 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:55.718 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1457197777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:55.718 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1387041967' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:55.718 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:55 smithi081 ceph-mon[29384]: from='client.35270 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:55.719 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1457197777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:55.719 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1387041967' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:55.719 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:55 smithi081 ceph-mon[39192]: from='client.35270 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:55.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:55.782 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 3 16 9367 9351 194.798 180.938 0.00316668 0.00510683 2024-09-21T18:58:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:55.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:55.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:55.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:56.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1457197777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:58:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1387041967' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:55 smithi104 ceph-mon[32438]: from='client.35270 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:58:56.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:56.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:56.203 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:56.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:56.342 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:56.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:56.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:56.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:56.617 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:56.701 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[29384]: pgmap v268: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[29384]: from='client.35270 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[29384]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[39192]: pgmap v268: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[39192]: from='client.35270 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:58:56.702 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:56 smithi081 ceph-mon[39192]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T18:58:56.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:56.750 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:56.781 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 4 16 12458 12442 194.391 193.188 0.00751817 0.00513706 2024-09-21T18:58:56.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:56.889 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:56.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:56.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:58:56.996 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:56.999 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:58:56.999 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:58:56.999 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:58:56.999 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:58:56.999 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:58:57.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:56 smithi104 ceph-mon[32438]: pgmap v268: 158 pgs: 158 active+clean; 583 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T18:58:57.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:56 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:58:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:56 smithi104 ceph-mon[32438]: from='client.35270 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:58:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:56 smithi104 ceph-mon[32438]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T18:58:57.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:58:57.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:57.170 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:57.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:57.301 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:57.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:58:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:58:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:58:57.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:58:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:58:57.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:57.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:57.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:57.713 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:57.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 5 16 15317 15301 191.061 178.688 0.00406339 0.00522637 2024-09-21T18:58:57.802 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:57 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T18:58:57.802 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:57 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T18:58:57.802 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:57 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T18:58:57.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:57 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T18:58:57.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:57 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T18:58:57.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:57 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T18:58:57.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:57.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:58.088 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:58.122 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:58.224 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:58.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:58:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:58:58.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:58.360 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:58.390 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:58.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T18:58:58.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:58:58.521 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:58:58.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[29384]: pgmap v270: 158 pgs: 158 active+clean; 765 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[39192]: pgmap v270: 158 pgs: 158 active+clean; 765 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T18:58:58.633 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T18:58:58.634 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:58.650 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:58.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:58.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:58:58.783 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:58.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 6 16 18002 17986 187.17 167.812 0.00476378 0.00533656 2024-09-21T18:58:58.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:58.915 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:58.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:59.004 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:58:59.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:58 smithi104 ceph-mon[32438]: pgmap v270: 158 pgs: 158 active+clean; 765 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-21T18:58:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:58 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T18:58:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:58 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T18:58:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:58:59.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:59.045 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:59.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:58:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.187 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:59.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:58:59.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:58:59.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:58:59.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:59.451 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:58:59.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:59.581 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:59.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:58:59.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:58:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.721 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:58:59.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:58:59.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 7 15 20411 20396 181.952 150.625 0.00777613 0.00548911 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[29384]: 4.15 scrub starts 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[29384]: 4.15 scrub ok 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[29384]: 5.13 scrub starts 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[29384]: 5.13 scrub ok 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[29384]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[39192]: 4.15 scrub starts 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[39192]: 4.15 scrub ok 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[39192]: 5.13 scrub starts 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[39192]: 5.13 scrub ok 2024-09-21T18:58:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:58:59 smithi081 ceph-mon[39192]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T18:58:59.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:58:59.848 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:58:59.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:58:59.982 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:58:59.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:58:59.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:59 smithi104 ceph-mon[32438]: 4.15 scrub starts 2024-09-21T18:59:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:59 smithi104 ceph-mon[32438]: 4.15 scrub ok 2024-09-21T18:59:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:59 smithi104 ceph-mon[32438]: 5.13 scrub starts 2024-09-21T18:59:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:59 smithi104 ceph-mon[32438]: 5.13 scrub ok 2024-09-21T18:59:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:58:59 smithi104 ceph-mon[32438]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T18:59:00.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:00.125 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:00.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:00.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:00.261 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:00.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:00.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:00.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:00.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:00.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:00.537 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:00.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:00.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:00.677 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:00.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:00.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:00.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 8 16 22857 22841 178.31 152.812 0.00349461 0.00560062 2024-09-21T18:59:00.814 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:00.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:00 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T18:59:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:00 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T18:59:00.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:00 smithi104 ceph-mon[32438]: pgmap v271: 158 pgs: 158 active+clean; 765 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-21T18:59:00.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201722 7.9 7:91094263:::benchmark_data_smithi081_73391_object62723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.568721+0000", 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 117.634567135, 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0597567990000001, 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.920 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201722 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568721+0000", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568721+0000", 2024-09-21T18:59:00.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568724+0000", 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5179999999999999e-06 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568729+0000", 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9000000000000003e-06 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568744+0000", 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4598e-05 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568751+0000", 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1899999999999998e-06 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568806+0000", 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4280000000000002e-05 2024-09-21T18:59:00.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568815+0000", 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2779999999999999e-06 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.604282+0000", 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0354674319999999 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.604409+0000", 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012624199999999999 2024-09-21T18:59:00.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.604524+0000", 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115437 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608000+0000", 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003475625 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608063+0000", 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3607000000000003e-05 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628368+0000", 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020305230000000001 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628428+0000", 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9450000000000002e-05 2024-09-21T18:59:00.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628478+0000", 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0012000000000002e-05 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201747 7.9 7:99156be5:::benchmark_data_smithi081_73391_object62748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T18:59:00.927 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.597481+0000", 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 117.605807008, 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0312291490000001, 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201747 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597481+0000", 2024-09-21T18:59:00.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597481+0000", 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597483+0000", 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1169999999999998e-06 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597490+0000", 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0559999999999999e-06 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597502+0000", 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1697e-05 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597510+0000", 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7670000000000003e-06 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597570+0000", 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0121999999999998e-05 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597579+0000", 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1819999999999995e-06 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608102+0000", 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0105227489999999 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608217+0000", 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114998 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608327+0000", 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010976600000000001 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614839+0000", 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0065126089999999999 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614897+0000", 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8202000000000003e-05 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628577+0000", 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013679358000000001 2024-09-21T18:59:00.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628676+0000", 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8945999999999998e-05 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628710+0000", 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4579999999999998e-05 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201757 7.9 7:9e7f9360:::benchmark_data_smithi081_73391_object62758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.608364+0000", 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 117.594924386, 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0210702650000001, 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201757 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.935 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608364+0000", 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608364+0000", 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608366+0000", 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0159999999999998e-06 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608372+0000", 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0009999999999998e-06 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608385+0000", 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2704000000000001e-05 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608391+0000", 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608495+0000", 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103872 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608504+0000", 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7779999999999996e-06 2024-09-21T18:59:00.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.611920+0000", 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0034158049999999 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.612052+0000", 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013184199999999999 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.612164+0000", 2024-09-21T18:59:00.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011197900000000001 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614934+0000", 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027704489999999999 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.617594+0000", 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002659809 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628761+0000", 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011167362 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629387+0000", 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062587899999999998 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629434+0000", 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6825e-05 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.941 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201759 7.9 7:99785370:::benchmark_data_smithi081_73391_object62760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.608408+0000", 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 117.594880291, 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0211269030000001, 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.942 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201759 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608408+0000", 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608408+0000", 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608408+0000", 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8299999999999997e-07 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608413+0000", 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2089999999999999e-06 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608419+0000", 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8839999999999999e-06 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608424+0000", 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1850000000000003e-06 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608526+0000", 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102303 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608532+0000", 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5799999999999999e-06 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614965+0000", 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.006433557 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.615116+0000", 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015114699999999999 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.615181+0000", 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4858999999999997e-05 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.620299+0000", 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051176720000000002 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.622872+0000", 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025732839999999999 2024-09-21T18:59:00.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629444+0000", 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0065717370000000002 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629495+0000", 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1044000000000001e-05 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629535+0000", 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9858999999999999e-05 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216350 7.6 7:693ca6d8:::benchmark_data_smithi081_73391_object77351:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518162+0000", 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.685125922, 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2616681970000001, 2024-09-21T18:59:00.949 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216350 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518164+0000", 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.593e-06 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518169+0000", 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6689999999999997e-06 2024-09-21T18:59:00.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518176+0000", 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8180000000000004e-06 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518182+0000", 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2549999999999997e-06 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518270+0000", 2024-09-21T18:59:00.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8738000000000003e-05 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518279+0000", 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779775+0000", 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.26149613 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779830+0000", 2024-09-21T18:59:00.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5727999999999999e-05 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216360 7.6 7:6a035ea2:::benchmark_data_smithi081_73391_object77361:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518188+0000", 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.68509986799999, 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2839865960000001, 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T18:59:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216360 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518189+0000", 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5400000000000002e-07 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518191+0000", 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518196+0000", 2024-09-21T18:59:00.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.622e-06 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518200+0000", 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6190000000000002e-06 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518299+0000", 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8587999999999999e-05 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518304+0000", 2024-09-21T18:59:00.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6210000000000001e-06 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780077+0000", 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2617725909999999 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780187+0000", 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110469 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802102+0000", 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021914330999999999 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802175+0000", 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3116999999999998e-05 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216394 7.6 7:6d24eb15:::benchmark_data_smithi081_73391_object77395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.959 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.559012+0000", 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.644275862, 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2297531310000001, 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216394 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559014+0000", 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0889999999999998e-06 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559020+0000", 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1669999999999998e-06 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559034+0000", 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4385e-05 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559041+0000", 2024-09-21T18:59:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.238e-06 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570552+0000", 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011510347000000001 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570561+0000", 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6209999999999999e-06 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780222+0000", 2024-09-21T18:59:00.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2096608149999999 2024-09-21T18:59:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780290+0000", 2024-09-21T18:59:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7988e-05 2024-09-21T18:59:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:00.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780361+0000", 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1239e-05 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788612+0000", 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082509909999999992 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788622+0000", 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159999999999999e-05 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788671+0000", 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8939e-05 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788709+0000", 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7295999999999997e-05 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788765+0000", 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6855999999999999e-05 2024-09-21T18:59:00.967 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216429 7.6 7:6000252a:::benchmark_data_smithi081_73391_object77430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.616476+0000", 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.58681219899999, 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1724721330000001, 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.968 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216429 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616478+0000", 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616484+0000", 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616495+0000", 2024-09-21T18:59:00.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1134999999999999e-05 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616502+0000", 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8569999999999997e-06 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616558+0000", 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6023999999999998e-05 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616567+0000", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8980000000000002e-06 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782338+0000", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.165771565 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782505+0000", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166558 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782607+0000", 2024-09-21T18:59:00.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102357 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788626+0000", 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006018921 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788801+0000", 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000174852 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788853+0000", 2024-09-21T18:59:00.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2006000000000003e-05 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788898+0000", 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4962000000000001e-05 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788948+0000", 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0006999999999997e-05 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.974 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216455 7.6 7:68c88556:::benchmark_data_smithi081_73391_object77456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.644975+0000", 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.558313423, 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.144373474, 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216455 2024-09-21T18:59:00.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644977+0000", 2024-09-21T18:59:00.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5500000000000001e-06 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644983+0000", 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0360000000000003e-06 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644995+0000", 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2006e-05 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645003+0000", 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9459999999999998e-06 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645059+0000", 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5337999999999998e-05 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645068+0000", 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4320000000000005e-06 2024-09-21T18:59:00.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.784935+0000", 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1398670499999999 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785006+0000", 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0840999999999995e-05 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785084+0000", 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8228999999999999e-05 2024-09-21T18:59:00.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788630+0000", 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035455730000000002 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789144+0000", 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00051452099999999999 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789199+0000", 2024-09-21T18:59:00.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4484000000000002e-05 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789251+0000", 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2633000000000002e-05 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789348+0000", 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6835000000000004e-05 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.981 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216459 7.6 7:66d9de63:::benchmark_data_smithi081_73391_object77460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.649564+0000", 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.553724622, 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1586775789999999, 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216459 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649566+0000", 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0540000000000002e-06 2024-09-21T18:59:00.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649586+0000", 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0472e-05 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649616+0000", 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9848e-05 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649637+0000", 2024-09-21T18:59:00.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1114000000000001e-05 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649894+0000", 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025646600000000001 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649902+0000", 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4589999999999998e-06 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789632+0000", 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1397303350000001 2024-09-21T18:59:00.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789788+0000", 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155852 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789900+0000", 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111909 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802209+0000", 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012309334 2024-09-21T18:59:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802251+0000", 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1421000000000002e-05 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808109+0000", 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058583660000000003 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808145+0000", 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5374999999999999e-05 2024-09-21T18:59:00.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808241+0000", 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6574000000000001e-05 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216478 7.6 7:69b4062c:::benchmark_data_smithi081_73391_object77479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.673950+0000", 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.529338016, 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.137844316, 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.988 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216478 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673953+0000", 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3910000000000001e-06 2024-09-21T18:59:00.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674028+0000", 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5197000000000005e-05 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674184+0000", 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156129 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674193+0000", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0410000000000003e-06 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674323+0000", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013050400000000001 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674368+0000", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4211000000000001e-05 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792228+0000", 2024-09-21T18:59:00.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.117859843 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792333+0000", 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105216 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792492+0000", 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015956400000000001 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802291+0000", 2024-09-21T18:59:00.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097984360000000006 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802325+0000", 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4235e-05 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811655+0000", 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093300610000000006 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811704+0000", 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8520999999999998e-05 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811795+0000", 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0966999999999998e-05 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216483 7.6 7:679f3a63:::benchmark_data_smithi081_73391_object77484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.680478+0000", 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.522810073, 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1314383640000001, 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216483 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:00.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680480+0000", 2024-09-21T18:59:00.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3010000000000001e-06 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680487+0000", 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8090000000000004e-06 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680532+0000", 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4369999999999997e-05 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:00.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680540+0000", 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5969999999999992e-06 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680600+0000", 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0275000000000001e-05 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680610+0000", 2024-09-21T18:59:00.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6090000000000007e-06 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798740+0000", 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1181301029999999 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798848+0000", 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010757 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798947+0000", 2024-09-21T18:59:01.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9349e-05 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802432+0000", 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034852860000000002 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808361+0000", 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059282689999999999 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811802+0000", 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003440869 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811840+0000", 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8197e-05 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811916+0000", 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6760000000000004e-05 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.002 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216512 7.6 7:6f163cff:::benchmark_data_smithi081_73391_object77513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.714817+0000", 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.488470781, 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0972389849999999, 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216512 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714820+0000", 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2450000000000001e-06 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714826+0000", 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1859999999999997e-06 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714841+0000", 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5288999999999999e-05 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714850+0000", 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6540000000000005e-06 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714963+0000", 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112893 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714971+0000", 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7120000000000006e-06 2024-09-21T18:59:01.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802465+0000", 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0874939400000001 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802564+0000", 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.904e-05 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802667+0000", 2024-09-21T18:59:01.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103101 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808099+0000", 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054315889999999997 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808411+0000", 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031148299999999999 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811922+0000", 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035117199999999999 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811958+0000", 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5766999999999998e-05 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812056+0000", 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8366e-05 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216545 7.6 7:6b69332e:::benchmark_data_smithi081_73391_object77546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.753189+0000", 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.45009927, 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0617420390000001, 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216545 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.010 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753191+0000", 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7680000000000001e-06 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753196+0000", 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1410000000000004e-06 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753207+0000", 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1143e-05 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753214+0000", 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.917e-06 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754192+0000", 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097769299999999996 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754200+0000", 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9400000000000002e-06 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.805892+0000", 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0516927620000001 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806005+0000", 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113143 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806125+0000", 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119331 2024-09-21T18:59:01.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808476+0000", 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023510919999999999 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812065+0000", 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035891429999999999 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814743+0000", 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026784489999999998 2024-09-21T18:59:01.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814839+0000", 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5358000000000004e-05 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814931+0000", 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2158999999999996e-05 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.016 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216552 7.6 7:63b154c8:::benchmark_data_smithi081_73391_object77553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.762041+0000", 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.441247164, 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.060947284, 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216552 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762043+0000", 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9889999999999999e-06 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762056+0000", 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3441999999999999e-05 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762068+0000", 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-05 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762075+0000", 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1169999999999998e-06 2024-09-21T18:59:01.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762135+0000", 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9759000000000002e-05 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762143+0000", 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808532+0000", 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0463883469999999 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808647+0000", 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011510600000000001 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808757+0000", 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109849 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814943+0000", 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006186558 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814980+0000", 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6729e-05 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822841+0000", 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078607529999999998 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822895+0000", 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4282e-05 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822988+0000", 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3422000000000001e-05 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.023 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216555 7.6 7:6310baba:::benchmark_data_smithi081_73391_object77556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.763367+0000", 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.439921325, 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0695222289999999, 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216555 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763369+0000", 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.635e-06 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763374+0000", 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.101e-06 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763503+0000", 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000129276 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763511+0000", 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5019999999999992e-06 2024-09-21T18:59:01.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763815+0000", 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030339199999999998 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763823+0000", 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7600000000000002e-06 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812093+0000", 2024-09-21T18:59:01.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0482702930000001 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812199+0000", 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010665100000000001 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812307+0000", 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107206 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819747+0000", 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074399499999999999 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819790+0000", 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3297999999999998e-05 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832684+0000", 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012894012 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832778+0000", 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4456999999999994e-05 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832889+0000", 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110696 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216569 7.6 7:67ad9aff:::benchmark_data_smithi081_73391_object77570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.798905+0000", 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 100.404383287, 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.034449366, 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216569 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.031 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798905+0000", 2024-09-21T18:59:01.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:00 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T18:59:01.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:00 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T18:59:01.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:00 smithi081 ceph-mon[39192]: pgmap v271: 158 pgs: 158 active+clean; 765 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798905+0000", 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798908+0000", 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.129e-06 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798917+0000", 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4929999999999998e-06 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798937+0000", 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0687e-05 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798947+0000", 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4690000000000003e-06 2024-09-21T18:59:01.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.799037+0000", 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0229999999999995e-05 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.799048+0000", 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0729e-05 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.815008+0000", 2024-09-21T18:59:01.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159601570000001 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.815112+0000", 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103707 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.815216+0000", 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104474 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819803+0000", 2024-09-21T18:59:01.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045866630000000004 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.820017+0000", 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021458499999999999 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.833215+0000", 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013198052 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.833269+0000", 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3511000000000003e-05 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.833354+0000", 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5480000000000002e-05 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:243094 7.4 7:2cb70af2:::benchmark_data_smithi081_73391_object104095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.728724+0000", 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 69.474564345999994, 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0326880460000001, 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243094 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728724+0000", 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728724+0000", 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728726+0000", 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.435e-06 2024-09-21T18:59:01.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728733+0000", 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6499999999999999e-06 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728749+0000", 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5698000000000001e-05 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728757+0000", 2024-09-21T18:59:01.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0639999999999994e-06 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.731046+0000", 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022893459999999998 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.731054+0000", 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0739999999999993e-06 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.756234+0000", 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0251801709999999 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.756338+0000", 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103612 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.756429+0000", 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1538000000000003e-05 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761266+0000", 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048370410000000003 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761304+0000", 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7122000000000001e-05 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761337+0000", 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3207999999999998e-05 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761383+0000", 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6294000000000001e-05 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761412+0000", 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8793e-05 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:243118 7.4 7:2a6c29ae:::benchmark_data_smithi081_73391_object104119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.750809+0000", 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 69.452479389000004, 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.010814289, 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243118 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.046 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750809+0000", 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750809+0000", 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750811+0000", 2024-09-21T18:59:01.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3319999999999999e-06 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750817+0000", 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9529999999999996e-06 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750829+0000", 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1429e-05 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750836+0000", 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3329999999999999e-06 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.754026+0000", 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003189983 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.754043+0000", 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7442e-05 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.758596+0000", 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0045523590000001 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.758673+0000", 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7152999999999998e-05 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.758736+0000", 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3042999999999999e-05 2024-09-21T18:59:01.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761485+0000", 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027495419999999998 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761522+0000", 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7098999999999998e-05 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761555+0000", 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2138999999999998e-05 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761601+0000", 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6740999999999999e-05 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761623+0000", 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1741000000000001e-05 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.052 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:243313 7.f 7:f737484f:::benchmark_data_smithi081_73391_object104314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:51.761627+0000", 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 68.441661072000002, 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0178787949999999, 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243313 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.053 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761627+0000", 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761627+0000", 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:01.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761628+0000", 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761633+0000", 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2549999999999999e-06 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761642+0000", 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0620000000000007e-06 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761648+0000", 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1330000000000004e-06 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761690+0000", 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2558999999999997e-05 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761697+0000", 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7209999999999997e-06 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.773488+0000", 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.01179102 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.773602+0000", 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114048 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.773709+0000", 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010694800000000001 2024-09-21T18:59:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779289+0000", 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055803789999999999 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779302+0000", 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2704000000000001e-05 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:01.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779412+0000", 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109616 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779469+0000", 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7438000000000001e-05 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779506+0000", 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6718999999999999e-05 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.059 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.060 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:01.060 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:01.060 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:01.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:00 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T18:59:01.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:00 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T18:59:01.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:00 smithi081 ceph-mon[29384]: pgmap v271: 158 pgs: 158 active+clean; 765 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-21T18:59:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:01.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:01.162 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:01.198 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:01.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:01.336 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:01.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:01.474 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:01.577 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.617 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:01.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:01.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:01.747 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:01.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 9 16 25377 25361 175.998 157.5 0.00526193 0.00567543 2024-09-21T18:59:01.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:01.887 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:01.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:01.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.022 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:02.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:02.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:02.149 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:02.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:02.285 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:02.387 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:02.423 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:02.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:02.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:02.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:02.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:02.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 10 16 27584 27568 172.193 137.938 0.00870856 0.00580229 2024-09-21T18:59:02.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:02.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:02 smithi104 ceph-mon[32438]: pgmap v273: 158 pgs: 158 active+clean; 819 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 734 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-21T18:59:02.874 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:02 smithi081 ceph-mon[39192]: pgmap v273: 158 pgs: 158 active+clean; 819 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 734 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-21T18:59:02.875 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:02 smithi081 ceph-mon[29384]: pgmap v273: 158 pgs: 158 active+clean; 819 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 734 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-21T18:59:02.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-21T18:59:02.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:02.963 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:02.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:02.966 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:02.970 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:03.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:03.108 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:03.210 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:03.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:03.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:03.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.510 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:03.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:03.611 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:03.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:03.777 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:03.787 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 11 16 29859 29843 169.465 142.188 0.0100479 0.00589484 2024-09-21T18:59:03.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T18:59:03.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:03.905 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:03.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:04.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.067 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:04.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:04.168 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:04.191 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:04.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:04.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:04.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:04.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:04.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:04.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:04.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:04.732 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[29384]: pgmap v274: 158 pgs: 158 active+clean; 1.5 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 197 MiB/s wr, 3.15k op/s 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[29384]: 5.1e scrub starts 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[29384]: 5.1e scrub ok 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[39192]: pgmap v274: 158 pgs: 158 active+clean; 1.5 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 197 MiB/s wr, 3.15k op/s 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[39192]: 5.1e scrub starts 2024-09-21T18:59:04.739 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:04 smithi081 ceph-mon[39192]: 5.1e scrub ok 2024-09-21T18:59:04.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 12 16 31770 31754 165.297 119.438 0.00843276 0.00604385 2024-09-21T18:59:04.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:04.912 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:04.926 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:04.926 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:04.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:04.977 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:04.977 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:04.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:05.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:04 smithi104 ceph-mon[32438]: pgmap v274: 158 pgs: 158 active+clean; 1.5 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 197 MiB/s wr, 3.15k op/s 2024-09-21T18:59:05.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:04 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T18:59:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:04 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T18:59:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:04 smithi104 ceph-mon[32438]: 5.1e scrub starts 2024-09-21T18:59:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:04 smithi104 ceph-mon[32438]: 5.1e scrub ok 2024-09-21T18:59:05.017 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:05.017 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:05.055 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:05.055 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:05.087 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:05.087 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:05.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:05.120 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:05.120 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:05.152 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:05.153 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:05.186 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:05.187 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:05.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:05.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:05.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:05.385 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:05.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:05.516 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:05.618 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:59:05.652 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:05.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:05.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 13 16 33708 33692 161.9 121.125 0.00342883 0.00616996 2024-09-21T18:59:05.789 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3939047222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/663988500' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3939047222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/663988500' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:05.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:05.924 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:05.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:06.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:05 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T18:59:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:05 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T18:59:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3939047222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/663988500' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:06.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:06.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:06.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:06.193 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:06.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:06.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:06.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:06.455 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:06.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.594 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:06.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:06.695 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:06.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:06.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 14 16 35727 35711 159.348 126.188 0.0055828 0.00626908 2024-09-21T18:59:06.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:06.870 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:06.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:06.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:06.993 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: pgmap v275: 158 pgs: 158 active+clean; 1.5 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-21T18:59:06.993 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:59:06.993 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:59:06.993 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/663988500' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:06.993 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: pgmap v275: 158 pgs: 158 active+clean; 1.5 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:59:06.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/663988500' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T18:59:06.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: pgmap v275: 158 pgs: 158 active+clean; 1.5 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/663988500' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T18:59:07.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T18:59:07.010 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:07.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:07.151 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:07.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:07.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:59:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:59:07.390 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:07.424 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:07.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:07.556 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:07.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:07.697 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:07.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 15 16 37573 37557 156.416 115.375 0.00215711 0.00636022 2024-09-21T18:59:07.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:07.834 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:07.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:07.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:08.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:59:08.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:08.129 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:08.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:08.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:59:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:59:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:07 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:07 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:07 smithi104 ceph-mon[32438]: 5.1 scrub starts 2024-09-21T18:59:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:07 smithi104 ceph-mon[32438]: 5.1 scrub ok 2024-09-21T18:59:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:08.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[29384]: 5.1 scrub starts 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[29384]: 5.1 scrub ok 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[39192]: 5.1 scrub starts 2024-09-21T18:59:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:07 smithi081 ceph-mon[39192]: 5.1 scrub ok 2024-09-21T18:59:08.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:08.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:08.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:08.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:08.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:08.670 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:08.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:08.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 16 16 39231 39215 153.117 103.625 0.00561889 0.00652583 2024-09-21T18:59:08.801 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:08.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:08.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:08.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:09.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:09.078 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:09.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:09.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:08 smithi104 ceph-mon[32438]: pgmap v277: 158 pgs: 158 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-09-21T18:59:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:09.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:09.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:08 smithi081 ceph-mon[39192]: pgmap v277: 158 pgs: 158 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-09-21T18:59:09.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:09.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:08 smithi081 ceph-mon[29384]: pgmap v277: 158 pgs: 158 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-09-21T18:59:09.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:09.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.492 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:09.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:09.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:09.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:09.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:09.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 17 16 41022 41006 150.695 111.938 0.00648107 0.00663147 2024-09-21T18:59:09.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:09.887 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:09.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:09.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:10.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:10.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:10.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:10.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:10.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:10.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:10.532 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:10.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:10.668 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:10.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:10.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 18 16 42341 42325 146.904 82.4375 0.00291948 0.00679479 2024-09-21T18:59:10.790 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:10.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[29384]: pgmap v278: 158 pgs: 158 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[29384]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[39192]: pgmap v278: 158 pgs: 158 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-09-21T18:59:10.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:10 smithi081 ceph-mon[39192]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T18:59:10.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:10.924 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:10.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:11.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:10 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T18:59:11.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:10 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T18:59:11.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:10 smithi104 ceph-mon[32438]: pgmap v278: 158 pgs: 158 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-09-21T18:59:11.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:10 smithi104 ceph-mon[32438]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T18:59:11.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:11.149 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:11.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:11.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:11.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:11.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:11.440 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:11.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:11.572 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:11.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:11.721 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:11.788 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 19 16 43451 43435 142.824 69.375 0.00259845 0.00698296 2024-09-21T18:59:11.823 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:11.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:11.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:11.976 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:12.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:12.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:12.141 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:12.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:12.374 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:12.404 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:12.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:12.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:12.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:12.789 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:59:12.787630+0000 min lat: 0.00110906 max lat: 0.19882 avg lat: 0.0070658 2024-09-21T18:59:12.789 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:59:12.789 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 20 16 45286 45270 141.416 114.688 0.00492839 0.0070658 2024-09-21T18:59:12.809 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:59:12.810 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:59:12.810 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:12 smithi081 ceph-mon[39192]: pgmap v280: 158 pgs: 158 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-21T18:59:12.810 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:12 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:12.810 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:12 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:12.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:12 smithi081 ceph-mon[29384]: pgmap v280: 158 pgs: 158 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-21T18:59:12.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:12 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:12.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:12 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:12.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:12.954 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:12.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:12.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:12 smithi104 ceph-mon[32438]: pgmap v280: 158 pgs: 158 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-21T18:59:12.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:12 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:12.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:12 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:13.055 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:13.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:13.226 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:13.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:13.374 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:13.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:13.475 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:13.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:13.637 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:13.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.774 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:13.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:13.789 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 21 15 47046 47031 139.923 110.062 0.0139312 0.00714098 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: 3.16 scrub starts 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: 3.16 scrub ok 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:13.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T18:59:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T18:59:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: 3.16 scrub starts 2024-09-21T18:59:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: 3.16 scrub ok 2024-09-21T18:59:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:59:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:59:13.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:13 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:13.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:13.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:13.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: 3.16 scrub starts 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: 3.16 scrub ok 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T18:59:14.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:13 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:14.016 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:14.019 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-21T18:59:14.041 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:14.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:14.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:14.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:14.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:14.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:14.578 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:14.629 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:14.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T18:59:14.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.723 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:14.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:14.789 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 22 16 48676 48660 138.191 101.812 0.00680059 0.00723053 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[29384]: pgmap v281: 158 pgs: 158 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[29384]: 6.2 scrub starts 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[29384]: 6.2 scrub ok 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T18:59:14.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[39192]: pgmap v281: 158 pgs: 158 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-21T18:59:14.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[39192]: 6.2 scrub starts 2024-09-21T18:59:14.818 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:14 smithi081 ceph-mon[39192]: 6.2 scrub ok 2024-09-21T18:59:14.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:14.857 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:14.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:14.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:14 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T18:59:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:14 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T18:59:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:14 smithi104 ceph-mon[32438]: pgmap v281: 158 pgs: 158 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-21T18:59:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:14 smithi104 ceph-mon[32438]: 6.2 scrub starts 2024-09-21T18:59:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:14 smithi104 ceph-mon[32438]: 6.2 scrub ok 2024-09-21T18:59:15.009 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:15.111 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:15.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:15.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:15.259 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:15.270 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:15.354 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:15.355 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:15.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:15.392 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:15.393 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:15.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:15.425 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:15.426 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:15.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:59:15.461 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:15.461 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:15.495 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:15.495 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:15.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:15.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:15.529 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:15.575 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:15.575 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:15.609 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:15.609 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:15.644 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[39192]: 3.10 scrub starts 2024-09-21T18:59:15.644 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[39192]: 3.10 scrub ok 2024-09-21T18:59:15.644 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[39192]: 4.1b scrub starts 2024-09-21T18:59:15.644 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[39192]: 4.1b scrub ok 2024-09-21T18:59:15.644 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3924082173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:15.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[29384]: 3.10 scrub starts 2024-09-21T18:59:15.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[29384]: 3.10 scrub ok 2024-09-21T18:59:15.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[29384]: 4.1b scrub starts 2024-09-21T18:59:15.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[29384]: 4.1b scrub ok 2024-09-21T18:59:15.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3924082173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:15.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:15.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:15.793 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 23 15 50186 50171 136.264 94.4375 0.0083993 0.0073333 2024-09-21T18:59:15.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:15.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:15.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:16.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:15 smithi104 ceph-mon[32438]: 3.10 scrub starts 2024-09-21T18:59:16.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:15 smithi104 ceph-mon[32438]: 3.10 scrub ok 2024-09-21T18:59:16.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:15 smithi104 ceph-mon[32438]: 4.1b scrub starts 2024-09-21T18:59:16.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:15 smithi104 ceph-mon[32438]: 4.1b scrub ok 2024-09-21T18:59:16.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3924082173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:16.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:16.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:16.241 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:16.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:16.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:16.379 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:16.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:16.518 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:16.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:16.654 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:16.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:16.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:16.793 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 24 16 51955 51939 135.19 110.5 0.00716773 0.00739167 2024-09-21T18:59:16.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:16 smithi081 ceph-mon[29384]: pgmap v282: 158 pgs: 158 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 115 MiB/s wr, 1.85k op/s 2024-09-21T18:59:16.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:16 smithi081 ceph-mon[29384]: 3.c scrub starts 2024-09-21T18:59:16.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:16 smithi081 ceph-mon[29384]: 3.c scrub ok 2024-09-21T18:59:16.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:16 smithi081 ceph-mon[39192]: pgmap v282: 158 pgs: 158 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 115 MiB/s wr, 1.85k op/s 2024-09-21T18:59:16.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:16 smithi081 ceph-mon[39192]: 3.c scrub starts 2024-09-21T18:59:16.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:16 smithi081 ceph-mon[39192]: 3.c scrub ok 2024-09-21T18:59:16.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:16.898 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:16.918 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:16.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:16.922 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-21T18:59:17.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:16 smithi104 ceph-mon[32438]: pgmap v282: 158 pgs: 158 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 115 MiB/s wr, 1.85k op/s 2024-09-21T18:59:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:16 smithi104 ceph-mon[32438]: 3.c scrub starts 2024-09-21T18:59:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:16 smithi104 ceph-mon[32438]: 3.c scrub ok 2024-09-21T18:59:17.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:17.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:17.189 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:17.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:17.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:17.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.449 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:17.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:17.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:17.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:59:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:59:17.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:17.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:17.707 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:17.793 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 25 16 53499 53483 133.643 96.5 0.0074003 0.00747753 2024-09-21T18:59:17.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:17.847 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:17.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T18:59:17.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T18:59:17.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-mon[32438]: 3.5 scrub starts 2024-09-21T18:59:17.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-mon[32438]: 3.5 scrub ok 2024-09-21T18:59:17.887 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-mon[32438]: 5.11 scrub starts 2024-09-21T18:59:17.887 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-mon[32438]: 5.11 scrub ok 2024-09-21T18:59:17.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:17.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[29384]: 3.5 scrub starts 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[29384]: 3.5 scrub ok 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[29384]: 5.11 scrub starts 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[29384]: 5.11 scrub ok 2024-09-21T18:59:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T18:59:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T18:59:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[39192]: 3.5 scrub starts 2024-09-21T18:59:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[39192]: 3.5 scrub ok 2024-09-21T18:59:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[39192]: 5.11 scrub starts 2024-09-21T18:59:18.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:17 smithi081 ceph-mon[39192]: 5.11 scrub ok 2024-09-21T18:59:18.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:18.121 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:18.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:18.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:59:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:59:18.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:18.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:18.397 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:18.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:18.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:18.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:18.660 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:18.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:18.793 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 26 16 55105 55089 132.364 100.375 0.0082761 0.00754984 2024-09-21T18:59:18.798 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:18.900 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:18.937 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:18.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: pgmap v283: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: 4.9 scrub starts 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: 4.9 scrub ok 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T18:59:19.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:18 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T18:59:19.039 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T18:59:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T18:59:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: pgmap v283: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-21T18:59:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: 4.9 scrub starts 2024-09-21T18:59:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: 4.9 scrub ok 2024-09-21T18:59:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: pgmap v283: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: 4.9 scrub starts 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: 4.9 scrub ok 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:59:19.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:59:19.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T18:59:19.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:18 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T18:59:19.071 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:19.172 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:19.200 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:19.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:19.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:19.576 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:19.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:19.794 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 27 16 56688 56672 131.126 98.9375 0.0104755 0.00762095 2024-09-21T18:59:19.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:19.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:19.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:19.882 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T18:59:19.882 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T18:59:19.882 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[39192]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T18:59:19.882 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[39192]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T18:59:19.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T18:59:19.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T18:59:19.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[29384]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T18:59:19.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:19 smithi081 ceph-mon[29384]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T18:59:19.894 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:19.952 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:19.952 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:59:19.953 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:59:19.953 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:59:19.953 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:19.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:20.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:20.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:19 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T18:59:20.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:19 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T18:59:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:19 smithi104 ceph-mon[32438]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T18:59:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:19 smithi104 ceph-mon[32438]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T18:59:20.105 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:20.135 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:20.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:20.374 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:20.408 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:20.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:20.606 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:20.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:20.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:20.744 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: pgmap v285: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 6.e scrub starts 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 6.e scrub ok 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 6.1f scrub starts 2024-09-21T18:59:20.745 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[39192]: 6.1f scrub ok 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: pgmap v285: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 6.e scrub starts 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 6.e scrub ok 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 6.1f scrub starts 2024-09-21T18:59:20.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:20 smithi081 ceph-mon[29384]: 6.1f scrub ok 2024-09-21T18:59:20.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:20.793 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 28 16 58220 58204 129.862 95.75 0.00497546 0.00769492 2024-09-21T18:59:20.821 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:20.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:20.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:21.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: pgmap v285: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 6.e scrub starts 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 6.e scrub ok 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 6.1f scrub starts 2024-09-21T18:59:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:20 smithi104 ceph-mon[32438]: 6.1f scrub ok 2024-09-21T18:59:21.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:21.190 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:21.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:21.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T18:59:21.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.389 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:21.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:21.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:59:21.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:21.545 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:21.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:21.647 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T18:59:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:21.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:21.732 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[39192]: 4.1 scrub starts 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[39192]: 4.1 scrub ok 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1169095442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[29384]: 4.1 scrub starts 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[29384]: 4.1 scrub ok 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1169095442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:59:21.733 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:21.794 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 29 16 59605 59589 128.368 86.5625 0.0134562 0.00778415 2024-09-21T18:59:21.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:21.852 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:21.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:21.986 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:21.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:22.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:21 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T18:59:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:21 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T18:59:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:21 smithi104 ceph-mon[32438]: 4.1 scrub starts 2024-09-21T18:59:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:21 smithi104 ceph-mon[32438]: 4.1 scrub ok 2024-09-21T18:59:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1169095442' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T18:59:22.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:22.088 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:22.121 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:22.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:22.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:22.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:22.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:22.398 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:22.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:22.537 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:22.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:22.664 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:22.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:22.794 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 30 16 61178 61162 127.365 98.3125 0.0123093 0.00784664 2024-09-21T18:59:22.800 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:22.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:22.933 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: pgmap v287: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1169095442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:59:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:22 smithi104 ceph-mon[32438]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T18:59:23.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: pgmap v287: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1169095442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[29384]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: pgmap v287: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T18:59:23.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T18:59:23.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T18:59:23.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:23.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1169095442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T18:59:23.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:22 smithi081 ceph-mon[39192]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T18:59:23.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:23.074 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:23.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:23.215 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:23.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.364 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:23.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:23.394 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-21T18:59:23.428 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:23.429 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:23.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:23.497 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:23.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:23.626 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:23.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:23.795 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 31 16 62678 62662 126.277 93.75 0.015073 0.00791473 2024-09-21T18:59:23.854 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:23.880 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:23.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:23.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:24.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:24.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:24.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:99377 7.9 7:9f989195:::benchmark_data_smithi081_73391_object99376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.51770922700001, 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106765947, 2024-09-21T18:59:24.230 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 99377 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707607+0000", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707610+0000", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.196e-06 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.707638+0000", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7880000000000001e-05 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763151+0000", 2024-09-21T18:59:24.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.055513121999999998 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763168+0000", 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6152000000000001e-05 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763470+0000", 2024-09-21T18:59:24.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030208399999999999 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763608+0000", 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013794999999999999 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.763999+0000", 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000391691 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.771410+0000", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074111979999999999 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.771550+0000", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013909999999999999 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814280+0000", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.042729984999999998 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814321+0000", 2024-09-21T18:59:24.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1594999999999997e-05 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:17.814373+0000", 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1993999999999997e-05 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101895 7.b 7:d6f8fed5:::benchmark_data_smithi081_73391_object101894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 244.31521687899999, 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10653628599999999, 2024-09-21T18:59:24.236 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101895 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910100+0000", 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910101+0000", 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8109999999999999e-06 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910123+0000", 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1500000000000001e-05 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910138+0000", 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4798e-05 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910145+0000", 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1790000000000002e-06 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910195+0000", 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9716000000000003e-05 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910299+0000", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104134 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.910439+0000", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001403 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.912887+0000", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024483080000000002 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916379+0000", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034916880000000002 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016537+0000", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.100158393 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016591+0000", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3170000000000001e-05 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.016636+0000", 2024-09-21T18:59:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5289e-05 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:101904 7.b 7:d7f209bc:::benchmark_data_smithi081_73391_object101903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 244.30843248599999, 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10347793, 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 101904 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916884+0000", 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916886+0000", 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916905+0000", 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9527000000000001e-05 2024-09-21T18:59:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916915+0000", 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8130000000000007e-06 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916920+0000", 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1420000000000001e-06 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.916958+0000", 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8059000000000003e-05 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.917042+0000", 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4251000000000004e-05 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.917130+0000", 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7626000000000004e-05 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.919036+0000", 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00190617 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:19.919594+0000", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055802599999999999 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020228+0000", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.100633629 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020322+0000", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4293e-05 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:20.020362+0000", 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9777999999999998e-05 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136125 7.b 7:df011645:::benchmark_data_smithi081_73391_object136124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.06433638999999, 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136450605, 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136125 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160980+0000", 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.160983+0000", 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4490000000000002e-06 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161014+0000", 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1038000000000003e-05 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161026+0000", 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2753e-05 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161035+0000", 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0040000000000005e-06 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161146+0000", 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110518 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161292+0000", 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001461 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161445+0000", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000153475 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170642+0000", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0091963119999999999 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170656+0000", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3947e-05 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297280+0000", 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12662463500000001 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297381+0000", 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101037 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.297431+0000", 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9336999999999999e-05 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136126 7.b 7:d13dafa2:::benchmark_data_smithi081_73391_object136125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.06425453099999, 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139333067, 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136126 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:59:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161063+0000", 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3500000000000005e-07 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161084+0000", 2024-09-21T18:59:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0755e-05 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161091+0000", 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2169999999999997e-06 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161097+0000", 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3230000000000003e-06 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164270+0000", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031733239999999999 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164391+0000", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120575 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164535+0000", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014399899999999999 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170661+0000", 2024-09-21T18:59:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061256419999999997 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170700+0000", 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9578e-05 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300264+0000", 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12956358800000001 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300355+0000", 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1219999999999997e-05 2024-09-21T18:59:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300395+0000", 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0011e-05 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136128 7.b 7:df396259:::benchmark_data_smithi081_73391_object136127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.062611984, 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139089409, 2024-09-21T18:59:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136128 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162707+0000", 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162731+0000", 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4485e-05 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162747+0000", 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5452999999999999e-05 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162755+0000", 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9920000000000007e-06 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167113+0000", 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043581749999999997 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167237+0000", 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012398699999999999 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167386+0000", 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149672 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170665+0000", 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032787960000000001 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170739+0000", 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4108000000000003e-05 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301654+0000", 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13091487199999999 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301749+0000", 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5187999999999997e-05 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301794+0000", 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4558000000000003e-05 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136146 7.b 7:da687baa:::benchmark_data_smithi081_73391_object136145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.04994227200001, 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12656247800000001, 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136146 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175374+0000", 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175376+0000", 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1040000000000001e-06 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175400+0000", 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4162e-05 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175412+0000", 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1802e-05 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175420+0000", 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7149999999999998e-06 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175494+0000", 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3688000000000003e-05 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175609+0000", 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115519 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.175745+0000", 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013621299999999999 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177851+0000", 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021052779999999999 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177986+0000", 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013539800000000001 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301842+0000", 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.123855964 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301899+0000", 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7426000000000002e-05 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301937+0000", 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7209000000000002e-05 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136148 7.b 7:d9fea987:::benchmark_data_smithi081_73391_object136147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.04738825499999, 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12981583899999999, 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136148 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177928+0000", 2024-09-21T18:59:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177930+0000", 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.545e-06 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177953+0000", 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.349e-05 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177962+0000", 2024-09-21T18:59:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8890000000000008e-06 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.177969+0000", 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8759999999999999e-06 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178007+0000", 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8126000000000001e-05 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178088+0000", 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0684000000000002e-05 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.178175+0000", 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7643000000000001e-05 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.179852+0000", 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001676774 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.179997+0000", 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014443799999999999 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307615+0000", 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12761845299999999 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307693+0000", 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8103999999999996e-05 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.307744+0000", 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0816999999999998e-05 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168555 7.7 e123/114 7:e5122ffb:::benchmark_data_smithi081_73391_object29556:head v 123'10492, mlcod=123'10490)", 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.113933+0000", 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 180.11138335300001, 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.111488108, 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113933+0000", 2024-09-21T18:59:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113933+0000", 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113934+0000", 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.001e-06 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113939+0000", 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1259999999999997e-06 2024-09-21T18:59:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113946+0000", 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2019999999999998e-06 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.113952+0000", 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4990000000000002e-06 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.117198+0000", 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032465110000000001 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.117222+0000", 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3991000000000001e-05 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225368+0000", 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.108145599 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225421+0000", 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3179e-05 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168557 7.7 e123/114 7:e8d3dc47:::benchmark_data_smithi081_73391_object29558:head v 123'10493, mlcod=123'10490)", 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 180.10663262599999, 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10676554100000001, 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:59:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118684+0000", 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.29e-07 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118687+0000", 2024-09-21T18:59:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3979999999999999e-06 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118692+0000", 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.233e-06 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.118695+0000", 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128618+0000", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0099226269999999998 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.128640+0000", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1719999999999999e-05 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225424+0000", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096784673000000002 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225449+0000", 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4953999999999999e-05 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168564 7.7 e123/114 7:e16a2db8:::benchmark_data_smithi081_73391_object29565:head v 123'10494, mlcod=123'10490)", 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.122367+0000", 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 180.10294924300001, 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.103158955, 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122367+0000", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122367+0000", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122368+0000", 2024-09-21T18:59:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.083e-06 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122374+0000", 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6969999999999998e-06 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122383+0000", 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5179999999999993e-06 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.122390+0000", 2024-09-21T18:59:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.105e-06 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.135024+0000", 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012634115 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.135044+0000", 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0098e-05 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225452+0000", 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.090408540999999995 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.225526+0000", 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3798000000000001e-05 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169000 7.b 7:de931d3b:::benchmark_data_smithi081_73391_object30001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637210+0000", 2024-09-21T18:59:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 179.58810600000001, 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106838799, 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169000 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637210+0000", 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637210+0000", 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637212+0000", 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637216+0000", 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8299999999999998e-06 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637224+0000", 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2039999999999997e-06 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637230+0000", 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0460000000000002e-06 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637377+0000", 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014751200000000001 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637486+0000", 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010838199999999999 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637585+0000", 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9575999999999997e-05 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646802+0000", 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0092170120000000001 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646840+0000", 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7957999999999998e-05 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.743910+0000", 2024-09-21T18:59:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097069595999999994 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744010+0000", 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9980000000000002e-05 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744049+0000", 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9313000000000001e-05 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169001 7.b 7:df6f9ae3:::benchmark_data_smithi081_73391_object30002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637246+0000", 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 179.58807031500001, 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.106883351, 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169001 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637246+0000", 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637246+0000", 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637247+0000", 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3099999999999997e-07 2024-09-21T18:59:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637249+0000", 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.74e-06 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637255+0000", 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4779999999999999e-06 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637260+0000", 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5639999999999998e-06 2024-09-21T18:59:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.640288+0000", 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030289150000000001 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.640393+0000", 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010474100000000001 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.640504+0000", 2024-09-21T18:59:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011120699999999999 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646873+0000", 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063683589999999997 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646907+0000", 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4659999999999997e-05 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744056+0000", 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097148935000000006 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744095+0000", 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9070999999999999e-05 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744129+0000", 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4050000000000001e-05 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169002 7.b 7:d5eb2253:::benchmark_data_smithi081_73391_object30003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 179.588051484, 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10702123099999999, 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169002 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637265+0000", 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6399999999999998e-07 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637268+0000", 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3939999999999999e-06 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637272+0000", 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3719999999999998e-06 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637276+0000", 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0130000000000004e-06 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.646949+0000", 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009673325 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.647050+0000", 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100342 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.647154+0000", 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010430899999999999 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744164+0000", 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097010258000000002 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744175+0000", 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0299000000000001e-05 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744209+0000", 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4576000000000002e-05 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744252+0000", 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2759000000000002e-05 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.744286+0000", 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4220000000000001e-05 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169003 7.9 7:9a50a309:::benchmark_data_smithi081_73391_object30004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637280+0000", 2024-09-21T18:59:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 179.58803601299999, 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.105220251, 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169003 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637280+0000", 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637280+0000", 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637281+0000", 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3000000000000002e-07 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637283+0000", 2024-09-21T18:59:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.858e-06 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637287+0000", 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3250000000000001e-06 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637291+0000", 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0080000000000004e-06 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637455+0000", 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016420700000000001 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637580+0000", 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000124595 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637989+0000", 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040934400000000002 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.738992+0000", 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10100292299999999 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739040+0000", 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8390000000000003e-05 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739085+0000", 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4425999999999998e-05 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742244+0000", 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031589750000000001 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742372+0000", 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012819799999999999 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742501+0000", 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128672 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:168997 7.4 e123/114 7:2a66f29b:::benchmark_data_smithi081_73391_object29998:head v 123'10730, mlcod=123'10727)", 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.637303+0000", 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 179.58801308, 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10184473600000001, 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637303+0000", 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637303+0000", 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637304+0000", 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1300000000000001e-07 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637308+0000", 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.002e-06 2024-09-21T18:59:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637313+0000", 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2290000000000002e-06 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.637317+0000", 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1150000000000004e-06 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.641772+0000", 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044551060000000003 2024-09-21T18:59:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.641794+0000", 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1693999999999999e-05 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.738772+0000", 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096977709999999995 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739148+0000", 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037646700000000001 2024-09-21T18:59:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:169011 7.9 7:9db9a5ae:::benchmark_data_smithi081_73391_object30012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:56:24.645238+0000", 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 179.580078902, 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.108457962, 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 169011 2024-09-21T18:59:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645238+0000", 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645238+0000", 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645240+0000", 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.086e-06 2024-09-21T18:59:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645244+0000", 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0749999999999999e-06 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645252+0000", 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4339999999999992e-06 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.645258+0000", 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8329999999999999e-06 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739378+0000", 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.094119797000000005 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739529+0000", 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015165299999999999 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.739662+0000", 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013272699999999999 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.742510+0000", 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028483240000000002 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.748769+0000", 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0062584529999999998 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.748856+0000", 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7122999999999999e-05 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.753525+0000", 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046688470000000003 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.753593+0000", 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7916999999999994e-05 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:56:24.753696+0000", 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102693 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242088 7.b 7:d4630f67:::benchmark_data_smithi081_73391_object103089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512627+0000", 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 94.712689214999997, 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.165892709, 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T18:59:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242088 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512627+0000", 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512627+0000", 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512629+0000", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0770000000000002e-06 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512636+0000", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.562e-06 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512649+0000", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3529e-05 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512657+0000", 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5639999999999999e-06 2024-09-21T18:59:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512723+0000", 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6489999999999995e-05 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512729+0000", 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0240000000000003e-06 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514154+0000", 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001424771 2024-09-21T18:59:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672882+0000", 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.158727757 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673023+0000", 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014103799999999999 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677654+0000", 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046308169999999997 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678082+0000", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000428572 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678431+0000", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034839400000000003 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678483+0000", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1996999999999997e-05 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678520+0000", 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7116999999999997e-05 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242087 7.d e149/114 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head v 149'15301, mlcod=148'15299)", 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512853+0000", 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 94.712463127000007, 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16432387000000001, 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512853+0000", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512853+0000", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512854+0000", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.052e-06 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512862+0000", 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3749999999999997e-06 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512871+0000", 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5920000000000007e-06 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512878+0000", 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6120000000000004e-06 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512943+0000", 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4738000000000003e-05 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512966+0000", 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3591000000000002e-05 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677136+0000", 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16416932200000001 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677177+0000", 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1588000000000002e-05 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242084 7.0 e149/79 7:0c19e5c0:::benchmark_data_smithi081_73391_object103085:head v 149'15371, mlcod=148'15368)", 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 94.711033322999995, 2024-09-21T18:59:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16410017099999999, 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5900000000000003e-07 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514287+0000", 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514294+0000", 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4200000000000004e-06 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674533+0000", 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.160238726 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674644+0000", 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111369 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674663+0000", 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9338e-05 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678317+0000", 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003654175 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678383+0000", 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5807000000000002e-05 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:24.340 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.341 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:24.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:24.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:24.475 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:24.577 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:24.606 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:24.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T18:59:24.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:24.726 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:24.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:24.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:24.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:24.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:24.736 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:24.736 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:24.736 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:24 smithi081 ceph-mon[39192]: pgmap v289: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:59:24.736 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3822037831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:24.736 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:24 smithi081 ceph-mon[39192]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T18:59:24.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:24 smithi081 ceph-mon[29384]: pgmap v289: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:59:24.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3822037831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:24.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:24 smithi081 ceph-mon[29384]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T18:59:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.738 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:24.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:24.795 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 32 16 63974 63958 124.862 81 0.00903284 0.00800338 2024-09-21T18:59:24.839 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:24.878 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:24.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:24.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:24 smithi104 ceph-mon[32438]: pgmap v289: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:59:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3822037831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:24 smithi104 ceph-mon[32438]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T18:59:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:25.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:25.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:25.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:25.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:25.451 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:25.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.596 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:25.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:25.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.748 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:25.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:25.796 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 33 16 65412 65396 123.802 89.875 0.00781948 0.00807256 2024-09-21T18:59:25.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:25.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:25.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:25.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:26.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:25 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:59:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:25 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:59:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.022 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:26.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:25 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:59:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:25 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:59:26.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:25 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:59:26.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:25 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:59:26.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:26.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T18:59:26.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:26.271 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:26.283 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:26.283 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:26.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:26.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:26.564 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:26.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:26.694 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:26.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:26.796 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 34 16 66771 66755 122.659 84.9375 0.00480118 0.00814734 2024-09-21T18:59:26.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:26 smithi081 ceph-mon[29384]: pgmap v291: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:59:26.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:26 smithi081 ceph-mon[29384]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T18:59:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:26 smithi081 ceph-mon[39192]: pgmap v291: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:59:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:26 smithi081 ceph-mon[39192]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T18:59:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:26.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:26.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:26.971 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:26.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:26 smithi104 ceph-mon[32438]: pgmap v291: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T18:59:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:26 smithi104 ceph-mon[32438]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T18:59:27.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.099 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:27.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:27.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:27.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:27.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:27.374 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:27.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:27.507 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:59:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:59:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:59:27.513 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:27.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:27.645 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:27.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:59:27.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:27.777 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:27.796 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 35 16 68091 68075 121.51 82.5 0.0146013 0.00822491 2024-09-21T18:59:27.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:27.939 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:27.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:28.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:28.077 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:28.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:28.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:59:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:59:28.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:28.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:28.473 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:28.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:28.612 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:28.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:28.743 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:28.796 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 36 16 69504 69488 120.588 88.3125 0.00639077 0.00828799 2024-09-21T18:59:28.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:28.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:28.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:28.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:28 smithi081 ceph-mon[29384]: pgmap v293: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-21T18:59:28.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:28.919 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:28 smithi081 ceph-mon[39192]: pgmap v293: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-21T18:59:28.919 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:28.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:28.983 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:29.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:28 smithi104 ceph-mon[32438]: pgmap v293: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-21T18:59:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:29.031 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:29.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:29.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:29.280 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:29.306 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:29.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:29.440 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:29.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:29.542 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T18:59:29.571 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:29.661 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:59:29.661 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:59:29.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:29.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:29.797 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 37 15 70722 70707 119.388 76.1875 0.00870899 0.00837132 2024-09-21T18:59:29.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1881693474' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: from='client.35336 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: from='client.35336 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:29.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[39192]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1881693474' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: from='client.35336 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: from='client.35336 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:29.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:29 smithi081 ceph-mon[29384]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T18:59:29.846 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:29.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:29.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:29.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1881693474' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: from='client.35336 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: from='client.35336 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:30.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:29 smithi104 ceph-mon[32438]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T18:59:30.058 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:30.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:30.129 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:30.231 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:30.264 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:30.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:30.396 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:30.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:30.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:30.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T18:59:30.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:30.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:30.796 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 38 16 72135 72119 118.569 88.25 0.0132446 0.00842893 2024-09-21T18:59:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:30.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:30.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:30 smithi081 ceph-mon[29384]: pgmap v294: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 651 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-21T18:59:30.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:30 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:59:30.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:30 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:59:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:30 smithi081 ceph-mon[39192]: pgmap v294: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 651 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-21T18:59:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:30 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:59:30.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:30 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:59:30.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:30.939 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:30.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:31.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:30 smithi104 ceph-mon[32438]: pgmap v294: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 651 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-21T18:59:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:30 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:59:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:30 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:59:31.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:31.082 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:31.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.213 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:31.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:31.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:31.381 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:31.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:31.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:31.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:31.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:31.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:31.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:31.510 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:31.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:31.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:31.650 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:31.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.784 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:31.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:31.796 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 39 16 73455 73439 117.644 82.5 0.00975649 0.00849503 2024-09-21T18:59:31.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:31.923 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:31.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:32.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:31 smithi104 ceph-mon[32438]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T18:59:32.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:31 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4239336617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:32.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:32.054 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:31 smithi081 ceph-mon[39192]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T18:59:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:31 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4239336617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:31 smithi081 ceph-mon[29384]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T18:59:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:31 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4239336617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:32.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:32.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:32.324 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:32.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:32.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.471 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:32.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:32.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:32.702 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:32.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:32.797 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:59:32.795634+0000 min lat: 0.00110906 max lat: 0.19882 avg lat: 0.00855986 2024-09-21T18:59:32.797 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:59:32.797 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 40 16 74766 74750 116.752 81.9375 0.00774533 0.00855986 2024-09-21T18:59:32.839 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:32.873 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:32.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:32.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:33.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:33.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:32 smithi104 ceph-mon[32438]: pgmap v297: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T18:59:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:32 smithi081 ceph-mon[39192]: pgmap v297: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T18:59:33.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:32 smithi081 ceph-mon[29384]: pgmap v297: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T18:59:33.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:33.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:33.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:33.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:33.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T18:59:33.391 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:33.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:33.539 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:33.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:33.720 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:59:33.720 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:59:33.720 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T18:59:33.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T18:59:33.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:33.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 41 16 75900 75884 115.631 70.875 0.00694724 0.00864318 2024-09-21T18:59:33.800 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:33.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:33.932 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:33.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:34.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:34.059 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:34.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:34.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:34.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:59:34.285 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:34.329 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:34.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.478 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:34.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:34.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:34.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:34.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.739 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:34.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:34.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 42 16 76882 76866 114.339 61.375 0.0102833 0.00874145 2024-09-21T18:59:34.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:34.874 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:34.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:34.997 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:34.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:35.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:34 smithi104 ceph-mon[32438]: pgmap v298: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-21T18:59:35.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:34 smithi104 ceph-mon[32438]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T18:59:35.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:34 smithi081 ceph-mon[39192]: pgmap v298: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-21T18:59:35.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:34 smithi081 ceph-mon[39192]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T18:59:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:34 smithi081 ceph-mon[29384]: pgmap v298: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-21T18:59:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:34 smithi081 ceph-mon[29384]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T18:59:35.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:35.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:35.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:35.276 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:35.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:35.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:35.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:35.541 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:35.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:35.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:35.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:35.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 43 15 78134 78119 113.502 78.3125 0.00783935 0.00880576 2024-09-21T18:59:35.821 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:35.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:35.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:36.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:36.093 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:36.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.217 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:36.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:36.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:36.358 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:36.459 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:36.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:36.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:36.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:36.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 44 16 79336 79320 112.628 75.0625 0.0143622 0.00887387 2024-09-21T18:59:36.849 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:36.849 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T18:59:36.849 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T18:59:36.849 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T18:59:36.849 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:36.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:36.873 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:36.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:36.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:36 smithi104 ceph-mon[32438]: pgmap v300: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-21T18:59:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:37.014 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:36 smithi081 ceph-mon[39192]: pgmap v300: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-21T18:59:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:36 smithi081 ceph-mon[29384]: pgmap v300: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-21T18:59:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T18:59:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:37.116 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:37.151 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:37.252 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.304 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:37.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:37.405 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:37.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:37.538 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:37.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:59:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:59:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:59:37.571 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:37.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:37.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:37.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 45 16 80646 80630 111.945 81.875 0.0193695 0.00892762 2024-09-21T18:59:37.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:37.810 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:37.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T18:59:37.850 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:37.899 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:37.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:38.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:38.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:38.152 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:38.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:59:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:59:38.280 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:38.313 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:38.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:38.460 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:38.531 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:38.531 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:38.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:38.567 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:38.606 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:38.606 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:38.611 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:38.641 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:38.641 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:38.674 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:38.711 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:38.713 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:38.744 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:38.744 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:38.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:38.787 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:38.787 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:38.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 46 16 81776 81760 111.047 70.625 0.00416746 0.00899888 2024-09-21T18:59:38.857 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:38.895 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:38.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:38.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:38 smithi081 ceph-mon[29384]: pgmap v301: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T18:59:38.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:38.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2404360330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:38.967 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:38 smithi081 ceph-mon[39192]: pgmap v301: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T18:59:38.968 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:38.968 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2404360330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:38.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:38 smithi104 ceph-mon[32438]: pgmap v301: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T18:59:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2404360330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:39.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:39.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.175 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:39.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:39.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.328 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:39.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:39.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:39.468 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:39.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 190, 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T18:59:34.531852+0000", 2024-09-21T18:59:39.590 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T18:52:58.333331+0000", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T18:55:50.490072+0000", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5806080, 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "nodeep-scrub", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "noscrub", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 18, 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T18:59:39.591 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 8, 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.592 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 11, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 11, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 11, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 11, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 11, 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.593 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "170", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "168", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:59:39.594 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.595 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.596 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.1800000667572021, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.1800000667572021, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-21T18:59:39.597 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.598 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.599 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.600 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.601 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.602 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6500000953674316, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6500000953674316, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:59:39.603 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.604 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.605 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.606 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.607 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.5, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.5, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T18:59:39.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.611 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.25, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.25, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.25, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.25, 2024-09-21T18:59:39.613 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.614 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.616 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.617 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.618 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.25, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.25, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.25, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.25, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.619 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.620 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.622 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.624 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.75, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.75, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.75, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.75, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.625 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 8, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_1", 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:58:42.335766+0000", 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T18:59:39.626 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 16, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 16, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 16, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 16, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 16, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "177", 2024-09-21T18:59:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T18:59:39.628 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T18:59:39.629 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T18:59:39.630 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.631 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 172, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.632 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.636 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:39.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 172, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T18:59:39.637 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.639 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.640 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 144, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.643 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 172, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.644 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T18:59:39.646 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T18:59:39.647 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 172, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.648 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.649 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.650 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.651 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 37, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 172, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.652 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/2569144231", 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/2569144231", 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/2569144231", 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/2569144231", 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.654 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 45, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 172, 2024-09-21T18:59:39.655 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 185975065 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/185975065", 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/185975065", 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/185975065", 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/185975065", 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.658 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T18:59:39.659 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 170, 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:59:39.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T18:59:39.662 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.663 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:59:18.069658+0000", 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:43.755530+0000", 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T18:59:39.664 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:59:29.694579+0000", 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.665 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:32.194729+0000", 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:59:24.922535+0000", 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:59:28.741565+0000", 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.667 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:36.602780+0000", 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T18:58:53.896362+0000", 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T18:59:39.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.7", 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.10", 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.13", 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T18:59:39.670 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.18", 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.1d", 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:39.671 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [], 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:59:39.672 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T18:59:39.673 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T18:59:39.674 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T18:59:39.675 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T18:59:39.675 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:39.675 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:39.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.779 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:39.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:39.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 47 16 82869 82853 110.137 68.3125 0.00829104 0.00907468 2024-09-21T18:59:39.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:39.920 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:39.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:40.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1163069419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T18:59:40.022 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:40.058 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:40.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1163069419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T18:59:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1163069419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T18:59:40.159 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:40.195 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:40.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:40.326 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:40.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:40.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:40.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:40.583 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:40.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:40.718 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T18:59:40.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 48 16 84202 84186 109.578 83.3125 0.0266376 0.00912109 2024-09-21T18:59:40.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:40.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:40.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:40.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:40.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:41.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:40 smithi104 ceph-mon[32438]: pgmap v302: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T18:59:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1927982084' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:59:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:40 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:59:41.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:40 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:59:41.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[29384]: pgmap v302: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1927982084' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[39192]: pgmap v302: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1927982084' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:59:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:40 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:59:41.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:41.123 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:41.225 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:41.252 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:41.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:41.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:41.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:41.511 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:41.613 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:41.652 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:41.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:41.787 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:41.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 49 16 85448 85432 108.931 77.875 0.0113261 0.0091754 2024-09-21T18:59:41.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:41.924 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:41.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:41 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1927982084' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:59:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:41 smithi104 ceph-mon[32438]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T18:59:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:41 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T18:59:42.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:41 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T18:59:42.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1927982084' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[29384]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1927982084' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[39192]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T18:59:42.031 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:41 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T18:59:42.058 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:42.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.197 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:42.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:42.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:42.335 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:42.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:59:42.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.474 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:42.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:42.575 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:42.600 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:42.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:42.738 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:42.798 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 50 15 86684 86669 108.299 77.3125 0.0143478 0.00922936 2024-09-21T18:59:42.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:42.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:42.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:42.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:43.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:43.016 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[39192]: pgmap v304: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 782 op/s 2024-09-21T18:59:43.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T18:59:43.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T18:59:43.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[39192]: 6.10 scrub starts 2024-09-21T18:59:43.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[39192]: 6.10 scrub ok 2024-09-21T18:59:43.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[29384]: pgmap v304: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 782 op/s 2024-09-21T18:59:43.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T18:59:43.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T18:59:43.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[29384]: 6.10 scrub starts 2024-09-21T18:59:43.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:42 smithi081 ceph-mon[29384]: 6.10 scrub ok 2024-09-21T18:59:43.068 INFO:tasks.thrashosds.thrasher:j is {'epoch': 190, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T18:59:34.531852+0000', 'last_up_change': '2024-09-21T18:52:58.333331+0000', 'last_in_change': '2024-09-21T18:55:50.490072+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '168', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-21T18:58:42.335766+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 2569144231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 2569144231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 2569144231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 2569144231}]}, 'public_addr': '172.21.15.104:6804/2569144231', 'cluster_addr': '172.21.15.104:6805/2569144231', 'heartbeat_back_addr': '172.21.15.104:6807/2569144231', 'heartbeat_front_addr': '172.21.15.104:6806/2569144231', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 185975065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 185975065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 185975065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 185975065}]}, 'public_addr': '172.21.15.104:6808/185975065', 'cluster_addr': '172.21.15.104:6809/185975065', 'heartbeat_back_addr': '172.21.15.104:6811/185975065', 'heartbeat_front_addr': '172.21.15.104:6810/185975065', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 170, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:59:18.069658+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:43.755530+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:59:29.694579+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:32.194729+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:59:24.922535+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:59:28.741565+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:36.602780+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T18:58:53.896362+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 4, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T18:59:43.068 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T18:59:43.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:43.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:43.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:42 smithi104 ceph-mon[32438]: pgmap v304: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 782 op/s 2024-09-21T18:59:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:42 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T18:59:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:42 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T18:59:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:42 smithi104 ceph-mon[32438]: 6.10 scrub starts 2024-09-21T18:59:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:42 smithi104 ceph-mon[32438]: 6.10 scrub ok 2024-09-21T18:59:43.277 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:43.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:43.411 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:43.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:43.546 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:43.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:43.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:43.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:43.799 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 51 16 88013 87997 107.802 83 0.00921934 0.00927129 2024-09-21T18:59:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:43.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:43.963 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:43.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:44.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:43 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T18:59:44.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:43 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T18:59:44.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:43 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T18:59:44.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:43 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T18:59:44.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:44.096 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:44.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:44.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:43 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T18:59:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:43 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T18:59:44.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:44.340 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:59:44.340 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:59:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:44.350 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:44.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:44.486 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:44.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:44.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:44.799 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 52 16 89256 89240 107.223 77.6875 0.00801643 0.00932193 2024-09-21T18:59:44.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:44.900 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:44.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:45.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.036 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:45.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[29384]: pgmap v305: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[29384]: 6.0 scrub starts 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[29384]: 6.0 scrub ok 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[29384]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[39192]: pgmap v305: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[39192]: 6.0 scrub starts 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[39192]: 6.0 scrub ok 2024-09-21T18:59:45.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:44 smithi081 ceph-mon[39192]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T18:59:45.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T18:59:45.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:45.169 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:44 smithi104 ceph-mon[32438]: pgmap v305: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T18:59:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:44 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T18:59:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:44 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T18:59:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:44 smithi104 ceph-mon[32438]: 6.0 scrub starts 2024-09-21T18:59:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:44 smithi104 ceph-mon[32438]: 6.0 scrub ok 2024-09-21T18:59:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:44 smithi104 ceph-mon[32438]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T18:59:45.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:45.296 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:45.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:45.458 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:45.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.600 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:45.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:45.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:45.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T18:59:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:45.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:45.760 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:45.760 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:45.799 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 53 16 90420 90404 106.573 72.75 0.012098 0.00937799 2024-09-21T18:59:45.809 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:45.848 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:45.848 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:45.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:45.881 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:45.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:45.889 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:45.889 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:45.923 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:45.924 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:45.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: 4.4 scrub starts 2024-09-21T18:59:45.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: 4.4 scrub ok 2024-09-21T18:59:45.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T18:59:45.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: 6.16 scrub starts 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: 6.16 scrub ok 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1361652013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: 4.4 scrub starts 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: 4.4 scrub ok 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: 6.16 scrub starts 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: 6.16 scrub ok 2024-09-21T18:59:45.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:45 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1361652013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:45.958 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:45.958 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:45.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:45.989 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:45.989 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:46.023 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:46.023 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:46.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:46.136 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:46.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: 4.4 scrub starts 2024-09-21T18:59:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: 4.4 scrub ok 2024-09-21T18:59:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T18:59:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T18:59:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: 6.16 scrub starts 2024-09-21T18:59:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: 6.16 scrub ok 2024-09-21T18:59:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:45 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1361652013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:46.277 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:46.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:46.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:46.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:46.550 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:46.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:46.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:46.800 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 54 16 91525 91509 105.876 69.0625 0.00714581 0.00943888 2024-09-21T18:59:46.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:46.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:46.951 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:46.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:47.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[29384]: pgmap v307: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[29384]: 1.7 scrub starts 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[29384]: 1.7 scrub ok 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[39192]: pgmap v307: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[39192]: 1.7 scrub starts 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[39192]: 1.7 scrub ok 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T18:59:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:46 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T18:59:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:47.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:47.229 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:46 smithi104 ceph-mon[32438]: pgmap v307: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 48 MiB/s wr, 767 op/s 2024-09-21T18:59:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:46 smithi104 ceph-mon[32438]: 1.7 scrub starts 2024-09-21T18:59:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:46 smithi104 ceph-mon[32438]: 1.7 scrub ok 2024-09-21T18:59:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:46 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T18:59:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:46 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T18:59:47.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:47.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:47.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:47.496 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:47.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:59:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:59:47.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:47.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:47.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:47.754 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:47.800 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 55 16 92724 92708 105.313 74.9375 0.0162235 0.00949088 2024-09-21T18:59:47.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:47.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:47.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:47.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:48.028 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T18:59:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 1.4 scrub starts 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[29384]: 1.4 scrub ok 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 1.4 scrub starts 2024-09-21T18:59:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:47 smithi081 ceph-mon[39192]: 1.4 scrub ok 2024-09-21T18:59:48.069 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:59:48.069 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:59:48.069 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T18:59:48.069 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T18:59:48.069 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T18:59:48.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T18:59:48.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:48.156 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:48.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 1.4 scrub starts 2024-09-21T18:59:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:47 smithi104 ceph-mon[32438]: 1.4 scrub ok 2024-09-21T18:59:48.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:48.292 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:48.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:48.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:48.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:48.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:48.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:48.717 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:93869 8.b 8:d3cce824:::benchmark_data_smithi081_105538_object93868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:59:48.784329+0000", 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0073102920000000004, 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0073414760000000004, 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:59:48.797 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 93869 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784329+0000", 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784329+0000", 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784332+0000", 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.362e-06 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784360+0000", 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7807999999999999e-05 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784377+0000", 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7138000000000001e-05 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784387+0000", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9119999999999993e-06 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784489+0000", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010255900000000001 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784617+0000", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012817800000000001 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T18:59:48.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.784759+0000", 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014153000000000001 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.788609+0000", 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038503610000000001 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789587+0000", 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097753199999999992 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:93871 8.c e192/172 8:3aa5a240:::benchmark_data_smithi081_105538_object93870:head v 192'5976, mlcod=192'5974)", 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:59:48.789356+0000", 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0022834550000000002, 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002390325, 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789356+0000", 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789356+0000", 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789357+0000", 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.176e-06 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789386+0000", 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8881999999999999e-05 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789396+0000", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0020999999999999e-05 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789403+0000", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3559999999999998e-06 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789666+0000", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026317400000000003 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:48.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.789686+0000", 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0458e-05 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:93873 8.0 8:0cbbe63b:::benchmark_data_smithi081_105538_object93872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:59:48.790416+0000", 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001223476, 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013833459999999999, 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:59:48.805 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 93873 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790416+0000", 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790416+0000", 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790419+0000", 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5299999999999999e-06 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790445+0000", 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6285e-05 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790465+0000", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0024999999999999e-05 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790473+0000", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3799999999999994e-06 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790634+0000", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016074700000000001 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T18:59:48.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790763+0000", 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012887700000000001 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790906+0000", 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014260799999999999 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:93876 8.0 8.d2ac2720 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:59:48.790493+0000", 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0011465259999999999, 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001367184, 2024-09-21T18:59:48.809 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 93876 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790493+0000", 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790493+0000", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790494+0000", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3600000000000002e-07 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790518+0000", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3643999999999999e-05 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790525+0000", 2024-09-21T18:59:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1160000000000001e-06 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790531+0000", 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1290000000000001e-06 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:93880 8.b 8.d67a794b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:59:48.790539+0000", 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0011006379999999999, 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013651760000000001, 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T18:59:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 93880 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790539+0000", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790539+0000", 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T18:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790540+0000", 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3e-07 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790563+0000", 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2955999999999998e-05 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790569+0000", 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5100000000000004e-06 2024-09-21T18:59:48.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:59:48.790575+0000", 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4310000000000002e-06 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 5 2024-09-21T18:59:48.815 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:48.816 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 56 16 93887 93871 104.731 72.6875 0.00634328 0.00954304 2024-09-21T18:59:48.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:48.826 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:48.847 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:48.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:48.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: pgmap v308: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 1.5 scrub starts 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 1.5 scrub ok 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: pgmap v308: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 1.5 scrub starts 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 1.5 scrub ok 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T18:59:49.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:48 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T18:59:49.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:49.123 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:49.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:49.256 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: pgmap v308: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 1.5 scrub starts 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 1.5 scrub ok 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T18:59:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:48 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T18:59:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:49.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:49.398 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:49.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:49.526 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:49.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:49.667 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:49.769 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:49.800 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 57 16 95147 95131 104.275 78.75 0.0269311 0.00958428 2024-09-21T18:59:49.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:49.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:49.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:49.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:50.042 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 3.1c scrub starts 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 3.1c scrub ok 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T18:59:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 3.7 scrub starts 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 3.7 scrub ok 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[39192]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 3.1c scrub starts 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 3.1c scrub ok 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T18:59:50.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 3.7 scrub starts 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 3.7 scrub ok 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T18:59:50.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:49 smithi081 ceph-mon[29384]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T18:59:50.070 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:50.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:50.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:50.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 3.1c scrub starts 2024-09-21T18:59:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 3.1c scrub ok 2024-09-21T18:59:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 3.7 scrub starts 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 3.7 scrub ok 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T18:59:50.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:49 smithi104 ceph-mon[32438]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T18:59:50.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:50.322 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:50.367 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T18:59:50.373 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:50.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:50.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:59:50.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:50.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:50.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:50.631 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T18:59:50.631 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T18:59:50.631 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T18:59:50.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:50.704 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:50.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:50.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 58 16 96287 96271 103.706 71.25 0.0279037 0.0096377 2024-09-21T18:59:50.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:50.834 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[29384]: pgmap v310: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T18:59:50.834 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[29384]: 1.2 scrub starts 2024-09-21T18:59:50.834 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[29384]: 1.2 scrub ok 2024-09-21T18:59:50.834 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[29384]: 3.1f scrub starts 2024-09-21T18:59:50.834 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[29384]: 3.1f scrub ok 2024-09-21T18:59:50.859 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:50.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:50.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:51.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:51.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:51.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[39192]: pgmap v310: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T18:59:51.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[39192]: 1.2 scrub starts 2024-09-21T18:59:51.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[39192]: 1.2 scrub ok 2024-09-21T18:59:51.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[39192]: 3.1f scrub starts 2024-09-21T18:59:51.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:50 smithi081 ceph-mon[39192]: 3.1f scrub ok 2024-09-21T18:59:51.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:50 smithi104 ceph-mon[32438]: pgmap v310: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T18:59:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:50 smithi104 ceph-mon[32438]: 1.2 scrub starts 2024-09-21T18:59:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:50 smithi104 ceph-mon[32438]: 1.2 scrub ok 2024-09-21T18:59:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:50 smithi104 ceph-mon[32438]: 3.1f scrub starts 2024-09-21T18:59:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:50 smithi104 ceph-mon[32438]: 3.1f scrub ok 2024-09-21T18:59:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:51.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:51.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:51.426 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:51.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.564 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:51.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:51.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.701 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:51.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:51.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 59 16 97554 97538 103.29 79.1875 0.0101938 0.00967608 2024-09-21T18:59:51.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:51.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.838 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:51.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:51.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:51.979 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:51.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:52.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.129 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:52.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:52.135 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:52.138 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:52.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 4.1a scrub starts 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 4.1a scrub ok 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 3.2 scrub starts 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 3.2 scrub ok 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 6.18 scrub starts 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 6.18 scrub ok 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: 8.5 scrub starts 2024-09-21T18:59:52.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:52.261 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 4.1a scrub starts 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 4.1a scrub ok 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T18:59:52.276 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 3.2 scrub starts 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 3.2 scrub ok 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 6.18 scrub starts 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 6.18 scrub ok 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: 8.5 scrub starts 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 4.1a scrub starts 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 4.1a scrub ok 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T18:59:52.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T18:59:52.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 3.2 scrub starts 2024-09-21T18:59:52.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 3.2 scrub ok 2024-09-21T18:59:52.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 6.18 scrub starts 2024-09-21T18:59:52.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 6.18 scrub ok 2024-09-21T18:59:52.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: 8.5 scrub starts 2024-09-21T18:59:52.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T18:59:52.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:52.403 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:52.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.539 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:52.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:52.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.665 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:52.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:52.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:52.781 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T18:59:52.781 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:59:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.794 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:52.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:52.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T18:59:52.799827+0000 min lat: 0.00110906 max lat: 0.19882 avg lat: 0.00973407 2024-09-21T18:59:52.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T18:59:52.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 60 16 98625 98609 102.684 66.9375 0.0229545 0.00973407 2024-09-21T18:59:52.816 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T18:59:52.816 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:59:52.856 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T18:59:52.856 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:59:52.856 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: pgmap v312: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 58 MiB/s wr, 912 op/s 2024-09-21T18:59:52.856 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: 6.6 scrub starts 2024-09-21T18:59:52.856 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: 6.6 scrub ok 2024-09-21T18:59:52.856 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: 8.6 scrub starts 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/735358189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: from='client.35369 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1992331497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: pgmap v312: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 58 MiB/s wr, 912 op/s 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: 6.6 scrub starts 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: 6.6 scrub ok 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: 8.6 scrub starts 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/735358189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: from='client.35369 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:52.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1992331497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:52.870 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T18:59:52.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:52.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T18:59:52.895 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:59:52.918 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:52.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:52.933 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T18:59:52.933 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:59:52.966 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T18:59:52.966 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:59:53.001 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T18:59:53.001 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:59:53.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:53.048 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T18:59:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:59:53.062 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:53.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:53.199 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:53.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: pgmap v312: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 58 MiB/s wr, 912 op/s 2024-09-21T18:59:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: 6.6 scrub starts 2024-09-21T18:59:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: 6.6 scrub ok 2024-09-21T18:59:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: 8.6 scrub starts 2024-09-21T18:59:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/735358189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: from='client.35369 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T18:59:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1992331497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T18:59:53.301 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:53.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:53.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:53.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:53.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:53.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:53.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:53.752 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:53.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 61 16 99727 99711 102.13 68.875 0.0108801 0.00978644 2024-09-21T18:59:53.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:53.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:53.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:54.023 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:54.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:54.164 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 8.5 scrub ok 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T18:59:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:53 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T18:59:54.266 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:54.293 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 8.5 scrub ok 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T18:59:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 8.5 scrub ok 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T18:59:54.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:53 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T18:59:54.395 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:54.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:54.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:54.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:54.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:54.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:54.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 62 15 100781 100766 101.546 65.9375 0.0087991 0.00984223 2024-09-21T18:59:54.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:54.839 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:54.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:54.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:55.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:55.116 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:55.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:55.256 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: 8.a scrub starts 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: pgmap v314: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: 4.c scrub starts 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: 4.c scrub ok 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: 3.6 scrub starts 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: 3.6 scrub ok 2024-09-21T18:59:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: 8.6 scrub ok 2024-09-21T18:59:55.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:54 smithi104 ceph-mon[32438]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: 8.a scrub starts 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: pgmap v314: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: 4.c scrub starts 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: 4.c scrub ok 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: 3.6 scrub starts 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: 3.6 scrub ok 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: 8.6 scrub ok 2024-09-21T18:59:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[29384]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: 8.a scrub starts 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: pgmap v314: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: 4.c scrub starts 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: 4.c scrub ok 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: 3.6 scrub starts 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: 3.6 scrub ok 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: 8.6 scrub ok 2024-09-21T18:59:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:54 smithi081 ceph-mon[39192]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T18:59:55.358 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:55.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:55.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:55.516 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:55.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:55.664 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:55.765 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:55.768 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:59:55.768 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:59:55.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:55.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 63 16 101919 101903 101.062 71.0625 0.0117948 0.00989051 2024-09-21T18:59:55.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:55.927 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:55.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:56.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[39192]: 3.4 scrub starts 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[39192]: 3.4 scrub ok 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[39192]: 8.c scrub starts 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[29384]: 3.4 scrub starts 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[29384]: 3.4 scrub ok 2024-09-21T18:59:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:55 smithi081 ceph-mon[29384]: 8.c scrub starts 2024-09-21T18:59:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.068 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:56.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:56.168 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:56.199 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:55 smithi104 ceph-mon[32438]: 3.4 scrub starts 2024-09-21T18:59:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:55 smithi104 ceph-mon[32438]: 3.4 scrub ok 2024-09-21T18:59:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:55 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T18:59:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:55 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T18:59:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:55 smithi104 ceph-mon[32438]: 8.c scrub starts 2024-09-21T18:59:56.301 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:56.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:56.465 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:56.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:56.620 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T18:59:56.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T18:59:56.649 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T18:59:56.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:56.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:56.801 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 64 16 103101 103085 100.638 73.875 0.0182341 0.00993123 2024-09-21T18:59:56.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T18:59:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:56.873 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:56.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T18:59:56.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T18:59:56.973 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:57.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:57.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:57.139 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:57.178 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[39192]: pgmap v316: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[39192]: 3.3 scrub starts 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[39192]: 3.3 scrub ok 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[39192]: 8.a scrub ok 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[39192]: 5.17 scrub starts 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[39192]: 5.17 scrub ok 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[29384]: pgmap v316: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[29384]: 3.3 scrub starts 2024-09-21T18:59:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[29384]: 3.3 scrub ok 2024-09-21T18:59:57.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[29384]: 8.a scrub ok 2024-09-21T18:59:57.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[29384]: 5.17 scrub starts 2024-09-21T18:59:57.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:56 smithi081 ceph-mon[29384]: 5.17 scrub ok 2024-09-21T18:59:57.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:56 smithi104 ceph-mon[32438]: pgmap v316: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T18:59:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:56 smithi104 ceph-mon[32438]: 3.3 scrub starts 2024-09-21T18:59:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:56 smithi104 ceph-mon[32438]: 3.3 scrub ok 2024-09-21T18:59:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:56 smithi104 ceph-mon[32438]: 8.a scrub ok 2024-09-21T18:59:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:56 smithi104 ceph-mon[32438]: 5.17 scrub starts 2024-09-21T18:59:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:56 smithi104 ceph-mon[32438]: 5.17 scrub ok 2024-09-21T18:59:57.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:57.383 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:57.413 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:57.460 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:18:59:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T18:59:57.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:59:57.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:57.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:57.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:57.670 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:57.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:57.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 65 16 104108 104092 100.057 62.9375 0.020143 0.00998955 2024-09-21T18:59:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:57.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:57.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[39192]: 5.9 scrub starts 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[39192]: 5.9 scrub ok 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[39192]: 8.c scrub ok 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T18:59:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T18:59:58.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[29384]: 5.9 scrub starts 2024-09-21T18:59:58.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[29384]: 5.9 scrub ok 2024-09-21T18:59:58.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:57 smithi081 ceph-mon[29384]: 8.c scrub ok 2024-09-21T18:59:58.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T18:59:58.139 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T18:59:58.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:58.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 18:59:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T18:59:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:57 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T18:59:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:57 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T18:59:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:57 smithi104 ceph-mon[32438]: 5.9 scrub starts 2024-09-21T18:59:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:57 smithi104 ceph-mon[32438]: 5.9 scrub ok 2024-09-21T18:59:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:57 smithi104 ceph-mon[32438]: 8.c scrub ok 2024-09-21T18:59:58.283 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:58.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:58.388 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T18:59:58.388 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:59:58.420 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:58.521 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T18:59:58.551 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T18:59:58.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:58.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:58.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 66 16 105268 105252 99.6393 72.5 0.0204931 0.0100317 2024-09-21T18:59:58.810 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:58.847 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:58.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:58.981 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:58.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: pgmap v317: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: 4.0 scrub starts 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: 4.0 scrub ok 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: 8.9 scrub starts 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4152309806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:59:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[39192]: from='client.35381 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: pgmap v317: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: 4.0 scrub starts 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: 4.0 scrub ok 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: 8.9 scrub starts 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4152309806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:58 smithi081 ceph-mon[29384]: from='client.35381 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:59:59.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:59.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:59.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:59.224 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T18:59:59.253 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: pgmap v317: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: 4.0 scrub starts 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: 4.0 scrub ok 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: 8.9 scrub starts 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4152309806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:59:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:58 smithi104 ceph-mon[32438]: from='client.35381 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T18:59:59.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T18:59:59.388 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T18:59:59.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T18:59:59.527 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T18:59:59.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T18:59:59.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T18:59:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.690 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:59.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T18:59:59.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T18:59:59.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 67 16 106484 106468 99.2864 76 0.0134446 0.0100669 2024-09-21T18:59:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.830 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T18:59:59.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T18:59:59.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:59 smithi104 ceph-mon[32438]: from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T18:59:59.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:59 smithi104 ceph-mon[32438]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T18:59:59.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:59 smithi104 ceph-mon[32438]: 6.13 scrub starts 2024-09-21T18:59:59.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:59 smithi104 ceph-mon[32438]: 6.13 scrub ok 2024-09-21T18:59:59.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 18:59:59 smithi104 ceph-mon[32438]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T18:59:59.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T18:59:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:59:59.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T18:59:59.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:00.042 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:00.057 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:00.064 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:00.092 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:00.094 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T19:00:00.133 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:00.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:00.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[39192]: from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:00:00.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[39192]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T19:00:00.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[39192]: 6.13 scrub starts 2024-09-21T19:00:00.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[39192]: 6.13 scrub ok 2024-09-21T19:00:00.212 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[39192]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T19:00:00.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[29384]: from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:00:00.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[29384]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T19:00:00.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[29384]: 6.13 scrub starts 2024-09-21T19:00:00.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[29384]: 6.13 scrub ok 2024-09-21T19:00:00.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 18:59:59 smithi081 ceph-mon[29384]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T19:00:00.230 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:00.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:00.379 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:00.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:00.518 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:00.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:00.630 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:00.630 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:00.653 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:00.755 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:00.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 68 16 107631 107615 98.8806 71.6875 0.0413705 0.010109 2024-09-21T19:00:00.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:00.922 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:00.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:01.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:01.058 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:01.143 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: pgmap v319: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 6.12 scrub starts 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 6.12 scrub ok 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 4.17 scrub starts 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 4.17 scrub ok 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: overall HEALTH_OK 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[39192]: 8.9 scrub ok 2024-09-21T19:00:01.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: pgmap v319: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 6.12 scrub starts 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 6.12 scrub ok 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 4.17 scrub starts 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 4.17 scrub ok 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: overall HEALTH_OK 2024-09-21T19:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:00 smithi081 ceph-mon[29384]: 8.9 scrub ok 2024-09-21T19:00:01.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:01.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:01.190 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:01.191 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: pgmap v319: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 6.12 scrub starts 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 6.12 scrub ok 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 4.17 scrub starts 2024-09-21T19:00:01.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 4.17 scrub ok 2024-09-21T19:00:01.202 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: overall HEALTH_OK 2024-09-21T19:00:01.202 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:00 smithi104 ceph-mon[32438]: 8.9 scrub ok 2024-09-21T19:00:01.238 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:01.239 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:01.286 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:01.286 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:01.289 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:01.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:01.325 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:01.325 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:01.345 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:01.360 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:01.360 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:01.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:00:01.399 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:01.399 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:01.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:01.436 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:01.436 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:01.464 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:01.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:01.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:01.741 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:01.796 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:01.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 69 16 108695 108679 98.4113 66.5 0.00948042 0.0101571 2024-09-21T19:00:01.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:01.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:01.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:01.945 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:00:01.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:02.016 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:02.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:02.149 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:02.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T19:00:02.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 3.1a scrub starts 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 3.1a scrub ok 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 5.4 scrub starts 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 5.4 scrub ok 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 4.11 scrub starts 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: 4.11 scrub ok 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2298564997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2326739307' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:02.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:01 smithi104 ceph-mon[32438]: from='client.25318 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:02.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:02.290 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:02.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T19:00:02.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T19:00:02.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 3.1a scrub starts 2024-09-21T19:00:02.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 3.1a scrub ok 2024-09-21T19:00:02.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 5.4 scrub starts 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 5.4 scrub ok 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 4.11 scrub starts 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: 4.11 scrub ok 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2298564997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2326739307' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[39192]: from='client.25318 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 3.1a scrub starts 2024-09-21T19:00:02.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 3.1a scrub ok 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 5.4 scrub starts 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 5.4 scrub ok 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 4.11 scrub starts 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: 4.11 scrub ok 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2298564997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2326739307' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:02.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:01 smithi081 ceph-mon[29384]: from='client.25318 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:02.384 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:00:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:00:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:00:02.384 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:00:02.385 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:00:02.392 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:02.430 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:02.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:02.570 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:02.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:02.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:02.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:02.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 70 16 109883 109867 98.0661 74.25 0.00462981 0.0101928 2024-09-21T19:00:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.832 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:02.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:02.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:02.961 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:02.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:03.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:03.092 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:03.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: pgmap v321: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 844 op/s 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: 3.8 scrub starts 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: 3.8 scrub ok 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:00:03.211 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[39192]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: pgmap v321: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 844 op/s 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: 3.8 scrub starts 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: 3.8 scrub ok 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:00:03.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:02 smithi081 ceph-mon[29384]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T19:00:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: pgmap v321: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 844 op/s 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: 3.8 scrub starts 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: 3.8 scrub ok 2024-09-21T19:00:03.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:00:03.237 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:02 smithi104 ceph-mon[32438]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T19:00:03.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:00:03.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:03.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:03.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:03.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:03.641 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:03.699 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:03.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:03.764 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:03.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 71 16 111055 111039 97.7166 73.25 0.0174765 0.0102294 2024-09-21T19:00:03.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:03.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:03.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T19:00:04.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:04.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:04.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3456301294' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:00:04.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:03 smithi104 ceph-mon[32438]: from='client.35399 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:00:04.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:04.166 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:04.239 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:00:04.239 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:00:04.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:00:04.239 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:00:04.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101885 7.4 7:270c3457:::benchmark_data_smithi081_73391_object101884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.904224+0000", 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 284.33072387200002, 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112901179, 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101885 2024-09-21T19:00:04.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.904224+0000", 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.904224+0000", 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.904226+0000", 2024-09-21T19:00:04.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5719999999999999e-06 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.904256+0000", 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0380999999999999e-05 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.904267+0000", 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0908e-05 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.904274+0000", 2024-09-21T19:00:04.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0219999999999999e-06 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.906134+0000", 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018600750000000001 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.906253+0000", 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011932199999999999 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.906389+0000", 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013550699999999999 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.911578+0000", 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051891059999999998 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996747+0000", 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085169295000000006 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.017045+0000", 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020297803 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.017092+0000", 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6456999999999999e-05 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.017125+0000", 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3730999999999999e-05 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.245 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101888 7.4 7:2c1b1b4b:::benchmark_data_smithi081_73391_object101887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.907623+0000", 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 284.32732515599997, 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10966002, 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.246 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101888 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.907623+0000", 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.907623+0000", 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.907625+0000", 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1619999999999998e-06 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.907666+0000", 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1140000000000003e-05 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.907679+0000", 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3174e-05 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.907689+0000", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1779999999999992e-06 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908793+0000", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011048939999999999 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908910+0000", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116469 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909043+0000", 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013259900000000001 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996434+0000", 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087391621000000003 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996785+0000", 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035119300000000002 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.017174+0000", 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0203886 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.017242+0000", 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7607999999999994e-05 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.017283+0000", 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1381999999999998e-05 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102355 7.4 7:2a73ca6f:::benchmark_data_smithi081_73391_object102354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.90990292100003, 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16875653400000001, 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.252 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102355 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:00:04.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325049+0000", 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.754e-06 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325092+0000", 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3551999999999999e-05 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325112+0000", 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9306000000000001e-05 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325122+0000", 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7960000000000007e-06 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325298+0000", 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000176014 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325430+0000", 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013203999999999999 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325589+0000", 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000159486 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.328322+0000", 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002732656 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329357+0000", 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010357400000000001 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493698+0000", 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16434100300000001 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493758+0000", 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9515999999999997e-05 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493802+0000", 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3671000000000003e-05 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102361 7.4 7:2b82beac:::benchmark_data_smithi081_73391_object102360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.258 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.90506259199998, 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.166344565, 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102361 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329887+0000", 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.545e-06 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329943+0000", 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5818e-05 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329952+0000", 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6729999999999998e-06 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329956+0000", 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8509999999999998e-06 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330052+0000", 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5827000000000003e-05 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330254+0000", 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020186800000000001 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330415+0000", 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016062699999999999 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338473+0000", 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080582750000000002 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341282+0000", 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028086040000000001 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496118+0000", 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15483660299999999 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496188+0000", 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9626e-05 2024-09-21T19:00:04.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496230+0000", 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2247999999999999e-05 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102367 7.4 7:2bc3d420:::benchmark_data_smithi081_73391_object102366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.90226143299998, 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16373927799999999, 2024-09-21T19:00:04.264 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102367 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332689+0000", 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0810000000000001e-06 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332717+0000", 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8388999999999999e-05 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332731+0000", 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4158999999999999e-05 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332739+0000", 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1159999999999999e-06 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332800+0000", 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0343999999999997e-05 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332928+0000", 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128341 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333068+0000", 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013955200000000001 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338482+0000", 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054145959999999998 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341320+0000", 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028377950000000002 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496287+0000", 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15496721699999999 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496382+0000", 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4477999999999999e-05 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496426+0000", 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4209999999999999e-05 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102368 7.4 7:29e74c0a:::benchmark_data_smithi081_73391_object102367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.90140065399999, 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16300020900000001, 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.270 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102368 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333549+0000", 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1930000000000001e-06 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333748+0000", 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019946800000000001 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333759+0000", 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0645000000000001e-05 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333766+0000", 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0219999999999999e-06 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335818+0000", 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002052356 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335940+0000", 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012176999999999999 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336078+0000", 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137639 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341538+0000", 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054607579999999996 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341553+0000", 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4975e-05 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496464+0000", 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.154911199 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496506+0000", 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1530999999999999e-05 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496548+0000", 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1653000000000003e-05 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102370 7.4 7:261d1db0:::benchmark_data_smithi081_73391_object102369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.89760088899999, 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15954671000000001, 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.276 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102370 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337352+0000", 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1070000000000004e-06 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337487+0000", 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134325 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337498+0000", 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1063999999999999e-05 2024-09-21T19:00:04.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337505+0000", 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3159999999999999e-06 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338628+0000", 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001123298 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338755+0000", 2024-09-21T19:00:04.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127059 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338910+0000", 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154213 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341560+0000", 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026500429999999999 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.342134+0000", 2024-09-21T19:00:04.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057463499999999999 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496815+0000", 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15468104099999999 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496862+0000", 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7197999999999998e-05 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496894+0000", 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1411000000000001e-05 2024-09-21T19:00:04.281 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102384 7.1 7:8c91c1ab:::benchmark_data_smithi081_73391_object102383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.88989050999999, 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.151452372, 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.282 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102384 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345059+0000", 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.714e-06 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345113+0000", 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3814999999999999e-05 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345121+0000", 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2479999999999996e-06 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345126+0000", 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9309999999999999e-06 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345176+0000", 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9310000000000001e-05 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345271+0000", 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4955000000000001e-05 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345358+0000", 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7319999999999997e-05 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347395+0000", 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020371719999999999 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.349879+0000", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002483519 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496421+0000", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14654283400000001 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496473+0000", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1610000000000002e-05 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496510+0000", 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6943999999999997e-05 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102385 7.1 7:812d7c66:::benchmark_data_smithi081_73391_object102384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.347553+0000", 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 283.88739526500001, 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.149076334, 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.288 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102385 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347553+0000", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347553+0000", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347555+0000", 2024-09-21T19:00:04.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.748e-06 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347598+0000", 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3829000000000003e-05 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347608+0000", 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4409999999999999e-06 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347613+0000", 2024-09-21T19:00:04.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4689999999999996e-06 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347717+0000", 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010415400000000001 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:04.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347839+0000", 2024-09-21T19:00:04.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012136199999999999 2024-09-21T19:00:04.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347977+0000", 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137937 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.349912+0000", 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019354419999999999 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.349980+0000", 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7976000000000001e-05 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496556+0000", 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14657595400000001 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496600+0000", 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3404e-05 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496629+0000", 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9618e-05 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.294 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136136 7.1 7:864bf247:::benchmark_data_smithi081_73391_object136135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.162908+0000", 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 256.07203987000003, 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13027375999999999, 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.295 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136136 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162908+0000", 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162908+0000", 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162910+0000", 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.148e-06 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162948+0000", 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7450999999999998e-05 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162959+0000", 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1259e-05 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.162966+0000", 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8360000000000003e-06 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163020+0000", 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3669999999999999e-05 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163146+0000", 2024-09-21T19:00:04.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000126193 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.163270+0000", 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012440500000000001 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166109+0000", 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002839224 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.166374+0000", 2024-09-21T19:00:04.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026452399999999998 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293071+0000", 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12669671599999999 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293135+0000", 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4146999999999994e-05 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.293182+0000", 2024-09-21T19:00:04.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7187000000000001e-05 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136141 7.1 7:86054b31:::benchmark_data_smithi081_73391_object136140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.172556+0000", 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 256.06239181500001, 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.122814698, 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136141 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172556+0000", 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172556+0000", 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172558+0000", 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0889999999999998e-06 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172656+0000", 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7421000000000001e-05 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172667+0000", 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1389999999999999e-05 2024-09-21T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172674+0000", 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5130000000000002e-06 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172721+0000", 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6922999999999998e-05 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172806+0000", 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5892999999999999e-05 2024-09-21T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.172897+0000", 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0019000000000007e-05 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.175411+0000", 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025140240000000001 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.176791+0000", 2024-09-21T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001380287 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.295269+0000", 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11847814500000001 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.295325+0000", 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6202000000000002e-05 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.295371+0000", 2024-09-21T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5791999999999998e-05 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:168997 7.4 7:2a66f29b:::benchmark_data_smithi081_73391_object29998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.636643+0000", 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 219.59830490100001, 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.102852716, 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 168997 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636643+0000", 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636643+0000", 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636645+0000", 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.587e-06 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636650+0000", 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4809999999999997e-06 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636661+0000", 2024-09-21T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.079e-05 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636668+0000", 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1369999999999996e-06 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636723+0000", 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4763999999999999e-05 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636849+0000", 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000125656 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,6", 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.636996+0000", 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000147241 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.640979+0000", 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039828040000000004 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.640994+0000", 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5012e-05 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.641360+0000", 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000365955 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739317+0000", 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.097957254999999993 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739379+0000", 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2303000000000003e-05 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739496+0000", 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116731 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211285 7.1 7:8dc95181:::benchmark_data_smithi081_73391_object72286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.782614+0000", 2024-09-21T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 170.45233357999999, 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100967134, 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211285 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782614+0000", 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782614+0000", 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782616+0000", 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.269e-06 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782619+0000", 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3670000000000001e-06 2024-09-21T19:00:04.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782627+0000", 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5639999999999999e-06 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782631+0000", 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3250000000000001e-06 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782674+0000", 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3291999999999998e-05 2024-09-21T19:00:04.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782754+0000", 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9612999999999998e-05 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782826+0000", 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2087000000000003e-05 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.786209+0000", 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033828410000000001 2024-09-21T19:00:04.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.786585+0000", 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037598299999999998 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.883483+0000", 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096897867999999998 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.883541+0000", 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8312e-05 2024-09-21T19:00:04.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.883582+0000", 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0612999999999999e-05 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211293 7.4 7:2a9b502b:::benchmark_data_smithi081_73391_object72294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.791022+0000", 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 170.44392588700001, 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095801541000000004, 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.319 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211293 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791022+0000", 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791022+0000", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791024+0000", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.725e-06 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791029+0000", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7949999999999998e-06 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791040+0000", 2024-09-21T19:00:04.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1556000000000001e-05 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791046+0000", 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0109999999999997e-06 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791116+0000", 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9533e-05 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791201+0000", 2024-09-21T19:00:04.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5575999999999995e-05 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,6", 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791283+0000", 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1629000000000001e-05 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.793467+0000", 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021843359999999998 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.793901+0000", 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00043319799999999998 2024-09-21T19:00:04.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.804034+0000", 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010133756000000001 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.886702+0000", 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082667737000000005 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.886742+0000", 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9560000000000001e-05 2024-09-21T19:00:04.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.886824+0000", 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2128999999999999e-05 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211294 7.1 7:85c3584d:::benchmark_data_smithi081_73391_object72295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.792450+0000", 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 170.44249776500001, 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103310706, 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.325 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211294 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792450+0000", 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792450+0000", 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792452+0000", 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3850000000000001e-06 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792457+0000", 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9790000000000001e-06 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792469+0000", 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2517000000000001e-05 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792476+0000", 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9009999999999997e-06 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792521+0000", 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5169000000000001e-05 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792595+0000", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4130999999999999e-05 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.792659+0000", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3598999999999999e-05 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.794884+0000", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002224738 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.795065+0000", 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018140300000000001 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895680+0000", 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100614685 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895721+0000", 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1091999999999999e-05 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895761+0000", 2024-09-21T19:00:04.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0107000000000001e-05 2024-09-21T19:00:04.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.331 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.332 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3456301294' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:00:04.332 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:03 smithi081 ceph-mon[39192]: from='client.35399 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:00:04.332 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3456301294' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:00:04.332 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:03 smithi081 ceph-mon[29384]: from='client.35399 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:00:04.332 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211483 7.1 7:8e2978cb:::benchmark_data_smithi081_73391_object72484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:00:04.332 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.060241+0000", 2024-09-21T19:00:04.332 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 170.17470683400001, 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.124388863, 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211483 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060241+0000", 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060241+0000", 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060244+0000", 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3549999999999999e-06 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060250+0000", 2024-09-21T19:00:04.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8469999999999998e-06 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060267+0000", 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6880000000000001e-05 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060276+0000", 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6610000000000006e-06 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060336+0000", 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9854e-05 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060448+0000", 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000112015 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.060571+0000", 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012271 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.063353+0000", 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027824590000000001 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.064086+0000", 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073266300000000004 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.184514+0000", 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.120428387 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.184580+0000", 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5543999999999995e-05 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.184630+0000", 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0488000000000002e-05 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211492 7.1 7:8a873050:::benchmark_data_smithi081_73391_object72493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.069155+0000", 2024-09-21T19:00:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 170.16579344199999, 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.115584282, 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211492 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069155+0000", 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069155+0000", 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069157+0000", 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9870000000000002e-06 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069163+0000", 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.207e-06 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069177+0000", 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4395e-05 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069185+0000", 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9810000000000003e-06 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069249+0000", 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4288999999999994e-05 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069375+0000", 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000125919 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.069503+0000", 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127138 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.072094+0000", 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025915000000000001 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.072274+0000", 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017972100000000001 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.184675+0000", 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11240117400000001 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.184705+0000", 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9709e-05 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.184739+0000", 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4261999999999998e-05 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242080 7.1 7:892cd020:::benchmark_data_smithi081_73391_object103081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 134.729239355, 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16669837000000001, 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.344 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242080 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505710+0000", 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5400000000000001e-06 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505716+0000", 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3170000000000001e-06 2024-09-21T19:00:04.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505725+0000", 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7899999999999994e-06 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505732+0000", 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9789999999999996e-06 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505867+0000", 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134523 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505964+0000", 2024-09-21T19:00:04.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6680000000000003e-05 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.506050+0000", 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6395000000000002e-05 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.509085+0000", 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030351779999999999 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.509739+0000", 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00065418699999999998 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672337+0000", 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16259812500000001 2024-09-21T19:00:04.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672376+0000", 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8399999999999998e-05 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672407+0000", 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1256e-05 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.355 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242082 7.1 7:83ef7caf:::benchmark_data_smithi081_73391_object103083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 134.72434019400001, 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16217166099999999, 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242082 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510609+0000", 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.345e-06 2024-09-21T19:00:04.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510614+0000", 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3030000000000001e-06 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510623+0000", 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3559999999999999e-06 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510630+0000", 2024-09-21T19:00:04.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9430000000000003e-06 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510673+0000", 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3180999999999999e-05 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510747+0000", 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4388999999999995e-05 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510818+0000", 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0288000000000002e-05 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515225+0000", 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044072699999999996 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515269+0000", 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4165000000000002e-05 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672707+0000", 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15743800899999999 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672742+0000", 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4878e-05 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672780+0000", 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7533999999999997e-05 2024-09-21T19:00:04.361 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242083 7.1 7:8f0bc0f0:::benchmark_data_smithi081_73391_object103084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 134.72299097499999, 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18721689799999999, 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:04.362 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242083 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511958+0000", 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7300000000000004e-07 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511962+0000", 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9199999999999997e-06 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511973+0000", 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0577000000000001e-05 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511980+0000", 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5310000000000004e-06 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512698+0000", 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071787200000000004 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512808+0000", 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110412 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512929+0000", 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012087999999999999 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515461+0000", 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002532191 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516028+0000", 2024-09-21T19:00:04.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056677599999999996 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699096+0000", 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18306810000000001 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699133+0000", 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6837e-05 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699174+0000", 2024-09-21T19:00:04.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0828999999999998e-05 2024-09-21T19:00:04.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.368 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:04.368 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:04.368 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:00:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:04.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:04.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:00:04.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:04.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:04.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:04.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:04.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 72 16 112204 112188 97.3567 71.8125 0.00967224 0.0102667 2024-09-21T19:00:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:04.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:04.974 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:04.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:05.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:05.107 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:05.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:05.242 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:05.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:04 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:00:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:04 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:00:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:04 smithi104 ceph-mon[32438]: pgmap v323: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 932 op/s 2024-09-21T19:00:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:04 smithi104 ceph-mon[32438]: from='client.35399 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:00:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:04 smithi104 ceph-mon[32438]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T19:00:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:00:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:00:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[29384]: pgmap v323: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 932 op/s 2024-09-21T19:00:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[29384]: from='client.35399 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:00:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[29384]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T19:00:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:00:05.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:00:05.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[39192]: pgmap v323: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 932 op/s 2024-09-21T19:00:05.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[39192]: from='client.35399 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:00:05.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:04 smithi081 ceph-mon[39192]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T19:00:05.344 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:05.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:00:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:05.503 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:05.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:05.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:05.735 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:05.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:00:05.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:05.802 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 73 16 113284 113268 96.9477 67.5 0.015856 0.0103104 2024-09-21T19:00:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.834 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:05.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:05.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:05.974 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:05.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:06.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:05 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:00:06.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:05 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:00:06.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:05 smithi081 ceph-mon[39192]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T19:00:06.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:05 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:00:06.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:05 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:00:06.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:05 smithi081 ceph-mon[29384]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T19:00:06.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:06.112 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:06.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:06.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:05 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:00:06.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:05 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:00:06.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:05 smithi104 ceph-mon[32438]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T19:00:06.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:06.255 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:06.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:06.309 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:00:06.309 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:00:06.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.309 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:113823 8.d 8:ba9b711f:::benchmark_data_smithi081_105538_object113822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T19:00:06.309 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:06.296697+0000", 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0067449679999999996, 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067763950000000002, 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 113823 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:06.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296697+0000", 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296697+0000", 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296699+0000", 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.328e-06 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:06.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296735+0000", 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6443999999999999e-05 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296748+0000", 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2962999999999999e-05 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296756+0000", 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5560000000000002e-06 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:06.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296806+0000", 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9706000000000001e-05 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.296901+0000", 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5830999999999998e-05 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,7", 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.297015+0000", 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113325 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.299871+0000", 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028562700000000002 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.300601+0000", 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072956799999999999 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:113830 8.d 8:b7104d25:::benchmark_data_smithi081_105538_object113829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T19:00:06.314 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:06.301553+0000", 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0018884290000000001, 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020075700000000002, 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 113830 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301553+0000", 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301553+0000", 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301555+0000", 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1119999999999999e-06 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301587+0000", 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1946000000000002e-05 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301602+0000", 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5162e-05 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301609+0000", 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1160000000000001e-06 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301658+0000", 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8204000000000001e-05 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301785+0000", 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127377 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,7", 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.301923+0000", 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013747600000000001 2024-09-21T19:00:06.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:113834 8.d 8.e0c666ed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:06.302431+0000", 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0010107009999999999, 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001186372, 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:00:06.319 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 113834 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302431+0000", 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302431+0000", 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302432+0000", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.265e-06 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302487+0000", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4786e-05 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302496+0000", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9020000000000005e-06 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302503+0000", 2024-09-21T19:00:06.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7170000000000002e-06 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.35258.0:113824 8.0 e201/172 8:0069da9d:::benchmark_data_smithi081_105538_object113823:head v 201'7045, mlcod=201'7043)", 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:06.302461+0000", 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0009803189999999999, 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011995969999999999, 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302461+0000", 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302461+0000", 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302463+0000", 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.891e-06 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302493+0000", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0052999999999998e-05 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302504+0000", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0407999999999999e-05 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302510+0000", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3500000000000002e-06 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:06.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302576+0000", 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5845000000000004e-05 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302593+0000", 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7621e-05 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.35258.0:113826 8.0 e201/172)", 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:06.302785+0000", 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00065689899999999996, 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091879099999999997, 2024-09-21T19:00:06.325 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302785+0000", 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302785+0000", 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302786+0000", 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4499999999999995e-07 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302845+0000", 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9490000000000001e-05 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302852+0000", 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2649999999999999e-06 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:06.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:06.302858+0000", 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4990000000000002e-06 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 5 2024-09-21T19:00:06.328 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:00:06.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:06.392 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:06.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:06.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:06.528 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:06.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:06.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:06.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:06.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:06.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T19:00:06.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:06.803 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 74 16 114440 114424 96.6139 72.25 0.0209383 0.0103459 2024-09-21T19:00:06.834 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:06.850 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:06.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:06.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:06.974 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:06 smithi081 ceph-mon[29384]: pgmap v326: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 856 B/s rd, 58 MiB/s wr, 936 op/s 2024-09-21T19:00:06.975 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:06.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:00:07.000 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:07.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.137 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:07.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:07.166 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:07.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:06 smithi104 ceph-mon[32438]: pgmap v326: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 856 B/s rd, 58 MiB/s wr, 936 op/s 2024-09-21T19:00:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:07.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:07.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:00:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:00:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:00:07.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:06 smithi081 ceph-mon[39192]: pgmap v326: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 856 B/s rd, 58 MiB/s wr, 936 op/s 2024-09-21T19:00:07.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:07.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:07.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:07.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:07.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:07.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:07.803 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 75 16 115544 115528 96.2448 69 0.00970251 0.0103853 2024-09-21T19:00:07.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:07.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:07.983 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:00:07.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:07.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:08.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:08.093 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:08.143 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:08.144 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:08.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1271461098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:08.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:07 smithi081 ceph-mon[29384]: from='client.35405 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:08.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1271461098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:08.144 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:07 smithi081 ceph-mon[39192]: from='client.35405 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:08.178 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:08.179 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:08.216 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:08.216 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:08.225 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:08.248 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:08.248 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:08.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:08.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:00:07 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:00:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1271461098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:07 smithi104 ceph-mon[32438]: from='client.35405 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:08.267 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:08.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:08.361 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:08.395 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:08.398 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:08.398 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:08.426 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:08.427 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:08.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:08.553 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:08.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.692 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:08.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:08.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:08.804 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 76 16 116600 116584 95.8465 66 0.00825959 0.010429 2024-09-21T19:00:08.830 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:08.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:08.963 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:08.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:09.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.100 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:09.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:09.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:08 smithi104 ceph-mon[32438]: pgmap v327: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T19:00:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:09 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:09 smithi104 ceph-mon[32438]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:09 smithi104 ceph-mon[32438]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T19:00:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2819364343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[39192]: pgmap v327: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[39192]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[39192]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2819364343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[29384]: pgmap v327: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[29384]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[29384]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T19:00:09.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2819364343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:09.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:09.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:09.489 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:09.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:09.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:09.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:09.750 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:09.804 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 77 15 117547 117532 95.3712 59.25 0.01242 0.0104812 2024-09-21T19:00:09.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:09.888 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:09.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:09.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:10.024 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:10.102 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:10.103 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:10.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:10.155 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:10.257 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:10.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:10.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:10.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:10.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:10.568 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:10.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.718 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:10.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:10.804 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 78 16 118794 118778 95.1469 77.875 0.00928493 0.0105056 2024-09-21T19:00:10.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:00:10.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:10.850 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:10.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:10.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:10.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:11.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:11.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:11.205 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:11.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:10 smithi104 ceph-mon[32438]: pgmap v329: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 963 op/s 2024-09-21T19:00:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:11.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:10 smithi081 ceph-mon[29384]: pgmap v329: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 963 op/s 2024-09-21T19:00:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:10 smithi081 ceph-mon[39192]: pgmap v329: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 963 op/s 2024-09-21T19:00:11.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:11.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:11.543 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:11.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:11.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:11.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:11.799 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:11.799 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:11.804 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 79 16 119992 119976 94.8903 74.875 0.00945187 0.010534 2024-09-21T19:00:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:11.922 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:11.958 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:11.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:12.005 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T19:00:12.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:12.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:12.227 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:12.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1929064946' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:00:12.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:12 smithi081 ceph-mon[29384]: from='client.25339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:00:12.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1929064946' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:00:12.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:12 smithi081 ceph-mon[39192]: from='client.25339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:00:12.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:12.365 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:12.382 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1929064946' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:00:12.382 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:12 smithi104 ceph-mon[32438]: from='client.25339 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:00:12.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:12.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:12.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:00:12.603 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:12.637 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:12.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:00:12.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:12.804 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:00:12.803450+0000 min lat: 0.00110906 max lat: 0.19882 avg lat: 0.010556 2024-09-21T19:00:12.805 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:00:12.805 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 80 16 121256 121240 94.6913 79 0.00474621 0.010556 2024-09-21T19:00:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:12.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:12.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T19:00:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:12.961 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:12.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:13.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:13 smithi081 ceph-mon[29384]: pgmap v330: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T19:00:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:13 smithi081 ceph-mon[29384]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:00:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:13 smithi081 ceph-mon[29384]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T19:00:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:13 smithi081 ceph-mon[39192]: pgmap v330: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T19:00:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:13 smithi081 ceph-mon[39192]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:00:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:13 smithi081 ceph-mon[39192]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T19:00:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:13.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:13.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:13.208 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:13.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:13.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:13 smithi104 ceph-mon[32438]: pgmap v330: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T19:00:13.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:13 smithi104 ceph-mon[32438]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:00:13.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:13 smithi104 ceph-mon[32438]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T19:00:13.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:13.375 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:13.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.518 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:13.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:13.529 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:13.529 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:13.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:13.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:00:13.659 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:13.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:13.787 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:13.806 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 81 16 122329 122313 94.3488 67.0625 0.00747234 0.0105936 2024-09-21T19:00:13.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:13.927 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:13.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:14.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:14.057 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:14.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:14.190 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:14.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:14.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:14.451 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:14.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:00:14.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:14.608 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:14.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:14.715 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:14.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:14.806 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 82 16 123273 123257 93.9177 59 0.0181839 0.0106431 2024-09-21T19:00:14.846 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:14.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:14.908 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:14.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:15.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:15.017 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T19:00:15.024 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:00:15.035 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:15.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:15.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: pgmap v332: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/880279835' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='client.25345 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:00:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3746674051' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:00:15.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:15.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: pgmap v332: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/880279835' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='client.25345 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:00:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3746674051' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: pgmap v332: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/880279835' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='client.25345 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:00:15.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3746674051' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:00:15.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:15.450 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:15.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:15.645 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:15.645 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:15.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.722 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:15.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:15.806 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 83 15 124386 124371 93.625 69.625 0.00623937 0.0106764 2024-09-21T19:00:15.823 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.859 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:15.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:15.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:15.998 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:15.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:16.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:16 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:16.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:16.242 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:16.242 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[39192]: from='client.25345 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:00:16.242 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3746674051' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:00:16.242 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[39192]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T19:00:16.242 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T19:00:16.242 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T19:00:16.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:16.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:16.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[29384]: from='client.25345 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:00:16.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3746674051' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:00:16.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[29384]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T19:00:16.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T19:00:16.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:16 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T19:00:16.277 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:16 smithi104 ceph-mon[32438]: from='client.25345 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:00:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3746674051' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:00:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:16 smithi104 ceph-mon[32438]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T19:00:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:16 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T19:00:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:16 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T19:00:16.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:16.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:00:16.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.418 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:16.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:16.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:16.550 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:16.652 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:16.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:16.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:16.806 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 84 16 125548 125532 93.3742 72.5625 0.0084627 0.0107052 2024-09-21T19:00:16.815 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:16.882 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:16.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:16.962 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:16.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:17.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:17.071 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:17.072 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:17.073 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:17.073 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:17.073 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:17.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:17.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:17.230 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:17.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:17 smithi104 ceph-mon[32438]: pgmap v334: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 678 B/s rd, 46 MiB/s wr, 736 op/s 2024-09-21T19:00:17.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:17 smithi104 ceph-mon[32438]: 8.d scrub starts 2024-09-21T19:00:17.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3319384476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-mon[39192]: pgmap v334: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 678 B/s rd, 46 MiB/s wr, 736 op/s 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-mon[39192]: 8.d scrub starts 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3319384476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-mon[29384]: pgmap v334: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 678 B/s rd, 46 MiB/s wr, 736 op/s 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-mon[29384]: 8.d scrub starts 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3319384476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:17.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:00:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:00:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:00:17.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:17.362 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:17.433 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:00:17.433 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102355 7.4 e105/82 7:2a73ca6f:::benchmark_data_smithi081_73391_object102354:head v 105'6448, mlcod=105'6446)", 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.327165+0000", 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.10197786700002, 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16628193299999999, 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327165+0000", 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327165+0000", 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327166+0000", 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5990000000000001e-06 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327197+0000", 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1355e-05 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327209+0000", 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1109e-05 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327217+0000", 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1960000000000008e-06 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327305+0000", 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7937999999999997e-05 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.327323+0000", 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8034999999999999e-05 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493402+0000", 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.166079438 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493446+0000", 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4263e-05 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102359 7.c 7:3c684ab9:::benchmark_data_smithi081_73391_object102358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.329664+0000", 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.09947825299997, 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16395372499999999, 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.438 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102359 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329664+0000", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329664+0000", 2024-09-21T19:00:17.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329666+0000", 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.477e-06 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329703+0000", 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7696e-05 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329715+0000", 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1430000000000001e-05 2024-09-21T19:00:17.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329722+0000", 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4100000000000002e-06 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329782+0000", 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9558000000000001e-05 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329907+0000", 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554600000000001 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330061+0000", 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015371300000000001 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337366+0000", 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073049380000000004 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493489+0000", 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15612330799999999 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493524+0000", 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4567999999999998e-05 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493575+0000", 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1141999999999999e-05 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493618+0000", 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2938999999999997e-05 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102362 7.d 7:b4fc79ed:::benchmark_data_smithi081_73391_object102361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.330143+0000", 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.098999585, 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16063384, 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102362 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330143+0000", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330143+0000", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330144+0000", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.361e-06 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330179+0000", 2024-09-21T19:00:17.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4944000000000003e-05 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330189+0000", 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9639999999999998e-06 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330196+0000", 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5259999999999999e-06 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333790+0000", 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035948120000000002 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333921+0000", 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000131051 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.334318+0000", 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000396463 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337942+0000", 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003624132 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489748+0000", 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15180602900000001 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490678+0000", 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093001499999999996 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490730+0000", 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1604000000000003e-05 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.490777+0000", 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6938999999999999e-05 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102363 7.c 7:396ad605:::benchmark_data_smithi081_73391_object102362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.330208+0000", 2024-09-21T19:00:17.449 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.09893483000002, 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.165470859, 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102363 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330208+0000", 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3300000000000002e-07 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330233+0000", 2024-09-21T19:00:17.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5241000000000001e-05 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330239+0000", 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9950000000000002e-06 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330244+0000", 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8119999999999998e-06 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333287+0000", 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030428870000000002 2024-09-21T19:00:17.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333401+0000", 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113414 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333571+0000", 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017085599999999999 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.492806+0000", 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15923499299999999 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493628+0000", 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082127600000000002 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495595+0000", 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001967445 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495642+0000", 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7024000000000002e-05 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495678+0000", 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6282999999999998e-05 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102361 7.4 e105/82 7:2b82beac:::benchmark_data_smithi081_73391_object102360:head v 105'6449, mlcod=105'6446)", 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333737+0000", 2024-09-21T19:00:17.455 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.09540517599999, 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.162036767, 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333737+0000", 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333737+0000", 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333739+0000", 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.364e-06 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333760+0000", 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1755000000000001e-05 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333770+0000", 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9699999999999994e-06 2024-09-21T19:00:17.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333777+0000", 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3740000000000003e-06 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335178+0000", 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014012409999999999 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335197+0000", 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8695e-05 2024-09-21T19:00:17.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495739+0000", 2024-09-21T19:00:17.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16054214 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495774+0000", 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5228000000000002e-05 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102367 7.4 e105/82 7:2bc3d420:::benchmark_data_smithi081_73391_object102366:head v 105'6450, mlcod=105'6446)", 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333834+0000", 2024-09-21T19:00:17.460 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.09530809900002, 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16196540800000001, 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333834+0000", 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333834+0000", 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333835+0000", 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6399999999999998e-07 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333854+0000", 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9420999999999999e-05 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333859+0000", 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1490000000000002e-06 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333863+0000", 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.16e-06 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337397+0000", 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00353382 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337414+0000", 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7051e-05 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495777+0000", 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.158363108 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495800+0000", 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2334999999999999e-05 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.464 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102368 7.4 e105/82 7:29e74c0a:::benchmark_data_smithi081_73391_object102367:head v 105'6451, mlcod=105'6446)", 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.336323+0000", 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.09281976400001, 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15951227800000001, 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336323+0000", 2024-09-21T19:00:17.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336323+0000", 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336324+0000", 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3260000000000001e-06 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336346+0000", 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1977e-05 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336355+0000", 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3090000000000001e-06 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336361+0000", 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1689999999999997e-06 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.491302+0000", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.154940193 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.491320+0000", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8844000000000001e-05 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495810+0000", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004489537 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495835+0000", 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4922999999999998e-05 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102371 7.7 7:e4277e62:::benchmark_data_smithi081_73391_object102370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.337854+0000", 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.091288459, 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15779875900000001, 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.469 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102371 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337854+0000", 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337854+0000", 2024-09-21T19:00:17.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337855+0000", 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5239999999999999e-06 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337887+0000", 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1814999999999999e-05 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337895+0000", 2024-09-21T19:00:17.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6639999999999998e-06 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337900+0000", 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.48e-06 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338644+0000", 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074344799999999996 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338755+0000", 2024-09-21T19:00:17.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011141099999999999 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489524+0000", 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15076826500000001 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495470+0000", 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059459880000000001 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495536+0000", 2024-09-21T19:00:17.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6666000000000002e-05 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495551+0000", 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5211000000000001e-05 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495608+0000", 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6668e-05 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495653+0000", 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4619000000000001e-05 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102375 7.c 7:3f5265de:::benchmark_data_smithi081_73391_object102374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.338287+0000", 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.09085514200001, 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15767785000000001, 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.475 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102375 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338287+0000", 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338287+0000", 2024-09-21T19:00:17.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338288+0000", 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.029e-06 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338354+0000", 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5264000000000004e-05 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338362+0000", 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.524e-06 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338369+0000", 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.421e-06 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489653+0000", 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15128423099999999 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489791+0000", 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013800800000000001 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489995+0000", 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000203773 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495713+0000", 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057185710000000004 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495809+0000", 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5754000000000005e-05 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495889+0000", 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9679999999999996e-05 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495934+0000", 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5609000000000003e-05 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495965+0000", 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0985999999999998e-05 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop_reply(client.34742.0:102362 7.d e105/82 ondisk, result = 0)", 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.338914+0000", 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.090228282, 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15084166600000001, 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.481 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338914+0000", 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338914+0000", 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338915+0000", 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2810000000000001e-06 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338920+0000", 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3749999999999996e-06 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489604+0000", 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150684605 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489671+0000", 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6698000000000004e-05 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489735+0000", 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4158999999999993e-05 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489742+0000", 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2150000000000004e-06 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489756+0000", 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3332999999999999e-05 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:102370 7.4 e105/82 7:261d1db0:::benchmark_data_smithi081_73391_object102369:head v 105'6452, mlcod=105'6446)", 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.339637+0000", 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 297.08950585899998, 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15622318700000001, 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339637+0000", 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339637+0000", 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339638+0000", 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1790000000000001e-06 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.339663+0000", 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5381999999999999e-05 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489911+0000", 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15024772 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.489917+0000", 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1090000000000003e-06 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493494+0000", 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035770260000000001 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493514+0000", 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9647999999999999e-05 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495837+0000", 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002323215 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.495860+0000", 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2908000000000002e-05 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:136121 7.c 7:3d398460:::benchmark_data_smithi081_73391_object136120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:48.155866+0000", 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 269.273276151, 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13614505299999999, 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 136121 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155866+0000", 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155866+0000", 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155868+0000", 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9180000000000001e-06 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155929+0000", 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0381999999999999e-05 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155939+0000", 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0815999999999999e-05 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155946+0000", 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9920000000000001e-06 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.155991+0000", 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4379999999999999e-05 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.156091+0000", 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010033 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.156182+0000", 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1057999999999995e-05 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.158936+0000", 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027542299999999999 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.159467+0000", 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053071599999999998 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291914+0000", 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13244737400000001 2024-09-21T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.291965+0000", 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0878999999999999e-05 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:48.292011+0000", 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5977999999999999e-05 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242081 7.7 7:eb350bf9:::benchmark_data_smithi081_73391_object103082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.508138+0000", 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 147.921004864, 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.194433776, 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242081 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508138+0000", 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508138+0000", 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508140+0000", 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0360000000000001e-06 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508146+0000", 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9549999999999997e-06 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508155+0000", 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6719999999999999e-06 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508162+0000", 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7309999999999996e-06 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508367+0000", 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000205353 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508471+0000", 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103912 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.508686+0000", 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000215054 2024-09-21T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511410+0000", 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027232530000000001 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699220+0000", 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18781029699999999 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.702479+0000", 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032595760000000001 2024-09-21T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.702539+0000", 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9150000000000001e-05 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.702571+0000", 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2787000000000003e-05 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242087 7.d 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.512359+0000", 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 147.91678377599999, 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20359369999999999, 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242087 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512359+0000", 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512359+0000", 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512360+0000", 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.519e-06 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512365+0000", 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7809999999999996e-06 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512377+0000", 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2211e-05 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512384+0000", 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2699999999999999e-06 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512443+0000", 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8335000000000003e-05 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512575+0000", 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013186299999999999 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512712+0000", 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013723899999999999 2024-09-21T19:00:17.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516039+0000", 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033274899999999998 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.677347+0000", 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16130797199999999 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.678572+0000", 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00122426 2024-09-21T19:00:17.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.715855+0000", 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.037282933999999997 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.715891+0000", 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6803e-05 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.715952+0000", 2024-09-21T19:00:17.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1023000000000002e-05 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243316 7.d 7:bf0cd40c:::benchmark_data_smithi081_73391_object104317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.765579+0000", 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.663563404, 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99853839899999997, 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.508 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243316 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765579+0000", 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765579+0000", 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765581+0000", 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9300000000000002e-06 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765586+0000", 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6840000000000004e-06 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765595+0000", 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.798e-06 2024-09-21T19:00:17.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765602+0000", 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9380000000000003e-06 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765644+0000", 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1282000000000002e-05 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.765649+0000", 2024-09-21T19:00:17.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.857e-06 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.756878+0000", 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99122877300000001 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.757004+0000", 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012556700000000001 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.757126+0000", 2024-09-21T19:00:17.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012181 2024-09-21T19:00:17.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.763931+0000", 2024-09-21T19:00:17.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068057559999999996 2024-09-21T19:00:17.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:17.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.763974+0000", 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2883999999999998e-05 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764026+0000", 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1981000000000003e-05 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764078+0000", 2024-09-21T19:00:17.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2176000000000003e-05 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764117+0000", 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8962999999999999e-05 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243350 7.d 7:bfd5d960:::benchmark_data_smithi081_73391_object104351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.796299+0000", 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.63284345700001, 2024-09-21T19:00:17.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97577692699999996, 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243350 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796299+0000", 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796299+0000", 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796300+0000", 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1540000000000001e-06 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796304+0000", 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1960000000000001e-06 2024-09-21T19:00:17.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796313+0000", 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4079999999999998e-06 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796319+0000", 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4400000000000002e-06 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.796397+0000", 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7651999999999994e-05 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.797134+0000", 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073711200000000005 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.759680+0000", 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96254598499999999 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.759744+0000", 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3629999999999999e-05 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.759811+0000", 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7885999999999993e-05 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764195+0000", 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043838039999999998 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764234+0000", 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9187999999999998e-05 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.771939+0000", 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0077048699999999999 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772013+0000", 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4122999999999995e-05 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772076+0000", 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2478999999999996e-05 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243371 7.d 7:b3855816:::benchmark_data_smithi081_73391_object104372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.814706+0000", 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.614436345, 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95757182200000002, 2024-09-21T19:00:17.523 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243371 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814706+0000", 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814706+0000", 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814708+0000", 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8759999999999999e-06 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814714+0000", 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1199999999999999e-06 2024-09-21T19:00:17.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814725+0000", 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1362000000000001e-05 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814733+0000", 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6079999999999998e-06 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814787+0000", 2024-09-21T19:00:17.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4459999999999997e-05 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.814796+0000", 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6519999999999995e-06 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.761141+0000", 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94634492999999997 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.761197+0000", 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5989000000000002e-05 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.761256+0000", 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8699000000000002e-05 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764201+0000", 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029451989999999999 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767108+0000", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029069629999999998 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772190+0000", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050818169999999998 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772240+0000", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0204999999999998e-05 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772278+0000", 2024-09-21T19:00:17.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7941999999999997e-05 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243372 7.d 7:b59e4c53:::benchmark_data_smithi081_73391_object104373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.815401+0000", 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.613741707, 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.956966714, 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243372 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815401+0000", 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815401+0000", 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815402+0000", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.443e-06 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815407+0000", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4789999999999998e-06 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815421+0000", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3917000000000001e-05 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815428+0000", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1300000000000003e-06 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815476+0000", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7919e-05 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.815483+0000", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9519999999999997e-06 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.762409+0000", 2024-09-21T19:00:17.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94692642900000001 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.762512+0000", 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102755 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.762629+0000", 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117614 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765635+0000", 2024-09-21T19:00:17.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030059599999999998 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767146+0000", 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015110869999999999 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772321+0000", 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005174583 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772349+0000", 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7605000000000001e-05 2024-09-21T19:00:17.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772367+0000", 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8841000000000001e-05 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243449 7.d 7:b7cc7edf:::benchmark_data_smithi081_73391_object104450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.919192+0000", 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.50995062699999, 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.85324788200000001, 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.536 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243449 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919192+0000", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919192+0000", 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919194+0000", 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.075e-06 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919199+0000", 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.485e-06 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919212+0000", 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2564e-05 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919221+0000", 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1589999999999999e-06 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919289+0000", 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8225999999999994e-05 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.919296+0000", 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6039999999999998e-06 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764271+0000", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.84497510200000003 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764379+0000", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010764099999999999 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.764479+0000", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100608 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767124+0000", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002644286 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.769391+0000", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022677740000000002 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772399+0000", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030081679999999999 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772422+0000", 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2461000000000001e-05 2024-09-21T19:00:17.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772440+0000", 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7728999999999999e-05 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:243457 7.d 7:b5e36520:::benchmark_data_smithi081_73391_object104458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:51.942755+0000", 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.48638773799999, 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.82976072599999995, 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:17.542 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 243457 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942755+0000", 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942755+0000", 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942758+0000", 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4800000000000001e-06 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942769+0000", 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0363999999999999e-05 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942795+0000", 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6043000000000001e-05 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942815+0000", 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0806e-05 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942928+0000", 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000112965 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:51.942934+0000", 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2440000000000001e-06 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765669+0000", 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.822735624 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765735+0000", 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6048000000000002e-05 2024-09-21T19:00:17.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.765796+0000", 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0699000000000003e-05 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.767208+0000", 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001412127 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.769429+0000", 2024-09-21T19:00:17.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022209420000000001 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772472+0000", 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030429020000000001 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772497+0000", 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.533e-05 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:52.772515+0000", 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8151999999999998e-05 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.548 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.549 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.549 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:17.549 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:17.549 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:00:17.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:17.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:17.688 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:17.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:17.806 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 85 16 126567 126551 93.0247 63.6875 0.0196182 0.0107453 2024-09-21T19:00:17.828 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:17.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:17.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:17.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:18.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:18.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:18.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:18.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:18.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:18.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:18 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:00:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:18 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:00:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:18 smithi104 ceph-mon[32438]: 8.1 scrub starts 2024-09-21T19:00:18.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:00:17 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:00:18.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:18 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:00:18.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:18 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:00:18.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:18 smithi081 ceph-mon[29384]: 8.1 scrub starts 2024-09-21T19:00:18.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:18 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:00:18.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:18 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:00:18.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:18 smithi081 ceph-mon[39192]: 8.1 scrub starts 2024-09-21T19:00:18.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:18.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:18.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:18.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:00:18.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:18.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:18.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:18.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:18.671 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:18.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:18.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:18.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T19:00:18.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:18.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 86 16 127738 127722 92.794 73.1875 0.00871454 0.0107719 2024-09-21T19:00:18.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:18.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:18.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:19.019 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:19.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:19.165 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:19 smithi104 ceph-mon[32438]: pgmap v335: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-21T19:00:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:19 smithi104 ceph-mon[32438]: 3.18 scrub starts 2024-09-21T19:00:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:19 smithi104 ceph-mon[32438]: 3.18 scrub ok 2024-09-21T19:00:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:19 smithi104 ceph-mon[32438]: 8.d scrub ok 2024-09-21T19:00:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:19 smithi104 ceph-mon[32438]: 8.4 scrub starts 2024-09-21T19:00:19.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:19.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:19.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.35258.0:128431 8.6 e204/172 8:69e1cc81:::benchmark_data_smithi081_105538_object128430:head v 204'7978, mlcod=204'7974)", 2024-09-21T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:19.464544+0000", 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0074796109999999997, 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075080839999999999, 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.464544+0000", 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.464544+0000", 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.464545+0000", 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2279999999999999e-06 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.464580+0000", 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4980000000000001e-05 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.464592+0000", 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1980999999999999e-05 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.464598+0000", 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8619999999999999e-06 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.466557+0000", 2024-09-21T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019586400000000002 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.466574+0000", 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7586999999999999e-05 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.35258.0:128438 8.6 e204/172 8:6cde8778:::benchmark_data_smithi081_105538_object128437:head v 204'7979, mlcod=204'7974)", 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:19.468355+0000", 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0036687209999999998, 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037505329999999999, 2024-09-21T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.468355+0000", 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.468355+0000", 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:00:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.468357+0000", 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.53e-06 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.468411+0000", 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4147999999999997e-05 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.468425+0000", 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4051e-05 2024-09-21T19:00:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.468432+0000", 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6109999999999996e-06 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.469965+0000", 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015329950000000001 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:19.470008+0000", 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2403e-05 2024-09-21T19:00:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:00:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:00:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:00:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 2 2024-09-21T19:00:19.482 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:00:19.541 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[39192]: pgmap v335: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-21T19:00:19.541 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[39192]: 3.18 scrub starts 2024-09-21T19:00:19.541 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[39192]: 3.18 scrub ok 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[39192]: 8.d scrub ok 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[39192]: 8.4 scrub starts 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[29384]: pgmap v335: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[29384]: 3.18 scrub starts 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[29384]: 3.18 scrub ok 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[29384]: 8.d scrub ok 2024-09-21T19:00:19.542 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:19 smithi081 ceph-mon[29384]: 8.4 scrub starts 2024-09-21T19:00:19.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.589 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:19.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:19.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.725 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:19.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:19.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 87 16 128843 128827 92.5213 69.0625 0.0131904 0.0108039 2024-09-21T19:00:19.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:19.867 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:19.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:19.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:20.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:20.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:20.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:20.235 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:20.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:00:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:20.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:20.409 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:20 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T19:00:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:20 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T19:00:20.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:20.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:20 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T19:00:20.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:20 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T19:00:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:20 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T19:00:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:20 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T19:00:20.577 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:20.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:20.685 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:20.728 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:20.730 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:20.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 88 16 130038 130022 92.3186 74.6875 0.00981051 0.0108275 2024-09-21T19:00:20.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:20.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:20.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:21.004 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:21.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:21.122 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[29384]: pgmap v336: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-21T19:00:21.122 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[29384]: 8.1 scrub ok 2024-09-21T19:00:21.122 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[29384]: 5.14 scrub starts 2024-09-21T19:00:21.122 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[29384]: 5.14 scrub ok 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[39192]: pgmap v336: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[39192]: 8.1 scrub ok 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[39192]: 5.14 scrub starts 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[39192]: 5.14 scrub ok 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T19:00:21.123 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:21 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T19:00:21.136 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:21.221 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:00:21.221 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:00:21.221 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:00:21.221 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:00:21.221 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:00:21.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:00:21.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:21.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:21.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:21.421 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:21 smithi104 ceph-mon[32438]: pgmap v336: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-21T19:00:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:21 smithi104 ceph-mon[32438]: 8.1 scrub ok 2024-09-21T19:00:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:21 smithi104 ceph-mon[32438]: 5.14 scrub starts 2024-09-21T19:00:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:21 smithi104 ceph-mon[32438]: 5.14 scrub ok 2024-09-21T19:00:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:21 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T19:00:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:21 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T19:00:21.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:21.548 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:21.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:21.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:21.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 89 16 131093 131077 92.0221 65.9375 0.0261626 0.0108624 2024-09-21T19:00:21.824 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:21.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:00:21.926 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:21.960 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:21.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:22.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:22.097 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T19:00:22.100 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:22.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:22.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:22.287 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:00:22.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:22.381 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:22.381 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T19:00:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T19:00:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: 8.4 scrub ok 2024-09-21T19:00:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: pgmap v337: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 722 op/s 2024-09-21T19:00:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4952364' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[29384]: from='client.25357 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: 8.4 scrub ok 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: pgmap v337: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 722 op/s 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4952364' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:00:22.382 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:22 smithi081 ceph-mon[39192]: from='client.25357 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:00:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.383 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:22.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:22.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T19:00:22.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T19:00:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: 8.4 scrub ok 2024-09-21T19:00:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: pgmap v337: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 722 op/s 2024-09-21T19:00:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4952364' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:00:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:22 smithi104 ceph-mon[32438]: from='client.25357 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:00:22.419 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:22.419 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:22.458 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:22.484 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:22.497 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:22.497 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:22.515 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:22.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:22.530 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:22.569 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:22.569 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:22.599 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:22.599 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:22.616 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:22.640 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.650 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:22.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:22.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:00:22.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:22.788 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total time run: 90.0041 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total writes made: 132229 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Write size: 65536 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object size: 65536 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Bandwidth (MB/sec): 91.8215 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Bandwidth: 33.7152 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max bandwidth (MB/sec): 213.688 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min bandwidth (MB/sec): 59 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average IOPS: 1469 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev IOPS: 539.443 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max IOPS: 3419 2024-09-21T19:00:22.807 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min IOPS: 944 2024-09-21T19:00:22.808 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average Latency(s): 0.0108869 2024-09-21T19:00:22.808 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Latency(s): 0.00860533 2024-09-21T19:00:22.808 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max latency(s): 0.19882 2024-09-21T19:00:22.808 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min latency(s): 0.00110906 2024-09-21T19:00:22.810 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T19:00:22.888 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:22.888 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:22.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:22.923 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:22.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:23.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:23.054 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:23.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:23.189 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:00:23.189 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T19:00:23.189 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T19:00:23.189 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T19:00:23.189 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2584353671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:23.189 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2584353671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:00:23.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:23 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:00:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.197 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:23.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:23.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T19:00:23.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.337 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:23.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:23.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:23.467 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:00:23.467 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T19:00:23.467 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T19:00:23.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T19:00:23.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2584353671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:23.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:00:23.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:23 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:00:23.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:23.596 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:23.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:00:23.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:23.734 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:23.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:23.873 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:23.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:23.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:24.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:24.098 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:24.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:24.152 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:24.157 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.159 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:24.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 5.0 scrub starts 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 5.0 scrub ok 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 4.a scrub starts 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: 4.a scrub ok 2024-09-21T19:00:24.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[39192]: pgmap v339: 158 pgs: 158 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 5.0 scrub starts 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 5.0 scrub ok 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 4.a scrub starts 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: 4.a scrub ok 2024-09-21T19:00:24.260 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:24 smithi081 ceph-mon[29384]: pgmap v339: 158 pgs: 158 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T19:00:24.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:24.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:24.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:24.424 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 5.0 scrub starts 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 5.0 scrub ok 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 4.a scrub starts 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: 4.a scrub ok 2024-09-21T19:00:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:24 smithi104 ceph-mon[32438]: pgmap v339: 158 pgs: 158 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T19:00:24.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:24.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:24.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:24.693 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:24.795 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:24.822 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:24.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:24.952 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:24.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:25.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:25.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:25.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:25.235 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:25.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:25.380 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:25.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:25.509 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T19:00:25.586 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:00:25.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:25.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:25.690 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:25.690 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:25.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:25.767 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[39192]: 6.1 scrub starts 2024-09-21T19:00:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[39192]: 6.1 scrub ok 2024-09-21T19:00:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1404140194' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[39192]: from='client.35447 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3293014374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:00:25.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[39192]: from='client.25369 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:00:25.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[29384]: 6.1 scrub starts 2024-09-21T19:00:25.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[29384]: 6.1 scrub ok 2024-09-21T19:00:25.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1404140194' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:25.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[29384]: from='client.35447 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:25.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3293014374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:00:25.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:25 smithi081 ceph-mon[29384]: from='client.25369 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:00:25.869 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:25.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:26.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:26.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:25 smithi104 ceph-mon[32438]: 6.1 scrub starts 2024-09-21T19:00:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:25 smithi104 ceph-mon[32438]: 6.1 scrub ok 2024-09-21T19:00:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1404140194' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:25 smithi104 ceph-mon[32438]: from='client.35447 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3293014374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:00:26.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:25 smithi104 ceph-mon[32438]: from='client.25369 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:00:26.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:00:26.034 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:26.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:26.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:26.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:26.316 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:26.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:26.450 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:26.551 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:26.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:26.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: 5.5 scrub starts 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: 5.5 scrub ok 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: pgmap v340: 158 pgs: 158 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: from='client.25369 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[39192]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: 5.5 scrub starts 2024-09-21T19:00:26.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: 5.5 scrub ok 2024-09-21T19:00:26.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: pgmap v340: 158 pgs: 158 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T19:00:26.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:00:26.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:00:26.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:26.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: from='client.25369 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:00:26.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:26 smithi081 ceph-mon[29384]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T19:00:26.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:26.848 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:26.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:26.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T19:00:26.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:26.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:27.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: 5.5 scrub starts 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: 5.5 scrub ok 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: pgmap v340: 158 pgs: 158 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: from='client.25369 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:00:27.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:26 smithi104 ceph-mon[32438]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T19:00:27.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.150 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:27.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:27.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:27.277 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:27.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:27.408 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:27.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:27.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:27.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:00:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:00:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:00:27.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:27.660 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:27.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:27.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:27.931 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:27.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:28.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:28.067 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:28.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:28.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:00:28.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:00:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:00:28.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:28.331 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:28.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:28.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:28.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:28.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:28.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:28.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:28.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:28.852 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:28.861 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:28.898 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:28.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:28.997 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[39192]: pgmap v342: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 52 MiB/s wr, 1.46k op/s 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[39192]: 3.1d scrub starts 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[39192]: 3.1d scrub ok 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[29384]: pgmap v342: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 52 MiB/s wr, 1.46k op/s 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[29384]: 3.1d scrub starts 2024-09-21T19:00:28.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:28 smithi081 ceph-mon[29384]: 3.1d scrub ok 2024-09-21T19:00:29.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:28 smithi104 ceph-mon[32438]: pgmap v342: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 52 MiB/s wr, 1.46k op/s 2024-09-21T19:00:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:28 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:00:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:28 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:00:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:28 smithi104 ceph-mon[32438]: 3.1d scrub starts 2024-09-21T19:00:29.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:28 smithi104 ceph-mon[32438]: 3.1d scrub ok 2024-09-21T19:00:29.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:29.047 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:29.149 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:29.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:29.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:29.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:29.588 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:29.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:29.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:29.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:29.860 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:29.962 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:29.994 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:29.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:30.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:30.130 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:30.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.253 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:30.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:30.257 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:30.257 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:30.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:30.393 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:30.494 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.521 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:30.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T19:00:30.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:30.651 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:30.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:30.786 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 4.14 scrub starts 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 4.14 scrub ok 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: pgmap v343: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 50 MiB/s wr, 1.42k op/s 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 3.11 scrub starts 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 3.11 scrub ok 2024-09-21T19:00:30.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2370900070' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 4.14 scrub starts 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 4.14 scrub ok 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: pgmap v343: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 50 MiB/s wr, 1.42k op/s 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 3.11 scrub starts 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 3.11 scrub ok 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2370900070' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T19:00:30.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:30 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T19:00:30.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:00:30.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:30.934 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:30.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:31.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 4.14 scrub starts 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 4.14 scrub ok 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: pgmap v343: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 50 MiB/s wr, 1.42k op/s 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 3.11 scrub starts 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 3.11 scrub ok 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2370900070' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T19:00:31.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:30 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T19:00:31.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.079 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:31.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:31.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:31.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:00:31.205 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:31.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.352 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:31.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:31.453 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:31.481 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:31.520 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:31.520 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:31.558 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:31.558 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:31.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:31.584 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:31.584 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:31.611 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:31.613 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:31.644 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:31.644 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:31.675 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:31.675 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:31.711 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:31.711 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:31.711 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:31 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2370900070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:00:31.711 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:31 smithi081 ceph-mon[39192]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T19:00:31.711 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:31 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1470275518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:31.712 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:31 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2370900070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:00:31.712 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:31 smithi081 ceph-mon[29384]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T19:00:31.712 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:31 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1470275518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:31.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:31.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:31.748 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:31.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:31.880 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:31.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:31.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:32.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:31 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2370900070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:00:32.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:31 smithi104 ceph-mon[32438]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T19:00:32.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:31 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1470275518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:32.017 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:32.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.155 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:32.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:32.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:32.292 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:32.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:32.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:32.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:32.556 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:32.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:32.795 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:32.823 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:32.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:32.956 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:32.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:32.969 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:32 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T19:00:32.969 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:32 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T19:00:32.969 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:32 smithi081 ceph-mon[39192]: pgmap v345: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 8.7 MiB/s wr, 808 op/s 2024-09-21T19:00:32.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:32 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T19:00:32.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:32 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T19:00:32.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:32 smithi081 ceph-mon[29384]: pgmap v345: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 8.7 MiB/s wr, 808 op/s 2024-09-21T19:00:33.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:32 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T19:00:33.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:32 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T19:00:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:32 smithi104 ceph-mon[32438]: pgmap v345: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 767 B/s rd, 8.7 MiB/s wr, 808 op/s 2024-09-21T19:00:33.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T19:00:33.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.094 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:33.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:33.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:33.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:33.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:33.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:33.477 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:33.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:33.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:33.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:33.743 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[39192]: 6.17 scrub starts 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[39192]: 6.17 scrub ok 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[29384]: 6.17 scrub starts 2024-09-21T19:00:33.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:33 smithi081 ceph-mon[29384]: 6.17 scrub ok 2024-09-21T19:00:33.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:33.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:33.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:34.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:33 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T19:00:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:33 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T19:00:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:33 smithi104 ceph-mon[32438]: 6.17 scrub starts 2024-09-21T19:00:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:33 smithi104 ceph-mon[32438]: 6.17 scrub ok 2024-09-21T19:00:34.011 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:34.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:34.117 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:34.123 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:34.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:34.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:34.290 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:34.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:34.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:34.557 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:34.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:34.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:34.705 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:00:34.705 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:00:34.705 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: pgmap v346: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 8.7 MiB/s wr, 1.54k op/s 2024-09-21T19:00:34.705 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: 4.16 scrub starts 2024-09-21T19:00:34.705 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: 4.16 scrub ok 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: 5.2 scrub starts 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[39192]: 5.2 scrub ok 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: pgmap v346: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 8.7 MiB/s wr, 1.54k op/s 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: 4.16 scrub starts 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: 4.16 scrub ok 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: 5.2 scrub starts 2024-09-21T19:00:34.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:34 smithi081 ceph-mon[29384]: 5.2 scrub ok 2024-09-21T19:00:34.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:34.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:34.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:34.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201722 7.9 7:91094263:::benchmark_data_smithi081_73391_object62723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.568721+0000", 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 212.382454561, 2024-09-21T19:00:34.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0597567990000001, 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201722 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:34.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568721+0000", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568721+0000", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568724+0000", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5179999999999999e-06 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568729+0000", 2024-09-21T19:00:34.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9000000000000003e-06 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568744+0000", 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4598e-05 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568751+0000", 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1899999999999998e-06 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568806+0000", 2024-09-21T19:00:34.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4280000000000002e-05 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.568815+0000", 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2779999999999999e-06 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.604282+0000", 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0354674319999999 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.604409+0000", 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012624199999999999 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.604524+0000", 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115437 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608000+0000", 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003475625 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608063+0000", 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3607000000000003e-05 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628368+0000", 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020305230000000001 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628428+0000", 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9450000000000002e-05 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628478+0000", 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0012000000000002e-05 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:34.969 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201747 7.9 7:99156be5:::benchmark_data_smithi081_73391_object62748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.597481+0000", 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 212.353694434, 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0312291490000001, 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201747 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.970 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597481+0000", 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597481+0000", 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597483+0000", 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1169999999999998e-06 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597490+0000", 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0559999999999999e-06 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597502+0000", 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1697e-05 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597510+0000", 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7670000000000003e-06 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597570+0000", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0121999999999998e-05 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.597579+0000", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1819999999999995e-06 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608102+0000", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0105227489999999 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608217+0000", 2024-09-21T19:00:34.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114998 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.608327+0000", 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010976600000000001 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614839+0000", 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0065126089999999999 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614897+0000", 2024-09-21T19:00:34.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8202000000000003e-05 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628577+0000", 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013679358000000001 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628676+0000", 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8945999999999998e-05 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628710+0000", 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4579999999999998e-05 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.975 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201757 7.9 7:9e7f9360:::benchmark_data_smithi081_73391_object62758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.608364+0000", 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 212.34281181200001, 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0210702650000001, 2024-09-21T19:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201757 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:34.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608364+0000", 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608364+0000", 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608366+0000", 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0159999999999998e-06 2024-09-21T19:00:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608372+0000", 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0009999999999998e-06 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608385+0000", 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2704000000000001e-05 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608391+0000", 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9439999999999999e-06 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608495+0000", 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103872 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608504+0000", 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7779999999999996e-06 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.611920+0000", 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0034158049999999 2024-09-21T19:00:34.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.612052+0000", 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013184199999999999 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.612164+0000", 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011197900000000001 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614934+0000", 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027704489999999999 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.617594+0000", 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002659809 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.628761+0000", 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011167362 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629387+0000", 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062587899999999998 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629434+0000", 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6825e-05 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:201759 7.9 7:99785370:::benchmark_data_smithi081_73391_object62760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:02.608408+0000", 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 212.34276771699999, 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0211269030000001, 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:34.983 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 201759 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608408+0000", 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608408+0000", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608408+0000", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8299999999999997e-07 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608413+0000", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2089999999999999e-06 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:34.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608419+0000", 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8839999999999999e-06 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608424+0000", 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1850000000000003e-06 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608526+0000", 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102303 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:34.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:02.608532+0000", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5799999999999999e-06 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.614965+0000", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.006433557 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.615116+0000", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015114699999999999 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.615181+0000", 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4858999999999997e-05 2024-09-21T19:00:34.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.620299+0000", 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051176720000000002 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.622872+0000", 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025732839999999999 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629444+0000", 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0065717370000000002 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629495+0000", 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1044000000000001e-05 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:03.629535+0000", 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9858999999999999e-05 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216350 7.6 7:693ca6d8:::benchmark_data_smithi081_73391_object77351:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:00:34.989 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.433013348, 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2616681970000001, 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216350 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.990 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518164+0000", 2024-09-21T19:00:34.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.593e-06 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518169+0000", 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6689999999999997e-06 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518176+0000", 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8180000000000004e-06 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518182+0000", 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2549999999999997e-06 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518270+0000", 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8738000000000003e-05 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518279+0000", 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2660000000000001e-06 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779775+0000", 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.26149613 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779830+0000", 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5727999999999999e-05 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216360 7.6 7:6a035ea2:::benchmark_data_smithi081_73391_object77361:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.43298729399999, 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2839865960000001, 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:00:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216360 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518189+0000", 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5400000000000002e-07 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518191+0000", 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518196+0000", 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.622e-06 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518200+0000", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6190000000000002e-06 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518299+0000", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8587999999999999e-05 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518304+0000", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6210000000000001e-06 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780077+0000", 2024-09-21T19:00:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2617725909999999 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780187+0000", 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110469 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802102+0000", 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021914330999999999 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:34.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802175+0000", 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3116999999999998e-05 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216394 7.6 7:6d24eb15:::benchmark_data_smithi081_73391_object77395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.39216328800001, 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2297531310000001, 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216394 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559014+0000", 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0889999999999998e-06 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559020+0000", 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1669999999999998e-06 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559034+0000", 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4385e-05 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559041+0000", 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.238e-06 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570552+0000", 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011510347000000001 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570561+0000", 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6209999999999999e-06 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780222+0000", 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2096608149999999 2024-09-21T19:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780290+0000", 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7988e-05 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780361+0000", 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1239e-05 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788612+0000", 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082509909999999992 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788622+0000", 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159999999999999e-05 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788671+0000", 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8939e-05 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788709+0000", 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7295999999999997e-05 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788765+0000", 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6855999999999999e-05 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216429 7.6 7:6000252a:::benchmark_data_smithi081_73391_object77430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.33469962500001, 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1724721330000001, 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.006 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216429 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616478+0000", 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616484+0000", 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616495+0000", 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1134999999999999e-05 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616502+0000", 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8569999999999997e-06 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616558+0000", 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6023999999999998e-05 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616567+0000", 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8980000000000002e-06 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782338+0000", 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.165771565 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782505+0000", 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166558 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782607+0000", 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102357 2024-09-21T19:00:35.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788626+0000", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006018921 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788801+0000", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000174852 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788853+0000", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2006000000000003e-05 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788898+0000", 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4962000000000001e-05 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788948+0000", 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0006999999999997e-05 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216455 7.6 7:68c88556:::benchmark_data_smithi081_73391_object77456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.30620084899999, 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.144373474, 2024-09-21T19:00:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216455 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:00:35.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644977+0000", 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5500000000000001e-06 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644983+0000", 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0360000000000003e-06 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644995+0000", 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2006e-05 2024-09-21T19:00:35.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645003+0000", 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9459999999999998e-06 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645059+0000", 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5337999999999998e-05 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645068+0000", 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4320000000000005e-06 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.784935+0000", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1398670499999999 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785006+0000", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0840999999999995e-05 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785084+0000", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8228999999999999e-05 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788630+0000", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035455730000000002 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789144+0000", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00051452099999999999 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789199+0000", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4484000000000002e-05 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789251+0000", 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2633000000000002e-05 2024-09-21T19:00:35.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789348+0000", 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6835000000000004e-05 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216459 7.6 7:66d9de63:::benchmark_data_smithi081_73391_object77460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.30161204800001, 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1586775789999999, 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.018 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216459 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649566+0000", 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0540000000000002e-06 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649586+0000", 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0472e-05 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649616+0000", 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9848e-05 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649637+0000", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1114000000000001e-05 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649894+0000", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025646600000000001 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649902+0000", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4589999999999998e-06 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789632+0000", 2024-09-21T19:00:35.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1397303350000001 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789788+0000", 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155852 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789900+0000", 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111909 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802209+0000", 2024-09-21T19:00:35.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012309334 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802251+0000", 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1421000000000002e-05 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808109+0000", 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058583660000000003 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808145+0000", 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5374999999999999e-05 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808241+0000", 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6574000000000001e-05 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.024 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216478 7.6 7:69b4062c:::benchmark_data_smithi081_73391_object77479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.277225442, 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.137844316, 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216478 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:00:35.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673953+0000", 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3910000000000001e-06 2024-09-21T19:00:35.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.027 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:35.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674028+0000", 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5197000000000005e-05 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674184+0000", 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156129 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674193+0000", 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0410000000000003e-06 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674323+0000", 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013050400000000001 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674368+0000", 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4211000000000001e-05 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792228+0000", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.117859843 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792333+0000", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105216 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792492+0000", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015956400000000001 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802291+0000", 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097984360000000006 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802325+0000", 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4235e-05 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811655+0000", 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093300610000000006 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811704+0000", 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8520999999999998e-05 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811795+0000", 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0966999999999998e-05 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216483 7.6 7:679f3a63:::benchmark_data_smithi081_73391_object77484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.27069749899999, 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1314383640000001, 2024-09-21T19:00:35.032 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216483 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680480+0000", 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3010000000000001e-06 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680487+0000", 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8090000000000004e-06 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680532+0000", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4369999999999997e-05 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680540+0000", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5969999999999992e-06 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680600+0000", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0275000000000001e-05 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680610+0000", 2024-09-21T19:00:35.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6090000000000007e-06 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798740+0000", 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1181301029999999 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798848+0000", 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010757 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798947+0000", 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9349e-05 2024-09-21T19:00:35.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802432+0000", 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034852860000000002 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808361+0000", 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059282689999999999 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811802+0000", 2024-09-21T19:00:35.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003440869 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811840+0000", 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8197e-05 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811916+0000", 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6760000000000004e-05 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216512 7.6 7:6f163cff:::benchmark_data_smithi081_73391_object77513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.038 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.23635820699999, 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0972389849999999, 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216512 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714820+0000", 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2450000000000001e-06 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714826+0000", 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1859999999999997e-06 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714841+0000", 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5288999999999999e-05 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714850+0000", 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6540000000000005e-06 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714963+0000", 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112893 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714971+0000", 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7120000000000006e-06 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802465+0000", 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0874939400000001 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802564+0000", 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.904e-05 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802667+0000", 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103101 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808099+0000", 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054315889999999997 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808411+0000", 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031148299999999999 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811922+0000", 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035117199999999999 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811958+0000", 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5766999999999998e-05 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812056+0000", 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8366e-05 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216545 7.6 7:6b69332e:::benchmark_data_smithi081_73391_object77546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.19798669599999, 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0617420390000001, 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.045 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216545 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753191+0000", 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7680000000000001e-06 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753196+0000", 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1410000000000004e-06 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753207+0000", 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1143e-05 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753214+0000", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.917e-06 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754192+0000", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097769299999999996 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754200+0000", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9400000000000002e-06 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.805892+0000", 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0516927620000001 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806005+0000", 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113143 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806125+0000", 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119331 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808476+0000", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023510919999999999 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812065+0000", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035891429999999999 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814743+0000", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026784489999999998 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814839+0000", 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5358000000000004e-05 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814931+0000", 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2158999999999996e-05 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216552 7.6 7:63b154c8:::benchmark_data_smithi081_73391_object77553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.18913459000001, 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.060947284, 2024-09-21T19:00:35.051 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216552 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:00:35.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762043+0000", 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9889999999999999e-06 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762056+0000", 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3441999999999999e-05 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762068+0000", 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-05 2024-09-21T19:00:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762075+0000", 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1169999999999998e-06 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762135+0000", 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9759000000000002e-05 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762143+0000", 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:00:35.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808532+0000", 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0463883469999999 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808647+0000", 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011510600000000001 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808757+0000", 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109849 2024-09-21T19:00:35.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814943+0000", 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006186558 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814980+0000", 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6729e-05 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822841+0000", 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078607529999999998 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822895+0000", 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4282e-05 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822988+0000", 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3422000000000001e-05 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.057 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216555 7.6 7:6310baba:::benchmark_data_smithi081_73391_object77556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.18780875100001, 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0695222289999999, 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216555 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:00:35.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763369+0000", 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.635e-06 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763374+0000", 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.101e-06 2024-09-21T19:00:35.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763503+0000", 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000129276 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763511+0000", 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5019999999999992e-06 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763815+0000", 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030339199999999998 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763823+0000", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7600000000000002e-06 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812093+0000", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0482702930000001 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812199+0000", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010665100000000001 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812307+0000", 2024-09-21T19:00:35.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107206 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819747+0000", 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074399499999999999 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819790+0000", 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3297999999999998e-05 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832684+0000", 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012894012 2024-09-21T19:00:35.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832778+0000", 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4456999999999994e-05 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832889+0000", 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110696 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216569 7.6 7:67ad9aff:::benchmark_data_smithi081_73391_object77570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.798905+0000", 2024-09-21T19:00:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 195.15227071300001, 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.034449366, 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216569 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798905+0000", 2024-09-21T19:00:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:00:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:00:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: pgmap v346: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 8.7 MiB/s wr, 1.54k op/s 2024-09-21T19:00:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: 4.16 scrub starts 2024-09-21T19:00:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: 4.16 scrub ok 2024-09-21T19:00:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: 5.2 scrub starts 2024-09-21T19:00:35.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:34 smithi104 ceph-mon[32438]: 5.2 scrub ok 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798905+0000", 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798908+0000", 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.129e-06 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798917+0000", 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4929999999999998e-06 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798937+0000", 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0687e-05 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.798947+0000", 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4690000000000003e-06 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.799037+0000", 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0229999999999995e-05 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.799048+0000", 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0729e-05 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.815008+0000", 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159601570000001 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.815112+0000", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103707 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.815216+0000", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104474 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819803+0000", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045866630000000004 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.820017+0000", 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021458499999999999 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.833215+0000", 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013198052 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.833269+0000", 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3511000000000003e-05 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.833354+0000", 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5480000000000002e-05 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:243094 7.4 7:2cb70af2:::benchmark_data_smithi081_73391_object104095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.728724+0000", 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 164.222451772, 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0326880460000001, 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.071 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243094 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728724+0000", 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728724+0000", 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728726+0000", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.435e-06 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728733+0000", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6499999999999999e-06 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728749+0000", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5698000000000001e-05 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.728757+0000", 2024-09-21T19:00:35.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0639999999999994e-06 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.731046+0000", 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022893459999999998 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.731054+0000", 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0739999999999993e-06 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.756234+0000", 2024-09-21T19:00:35.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0251801709999999 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.756338+0000", 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103612 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.756429+0000", 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1538000000000003e-05 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761266+0000", 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048370410000000003 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761304+0000", 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7122000000000001e-05 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761337+0000", 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3207999999999998e-05 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761383+0000", 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6294000000000001e-05 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761412+0000", 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8793e-05 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:243118 7.4 7:2a6c29ae:::benchmark_data_smithi081_73391_object104119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.750809+0000", 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 164.200366815, 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.010814289, 2024-09-21T19:00:35.077 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243118 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750809+0000", 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.078 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750809+0000", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750811+0000", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3319999999999999e-06 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750817+0000", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9529999999999996e-06 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750829+0000", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1429e-05 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.750836+0000", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3329999999999999e-06 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.754026+0000", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003189983 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.754043+0000", 2024-09-21T19:00:35.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7442e-05 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.758596+0000", 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0045523590000001 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.758673+0000", 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7152999999999998e-05 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.758736+0000", 2024-09-21T19:00:35.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3042999999999999e-05 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761485+0000", 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027495419999999998 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761522+0000", 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7098999999999998e-05 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761555+0000", 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2138999999999998e-05 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761601+0000", 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6740999999999999e-05 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761623+0000", 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1741000000000001e-05 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.083 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:243313 7.f 7:f737484f:::benchmark_data_smithi081_73391_object104314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:51.761627+0000", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 163.18954849799999, 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0178787949999999, 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243313 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761627+0000", 2024-09-21T19:00:35.084 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761627+0000", 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761628+0000", 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1930000000000001e-06 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761633+0000", 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2549999999999999e-06 2024-09-21T19:00:35.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761642+0000", 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0620000000000007e-06 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761648+0000", 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1330000000000004e-06 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761690+0000", 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2558999999999997e-05 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:51.761697+0000", 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7209999999999997e-06 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.773488+0000", 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.01179102 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.773602+0000", 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114048 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.773709+0000", 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010694800000000001 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779289+0000", 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055803789999999999 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779302+0000", 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2704000000000001e-05 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779412+0000", 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109616 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779469+0000", 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7438000000000001e-05 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:52.779506+0000", 2024-09-21T19:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6718999999999999e-05 2024-09-21T19:00:35.090 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.090 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.090 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.090 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:35.090 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:35.090 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:00:35.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.166 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:35.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:35.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:35.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.438 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:35.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:35.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:35.562 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:35.632 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:00:35.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:35.693 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:35.795 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:35.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:35.963 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:35.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: 4.19 scrub starts 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: 4.19 scrub ok 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: 3.14 scrub starts 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: 3.14 scrub ok 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2325738722' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:36.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:35 smithi104 ceph-mon[32438]: from='client.35465 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:36.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: 4.19 scrub starts 2024-09-21T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: 4.19 scrub ok 2024-09-21T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: 3.14 scrub starts 2024-09-21T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: 3.14 scrub ok 2024-09-21T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2325738722' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[29384]: from='client.35465 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: 4.19 scrub starts 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: 4.19 scrub ok 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: 3.14 scrub starts 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: 3.14 scrub ok 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2325738722' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:36.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:35 smithi081 ceph-mon[39192]: from='client.35465 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:00:36.099 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:36.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:00:36.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:36.228 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:36.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:36.355 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:36.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:00:36.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:36.585 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.627 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:36.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:36.728 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: 6.b scrub starts 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: 6.b scrub ok 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: pgmap v347: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.9 MiB/s wr, 1.23k op/s 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: from='client.35465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: 6.b scrub starts 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: 6.b scrub ok 2024-09-21T19:00:36.730 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: pgmap v347: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.9 MiB/s wr, 1.23k op/s 2024-09-21T19:00:36.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:36.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: from='client.35465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:00:36.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T19:00:36.731 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:36.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:36.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:36.917 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:36.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: 6.b scrub starts 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: 6.b scrub ok 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: pgmap v347: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.9 MiB/s wr, 1.23k op/s 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: from='client.35465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T19:00:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:37.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:37.055 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:37.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:37.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:37.242 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:37.262 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:37.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:37.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:00:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:00:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:00:37.343 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:37.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:37.487 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:37.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.636 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:37.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:37.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:37.770 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:37.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:37.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:37.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:38.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:38.047 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:38.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:38.184 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:38.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:00:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:00:38.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:38.323 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:38.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:38.463 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:38.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:38.594 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:38.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:38.716 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[39192]: pgmap v349: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4174104160' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[39192]: from='client.35471 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[29384]: pgmap v349: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4174104160' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:00:38.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:38 smithi081 ceph-mon[29384]: from='client.35471 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:00:38.818 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:38.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:38.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:38.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:39.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:38 smithi104 ceph-mon[32438]: pgmap v349: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-21T19:00:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4174104160' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:00:39.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:38 smithi104 ceph-mon[32438]: from='client.35471 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:00:39.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:39.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:00:39.098 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:39.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:39.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:39.367 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:39.386 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T19:00:39.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.506 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:39.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:39.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:39.637 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:39.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:39.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:39.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:39.899 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:39.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:40.000 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:40.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:39 smithi104 ceph-mon[32438]: from='client.35471 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:00:40.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:39 smithi104 ceph-mon[32438]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T19:00:40.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:00:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:00:40.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[39192]: from='client.35471 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[39192]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[29384]: from='client.35471 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[29384]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:00:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:00:40.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:40.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:40.387 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:40.419 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:40.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.561 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:40.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:40.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.697 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:40.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:40.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:40.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:40.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:40 smithi104 ceph-mon[32438]: pgmap v351: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:00:40.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:40 smithi104 ceph-mon[32438]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T19:00:40.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:40.966 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:40.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:40 smithi081 ceph-mon[39192]: pgmap v351: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:00:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:40 smithi081 ceph-mon[39192]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T19:00:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:40 smithi081 ceph-mon[29384]: pgmap v351: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:00:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:40 smithi081 ceph-mon[29384]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T19:00:41.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.099 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:41.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:41.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:41.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:41.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.378 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:41.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:41.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:41.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:41.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.646 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:41.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:41.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.774 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:41.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:41.875 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:41.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:41.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:42.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:42.043 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:42.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:42.177 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:42.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:42.314 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:42.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:42.451 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:42.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:42.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:42.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:42.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:42.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:42 smithi081 ceph-mon[39192]: pgmap v353: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-21T19:00:42.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:42 smithi081 ceph-mon[29384]: pgmap v353: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-21T19:00:42.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:42.864 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:42.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:42 smithi104 ceph-mon[32438]: pgmap v353: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-21T19:00:42.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:42.992 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:42.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:43.094 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:43.118 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:43.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:43.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:43.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:43.278 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T19:00:43.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:00:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:43.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:43.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:43.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:43.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:43.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:43.559 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:43.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:43.584 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:43.584 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:43.618 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:43.618 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:43.658 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:43.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:43.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:43.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:43.693 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:43.694 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:43.727 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:43.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:43.758 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:43.758 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:43.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:43.791 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:43.792 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:43.792 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:43 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:00:43.792 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:43 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:00:43.792 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:43 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1703794261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:43.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:43 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:00:43.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:43 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:00:43.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:43 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1703794261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:43.816 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:43.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:43.832 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:43.896 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:44.049 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:44.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:44.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:43 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:00:44.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:43 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:00:44.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:43 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1703794261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:44.090 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:44.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:44.226 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:44.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:44.360 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:44.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:44.491 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:44.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:44.628 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:44.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:44.763 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:44.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:44.898 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:44.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:44.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:45.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:44 smithi104 ceph-mon[32438]: pgmap v354: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-21T19:00:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:44 smithi104 ceph-mon[32438]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T19:00:45.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:45.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:44 smithi081 ceph-mon[39192]: pgmap v354: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-21T19:00:45.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:44 smithi081 ceph-mon[39192]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T19:00:45.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:44 smithi081 ceph-mon[29384]: pgmap v354: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-21T19:00:45.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:44 smithi081 ceph-mon[29384]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T19:00:45.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:00:45.104 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:45.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:45.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:45.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:45.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:45.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.451 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:45.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:45.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:45.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:45.686 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:45.715 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:45.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:45.846 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:45.948 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:45.975 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:45.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:46.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:46.134 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:46.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:46.207 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:46.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:46.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:46.386 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:46.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:46.513 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:46.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:46.645 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:46.698 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:00:46.737 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:46 smithi081 ceph-mon[39192]: pgmap v356: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 845 op/s 2024-09-21T19:00:46.737 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3367761060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:46.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:46 smithi081 ceph-mon[29384]: pgmap v356: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 845 op/s 2024-09-21T19:00:46.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3367761060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:46.747 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:46.779 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:46.881 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:46.915 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:46.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:46 smithi104 ceph-mon[32438]: pgmap v356: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 845 op/s 2024-09-21T19:00:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3367761060' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:00:47.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:47.052 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:47.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:47.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:47.213 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:47.213 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:47.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:47.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:00:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:00:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:47.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:47.324 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:00:47.328 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:00:47.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:47.454 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:47.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:47.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:00:47.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:47.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:47.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:47.883 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:47.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:47.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:47.987 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:47 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:47.987 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3367761060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:47.987 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:47 smithi104 ceph-mon[32438]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T19:00:47.987 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:47 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:00:47.987 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:47 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:00:48.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3367761060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[29384]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T19:00:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:00:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:00:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3367761060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:00:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[39192]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T19:00:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:00:48.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:47 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:00:48.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:48.168 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:48.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:00:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:00:48.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:48.300 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:48.401 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:48.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:48.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:48.575 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:48.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:48.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:48.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:48.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:48.845 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:48.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:48.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:48.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:49.001 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[29384]: from='client.25402 v1:172.21.15.81:0/764567568' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:00:49.001 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[29384]: pgmap v358: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:00:49.001 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[29384]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T19:00:49.001 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:49.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[39192]: from='client.25402 v1:172.21.15.81:0/764567568' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:00:49.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[39192]: pgmap v358: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:00:49.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[39192]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T19:00:49.002 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:49.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:48 smithi104 ceph-mon[32438]: from='client.25402 v1:172.21.15.81:0/764567568' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:00:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:48 smithi104 ceph-mon[32438]: pgmap v358: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:00:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:48 smithi104 ceph-mon[32438]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T19:00:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:49.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:00:49.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:49.141 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:49.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:49.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:49.282 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:49.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:49.383 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:49.417 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:49.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:49.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:49.549 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:49.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:49.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:49.660 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-21T19:00:46.692773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6969825843,"num_objects":106542,"num_object_clones":0,"num_object_copies":319626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106542,"num_whiteouts":0,"num_read":1166,"num_read_kb":918,"num_write":158553,"num_write_kb":8462702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":6230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82465,"ondisk_log_size":82465,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22820316,"kb_used_data":20452184,"kb_used_omap":12,"kb_used_meta":2367859,"kb_avail":727140900,"statfs":{"total":767960285184,"available":744592281600,"internally_reserved":0,"allocated":20943036416,"data_stored":20939875437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":2424688180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,3,6,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-538312704,"num_objects":-8214,"num_object_clones":0,"num_object_copies":-24642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8214,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8214,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.154866"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":222,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.667048+0000","last_change":"2024-09-21T18:59:50.144510+0000","last_active":"2024-09-21T19:00:39.667048+0000","last_peered":"2024-09-21T19:00:39.667048+0000","last_clean":"2024-09-21T19:00:39.667048+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:00:39.667048+0000","last_undegraded":"2024-09-21T19:00:39.667048+0000","last_fullsized":"2024-09-21T19:00:39.667048+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.144434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.144434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:15.699799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":365,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687033+0000","last_change":"2024-09-21T18:58:35.867068+0000","last_active":"2024-09-21T19:00:43.687033+0000","last_peered":"2024-09-21T19:00:43.687033+0000","last_clean":"2024-09-21T19:00:43.687033+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T19:00:43.687033+0000","last_undegraded":"2024-09-21T19:00:43.687033+0000","last_fullsized":"2024-09-21T19:00:43.687033+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:35.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:35.867013+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.705270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":334,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.662919+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:00:39.662919+0000","last_peered":"2024-09-21T19:00:39.662919+0000","last_clean":"2024-09-21T19:00:39.662919+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:00:39.662919+0000","last_undegraded":"2024-09-21T19:00:39.662919+0000","last_fullsized":"2024-09-21T19:00:39.662919+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.393494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":316,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687793+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T19:00:43.687793+0000","last_peered":"2024-09-21T19:00:43.687793+0000","last_clean":"2024-09-21T19:00:43.687793+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:00:43.687793+0000","last_undegraded":"2024-09-21T19:00:43.687793+0000","last_fullsized":"2024-09-21T19:00:43.687793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.765211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":322,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.661032+0000","last_change":"2024-09-21T18:58:35.022059+0000","last_active":"2024-09-21T19:00:39.661032+0000","last_peered":"2024-09-21T19:00:39.661032+0000","last_clean":"2024-09-21T19:00:39.661032+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T19:00:39.661032+0000","last_undegraded":"2024-09-21T19:00:39.661032+0000","last_fullsized":"2024-09-21T19:00:39.661032+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:35.022013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:35.022013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.887695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":335,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.654089+0000","last_change":"2024-09-21T18:58:36.978527+0000","last_active":"2024-09-21T19:00:39.654089+0000","last_peered":"2024-09-21T19:00:39.654089+0000","last_clean":"2024-09-21T19:00:39.654089+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T19:00:39.654089+0000","last_undegraded":"2024-09-21T19:00:39.654089+0000","last_fullsized":"2024-09-21T19:00:39.654089+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:36.978471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:36.978471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.858664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":394,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.654484+0000","last_change":"2024-09-21T19:00:33.910621+0000","last_active":"2024-09-21T19:00:39.654484+0000","last_peered":"2024-09-21T19:00:39.654484+0000","last_clean":"2024-09-21T19:00:39.654484+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:00:39.654484+0000","last_undegraded":"2024-09-21T19:00:39.654484+0000","last_fullsized":"2024-09-21T19:00:39.654484+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:33.910535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.910535+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.545163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":328,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.686997+0000","last_change":"2024-09-21T18:59:48.790074+0000","last_active":"2024-09-21T19:00:43.686997+0000","last_peered":"2024-09-21T19:00:43.686997+0000","last_clean":"2024-09-21T19:00:43.686997+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T19:00:43.686997+0000","last_undegraded":"2024-09-21T19:00:43.686997+0000","last_fullsized":"2024-09-21T19:00:43.686997+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.789998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.789998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:00.025184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683958+0000","last_change":"2024-09-21T19:00:28.036188+0000","last_active":"2024-09-21T19:00:43.683958+0000","last_peered":"2024-09-21T19:00:43.683958+0000","last_clean":"2024-09-21T19:00:43.683958+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T19:00:43.683958+0000","last_undegraded":"2024-09-21T19:00:43.683958+0000","last_fullsized":"2024-09-21T19:00:43.683958+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:28.036123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.036123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.098067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":390,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688103+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:00:43.688103+0000","last_peered":"2024-09-21T19:00:43.688103+0000","last_clean":"2024-09-21T19:00:43.688103+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:00:43.688103+0000","last_undegraded":"2024-09-21T19:00:43.688103+0000","last_fullsized":"2024-09-21T19:00:43.688103+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:04.701591+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":350,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683312+0000","last_change":"2024-09-21T18:59:20.569305+0000","last_active":"2024-09-21T19:00:43.683312+0000","last_peered":"2024-09-21T19:00:43.683312+0000","last_clean":"2024-09-21T19:00:43.683312+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:00:43.683312+0000","last_undegraded":"2024-09-21T19:00:43.683312+0000","last_fullsized":"2024-09-21T19:00:43.683312+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:20.569235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.569235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.666145+0000","last_change":"2024-09-21T18:59:51.189725+0000","last_active":"2024-09-21T19:00:39.666145+0000","last_peered":"2024-09-21T19:00:39.666145+0000","last_clean":"2024-09-21T19:00:39.666145+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:00:39.666145+0000","last_undegraded":"2024-09-21T19:00:39.666145+0000","last_fullsized":"2024-09-21T19:00:39.666145+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.189659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.189659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.202774+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684707+0000","last_change":"2024-09-21T18:59:48.531625+0000","last_active":"2024-09-21T19:00:43.684707+0000","last_peered":"2024-09-21T19:00:43.684707+0000","last_clean":"2024-09-21T19:00:43.684707+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T19:00:43.684707+0000","last_undegraded":"2024-09-21T19:00:43.684707+0000","last_fullsized":"2024-09-21T19:00:43.684707+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.531568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.531568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.572428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":342,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687900+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:00:43.687900+0000","last_peered":"2024-09-21T19:00:43.687900+0000","last_clean":"2024-09-21T19:00:43.687900+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:00:43.687900+0000","last_undegraded":"2024-09-21T19:00:43.687900+0000","last_fullsized":"2024-09-21T19:00:43.687900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.764044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692489+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T19:00:43.692489+0000","last_peered":"2024-09-21T19:00:43.692489+0000","last_clean":"2024-09-21T19:00:43.692489+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:00:43.692489+0000","last_undegraded":"2024-09-21T19:00:43.692489+0000","last_fullsized":"2024-09-21T19:00:43.692489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.462353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":332,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684425+0000","last_change":"2024-09-21T18:59:57.720813+0000","last_active":"2024-09-21T19:00:43.684425+0000","last_peered":"2024-09-21T19:00:43.684425+0000","last_clean":"2024-09-21T19:00:43.684425+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T19:00:43.684425+0000","last_undegraded":"2024-09-21T19:00:43.684425+0000","last_fullsized":"2024-09-21T19:00:43.684425+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:57.720747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.720747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.304188+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688713+0000","last_change":"2024-09-21T18:58:32.943602+0000","last_active":"2024-09-21T19:00:43.688713+0000","last_peered":"2024-09-21T19:00:43.688713+0000","last_clean":"2024-09-21T19:00:43.688713+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T19:00:43.688713+0000","last_undegraded":"2024-09-21T19:00:43.688713+0000","last_fullsized":"2024-09-21T19:00:43.688713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:32.943549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:32.943549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:10.666685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":213,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.666906+0000","last_change":"2024-09-21T18:59:46.218922+0000","last_active":"2024-09-21T19:00:39.666906+0000","last_peered":"2024-09-21T19:00:39.666906+0000","last_clean":"2024-09-21T19:00:39.666906+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:00:39.666906+0000","last_undegraded":"2024-09-21T19:00:39.666906+0000","last_fullsized":"2024-09-21T19:00:39.666906+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.218852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.218852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.131850+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":363,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.659129+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:00:39.659129+0000","last_peered":"2024-09-21T19:00:39.659129+0000","last_clean":"2024-09-21T19:00:39.659129+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:00:39.659129+0000","last_undegraded":"2024-09-21T19:00:39.659129+0000","last_fullsized":"2024-09-21T19:00:39.659129+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.239922+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":348,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.665849+0000","last_change":"2024-09-21T19:00:15.241392+0000","last_active":"2024-09-21T19:00:39.665849+0000","last_peered":"2024-09-21T19:00:39.665849+0000","last_clean":"2024-09-21T19:00:39.665849+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:00:39.665849+0000","last_undegraded":"2024-09-21T19:00:39.665849+0000","last_fullsized":"2024-09-21T19:00:39.665849+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:15.241320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:15.241320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.208748+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":315,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.663428+0000","last_change":"2024-09-21T18:59:19.930405+0000","last_active":"2024-09-21T19:00:39.663428+0000","last_peered":"2024-09-21T19:00:39.663428+0000","last_clean":"2024-09-21T19:00:39.663428+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T19:00:39.663428+0000","last_undegraded":"2024-09-21T19:00:39.663428+0000","last_fullsized":"2024-09-21T19:00:39.663428+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688214+0000","last_change":"2024-09-21T19:00:00.649580+0000","last_active":"2024-09-21T19:00:43.688214+0000","last_peered":"2024-09-21T19:00:43.688214+0000","last_clean":"2024-09-21T19:00:43.688214+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:00:43.688214+0000","last_undegraded":"2024-09-21T19:00:43.688214+0000","last_fullsized":"2024-09-21T19:00:43.688214+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.649397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.649397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:24.015113+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":381,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653569+0000","last_change":"2024-09-21T19:00:18.857976+0000","last_active":"2024-09-21T19:00:39.653569+0000","last_peered":"2024-09-21T19:00:39.653569+0000","last_clean":"2024-09-21T19:00:39.653569+0000","last_became_active":"2024-09-21T18:58:23.993245+0000","last_became_peered":"2024-09-21T18:58:23.993245+0000","last_unstale":"2024-09-21T19:00:39.653569+0000","last_undegraded":"2024-09-21T19:00:39.653569+0000","last_fullsized":"2024-09-21T19:00:39.653569+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:00:18.857901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.857901+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.470039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":328,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688122+0000","last_change":"2024-09-21T18:58:28.912619+0000","last_active":"2024-09-21T19:00:43.688122+0000","last_peered":"2024-09-21T19:00:43.688122+0000","last_clean":"2024-09-21T19:00:43.688122+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:00:43.688122+0000","last_undegraded":"2024-09-21T19:00:43.688122+0000","last_fullsized":"2024-09-21T19:00:43.688122+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:28.912554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:28.912554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":332,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.693017+0000","last_change":"2024-09-21T18:58:38.110564+0000","last_active":"2024-09-21T19:00:43.693017+0000","last_peered":"2024-09-21T19:00:43.693017+0000","last_clean":"2024-09-21T19:00:43.693017+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:00:43.693017+0000","last_undegraded":"2024-09-21T19:00:43.693017+0000","last_fullsized":"2024-09-21T19:00:43.693017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:58:38.110509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:38.110509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.724266+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":346,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.662169+0000","last_change":"2024-09-21T19:00:23.223251+0000","last_active":"2024-09-21T19:00:39.662169+0000","last_peered":"2024-09-21T19:00:39.662169+0000","last_clean":"2024-09-21T19:00:39.662169+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:00:39.662169+0000","last_undegraded":"2024-09-21T19:00:39.662169+0000","last_fullsized":"2024-09-21T19:00:39.662169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:23.223202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.223202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:35.046166+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":234,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.669566+0000","last_change":"2024-09-21T19:00:22.256583+0000","last_active":"2024-09-21T19:00:39.669566+0000","last_peered":"2024-09-21T19:00:39.669566+0000","last_clean":"2024-09-21T19:00:39.669566+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:00:39.669566+0000","last_undegraded":"2024-09-21T19:00:39.669566+0000","last_fullsized":"2024-09-21T19:00:39.669566+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:22.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.256498+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.316225+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":438,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.689358+0000","last_change":"2024-09-21T19:00:35.363014+0000","last_active":"2024-09-21T19:00:43.689358+0000","last_peered":"2024-09-21T19:00:43.689358+0000","last_clean":"2024-09-21T19:00:43.689358+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T19:00:43.689358+0000","last_undegraded":"2024-09-21T19:00:43.689358+0000","last_fullsized":"2024-09-21T19:00:43.689358+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.428300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"210'9627","reported_seq":9714,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.950255+0000","last_change":"2024-09-21T18:58:44.098981+0000","last_active":"2024-09-21T19:00:41.950255+0000","last_peered":"2024-09-21T19:00:41.950255+0000","last_clean":"2024-09-21T19:00:41.950255+0000","last_became_active":"2024-09-21T18:58:44.098681+0000","last_became_peered":"2024-09-21T18:58:44.098681+0000","last_unstale":"2024-09-21T19:00:41.950255+0000","last_undegraded":"2024-09-21T19:00:41.950255+0000","last_fullsized":"2024-09-21T19:00:41.950255+0000","mapping_epoch":172,"log_start":"189'4600","ondisk_log_start":"189'4600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.272947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":522752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":100,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684599+0000","last_change":"2024-09-21T18:59:46.479423+0000","last_active":"2024-09-21T19:00:43.684599+0000","last_peered":"2024-09-21T19:00:43.684599+0000","last_clean":"2024-09-21T19:00:43.684599+0000","last_became_active":"2024-09-21T18:58:40.917818+0000","last_became_peered":"2024-09-21T18:58:40.917818+0000","last_unstale":"2024-09-21T19:00:43.684599+0000","last_undegraded":"2024-09-21T19:00:43.684599+0000","last_fullsized":"2024-09-21T19:00:43.684599+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.479379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.479379+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:48.562642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":355,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.665929+0000","last_change":"2024-09-21T18:59:21.369411+0000","last_active":"2024-09-21T19:00:39.665929+0000","last_peered":"2024-09-21T19:00:39.665929+0000","last_clean":"2024-09-21T19:00:39.665929+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:00:39.665929+0000","last_undegraded":"2024-09-21T19:00:39.665929+0000","last_fullsized":"2024-09-21T19:00:39.665929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:59:21.369328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.369328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":343,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.657396+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:00:39.657396+0000","last_peered":"2024-09-21T19:00:39.657396+0000","last_clean":"2024-09-21T19:00:39.657396+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:00:39.657396+0000","last_undegraded":"2024-09-21T19:00:39.657396+0000","last_fullsized":"2024-09-21T19:00:39.657396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:17.164534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":450,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688762+0000","last_change":"2024-09-21T18:59:54.654178+0000","last_active":"2024-09-21T19:00:43.688762+0000","last_peered":"2024-09-21T19:00:43.688762+0000","last_clean":"2024-09-21T19:00:43.688762+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:00:43.688762+0000","last_undegraded":"2024-09-21T19:00:43.688762+0000","last_fullsized":"2024-09-21T19:00:43.688762+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:59:54.654112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.654112+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:58.509872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":346,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683514+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T19:00:43.683514+0000","last_peered":"2024-09-21T19:00:43.683514+0000","last_clean":"2024-09-21T19:00:43.683514+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T19:00:43.683514+0000","last_undegraded":"2024-09-21T19:00:43.683514+0000","last_fullsized":"2024-09-21T19:00:43.683514+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.873896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"210'9818","reported_seq":9980,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.713807+0000","last_change":"2024-09-21T18:59:57.437732+0000","last_active":"2024-09-21T19:00:41.713807+0000","last_peered":"2024-09-21T19:00:41.713807+0000","last_clean":"2024-09-21T19:00:41.713807+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:00:41.713807+0000","last_undegraded":"2024-09-21T19:00:41.713807+0000","last_fullsized":"2024-09-21T19:00:41.713807+0000","mapping_epoch":172,"log_start":"187'4300","ondisk_log_start":"187'4300","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'6593","last_scrub_stamp":"2024-09-21T18:59:57.437417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.437417+0000","objects_scrubbed":6510,"log_size":5518,"log_dups_size":0,"ondisk_log_size":5518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.144644+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":534464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":101,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688085+0000","last_change":"2024-09-21T18:59:47.823699+0000","last_active":"2024-09-21T19:00:43.688085+0000","last_peered":"2024-09-21T19:00:43.688085+0000","last_clean":"2024-09-21T19:00:43.688085+0000","last_became_active":"2024-09-21T18:58:40.916167+0000","last_became_peered":"2024-09-21T18:58:40.916167+0000","last_unstale":"2024-09-21T19:00:43.688085+0000","last_undegraded":"2024-09-21T19:00:43.688085+0000","last_fullsized":"2024-09-21T19:00:43.688085+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.823624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.823624+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:16.932400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":413,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.657202+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:00:39.657202+0000","last_peered":"2024-09-21T19:00:39.657202+0000","last_clean":"2024-09-21T19:00:39.657202+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:00:39.657202+0000","last_undegraded":"2024-09-21T19:00:39.657202+0000","last_fullsized":"2024-09-21T19:00:39.657202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.575046+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":349,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.663612+0000","last_change":"2024-09-21T18:59:48.973683+0000","last_active":"2024-09-21T19:00:39.663612+0000","last_peered":"2024-09-21T19:00:39.663612+0000","last_clean":"2024-09-21T19:00:39.663612+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:00:39.663612+0000","last_undegraded":"2024-09-21T19:00:39.663612+0000","last_fullsized":"2024-09-21T19:00:39.663612+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.973616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.973616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.516500+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":328,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687136+0000","last_change":"2024-09-21T18:59:06.911709+0000","last_active":"2024-09-21T19:00:43.687136+0000","last_peered":"2024-09-21T19:00:43.687136+0000","last_clean":"2024-09-21T19:00:43.687136+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:00:43.687136+0000","last_undegraded":"2024-09-21T19:00:43.687136+0000","last_fullsized":"2024-09-21T19:00:43.687136+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.911638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.911638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.735119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":316,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688020+0000","last_change":"2024-09-21T18:59:13.953988+0000","last_active":"2024-09-21T19:00:43.688020+0000","last_peered":"2024-09-21T19:00:43.688020+0000","last_clean":"2024-09-21T19:00:43.688020+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:00:43.688020+0000","last_undegraded":"2024-09-21T19:00:43.688020+0000","last_fullsized":"2024-09-21T19:00:43.688020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:13.953912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:13.953912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"211'10137","reported_seq":10315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:46.538162+0000","last_change":"2024-09-21T19:00:18.040765+0000","last_active":"2024-09-21T19:00:46.538162+0000","last_peered":"2024-09-21T19:00:46.538162+0000","last_clean":"2024-09-21T19:00:46.538162+0000","last_became_active":"2024-09-21T18:58:44.184359+0000","last_became_peered":"2024-09-21T18:58:44.184359+0000","last_unstale":"2024-09-21T19:00:46.538162+0000","last_undegraded":"2024-09-21T19:00:46.538162+0000","last_fullsized":"2024-09-21T19:00:46.538162+0000","mapping_epoch":172,"log_start":"190'5200","ondisk_log_start":"190'5200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7943","last_scrub_stamp":"2024-09-21T19:00:18.040678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.040678+0000","objects_scrubbed":7868,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.698500+0000","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422838296,"num_objects":6453,"num_object_clones":0,"num_object_copies":19359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10137,"num_write_kb":530817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":103,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653209+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:00:39.653209+0000","last_peered":"2024-09-21T19:00:39.653209+0000","last_clean":"2024-09-21T19:00:39.653209+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:00:39.653209+0000","last_undegraded":"2024-09-21T19:00:39.653209+0000","last_fullsized":"2024-09-21T19:00:39.653209+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:48.376905+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":353,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687263+0000","last_change":"2024-09-21T18:59:20.963196+0000","last_active":"2024-09-21T19:00:43.687263+0000","last_peered":"2024-09-21T19:00:43.687263+0000","last_clean":"2024-09-21T19:00:43.687263+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T19:00:43.687263+0000","last_undegraded":"2024-09-21T19:00:43.687263+0000","last_fullsized":"2024-09-21T19:00:43.687263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:59:20.962753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.962753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":226,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.669828+0000","last_change":"2024-09-21T18:59:54.225013+0000","last_active":"2024-09-21T19:00:39.669828+0000","last_peered":"2024-09-21T19:00:39.669828+0000","last_clean":"2024-09-21T19:00:39.669828+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:00:39.669828+0000","last_undegraded":"2024-09-21T19:00:39.669828+0000","last_fullsized":"2024-09-21T19:00:39.669828+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:54.224944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.224944+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.892818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":418,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.657043+0000","last_change":"2024-09-21T19:00:22.845024+0000","last_active":"2024-09-21T19:00:39.657043+0000","last_peered":"2024-09-21T19:00:39.657043+0000","last_clean":"2024-09-21T19:00:39.657043+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:00:39.657043+0000","last_undegraded":"2024-09-21T19:00:39.657043+0000","last_fullsized":"2024-09-21T19:00:39.657043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:22.844953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.844953+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:26.214253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692793+0000","last_change":"2024-09-21T18:58:00.240553+0000","last_active":"2024-09-21T19:00:43.692793+0000","last_peered":"2024-09-21T19:00:43.692793+0000","last_clean":"2024-09-21T19:00:43.692793+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T19:00:43.692793+0000","last_undegraded":"2024-09-21T19:00:43.692793+0000","last_fullsized":"2024-09-21T19:00:43.692793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.240476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.240476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:05.824646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"210'9800","reported_seq":9886,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.756005+0000","last_change":"2024-09-21T18:58:44.097039+0000","last_active":"2024-09-21T19:00:41.756005+0000","last_peered":"2024-09-21T19:00:41.756005+0000","last_clean":"2024-09-21T19:00:41.756005+0000","last_became_active":"2024-09-21T18:58:44.096773+0000","last_became_peered":"2024-09-21T18:58:44.096773+0000","last_unstale":"2024-09-21T19:00:41.756005+0000","last_undegraded":"2024-09-21T19:00:41.756005+0000","last_fullsized":"2024-09-21T19:00:41.756005+0000","mapping_epoch":172,"log_start":"187'4300","ondisk_log_start":"187'4300","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.241385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447217664,"num_objects":6824,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9800,"num_write_kb":531968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":184,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.689159+0000","last_change":"2024-09-21T18:59:46.185760+0000","last_active":"2024-09-21T19:00:43.689159+0000","last_peered":"2024-09-21T19:00:43.689159+0000","last_clean":"2024-09-21T19:00:43.689159+0000","last_became_active":"2024-09-21T18:58:40.530131+0000","last_became_peered":"2024-09-21T18:58:40.530131+0000","last_unstale":"2024-09-21T19:00:43.689159+0000","last_undegraded":"2024-09-21T19:00:43.689159+0000","last_fullsized":"2024-09-21T19:00:43.689159+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.185693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.185693+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.587905+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":337,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.664221+0000","last_change":"2024-09-21T18:59:17.346990+0000","last_active":"2024-09-21T19:00:39.664221+0000","last_peered":"2024-09-21T19:00:39.664221+0000","last_clean":"2024-09-21T19:00:39.664221+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:00:39.664221+0000","last_undegraded":"2024-09-21T19:00:39.664221+0000","last_fullsized":"2024-09-21T19:00:39.664221+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:17.346414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.346414+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:44.883131+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684031+0000","last_change":"2024-09-21T18:59:16.589986+0000","last_active":"2024-09-21T19:00:43.684031+0000","last_peered":"2024-09-21T19:00:43.684031+0000","last_clean":"2024-09-21T19:00:43.684031+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T19:00:43.684031+0000","last_undegraded":"2024-09-21T19:00:43.684031+0000","last_fullsized":"2024-09-21T19:00:43.684031+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.589912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.589912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":268,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.668174+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T19:00:39.668174+0000","last_peered":"2024-09-21T19:00:39.668174+0000","last_clean":"2024-09-21T19:00:39.668174+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T19:00:39.668174+0000","last_undegraded":"2024-09-21T19:00:39.668174+0000","last_fullsized":"2024-09-21T19:00:39.668174+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.258988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.667992+0000","last_change":"2024-09-21T18:59:44.256590+0000","last_active":"2024-09-21T19:00:39.667992+0000","last_peered":"2024-09-21T19:00:39.667992+0000","last_clean":"2024-09-21T19:00:39.667992+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:00:39.667992+0000","last_undegraded":"2024-09-21T19:00:39.667992+0000","last_fullsized":"2024-09-21T19:00:39.667992+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:44.256513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.256513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:45.397701+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"210'9799","reported_seq":9886,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:42.332795+0000","last_change":"2024-09-21T18:58:44.097045+0000","last_active":"2024-09-21T19:00:42.332795+0000","last_peered":"2024-09-21T19:00:42.332795+0000","last_clean":"2024-09-21T19:00:42.332795+0000","last_became_active":"2024-09-21T18:58:44.096634+0000","last_became_peered":"2024-09-21T18:58:44.096634+0000","last_unstale":"2024-09-21T19:00:42.332795+0000","last_undegraded":"2024-09-21T19:00:42.332795+0000","last_fullsized":"2024-09-21T19:00:42.332795+0000","mapping_epoch":172,"log_start":"191'5300","ondisk_log_start":"191'5300","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443088896,"num_objects":6761,"num_object_clones":0,"num_object_copies":20283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9799,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":184,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.689379+0000","last_change":"2024-09-21T18:59:41.296627+0000","last_active":"2024-09-21T19:00:43.689379+0000","last_peered":"2024-09-21T19:00:43.689379+0000","last_clean":"2024-09-21T19:00:43.689379+0000","last_became_active":"2024-09-21T18:58:40.528964+0000","last_became_peered":"2024-09-21T18:58:40.528964+0000","last_unstale":"2024-09-21T19:00:43.689379+0000","last_undegraded":"2024-09-21T19:00:43.689379+0000","last_fullsized":"2024-09-21T19:00:43.689379+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:41.296558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:41.296558+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:59.947419+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":240,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.666992+0000","last_change":"2024-09-21T18:59:48.165996+0000","last_active":"2024-09-21T19:00:39.666992+0000","last_peered":"2024-09-21T19:00:39.666992+0000","last_clean":"2024-09-21T19:00:39.666992+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:00:39.666992+0000","last_undegraded":"2024-09-21T19:00:39.666992+0000","last_fullsized":"2024-09-21T19:00:39.666992+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:59:48.165907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.165907+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:55.224625+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":334,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.665331+0000","last_change":"2024-09-21T18:59:54.420959+0000","last_active":"2024-09-21T19:00:39.665331+0000","last_peered":"2024-09-21T19:00:39.665331+0000","last_clean":"2024-09-21T19:00:39.665331+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:00:39.665331+0000","last_undegraded":"2024-09-21T19:00:39.665331+0000","last_fullsized":"2024-09-21T19:00:39.665331+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:54.420888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.420888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:54.568945+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":341,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688161+0000","last_change":"2024-09-21T19:00:34.409013+0000","last_active":"2024-09-21T19:00:43.688161+0000","last_peered":"2024-09-21T19:00:43.688161+0000","last_clean":"2024-09-21T19:00:43.688161+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T19:00:43.688161+0000","last_undegraded":"2024-09-21T19:00:43.688161+0000","last_fullsized":"2024-09-21T19:00:43.688161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.816307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":322,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.659005+0000","last_change":"2024-09-21T19:00:25.249425+0000","last_active":"2024-09-21T19:00:39.659005+0000","last_peered":"2024-09-21T19:00:39.659005+0000","last_clean":"2024-09-21T19:00:39.659005+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T19:00:39.659005+0000","last_undegraded":"2024-09-21T19:00:39.659005+0000","last_fullsized":"2024-09-21T19:00:39.659005+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.249360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.249360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:32.354078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"211'9802","reported_seq":9891,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:46.568371+0000","last_change":"2024-09-21T18:58:44.183202+0000","last_active":"2024-09-21T19:00:46.568371+0000","last_peered":"2024-09-21T19:00:46.568371+0000","last_clean":"2024-09-21T19:00:46.568371+0000","last_became_active":"2024-09-21T18:58:44.182910+0000","last_became_peered":"2024-09-21T18:58:44.182910+0000","last_unstale":"2024-09-21T19:00:46.568371+0000","last_undegraded":"2024-09-21T19:00:46.568371+0000","last_fullsized":"2024-09-21T19:00:46.568371+0000","mapping_epoch":172,"log_start":"190'4800","ondisk_log_start":"190'4800","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.830014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410779648,"num_objects":6268,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9802,"num_write_kb":514240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":101,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684231+0000","last_change":"2024-09-21T18:59:41.588294+0000","last_active":"2024-09-21T19:00:43.684231+0000","last_peered":"2024-09-21T19:00:43.684231+0000","last_clean":"2024-09-21T19:00:43.684231+0000","last_became_active":"2024-09-21T18:58:40.916397+0000","last_became_peered":"2024-09-21T18:58:40.916397+0000","last_unstale":"2024-09-21T19:00:43.684231+0000","last_undegraded":"2024-09-21T19:00:43.684231+0000","last_fullsized":"2024-09-21T19:00:43.684231+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:41.588231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:41.588231+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":469,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.665197+0000","last_change":"2024-09-21T18:59:44.360729+0000","last_active":"2024-09-21T19:00:39.665197+0000","last_peered":"2024-09-21T19:00:39.665197+0000","last_clean":"2024-09-21T19:00:39.665197+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:00:39.665197+0000","last_undegraded":"2024-09-21T19:00:39.665197+0000","last_fullsized":"2024-09-21T19:00:39.665197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:59:44.360619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.360619+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:45.224923+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":358,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688237+0000","last_change":"2024-09-21T18:59:55.645493+0000","last_active":"2024-09-21T19:00:43.688237+0000","last_peered":"2024-09-21T19:00:43.688237+0000","last_clean":"2024-09-21T19:00:43.688237+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T19:00:43.688237+0000","last_undegraded":"2024-09-21T19:00:43.688237+0000","last_fullsized":"2024-09-21T19:00:43.688237+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:55.645419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.645419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:19.840924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":354,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.685120+0000","last_change":"2024-09-21T18:59:19.567802+0000","last_active":"2024-09-21T19:00:43.685120+0000","last_peered":"2024-09-21T19:00:43.685120+0000","last_clean":"2024-09-21T19:00:43.685120+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T19:00:43.685120+0000","last_undegraded":"2024-09-21T19:00:43.685120+0000","last_fullsized":"2024-09-21T19:00:43.685120+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:59:19.567649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.567649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.667304+0000","last_change":"2024-09-21T19:00:25.316275+0000","last_active":"2024-09-21T19:00:39.667304+0000","last_peered":"2024-09-21T19:00:39.667304+0000","last_clean":"2024-09-21T19:00:39.667304+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:00:39.667304+0000","last_undegraded":"2024-09-21T19:00:39.667304+0000","last_fullsized":"2024-09-21T19:00:39.667304+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.314718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.314718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.602505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":323,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653845+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:00:39.653845+0000","last_peered":"2024-09-21T19:00:39.653845+0000","last_clean":"2024-09-21T19:00:39.653845+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:00:39.653845+0000","last_undegraded":"2024-09-21T19:00:39.653845+0000","last_fullsized":"2024-09-21T19:00:39.653845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:05.377423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"210'9900","reported_seq":9987,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:42.331642+0000","last_change":"2024-09-21T18:58:44.100975+0000","last_active":"2024-09-21T19:00:42.331642+0000","last_peered":"2024-09-21T19:00:42.331642+0000","last_clean":"2024-09-21T19:00:42.331642+0000","last_became_active":"2024-09-21T18:58:44.100676+0000","last_became_peered":"2024-09-21T18:58:44.100676+0000","last_unstale":"2024-09-21T19:00:42.331642+0000","last_undegraded":"2024-09-21T19:00:42.331642+0000","last_fullsized":"2024-09-21T19:00:42.331642+0000","mapping_epoch":172,"log_start":"191'5400","ondisk_log_start":"191'5400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.859120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445120512,"num_objects":6792,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":414,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692439+0000","last_change":"2024-09-21T18:59:49.845480+0000","last_active":"2024-09-21T19:00:43.692439+0000","last_peered":"2024-09-21T19:00:43.692439+0000","last_clean":"2024-09-21T19:00:43.692439+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:00:43.692439+0000","last_undegraded":"2024-09-21T19:00:43.692439+0000","last_fullsized":"2024-09-21T19:00:43.692439+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:49.845397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.845397+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:01.762366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":301,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.660857+0000","last_change":"2024-09-21T18:59:48.817751+0000","last_active":"2024-09-21T19:00:39.660857+0000","last_peered":"2024-09-21T19:00:39.660857+0000","last_clean":"2024-09-21T19:00:39.660857+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:00:39.660857+0000","last_undegraded":"2024-09-21T19:00:39.660857+0000","last_fullsized":"2024-09-21T19:00:39.660857+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:59:48.817656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.817656+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:59.596604+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.659620+0000","last_change":"2024-09-21T18:59:50.378784+0000","last_active":"2024-09-21T19:00:39.659620+0000","last_peered":"2024-09-21T19:00:39.659620+0000","last_clean":"2024-09-21T19:00:39.659620+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T19:00:39.659620+0000","last_undegraded":"2024-09-21T19:00:39.659620+0000","last_fullsized":"2024-09-21T19:00:39.659620+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:50.378581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.378581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:57.041830+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":381,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.661216+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:00:39.661216+0000","last_peered":"2024-09-21T19:00:39.661216+0000","last_clean":"2024-09-21T19:00:39.661216+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:00:39.661216+0000","last_undegraded":"2024-09-21T19:00:39.661216+0000","last_fullsized":"2024-09-21T19:00:39.661216+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.686451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":334,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.656251+0000","last_change":"2024-09-21T19:00:29.002389+0000","last_active":"2024-09-21T19:00:39.656251+0000","last_peered":"2024-09-21T19:00:39.656251+0000","last_clean":"2024-09-21T19:00:39.656251+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:00:39.656251+0000","last_undegraded":"2024-09-21T19:00:39.656251+0000","last_fullsized":"2024-09-21T19:00:39.656251+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.002321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.002321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:56.121924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692994+0000","last_change":"2024-09-21T19:00:32.573881+0000","last_active":"2024-09-21T19:00:43.692994+0000","last_peered":"2024-09-21T19:00:43.692994+0000","last_clean":"2024-09-21T19:00:43.692994+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T19:00:43.692994+0000","last_undegraded":"2024-09-21T19:00:43.692994+0000","last_fullsized":"2024-09-21T19:00:43.692994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:00:32.573704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.573704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.007025+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"210'9683","reported_seq":9841,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.932547+0000","last_change":"2024-09-21T18:59:55.688614+0000","last_active":"2024-09-21T19:00:41.932547+0000","last_peered":"2024-09-21T19:00:41.932547+0000","last_clean":"2024-09-21T19:00:41.932547+0000","last_became_active":"2024-09-21T18:58:44.099439+0000","last_became_peered":"2024-09-21T18:58:44.099439+0000","last_unstale":"2024-09-21T19:00:41.932547+0000","last_undegraded":"2024-09-21T19:00:41.932547+0000","last_fullsized":"2024-09-21T19:00:41.932547+0000","mapping_epoch":172,"log_start":"189'4600","ondisk_log_start":"189'4600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'6308","last_scrub_stamp":"2024-09-21T18:59:55.688429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.688429+0000","objects_scrubbed":6231,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:56.502643+0000","scrub_duration":2287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439156736,"num_objects":6701,"num_object_clones":0,"num_object_copies":20103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":98,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.662746+0000","last_change":"2024-09-21T18:59:42.333738+0000","last_active":"2024-09-21T19:00:39.662746+0000","last_peered":"2024-09-21T19:00:39.662746+0000","last_clean":"2024-09-21T19:00:39.662746+0000","last_became_active":"2024-09-21T18:58:40.528545+0000","last_became_peered":"2024-09-21T18:58:40.528545+0000","last_unstale":"2024-09-21T19:00:39.662746+0000","last_undegraded":"2024-09-21T19:00:39.662746+0000","last_fullsized":"2024-09-21T19:00:39.662746+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:42.333677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.333677+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:55.111757+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":236,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.669660+0000","last_change":"2024-09-21T18:59:53.225904+0000","last_active":"2024-09-21T19:00:39.669660+0000","last_peered":"2024-09-21T19:00:39.669660+0000","last_clean":"2024-09-21T19:00:39.669660+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:00:39.669660+0000","last_undegraded":"2024-09-21T19:00:39.669660+0000","last_fullsized":"2024-09-21T19:00:39.669660+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:53.225823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.225823+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:05.051041+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":222,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.668399+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:00:39.668399+0000","last_peered":"2024-09-21T19:00:39.668399+0000","last_clean":"2024-09-21T19:00:39.668399+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:00:39.668399+0000","last_undegraded":"2024-09-21T19:00:39.668399+0000","last_fullsized":"2024-09-21T19:00:39.668399+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.948040+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":821,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:44.917860+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T19:00:44.917860+0000","last_peered":"2024-09-21T19:00:44.917860+0000","last_clean":"2024-09-21T19:00:44.917860+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:00:44.917860+0000","last_undegraded":"2024-09-21T19:00:44.917860+0000","last_fullsized":"2024-09-21T19:00:44.917860+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.500330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":358,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683731+0000","last_change":"2024-09-21T18:59:21.571613+0000","last_active":"2024-09-21T19:00:43.683731+0000","last_peered":"2024-09-21T19:00:43.683731+0000","last_clean":"2024-09-21T19:00:43.683731+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:00:43.683731+0000","last_undegraded":"2024-09-21T19:00:43.683731+0000","last_fullsized":"2024-09-21T19:00:43.683731+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:21.571551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.571551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:44.417480+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":322,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.662160+0000","last_change":"2024-09-21T19:00:33.443101+0000","last_active":"2024-09-21T19:00:39.662160+0000","last_peered":"2024-09-21T19:00:39.662160+0000","last_clean":"2024-09-21T19:00:39.662160+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:00:39.662160+0000","last_undegraded":"2024-09-21T19:00:39.662160+0000","last_fullsized":"2024-09-21T19:00:39.662160+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:33.443040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.443040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.013096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"211'10315","reported_seq":10479,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:46.530143+0000","last_change":"2024-09-21T19:00:00.245865+0000","last_active":"2024-09-21T19:00:46.530143+0000","last_peered":"2024-09-21T19:00:46.530143+0000","last_clean":"2024-09-21T19:00:46.530143+0000","last_became_active":"2024-09-21T18:58:44.096509+0000","last_became_peered":"2024-09-21T18:58:44.096509+0000","last_unstale":"2024-09-21T19:00:46.530143+0000","last_undegraded":"2024-09-21T19:00:46.530143+0000","last_fullsized":"2024-09-21T19:00:46.530143+0000","mapping_epoch":172,"log_start":"189'4700","ondisk_log_start":"189'4700","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'6787","last_scrub_stamp":"2024-09-21T19:00:00.245707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.245707+0000","objects_scrubbed":6706,"log_size":5615,"log_dups_size":0,"ondisk_log_size":5615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.145095+0000","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428146688,"num_objects":6533,"num_object_clones":0,"num_object_copies":19599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":445,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.689106+0000","last_change":"2024-09-21T18:59:12.162986+0000","last_active":"2024-09-21T19:00:43.689106+0000","last_peered":"2024-09-21T19:00:43.689106+0000","last_clean":"2024-09-21T19:00:43.689106+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T19:00:43.689106+0000","last_undegraded":"2024-09-21T19:00:43.689106+0000","last_fullsized":"2024-09-21T19:00:43.689106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:59:12.162725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.162725+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.317843+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":335,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.654786+0000","last_change":"2024-09-21T18:59:50.993922+0000","last_active":"2024-09-21T19:00:39.654786+0000","last_peered":"2024-09-21T19:00:39.654786+0000","last_clean":"2024-09-21T19:00:39.654786+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T19:00:39.654786+0000","last_undegraded":"2024-09-21T19:00:39.654786+0000","last_fullsized":"2024-09-21T19:00:39.654786+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.712596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":408,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692580+0000","last_change":"2024-09-21T18:59:44.783805+0000","last_active":"2024-09-21T19:00:43.692580+0000","last_peered":"2024-09-21T19:00:43.692580+0000","last_clean":"2024-09-21T19:00:43.692580+0000","last_became_active":"2024-09-21T18:58:39.058347+0000","last_became_peered":"2024-09-21T18:58:39.058347+0000","last_unstale":"2024-09-21T19:00:43.692580+0000","last_undegraded":"2024-09-21T19:00:43.692580+0000","last_fullsized":"2024-09-21T19:00:43.692580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:44.783735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.783735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:56.772817+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.694687+0000","last_change":"2024-09-21T19:00:00.889227+0000","last_active":"2024-09-21T19:00:43.694687+0000","last_peered":"2024-09-21T19:00:43.694687+0000","last_clean":"2024-09-21T19:00:43.694687+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:00:43.694687+0000","last_undegraded":"2024-09-21T19:00:43.694687+0000","last_fullsized":"2024-09-21T19:00:43.694687+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.889156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.889156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:16.434600+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683586+0000","last_change":"2024-09-21T18:58:30.445025+0000","last_active":"2024-09-21T19:00:43.683586+0000","last_peered":"2024-09-21T19:00:43.683586+0000","last_clean":"2024-09-21T19:00:43.683586+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T19:00:43.683586+0000","last_undegraded":"2024-09-21T19:00:43.683586+0000","last_fullsized":"2024-09-21T19:00:43.683586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:30.444954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:30.444954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.851328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"211'10193","reported_seq":10282,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:46.460332+0000","last_change":"2024-09-21T18:58:44.097275+0000","last_active":"2024-09-21T19:00:46.460332+0000","last_peered":"2024-09-21T19:00:46.460332+0000","last_clean":"2024-09-21T19:00:46.460332+0000","last_became_active":"2024-09-21T18:58:44.096986+0000","last_became_peered":"2024-09-21T18:58:44.096986+0000","last_unstale":"2024-09-21T19:00:46.460332+0000","last_undegraded":"2024-09-21T19:00:46.460332+0000","last_fullsized":"2024-09-21T19:00:46.460332+0000","mapping_epoch":172,"log_start":"191'5500","ondisk_log_start":"191'5500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.628491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417398784,"num_objects":6369,"num_object_clones":0,"num_object_copies":19107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":359,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688809+0000","last_change":"2024-09-21T19:00:01.693318+0000","last_active":"2024-09-21T19:00:43.688809+0000","last_peered":"2024-09-21T19:00:43.688809+0000","last_clean":"2024-09-21T19:00:43.688809+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T19:00:43.688809+0000","last_undegraded":"2024-09-21T19:00:43.688809+0000","last_fullsized":"2024-09-21T19:00:43.688809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:01.693221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.693221+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.644648+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687515+0000","last_change":"2024-09-21T19:00:27.721451+0000","last_active":"2024-09-21T19:00:43.687515+0000","last_peered":"2024-09-21T19:00:43.687515+0000","last_clean":"2024-09-21T19:00:43.687515+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:00:43.687515+0000","last_undegraded":"2024-09-21T19:00:43.687515+0000","last_fullsized":"2024-09-21T19:00:43.687515+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:27.721385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:27.721385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.383225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683434+0000","last_change":"2024-09-21T18:59:52.640497+0000","last_active":"2024-09-21T19:00:43.683434+0000","last_peered":"2024-09-21T19:00:43.683434+0000","last_clean":"2024-09-21T19:00:43.683434+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T19:00:43.683434+0000","last_undegraded":"2024-09-21T19:00:43.683434+0000","last_fullsized":"2024-09-21T19:00:43.683434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.640432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.640432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:53.254498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":307,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653385+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T19:00:39.653385+0000","last_peered":"2024-09-21T19:00:39.653385+0000","last_clean":"2024-09-21T19:00:39.653385+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T19:00:39.653385+0000","last_undegraded":"2024-09-21T19:00:39.653385+0000","last_fullsized":"2024-09-21T19:00:39.653385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"210'9766","reported_seq":9941,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.957812+0000","last_change":"2024-09-21T19:00:19.801173+0000","last_active":"2024-09-21T19:00:41.957812+0000","last_peered":"2024-09-21T19:00:41.957812+0000","last_clean":"2024-09-21T19:00:41.957812+0000","last_became_active":"2024-09-21T18:58:44.097827+0000","last_became_peered":"2024-09-21T18:58:44.097827+0000","last_unstale":"2024-09-21T19:00:41.957812+0000","last_undegraded":"2024-09-21T19:00:41.957812+0000","last_fullsized":"2024-09-21T19:00:41.957812+0000","mapping_epoch":172,"log_start":"189'4700","ondisk_log_start":"189'4700","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8051","last_scrub_stamp":"2024-09-21T19:00:19.800992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.800992+0000","objects_scrubbed":7979,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.556183+0000","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444727296,"num_objects":6786,"num_object_clones":0,"num_object_copies":20358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9766,"num_write_kb":529664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":107,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684522+0000","last_change":"2024-09-21T18:59:47.518847+0000","last_active":"2024-09-21T19:00:43.684522+0000","last_peered":"2024-09-21T19:00:43.684522+0000","last_clean":"2024-09-21T19:00:43.684522+0000","last_became_active":"2024-09-21T18:58:40.917793+0000","last_became_peered":"2024-09-21T18:58:40.917793+0000","last_unstale":"2024-09-21T19:00:43.684522+0000","last_undegraded":"2024-09-21T19:00:43.684522+0000","last_fullsized":"2024-09-21T19:00:43.684522+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.518777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.518777+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.628241+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":360,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687900+0000","last_change":"2024-09-21T18:59:03.883391+0000","last_active":"2024-09-21T19:00:43.687900+0000","last_peered":"2024-09-21T19:00:43.687900+0000","last_clean":"2024-09-21T19:00:43.687900+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:00:43.687900+0000","last_undegraded":"2024-09-21T19:00:43.687900+0000","last_fullsized":"2024-09-21T19:00:43.687900+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:59:03.883306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:03.883306+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688437+0000","last_change":"2024-09-21T18:59:06.194777+0000","last_active":"2024-09-21T19:00:43.688437+0000","last_peered":"2024-09-21T19:00:43.688437+0000","last_clean":"2024-09-21T19:00:43.688437+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T19:00:43.688437+0000","last_undegraded":"2024-09-21T19:00:43.688437+0000","last_fullsized":"2024-09-21T19:00:43.688437+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.194722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.194722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.287924+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":332,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.662259+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T19:00:39.662259+0000","last_peered":"2024-09-21T19:00:39.662259+0000","last_clean":"2024-09-21T19:00:39.662259+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:00:39.662259+0000","last_undegraded":"2024-09-21T19:00:39.662259+0000","last_fullsized":"2024-09-21T19:00:39.662259+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.313883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":322,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.660997+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:00:39.660997+0000","last_peered":"2024-09-21T19:00:39.660997+0000","last_clean":"2024-09-21T19:00:39.660997+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:00:39.660997+0000","last_undegraded":"2024-09-21T19:00:39.660997+0000","last_fullsized":"2024-09-21T19:00:39.660997+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:47.425210+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"210'9672","reported_seq":9759,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:42.297485+0000","last_change":"2024-09-21T18:58:44.096415+0000","last_active":"2024-09-21T19:00:42.297485+0000","last_peered":"2024-09-21T19:00:42.297485+0000","last_clean":"2024-09-21T19:00:42.297485+0000","last_became_active":"2024-09-21T18:58:44.096142+0000","last_became_peered":"2024-09-21T18:58:44.096142+0000","last_unstale":"2024-09-21T19:00:42.297485+0000","last_undegraded":"2024-09-21T19:00:42.297485+0000","last_fullsized":"2024-09-21T19:00:42.297485+0000","mapping_epoch":172,"log_start":"190'5100","ondisk_log_start":"190'5100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:07.897929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438829056,"num_objects":6696,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":523776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":413,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692644+0000","last_change":"2024-09-21T18:59:50.870015+0000","last_active":"2024-09-21T19:00:43.692644+0000","last_peered":"2024-09-21T19:00:43.692644+0000","last_clean":"2024-09-21T19:00:43.692644+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:00:43.692644+0000","last_undegraded":"2024-09-21T19:00:43.692644+0000","last_fullsized":"2024-09-21T19:00:43.692644+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:50.869945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.869945+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:07.279297+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":347,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688175+0000","last_change":"2024-09-21T18:59:53.699627+0000","last_active":"2024-09-21T19:00:43.688175+0000","last_peered":"2024-09-21T19:00:43.688175+0000","last_clean":"2024-09-21T19:00:43.688175+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T19:00:43.688175+0000","last_undegraded":"2024-09-21T19:00:43.688175+0000","last_fullsized":"2024-09-21T19:00:43.688175+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.929430+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":363,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.661381+0000","last_change":"2024-09-21T18:58:30.050910+0000","last_active":"2024-09-21T19:00:39.661381+0000","last_peered":"2024-09-21T19:00:39.661381+0000","last_clean":"2024-09-21T19:00:39.661381+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:00:39.661381+0000","last_undegraded":"2024-09-21T19:00:39.661381+0000","last_fullsized":"2024-09-21T19:00:39.661381+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:58:30.050859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:30.050859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.904611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":441,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.660775+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:00:39.660775+0000","last_peered":"2024-09-21T19:00:39.660775+0000","last_clean":"2024-09-21T19:00:39.660775+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:00:39.660775+0000","last_undegraded":"2024-09-21T19:00:39.660775+0000","last_fullsized":"2024-09-21T19:00:39.660775+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.071423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688666+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:00:43.688666+0000","last_peered":"2024-09-21T19:00:43.688666+0000","last_clean":"2024-09-21T19:00:43.688666+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:00:43.688666+0000","last_undegraded":"2024-09-21T19:00:43.688666+0000","last_fullsized":"2024-09-21T19:00:43.688666+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.791557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"210'9804","reported_seq":9891,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.941624+0000","last_change":"2024-09-21T18:58:44.101043+0000","last_active":"2024-09-21T19:00:41.941624+0000","last_peered":"2024-09-21T19:00:41.941624+0000","last_clean":"2024-09-21T19:00:41.941624+0000","last_became_active":"2024-09-21T18:58:44.100559+0000","last_became_peered":"2024-09-21T18:58:44.100559+0000","last_unstale":"2024-09-21T19:00:41.941624+0000","last_undegraded":"2024-09-21T19:00:41.941624+0000","last_fullsized":"2024-09-21T19:00:41.941624+0000","mapping_epoch":172,"log_start":"189'4800","ondisk_log_start":"189'4800","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452591616,"num_objects":6906,"num_object_clones":0,"num_object_copies":20718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":534720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":216,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.667074+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T19:00:39.667074+0000","last_peered":"2024-09-21T19:00:39.667074+0000","last_clean":"2024-09-21T19:00:39.667074+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:00:39.667074+0000","last_undegraded":"2024-09-21T19:00:39.667074+0000","last_fullsized":"2024-09-21T19:00:39.667074+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.261047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684638+0000","last_change":"2024-09-21T18:59:15.560811+0000","last_active":"2024-09-21T19:00:43.684638+0000","last_peered":"2024-09-21T19:00:43.684638+0000","last_clean":"2024-09-21T19:00:43.684638+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T19:00:43.684638+0000","last_undegraded":"2024-09-21T19:00:43.684638+0000","last_fullsized":"2024-09-21T19:00:43.684638+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:15.560713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:15.560713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":326,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.655940+0000","last_change":"2024-09-21T18:59:43.841286+0000","last_active":"2024-09-21T19:00:39.655940+0000","last_peered":"2024-09-21T19:00:39.655940+0000","last_clean":"2024-09-21T19:00:39.655940+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:00:39.655940+0000","last_undegraded":"2024-09-21T19:00:39.655940+0000","last_fullsized":"2024-09-21T19:00:39.655940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:43.841222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:43.841222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:45.452296+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.668321+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:00:39.668321+0000","last_peered":"2024-09-21T19:00:39.668321+0000","last_clean":"2024-09-21T19:00:39.668321+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:00:39.668321+0000","last_undegraded":"2024-09-21T19:00:39.668321+0000","last_fullsized":"2024-09-21T19:00:39.668321+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"210'9675","reported_seq":9832,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:41.716882+0000","last_change":"2024-09-21T18:59:54.271364+0000","last_active":"2024-09-21T19:00:41.716882+0000","last_peered":"2024-09-21T19:00:41.716882+0000","last_clean":"2024-09-21T19:00:41.716882+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:00:41.716882+0000","last_undegraded":"2024-09-21T19:00:41.716882+0000","last_fullsized":"2024-09-21T19:00:41.716882+0000","mapping_epoch":172,"log_start":"187'4200","ondisk_log_start":"187'4200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'6188","last_scrub_stamp":"2024-09-21T18:59:54.271275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.271275+0000","objects_scrubbed":6120,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:55.967625+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443875328,"num_objects":6773,"num_object_clones":0,"num_object_copies":20319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9675,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":362,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688246+0000","last_change":"2024-09-21T19:00:23.328177+0000","last_active":"2024-09-21T19:00:43.688246+0000","last_peered":"2024-09-21T19:00:43.688246+0000","last_clean":"2024-09-21T19:00:43.688246+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T19:00:43.688246+0000","last_undegraded":"2024-09-21T19:00:43.688246+0000","last_fullsized":"2024-09-21T19:00:43.688246+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.465067+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.693251+0000","last_change":"2024-09-21T18:58:37.143097+0000","last_active":"2024-09-21T19:00:43.693251+0000","last_peered":"2024-09-21T19:00:43.693251+0000","last_clean":"2024-09-21T19:00:43.693251+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T19:00:43.693251+0000","last_undegraded":"2024-09-21T19:00:43.693251+0000","last_fullsized":"2024-09-21T19:00:43.693251+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:37.143050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:37.143050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.764335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.655769+0000","last_change":"2024-09-21T19:00:30.996323+0000","last_active":"2024-09-21T19:00:39.655769+0000","last_peered":"2024-09-21T19:00:39.655769+0000","last_clean":"2024-09-21T19:00:39.655769+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:00:39.655769+0000","last_undegraded":"2024-09-21T19:00:39.655769+0000","last_fullsized":"2024-09-21T19:00:39.655769+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:30.996260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:30.996260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:34.407198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692473+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:00:43.692473+0000","last_peered":"2024-09-21T19:00:43.692473+0000","last_clean":"2024-09-21T19:00:43.692473+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:00:43.692473+0000","last_undegraded":"2024-09-21T19:00:43.692473+0000","last_fullsized":"2024-09-21T19:00:43.692473+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"211'10111","reported_seq":10270,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:46.522098+0000","last_change":"2024-09-21T18:59:52.633622+0000","last_active":"2024-09-21T19:00:46.522098+0000","last_peered":"2024-09-21T19:00:46.522098+0000","last_clean":"2024-09-21T19:00:46.522098+0000","last_became_active":"2024-09-21T18:58:44.184048+0000","last_became_peered":"2024-09-21T18:58:44.184048+0000","last_unstale":"2024-09-21T19:00:46.522098+0000","last_undegraded":"2024-09-21T19:00:46.522098+0000","last_fullsized":"2024-09-21T19:00:46.522098+0000","mapping_epoch":172,"log_start":"189'4500","ondisk_log_start":"189'4500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'6240","last_scrub_stamp":"2024-09-21T18:59:52.633386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.633386+0000","objects_scrubbed":6189,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:00.395113+0000","scrub_duration":1379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426180608,"num_objects":6503,"num_object_clones":0,"num_object_copies":19509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10111,"num_write_kb":531648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":374,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687384+0000","last_change":"2024-09-21T18:59:17.862166+0000","last_active":"2024-09-21T19:00:43.687384+0000","last_peered":"2024-09-21T19:00:43.687384+0000","last_clean":"2024-09-21T19:00:43.687384+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:00:43.687384+0000","last_undegraded":"2024-09-21T19:00:43.687384+0000","last_fullsized":"2024-09-21T19:00:43.687384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:17.862105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.862105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":343,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.693240+0000","last_change":"2024-09-21T19:00:22.616468+0000","last_active":"2024-09-21T19:00:43.693240+0000","last_peered":"2024-09-21T19:00:43.693240+0000","last_clean":"2024-09-21T19:00:43.693240+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:00:43.693240+0000","last_undegraded":"2024-09-21T19:00:43.693240+0000","last_fullsized":"2024-09-21T19:00:43.693240+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:22.616386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.616386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:52.426205+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":348,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.655496+0000","last_change":"2024-09-21T18:59:16.539031+0000","last_active":"2024-09-21T19:00:39.655496+0000","last_peered":"2024-09-21T19:00:39.655496+0000","last_clean":"2024-09-21T19:00:39.655496+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:00:39.655496+0000","last_undegraded":"2024-09-21T19:00:39.655496+0000","last_fullsized":"2024-09-21T19:00:39.655496+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.538853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.538853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:43.935869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":315,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.661109+0000","last_change":"2024-09-21T19:00:34.960755+0000","last_active":"2024-09-21T19:00:39.661109+0000","last_peered":"2024-09-21T19:00:39.661109+0000","last_clean":"2024-09-21T19:00:39.661109+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:00:39.661109+0000","last_undegraded":"2024-09-21T19:00:39.661109+0000","last_fullsized":"2024-09-21T19:00:39.661109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.960689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.960689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:57.667040+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"211'10015","reported_seq":10188,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:46.572580+0000","last_change":"2024-09-21T19:00:21.263588+0000","last_active":"2024-09-21T19:00:46.572580+0000","last_peered":"2024-09-21T19:00:46.572580+0000","last_clean":"2024-09-21T19:00:46.572580+0000","last_became_active":"2024-09-21T18:58:44.094697+0000","last_became_peered":"2024-09-21T18:58:44.094697+0000","last_unstale":"2024-09-21T19:00:46.572580+0000","last_undegraded":"2024-09-21T19:00:46.572580+0000","last_fullsized":"2024-09-21T19:00:46.572580+0000","mapping_epoch":172,"log_start":"190'5000","ondisk_log_start":"190'5000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8114","last_scrub_stamp":"2024-09-21T19:00:21.263245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.263245+0000","objects_scrubbed":8036,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.729679+0000","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418447360,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":365,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684684+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T19:00:43.684684+0000","last_peered":"2024-09-21T19:00:43.684684+0000","last_clean":"2024-09-21T19:00:43.684684+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T19:00:43.684684+0000","last_undegraded":"2024-09-21T19:00:43.684684+0000","last_fullsized":"2024-09-21T19:00:43.684684+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.863281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":365,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.693326+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T19:00:43.693326+0000","last_peered":"2024-09-21T19:00:43.693326+0000","last_clean":"2024-09-21T19:00:43.693326+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:00:43.693326+0000","last_undegraded":"2024-09-21T19:00:43.693326+0000","last_fullsized":"2024-09-21T19:00:43.693326+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":438,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692501+0000","last_change":"2024-09-21T18:59:56.831306+0000","last_active":"2024-09-21T19:00:43.692501+0000","last_peered":"2024-09-21T19:00:43.692501+0000","last_clean":"2024-09-21T19:00:43.692501+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T19:00:43.692501+0000","last_undegraded":"2024-09-21T19:00:43.692501+0000","last_fullsized":"2024-09-21T19:00:43.692501+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:59:56.831225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.831225+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.230252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":346,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684796+0000","last_change":"2024-09-21T18:59:59.767058+0000","last_active":"2024-09-21T19:00:43.684796+0000","last_peered":"2024-09-21T19:00:43.684796+0000","last_clean":"2024-09-21T19:00:43.684796+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T19:00:43.684796+0000","last_undegraded":"2024-09-21T19:00:43.684796+0000","last_fullsized":"2024-09-21T19:00:43.684796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.766985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.766985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:00.540870+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":349,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687901+0000","last_change":"2024-09-21T18:59:14.180429+0000","last_active":"2024-09-21T19:00:43.687901+0000","last_peered":"2024-09-21T19:00:43.687901+0000","last_clean":"2024-09-21T19:00:43.687901+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T19:00:43.687901+0000","last_undegraded":"2024-09-21T19:00:43.687901+0000","last_fullsized":"2024-09-21T19:00:43.687901+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:44.041103+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.663426+0000","last_change":"2024-09-21T18:59:59.958853+0000","last_active":"2024-09-21T19:00:39.663426+0000","last_peered":"2024-09-21T19:00:39.663426+0000","last_clean":"2024-09-21T19:00:39.663426+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:00:39.663426+0000","last_undegraded":"2024-09-21T19:00:39.663426+0000","last_fullsized":"2024-09-21T19:00:39.663426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T18:59:59.958609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.958609+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:01.002394+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683984+0000","last_change":"2024-09-21T19:00:29.955243+0000","last_active":"2024-09-21T19:00:43.683984+0000","last_peered":"2024-09-21T19:00:43.683984+0000","last_clean":"2024-09-21T19:00:43.683984+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T19:00:43.683984+0000","last_undegraded":"2024-09-21T19:00:43.683984+0000","last_fullsized":"2024-09-21T19:00:43.683984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.955184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.955184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.019605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692977+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T19:00:43.692977+0000","last_peered":"2024-09-21T19:00:43.692977+0000","last_clean":"2024-09-21T19:00:43.692977+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T19:00:43.692977+0000","last_undegraded":"2024-09-21T19:00:43.692977+0000","last_fullsized":"2024-09-21T19:00:43.692977+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":224,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.669786+0000","last_change":"2024-09-21T19:00:34.345923+0000","last_active":"2024-09-21T19:00:39.669786+0000","last_peered":"2024-09-21T19:00:39.669786+0000","last_clean":"2024-09-21T19:00:39.669786+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:00:39.669786+0000","last_undegraded":"2024-09-21T19:00:39.669786+0000","last_fullsized":"2024-09-21T19:00:39.669786+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:00:34.345792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.345792+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:58.340045+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692740+0000","last_change":"2024-09-21T19:00:29.586470+0000","last_active":"2024-09-21T19:00:43.692740+0000","last_peered":"2024-09-21T19:00:43.692740+0000","last_clean":"2024-09-21T19:00:43.692740+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T19:00:43.692740+0000","last_undegraded":"2024-09-21T19:00:43.692740+0000","last_fullsized":"2024-09-21T19:00:43.692740+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.586411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.586411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.139362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":327,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.663166+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:00:39.663166+0000","last_peered":"2024-09-21T19:00:39.663166+0000","last_clean":"2024-09-21T19:00:39.663166+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:00:39.663166+0000","last_undegraded":"2024-09-21T19:00:39.663166+0000","last_fullsized":"2024-09-21T19:00:39.663166+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.620068+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":322,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.656165+0000","last_change":"2024-09-21T18:59:47.825900+0000","last_active":"2024-09-21T19:00:39.656165+0000","last_peered":"2024-09-21T19:00:39.656165+0000","last_clean":"2024-09-21T19:00:39.656165+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:00:39.656165+0000","last_undegraded":"2024-09-21T19:00:39.656165+0000","last_fullsized":"2024-09-21T19:00:39.656165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:47.825835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.825835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.012332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":348,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683841+0000","last_change":"2024-09-21T18:58:58.524782+0000","last_active":"2024-09-21T19:00:43.683841+0000","last_peered":"2024-09-21T19:00:43.683841+0000","last_clean":"2024-09-21T19:00:43.683841+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T19:00:43.683841+0000","last_undegraded":"2024-09-21T19:00:43.683841+0000","last_fullsized":"2024-09-21T19:00:43.683841+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:58.524708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.524708+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.667781+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T19:00:39.667781+0000","last_peered":"2024-09-21T19:00:39.667781+0000","last_clean":"2024-09-21T19:00:39.667781+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:00:39.667781+0000","last_undegraded":"2024-09-21T19:00:39.667781+0000","last_fullsized":"2024-09-21T19:00:39.667781+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":434,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653610+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:00:39.653610+0000","last_peered":"2024-09-21T19:00:39.653610+0000","last_clean":"2024-09-21T19:00:39.653610+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:00:39.653610+0000","last_undegraded":"2024-09-21T19:00:39.653610+0000","last_fullsized":"2024-09-21T19:00:39.653610+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:22.257098+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688454+0000","last_change":"2024-09-21T19:00:32.753965+0000","last_active":"2024-09-21T19:00:43.688454+0000","last_peered":"2024-09-21T19:00:43.688454+0000","last_clean":"2024-09-21T19:00:43.688454+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:00:43.688454+0000","last_undegraded":"2024-09-21T19:00:43.688454+0000","last_fullsized":"2024-09-21T19:00:43.688454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:32.753902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.753902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.288209+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":338,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653845+0000","last_change":"2024-09-21T19:00:28.893623+0000","last_active":"2024-09-21T19:00:39.653845+0000","last_peered":"2024-09-21T19:00:39.653845+0000","last_clean":"2024-09-21T19:00:39.653845+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:00:39.653845+0000","last_undegraded":"2024-09-21T19:00:39.653845+0000","last_fullsized":"2024-09-21T19:00:39.653845+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:28.893403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.893403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.904866+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692964+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:00:43.692964+0000","last_peered":"2024-09-21T19:00:43.692964+0000","last_clean":"2024-09-21T19:00:43.692964+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:00:43.692964+0000","last_undegraded":"2024-09-21T19:00:43.692964+0000","last_fullsized":"2024-09-21T19:00:43.692964+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.507018+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":442,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.683662+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T19:00:43.683662+0000","last_peered":"2024-09-21T19:00:43.683662+0000","last_clean":"2024-09-21T19:00:43.683662+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T19:00:43.683662+0000","last_undegraded":"2024-09-21T19:00:43.683662+0000","last_fullsized":"2024-09-21T19:00:43.683662+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.883299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.670090+0000","last_change":"2024-09-21T18:59:45.212437+0000","last_active":"2024-09-21T19:00:39.670090+0000","last_peered":"2024-09-21T19:00:39.670090+0000","last_clean":"2024-09-21T19:00:39.670090+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:00:39.670090+0000","last_undegraded":"2024-09-21T19:00:39.670090+0000","last_fullsized":"2024-09-21T19:00:39.670090+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.212362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.212362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:48.818398+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":370,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687942+0000","last_change":"2024-09-21T18:59:56.667414+0000","last_active":"2024-09-21T19:00:43.687942+0000","last_peered":"2024-09-21T19:00:43.687942+0000","last_clean":"2024-09-21T19:00:43.687942+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T19:00:43.687942+0000","last_undegraded":"2024-09-21T19:00:43.687942+0000","last_fullsized":"2024-09-21T19:00:43.687942+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:56.667316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.667316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.881734+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.688380+0000","last_change":"2024-09-21T19:00:34.746168+0000","last_active":"2024-09-21T19:00:43.688380+0000","last_peered":"2024-09-21T19:00:43.688380+0000","last_clean":"2024-09-21T19:00:43.688380+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T19:00:43.688380+0000","last_undegraded":"2024-09-21T19:00:43.688380+0000","last_fullsized":"2024-09-21T19:00:43.688380+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.746126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.746126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":334,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.664096+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:00:39.664096+0000","last_peered":"2024-09-21T19:00:39.664096+0000","last_clean":"2024-09-21T19:00:39.664096+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:00:39.664096+0000","last_undegraded":"2024-09-21T19:00:39.664096+0000","last_fullsized":"2024-09-21T19:00:39.664096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:59.880814+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":329,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.653136+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T19:00:39.653136+0000","last_peered":"2024-09-21T19:00:39.653136+0000","last_clean":"2024-09-21T19:00:39.653136+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:00:39.653136+0000","last_undegraded":"2024-09-21T19:00:39.653136+0000","last_fullsized":"2024-09-21T19:00:39.653136+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":354,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.664340+0000","last_change":"2024-09-21T18:58:59.379970+0000","last_active":"2024-09-21T19:00:39.664340+0000","last_peered":"2024-09-21T19:00:39.664340+0000","last_clean":"2024-09-21T19:00:39.664340+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T19:00:39.664340+0000","last_undegraded":"2024-09-21T19:00:39.664340+0000","last_fullsized":"2024-09-21T19:00:39.664340+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:59.379849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:59.379849+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.273644+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":343,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.693188+0000","last_change":"2024-09-21T18:58:34.182624+0000","last_active":"2024-09-21T19:00:43.693188+0000","last_peered":"2024-09-21T19:00:43.693188+0000","last_clean":"2024-09-21T19:00:43.693188+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:00:43.693188+0000","last_undegraded":"2024-09-21T19:00:43.693188+0000","last_fullsized":"2024-09-21T19:00:43.693188+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:34.182578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:34.182578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.687690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":341,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.661180+0000","last_change":"2024-09-21T18:58:58.952448+0000","last_active":"2024-09-21T19:00:39.661180+0000","last_peered":"2024-09-21T19:00:39.661180+0000","last_clean":"2024-09-21T19:00:39.661180+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:00:39.661180+0000","last_undegraded":"2024-09-21T19:00:39.661180+0000","last_fullsized":"2024-09-21T19:00:39.661180+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:58.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":222,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.668095+0000","last_change":"2024-09-21T18:59:42.223794+0000","last_active":"2024-09-21T19:00:39.668095+0000","last_peered":"2024-09-21T19:00:39.668095+0000","last_clean":"2024-09-21T19:00:39.668095+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:00:39.668095+0000","last_undegraded":"2024-09-21T19:00:39.668095+0000","last_fullsized":"2024-09-21T19:00:39.668095+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:42.223731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.223731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.039045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":350,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.654237+0000","last_change":"2024-09-21T19:00:00.939564+0000","last_active":"2024-09-21T19:00:39.654237+0000","last_peered":"2024-09-21T19:00:39.654237+0000","last_clean":"2024-09-21T19:00:39.654237+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T19:00:39.654237+0000","last_undegraded":"2024-09-21T19:00:39.654237+0000","last_fullsized":"2024-09-21T19:00:39.654237+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:20.583806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.684059+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:00:43.684059+0000","last_peered":"2024-09-21T19:00:43.684059+0000","last_clean":"2024-09-21T19:00:43.684059+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:00:43.684059+0000","last_undegraded":"2024-09-21T19:00:43.684059+0000","last_fullsized":"2024-09-21T19:00:43.684059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692242+0000","last_change":"2024-09-21T19:00:16.651104+0000","last_active":"2024-09-21T19:00:43.692242+0000","last_peered":"2024-09-21T19:00:43.692242+0000","last_clean":"2024-09-21T19:00:43.692242+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T19:00:43.692242+0000","last_undegraded":"2024-09-21T19:00:43.692242+0000","last_fullsized":"2024-09-21T19:00:43.692242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:16.650906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:16.650906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.021875+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":337,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.661306+0000","last_change":"2024-09-21T18:59:58.945778+0000","last_active":"2024-09-21T19:00:39.661306+0000","last_peered":"2024-09-21T19:00:39.661306+0000","last_clean":"2024-09-21T19:00:39.661306+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T19:00:39.661306+0000","last_undegraded":"2024-09-21T19:00:39.661306+0000","last_fullsized":"2024-09-21T19:00:39.661306+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:58.945631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:58.945631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:10.452414+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":334,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.657124+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T19:00:39.657124+0000","last_peered":"2024-09-21T19:00:39.657124+0000","last_clean":"2024-09-21T19:00:39.657124+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T19:00:39.657124+0000","last_undegraded":"2024-09-21T19:00:39.657124+0000","last_fullsized":"2024-09-21T19:00:39.657124+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.954764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":214,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.666959+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T19:00:39.666959+0000","last_peered":"2024-09-21T19:00:39.666959+0000","last_clean":"2024-09-21T19:00:39.666959+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:00:39.666959+0000","last_undegraded":"2024-09-21T19:00:39.666959+0000","last_fullsized":"2024-09-21T19:00:39.666959+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.291673+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.687691+0000","last_change":"2024-09-21T18:59:17.117569+0000","last_active":"2024-09-21T19:00:43.687691+0000","last_peered":"2024-09-21T19:00:43.687691+0000","last_clean":"2024-09-21T19:00:43.687691+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T19:00:43.687691+0000","last_undegraded":"2024-09-21T19:00:43.687691+0000","last_fullsized":"2024-09-21T19:00:43.687691+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.692460+0000","last_change":"2024-09-21T18:59:59.934497+0000","last_active":"2024-09-21T19:00:43.692460+0000","last_peered":"2024-09-21T19:00:43.692460+0000","last_clean":"2024-09-21T19:00:43.692460+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:00:43.692460+0000","last_undegraded":"2024-09-21T19:00:43.692460+0000","last_fullsized":"2024-09-21T19:00:43.692460+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.934416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.934416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.394867+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":321,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.665553+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:00:39.665553+0000","last_peered":"2024-09-21T19:00:39.665553+0000","last_clean":"2024-09-21T19:00:39.665553+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:00:39.665553+0000","last_undegraded":"2024-09-21T19:00:39.665553+0000","last_fullsized":"2024-09-21T19:00:39.665553+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:21.108434+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":357,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.657326+0000","last_change":"2024-09-21T19:00:17.839077+0000","last_active":"2024-09-21T19:00:39.657326+0000","last_peered":"2024-09-21T19:00:39.657326+0000","last_clean":"2024-09-21T19:00:39.657326+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:00:39.657326+0000","last_undegraded":"2024-09-21T19:00:39.657326+0000","last_fullsized":"2024-09-21T19:00:39.657326+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:17.838903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:17.838903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.006877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":350,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-21T19:00:43.689465+0000","last_change":"2024-09-21T18:59:18.108977+0000","last_active":"2024-09-21T19:00:43.689465+0000","last_peered":"2024-09-21T19:00:43.689465+0000","last_clean":"2024-09-21T19:00:43.689465+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T19:00:43.689465+0000","last_undegraded":"2024-09-21T19:00:43.689465+0000","last_fullsized":"2024-09-21T19:00:43.689465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:18.108904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:18.108904+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.281496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":222,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-21T19:00:39.669985+0000","last_change":"2024-09-21T18:59:04.021340+0000","last_active":"2024-09-21T19:00:39.669985+0000","last_peered":"2024-09-21T19:00:39.669985+0000","last_clean":"2024-09-21T19:00:39.669985+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:00:39.669985+0000","last_undegraded":"2024-09-21T19:00:39.669985+0000","last_fullsized":"2024-09-21T19:00:39.669985+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:04.019934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:04.019934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.208611+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6969229336,"num_objects":106343,"num_object_clones":0,"num_object_copies":319029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106343,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158117,"num_write_kb":8462657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20801728512,"data_stored":20801716296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81117,"ondisk_log_size":81117,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":448,"num_read_kb":443,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266784,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2306992,"kb_used_data":2059232,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91438160,"statfs":{"total":95995035648,"available":93632675840,"internally_reserved":0,"allocated":2108653568,"data_stored":2108256482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528421,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2326564,"kb_used_data":2074644,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91418588,"statfs":{"total":95995035648,"available":93612634112,"internally_reserved":0,"allocated":2124435456,"data_stored":2124050205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790056,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3581224,"kb_used_data":3293208,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":90163928,"statfs":{"total":95995035648,"available":92327862272,"internally_reserved":0,"allocated":3372244992,"data_stored":3371856123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018990,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2333616,"kb_used_data":2081692,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91411536,"statfs":{"total":95995035648,"available":93605412864,"internally_reserved":0,"allocated":2131652608,"data_stored":2131259874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182514,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2965944,"kb_used_data":2609832,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":90779208,"statfs":{"total":95995035648,"available":92957908992,"internally_reserved":0,"allocated":2672467968,"data_stored":2672057016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364575178},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2909604,"kb_used_data":2639636,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90835548,"statfs":{"total":95995035648,"available":93015601152,"internally_reserved":0,"allocated":2702987264,"data_stored":2702596053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3405616,"kb_used_data":3059296,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90339536,"statfs":{"total":95995035648,"available":92507684864,"internally_reserved":0,"allocated":3132719104,"data_stored":3132315354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2990756,"kb_used_data":2634644,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":90754396,"statfs":{"total":95995035648,"available":92932501504,"internally_reserved":0,"allocated":2697875456,"data_stored":2697484330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2682454016,"data_stored":2682454016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3109355520,"data_stored":3109355520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2672955392,"data_stored":2672951320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2657746944,"data_stored":2657746944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2111045632,"data_stored":2111045632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3343781888,"data_stored":3343777816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2119368704,"data_stored":2119368704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2105020416,"data_stored":2105016344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:00:49.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:00:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:49.706 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:49.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:49.805 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:49.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:49.859 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:49.870 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:49.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:49.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:50.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:50.144 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:50.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:50.274 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:50.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:00:50.376 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:50.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:50.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:50.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:50.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:50.713 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:50.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:50 smithi081 ceph-mon[29384]: pgmap v360: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-21T19:00:50.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:50 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:00:50.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:50 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:00:50.790 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:50 smithi081 ceph-mon[39192]: pgmap v360: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-21T19:00:50.790 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:50 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:00:50.790 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:50 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:00:50.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:50.845 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:00:50.845 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:00:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:50.858 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:50.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:50.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:50 smithi104 ceph-mon[32438]: pgmap v360: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-21T19:00:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:50 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:00:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:50 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:00:51.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:51.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:51.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.299 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:51.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:51.374 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:00:51.374 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:00:51.374 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:00:51.374 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:00:51.374 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:00:51.375 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:00:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:00:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:00:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:00:51.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:51.434 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:51.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:51.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:51.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:51.710 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:51.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:51.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:51.843 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:51.945 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:51.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:51.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:52.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:51 smithi104 ceph-mon[32438]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T19:00:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:51 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/505109267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:51 smithi081 ceph-mon[39192]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T19:00:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:51 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/505109267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:51 smithi081 ceph-mon[29384]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T19:00:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:51 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/505109267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:00:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:00:52.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:52.117 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:52.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:00:52.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:52.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:52.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:52.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:52.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:52.541 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:52.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:52.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:52.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:52.814 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:52.818 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:52.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:52.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:52.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:53.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:52 smithi104 ceph-mon[32438]: pgmap v362: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T19:00:53.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:52 smithi081 ceph-mon[39192]: pgmap v362: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T19:00:53.010 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:52 smithi081 ceph-mon[29384]: pgmap v362: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T19:00:53.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:53.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:53.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:53.323 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:53.360 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:53.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:53.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:53.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:53.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:53.769 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:53.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:53.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:54.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:54.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:54.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:54.193 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:00:54.286 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:00:54.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:54.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:54.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:54.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:54.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:54.576 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:54.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.709 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:00:54.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:00:54.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:54.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:54.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:54.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:54.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:54 smithi104 ceph-mon[32438]: pgmap v363: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T19:00:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:54 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1408510203' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:55.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:54 smithi081 ceph-mon[29384]: pgmap v363: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T19:00:55.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:54 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1408510203' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:55.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:54 smithi081 ceph-mon[39192]: pgmap v363: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T19:00:55.015 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:54 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1408510203' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:00:55.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:172314 8.1 e215/172 8:8c27992b:::benchmark_data_smithi081_105538_object40082:head v 215'10771, mlcod=215'10769)", 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:00:55.092695+0000", 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.000202468, 2024-09-21T19:00:55.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022946800000000001, 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092695+0000", 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092695+0000", 2024-09-21T19:00:55.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092697+0000", 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5940000000000001e-06 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092707+0000", 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4560000000000006e-06 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:00:55.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092726+0000", 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9581000000000001e-05 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092735+0000", 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5890000000000003e-06 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092830+0000", 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5252000000000002e-05 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:00:55.106 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:00:55.092849+0000", 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.906e-05 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 1 2024-09-21T19:00:55.107 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:00:55.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:55.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:55.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:55.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:55.507 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:55.608 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:55.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:55.697 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-21T19:00:53.542788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6300113459,"num_objects":96323,"num_object_clones":0,"num_object_copies":288969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96323,"num_whiteouts":0,"num_read":1171,"num_read_kb":923,"num_write":168772,"num_write_kb":8462702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":6230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82284,"ondisk_log_size":82284,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21044544,"kb_used_data":18617288,"kb_used_omap":12,"kb_used_meta":2426163,"kb_avail":728916672,"statfs":{"total":767960285184,"available":746410672128,"internally_reserved":0,"allocated":19064102912,"data_stored":19061263697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":2484391476},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-669712384,"num_objects":-10219,"num_object_clones":0,"num_object_copies":-30657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10219,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":10219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.850014"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":230,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.724711+0000","last_change":"2024-09-21T18:59:50.144510+0000","last_active":"2024-09-21T19:00:50.724711+0000","last_peered":"2024-09-21T19:00:50.724711+0000","last_clean":"2024-09-21T19:00:50.724711+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:00:50.724711+0000","last_undegraded":"2024-09-21T19:00:50.724711+0000","last_fullsized":"2024-09-21T19:00:50.724711+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.144434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.144434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:15.699799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":371,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725865+0000","last_change":"2024-09-21T18:58:35.867068+0000","last_active":"2024-09-21T19:00:50.725865+0000","last_peered":"2024-09-21T19:00:50.725865+0000","last_clean":"2024-09-21T19:00:50.725865+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T19:00:50.725865+0000","last_undegraded":"2024-09-21T19:00:50.725865+0000","last_fullsized":"2024-09-21T19:00:50.725865+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:35.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:35.867013+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.705270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.735905+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:00:50.735905+0000","last_peered":"2024-09-21T19:00:50.735905+0000","last_clean":"2024-09-21T19:00:50.735905+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:00:50.735905+0000","last_undegraded":"2024-09-21T19:00:50.735905+0000","last_fullsized":"2024-09-21T19:00:50.735905+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.393494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":322,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726833+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T19:00:50.726833+0000","last_peered":"2024-09-21T19:00:50.726833+0000","last_clean":"2024-09-21T19:00:50.726833+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:00:50.726833+0000","last_undegraded":"2024-09-21T19:00:50.726833+0000","last_fullsized":"2024-09-21T19:00:50.726833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.765211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":331,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728210+0000","last_change":"2024-09-21T18:58:35.022059+0000","last_active":"2024-09-21T19:00:50.728210+0000","last_peered":"2024-09-21T19:00:50.728210+0000","last_clean":"2024-09-21T19:00:50.728210+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T19:00:50.728210+0000","last_undegraded":"2024-09-21T19:00:50.728210+0000","last_fullsized":"2024-09-21T19:00:50.728210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:35.022013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:35.022013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.887695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":343,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.730717+0000","last_change":"2024-09-21T18:58:36.978527+0000","last_active":"2024-09-21T19:00:50.730717+0000","last_peered":"2024-09-21T19:00:50.730717+0000","last_clean":"2024-09-21T19:00:50.730717+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T19:00:50.730717+0000","last_undegraded":"2024-09-21T19:00:50.730717+0000","last_fullsized":"2024-09-21T19:00:50.730717+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:36.978471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:36.978471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.858664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":402,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728499+0000","last_change":"2024-09-21T19:00:33.910621+0000","last_active":"2024-09-21T19:00:50.728499+0000","last_peered":"2024-09-21T19:00:50.728499+0000","last_clean":"2024-09-21T19:00:50.728499+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:00:50.728499+0000","last_undegraded":"2024-09-21T19:00:50.728499+0000","last_fullsized":"2024-09-21T19:00:50.728499+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:33.910535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.910535+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.545163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":334,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725812+0000","last_change":"2024-09-21T18:59:48.790074+0000","last_active":"2024-09-21T19:00:50.725812+0000","last_peered":"2024-09-21T19:00:50.725812+0000","last_clean":"2024-09-21T19:00:50.725812+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T19:00:50.725812+0000","last_undegraded":"2024-09-21T19:00:50.725812+0000","last_fullsized":"2024-09-21T19:00:50.725812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.789998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.789998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:00.025184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736797+0000","last_change":"2024-09-21T19:00:28.036188+0000","last_active":"2024-09-21T19:00:50.736797+0000","last_peered":"2024-09-21T19:00:50.736797+0000","last_clean":"2024-09-21T19:00:50.736797+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T19:00:50.736797+0000","last_undegraded":"2024-09-21T19:00:50.736797+0000","last_fullsized":"2024-09-21T19:00:50.736797+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:28.036123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.036123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.098067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":396,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727124+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:00:50.727124+0000","last_peered":"2024-09-21T19:00:50.727124+0000","last_clean":"2024-09-21T19:00:50.727124+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:00:50.727124+0000","last_undegraded":"2024-09-21T19:00:50.727124+0000","last_fullsized":"2024-09-21T19:00:50.727124+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:04.701591+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":356,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736436+0000","last_change":"2024-09-21T18:59:20.569305+0000","last_active":"2024-09-21T19:00:50.736436+0000","last_peered":"2024-09-21T19:00:50.736436+0000","last_clean":"2024-09-21T19:00:50.736436+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:00:50.736436+0000","last_undegraded":"2024-09-21T19:00:50.736436+0000","last_fullsized":"2024-09-21T19:00:50.736436+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:20.569235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.569235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723084+0000","last_change":"2024-09-21T18:59:51.189725+0000","last_active":"2024-09-21T19:00:50.723084+0000","last_peered":"2024-09-21T19:00:50.723084+0000","last_clean":"2024-09-21T19:00:50.723084+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:00:50.723084+0000","last_undegraded":"2024-09-21T19:00:50.723084+0000","last_fullsized":"2024-09-21T19:00:50.723084+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.189659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.189659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.202774+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.737221+0000","last_change":"2024-09-21T18:59:48.531625+0000","last_active":"2024-09-21T19:00:50.737221+0000","last_peered":"2024-09-21T19:00:50.737221+0000","last_clean":"2024-09-21T19:00:50.737221+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T19:00:50.737221+0000","last_undegraded":"2024-09-21T19:00:50.737221+0000","last_fullsized":"2024-09-21T19:00:50.737221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.531568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.531568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.572428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":348,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726631+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:00:50.726631+0000","last_peered":"2024-09-21T19:00:50.726631+0000","last_clean":"2024-09-21T19:00:50.726631+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:00:50.726631+0000","last_undegraded":"2024-09-21T19:00:50.726631+0000","last_fullsized":"2024-09-21T19:00:50.726631+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.764044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":333,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744765+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T19:00:50.744765+0000","last_peered":"2024-09-21T19:00:50.744765+0000","last_clean":"2024-09-21T19:00:50.744765+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:00:50.744765+0000","last_undegraded":"2024-09-21T19:00:50.744765+0000","last_fullsized":"2024-09-21T19:00:50.744765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.462353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":338,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736414+0000","last_change":"2024-09-21T18:59:57.720813+0000","last_active":"2024-09-21T19:00:50.736414+0000","last_peered":"2024-09-21T19:00:50.736414+0000","last_clean":"2024-09-21T19:00:50.736414+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T19:00:50.736414+0000","last_undegraded":"2024-09-21T19:00:50.736414+0000","last_fullsized":"2024-09-21T19:00:50.736414+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:57.720747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.720747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.304188+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726566+0000","last_change":"2024-09-21T18:58:32.943602+0000","last_active":"2024-09-21T19:00:50.726566+0000","last_peered":"2024-09-21T19:00:50.726566+0000","last_clean":"2024-09-21T19:00:50.726566+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T19:00:50.726566+0000","last_undegraded":"2024-09-21T19:00:50.726566+0000","last_fullsized":"2024-09-21T19:00:50.726566+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:32.943549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:32.943549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:10.666685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723849+0000","last_change":"2024-09-21T18:59:46.218922+0000","last_active":"2024-09-21T19:00:50.723849+0000","last_peered":"2024-09-21T19:00:50.723849+0000","last_clean":"2024-09-21T19:00:50.723849+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:00:50.723849+0000","last_undegraded":"2024-09-21T19:00:50.723849+0000","last_fullsized":"2024-09-21T19:00:50.723849+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.218852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.218852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.131850+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":371,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723805+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:00:50.723805+0000","last_peered":"2024-09-21T19:00:50.723805+0000","last_clean":"2024-09-21T19:00:50.723805+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:00:50.723805+0000","last_undegraded":"2024-09-21T19:00:50.723805+0000","last_fullsized":"2024-09-21T19:00:50.723805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.239922+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":356,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733779+0000","last_change":"2024-09-21T19:00:15.241392+0000","last_active":"2024-09-21T19:00:50.733779+0000","last_peered":"2024-09-21T19:00:50.733779+0000","last_clean":"2024-09-21T19:00:50.733779+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:00:50.733779+0000","last_undegraded":"2024-09-21T19:00:50.733779+0000","last_fullsized":"2024-09-21T19:00:50.733779+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:15.241320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:15.241320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.208748+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":323,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.730707+0000","last_change":"2024-09-21T18:59:19.930405+0000","last_active":"2024-09-21T19:00:50.730707+0000","last_peered":"2024-09-21T19:00:50.730707+0000","last_clean":"2024-09-21T19:00:50.730707+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T19:00:50.730707+0000","last_undegraded":"2024-09-21T19:00:50.730707+0000","last_fullsized":"2024-09-21T19:00:50.730707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.867337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":350,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727232+0000","last_change":"2024-09-21T19:00:00.649580+0000","last_active":"2024-09-21T19:00:50.727232+0000","last_peered":"2024-09-21T19:00:50.727232+0000","last_clean":"2024-09-21T19:00:50.727232+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:00:50.727232+0000","last_undegraded":"2024-09-21T19:00:50.727232+0000","last_fullsized":"2024-09-21T19:00:50.727232+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.649397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.649397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:24.015113+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":389,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729516+0000","last_change":"2024-09-21T19:00:18.857976+0000","last_active":"2024-09-21T19:00:50.729516+0000","last_peered":"2024-09-21T19:00:50.729516+0000","last_clean":"2024-09-21T19:00:50.729516+0000","last_became_active":"2024-09-21T18:58:23.993245+0000","last_became_peered":"2024-09-21T18:58:23.993245+0000","last_unstale":"2024-09-21T19:00:50.729516+0000","last_undegraded":"2024-09-21T19:00:50.729516+0000","last_fullsized":"2024-09-21T19:00:50.729516+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:00:18.857901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.857901+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.470039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":334,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726451+0000","last_change":"2024-09-21T18:58:28.912619+0000","last_active":"2024-09-21T19:00:50.726451+0000","last_peered":"2024-09-21T19:00:50.726451+0000","last_clean":"2024-09-21T19:00:50.726451+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:00:50.726451+0000","last_undegraded":"2024-09-21T19:00:50.726451+0000","last_fullsized":"2024-09-21T19:00:50.726451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:28.912554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:28.912554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":338,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745159+0000","last_change":"2024-09-21T18:58:38.110564+0000","last_active":"2024-09-21T19:00:50.745159+0000","last_peered":"2024-09-21T19:00:50.745159+0000","last_clean":"2024-09-21T19:00:50.745159+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:00:50.745159+0000","last_undegraded":"2024-09-21T19:00:50.745159+0000","last_fullsized":"2024-09-21T19:00:50.745159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:58:38.110509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:38.110509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.724266+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":354,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.735945+0000","last_change":"2024-09-21T19:00:23.223251+0000","last_active":"2024-09-21T19:00:50.735945+0000","last_peered":"2024-09-21T19:00:50.735945+0000","last_clean":"2024-09-21T19:00:50.735945+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:00:50.735945+0000","last_undegraded":"2024-09-21T19:00:50.735945+0000","last_fullsized":"2024-09-21T19:00:50.735945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:23.223202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.223202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:35.046166+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":242,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.724950+0000","last_change":"2024-09-21T19:00:22.256583+0000","last_active":"2024-09-21T19:00:50.724950+0000","last_peered":"2024-09-21T19:00:50.724950+0000","last_clean":"2024-09-21T19:00:50.724950+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:00:50.724950+0000","last_undegraded":"2024-09-21T19:00:50.724950+0000","last_fullsized":"2024-09-21T19:00:50.724950+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:22.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.256498+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.316225+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":445,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728617+0000","last_change":"2024-09-21T19:00:35.363014+0000","last_active":"2024-09-21T19:00:50.728617+0000","last_peered":"2024-09-21T19:00:50.728617+0000","last_clean":"2024-09-21T19:00:50.728617+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T19:00:50.728617+0000","last_undegraded":"2024-09-21T19:00:50.728617+0000","last_fullsized":"2024-09-21T19:00:50.728617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.428300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"214'10433","reported_seq":10528,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.983906+0000","last_change":"2024-09-21T18:58:44.098981+0000","last_active":"2024-09-21T19:00:51.983906+0000","last_peered":"2024-09-21T19:00:51.983906+0000","last_clean":"2024-09-21T19:00:51.983906+0000","last_became_active":"2024-09-21T18:58:44.098681+0000","last_became_peered":"2024-09-21T18:58:44.098681+0000","last_unstale":"2024-09-21T19:00:51.983906+0000","last_undegraded":"2024-09-21T19:00:51.983906+0000","last_fullsized":"2024-09-21T19:00:51.983906+0000","mapping_epoch":172,"log_start":"191'5400","ondisk_log_start":"191'5400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.272947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386859008,"num_objects":5903,"num_object_clones":0,"num_object_copies":17709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10433,"num_write_kb":522752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":107,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736362+0000","last_change":"2024-09-21T18:59:46.479423+0000","last_active":"2024-09-21T19:00:50.736362+0000","last_peered":"2024-09-21T19:00:50.736362+0000","last_clean":"2024-09-21T19:00:50.736362+0000","last_became_active":"2024-09-21T18:58:40.917818+0000","last_became_peered":"2024-09-21T18:58:40.917818+0000","last_unstale":"2024-09-21T19:00:50.736362+0000","last_undegraded":"2024-09-21T19:00:50.736362+0000","last_fullsized":"2024-09-21T19:00:50.736362+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.479379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.479379+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":363,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.735204+0000","last_change":"2024-09-21T18:59:21.369411+0000","last_active":"2024-09-21T19:00:50.735204+0000","last_peered":"2024-09-21T19:00:50.735204+0000","last_clean":"2024-09-21T19:00:50.735204+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:00:50.735204+0000","last_undegraded":"2024-09-21T19:00:50.735204+0000","last_fullsized":"2024-09-21T19:00:50.735204+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:59:21.369328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.369328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.377767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":351,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.732361+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:00:50.732361+0000","last_peered":"2024-09-21T19:00:50.732361+0000","last_clean":"2024-09-21T19:00:50.732361+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:00:50.732361+0000","last_undegraded":"2024-09-21T19:00:50.732361+0000","last_fullsized":"2024-09-21T19:00:50.732361+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:17.164534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":457,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726660+0000","last_change":"2024-09-21T18:59:54.654178+0000","last_active":"2024-09-21T19:00:50.726660+0000","last_peered":"2024-09-21T19:00:50.726660+0000","last_clean":"2024-09-21T19:00:50.726660+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:00:50.726660+0000","last_undegraded":"2024-09-21T19:00:50.726660+0000","last_fullsized":"2024-09-21T19:00:50.726660+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:59:54.654112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.654112+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:58.509872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":352,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736525+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T19:00:50.736525+0000","last_peered":"2024-09-21T19:00:50.736525+0000","last_clean":"2024-09-21T19:00:50.736525+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T19:00:50.736525+0000","last_undegraded":"2024-09-21T19:00:50.736525+0000","last_fullsized":"2024-09-21T19:00:50.736525+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.873896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"214'10615","reported_seq":10785,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.802258+0000","last_change":"2024-09-21T18:59:57.437732+0000","last_active":"2024-09-21T19:00:51.802258+0000","last_peered":"2024-09-21T19:00:51.802258+0000","last_clean":"2024-09-21T19:00:51.802258+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:00:51.802258+0000","last_undegraded":"2024-09-21T19:00:51.802258+0000","last_fullsized":"2024-09-21T19:00:51.802258+0000","mapping_epoch":172,"log_start":"190'5100","ondisk_log_start":"190'5100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'6593","last_scrub_stamp":"2024-09-21T18:59:57.437417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.437417+0000","objects_scrubbed":6510,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.144644+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398917632,"num_objects":6087,"num_object_clones":0,"num_object_copies":18261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":534464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":107,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727188+0000","last_change":"2024-09-21T18:59:47.823699+0000","last_active":"2024-09-21T19:00:50.727188+0000","last_peered":"2024-09-21T19:00:50.727188+0000","last_clean":"2024-09-21T19:00:50.727188+0000","last_became_active":"2024-09-21T18:58:40.916167+0000","last_became_peered":"2024-09-21T18:58:40.916167+0000","last_unstale":"2024-09-21T19:00:50.727188+0000","last_undegraded":"2024-09-21T19:00:50.727188+0000","last_fullsized":"2024-09-21T19:00:50.727188+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.823624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.823624+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:16.932400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":421,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.732527+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:00:50.732527+0000","last_peered":"2024-09-21T19:00:50.732527+0000","last_clean":"2024-09-21T19:00:50.732527+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:00:50.732527+0000","last_undegraded":"2024-09-21T19:00:50.732527+0000","last_fullsized":"2024-09-21T19:00:50.732527+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.575046+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":357,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.731197+0000","last_change":"2024-09-21T18:59:48.973683+0000","last_active":"2024-09-21T19:00:50.731197+0000","last_peered":"2024-09-21T19:00:50.731197+0000","last_clean":"2024-09-21T19:00:50.731197+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:00:50.731197+0000","last_undegraded":"2024-09-21T19:00:50.731197+0000","last_fullsized":"2024-09-21T19:00:50.731197+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.973616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.973616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.516500+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":334,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726078+0000","last_change":"2024-09-21T18:59:06.911709+0000","last_active":"2024-09-21T19:00:50.726078+0000","last_peered":"2024-09-21T19:00:50.726078+0000","last_clean":"2024-09-21T19:00:50.726078+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:00:50.726078+0000","last_undegraded":"2024-09-21T19:00:50.726078+0000","last_fullsized":"2024-09-21T19:00:50.726078+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.911638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.911638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.735119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":322,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727036+0000","last_change":"2024-09-21T18:59:13.953988+0000","last_active":"2024-09-21T19:00:50.727036+0000","last_peered":"2024-09-21T19:00:50.727036+0000","last_clean":"2024-09-21T19:00:50.727036+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:00:50.727036+0000","last_undegraded":"2024-09-21T19:00:50.727036+0000","last_fullsized":"2024-09-21T19:00:50.727036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:13.953912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:13.953912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"214'10539","reported_seq":10723,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.508747+0000","last_change":"2024-09-21T19:00:18.040765+0000","last_active":"2024-09-21T19:00:51.508747+0000","last_peered":"2024-09-21T19:00:51.508747+0000","last_clean":"2024-09-21T19:00:51.508747+0000","last_became_active":"2024-09-21T18:58:44.184359+0000","last_became_peered":"2024-09-21T18:58:44.184359+0000","last_unstale":"2024-09-21T19:00:51.508747+0000","last_undegraded":"2024-09-21T19:00:51.508747+0000","last_fullsized":"2024-09-21T19:00:51.508747+0000","mapping_epoch":172,"log_start":"191'5600","ondisk_log_start":"191'5600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7943","last_scrub_stamp":"2024-09-21T19:00:18.040678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.040678+0000","objects_scrubbed":7868,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.698500+0000","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396492824,"num_objects":6051,"num_object_clones":0,"num_object_copies":18153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10539,"num_write_kb":530817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":112,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729531+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:00:50.729531+0000","last_peered":"2024-09-21T19:00:50.729531+0000","last_clean":"2024-09-21T19:00:50.729531+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:00:50.729531+0000","last_undegraded":"2024-09-21T19:00:50.729531+0000","last_fullsized":"2024-09-21T19:00:50.729531+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:49.951550+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":359,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726062+0000","last_change":"2024-09-21T18:59:20.963196+0000","last_active":"2024-09-21T19:00:50.726062+0000","last_peered":"2024-09-21T19:00:50.726062+0000","last_clean":"2024-09-21T19:00:50.726062+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T19:00:50.726062+0000","last_undegraded":"2024-09-21T19:00:50.726062+0000","last_fullsized":"2024-09-21T19:00:50.726062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:59:20.962753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.962753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":234,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725104+0000","last_change":"2024-09-21T18:59:54.225013+0000","last_active":"2024-09-21T19:00:50.725104+0000","last_peered":"2024-09-21T19:00:50.725104+0000","last_clean":"2024-09-21T19:00:50.725104+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:00:50.725104+0000","last_undegraded":"2024-09-21T19:00:50.725104+0000","last_fullsized":"2024-09-21T19:00:50.725104+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:54.224944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.224944+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.892818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":428,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.732024+0000","last_change":"2024-09-21T19:00:22.845024+0000","last_active":"2024-09-21T19:00:50.732024+0000","last_peered":"2024-09-21T19:00:50.732024+0000","last_clean":"2024-09-21T19:00:50.732024+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:00:50.732024+0000","last_undegraded":"2024-09-21T19:00:50.732024+0000","last_fullsized":"2024-09-21T19:00:50.732024+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:22.844953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.844953+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:26.214253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":321,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744914+0000","last_change":"2024-09-21T18:58:00.240553+0000","last_active":"2024-09-21T19:00:50.744914+0000","last_peered":"2024-09-21T19:00:50.744914+0000","last_clean":"2024-09-21T19:00:50.744914+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T19:00:50.744914+0000","last_undegraded":"2024-09-21T19:00:50.744914+0000","last_fullsized":"2024-09-21T19:00:50.744914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.240476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.240476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:05.824646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"214'10581","reported_seq":10674,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.801842+0000","last_change":"2024-09-21T18:58:44.097039+0000","last_active":"2024-09-21T19:00:51.801842+0000","last_peered":"2024-09-21T19:00:51.801842+0000","last_clean":"2024-09-21T19:00:51.801842+0000","last_became_active":"2024-09-21T18:58:44.096773+0000","last_became_peered":"2024-09-21T18:58:44.096773+0000","last_unstale":"2024-09-21T19:00:51.801842+0000","last_undegraded":"2024-09-21T19:00:51.801842+0000","last_fullsized":"2024-09-21T19:00:51.801842+0000","mapping_epoch":172,"log_start":"190'5100","ondisk_log_start":"190'5100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.241385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396034048,"num_objects":6043,"num_object_clones":0,"num_object_copies":18129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10581,"num_write_kb":531968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":191,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728540+0000","last_change":"2024-09-21T18:59:46.185760+0000","last_active":"2024-09-21T19:00:50.728540+0000","last_peered":"2024-09-21T19:00:50.728540+0000","last_clean":"2024-09-21T19:00:50.728540+0000","last_became_active":"2024-09-21T18:58:40.530131+0000","last_became_peered":"2024-09-21T18:58:40.530131+0000","last_unstale":"2024-09-21T19:00:50.728540+0000","last_undegraded":"2024-09-21T19:00:50.728540+0000","last_fullsized":"2024-09-21T19:00:50.728540+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.185693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.185693+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.587905+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":345,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733863+0000","last_change":"2024-09-21T18:59:17.346990+0000","last_active":"2024-09-21T19:00:50.733863+0000","last_peered":"2024-09-21T19:00:50.733863+0000","last_clean":"2024-09-21T19:00:50.733863+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:00:50.733863+0000","last_undegraded":"2024-09-21T19:00:50.733863+0000","last_fullsized":"2024-09-21T19:00:50.733863+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:17.346414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.346414+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.396410+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":350,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736881+0000","last_change":"2024-09-21T18:59:16.589986+0000","last_active":"2024-09-21T19:00:50.736881+0000","last_peered":"2024-09-21T19:00:50.736881+0000","last_clean":"2024-09-21T19:00:50.736881+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T19:00:50.736881+0000","last_undegraded":"2024-09-21T19:00:50.736881+0000","last_fullsized":"2024-09-21T19:00:50.736881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.589912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.589912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.916495+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":278,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723705+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T19:00:50.723705+0000","last_peered":"2024-09-21T19:00:50.723705+0000","last_clean":"2024-09-21T19:00:50.723705+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T19:00:50.723705+0000","last_undegraded":"2024-09-21T19:00:50.723705+0000","last_fullsized":"2024-09-21T19:00:50.723705+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.258988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723229+0000","last_change":"2024-09-21T18:59:44.256590+0000","last_active":"2024-09-21T19:00:50.723229+0000","last_peered":"2024-09-21T19:00:50.723229+0000","last_clean":"2024-09-21T19:00:50.723229+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:00:50.723229+0000","last_undegraded":"2024-09-21T19:00:50.723229+0000","last_fullsized":"2024-09-21T19:00:50.723229+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:44.256513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.256513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"214'10608","reported_seq":10703,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:52.309970+0000","last_change":"2024-09-21T18:58:44.097045+0000","last_active":"2024-09-21T19:00:52.309970+0000","last_peered":"2024-09-21T19:00:52.309970+0000","last_clean":"2024-09-21T19:00:52.309970+0000","last_became_active":"2024-09-21T18:58:44.096634+0000","last_became_peered":"2024-09-21T18:58:44.096634+0000","last_unstale":"2024-09-21T19:00:52.309970+0000","last_undegraded":"2024-09-21T19:00:52.309970+0000","last_fullsized":"2024-09-21T19:00:52.309970+0000","mapping_epoch":172,"log_start":"194'6100","ondisk_log_start":"194'6100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.889384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390070272,"num_objects":5952,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":191,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728615+0000","last_change":"2024-09-21T18:59:41.296627+0000","last_active":"2024-09-21T19:00:50.728615+0000","last_peered":"2024-09-21T19:00:50.728615+0000","last_clean":"2024-09-21T19:00:50.728615+0000","last_became_active":"2024-09-21T18:58:40.528964+0000","last_became_peered":"2024-09-21T18:58:40.528964+0000","last_unstale":"2024-09-21T19:00:50.728615+0000","last_undegraded":"2024-09-21T19:00:50.728615+0000","last_fullsized":"2024-09-21T19:00:50.728615+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:41.296558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:41.296558+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:59.947419+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":248,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723896+0000","last_change":"2024-09-21T18:59:48.165996+0000","last_active":"2024-09-21T19:00:50.723896+0000","last_peered":"2024-09-21T19:00:50.723896+0000","last_clean":"2024-09-21T19:00:50.723896+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:00:50.723896+0000","last_undegraded":"2024-09-21T19:00:50.723896+0000","last_fullsized":"2024-09-21T19:00:50.723896+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T18:59:48.165907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.165907+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:55.224625+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.734482+0000","last_change":"2024-09-21T18:59:54.420959+0000","last_active":"2024-09-21T19:00:50.734482+0000","last_peered":"2024-09-21T19:00:50.734482+0000","last_clean":"2024-09-21T19:00:50.734482+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:00:50.734482+0000","last_undegraded":"2024-09-21T19:00:50.734482+0000","last_fullsized":"2024-09-21T19:00:50.734482+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:54.420888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.420888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:54.568945+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":347,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728180+0000","last_change":"2024-09-21T19:00:34.409013+0000","last_active":"2024-09-21T19:00:50.728180+0000","last_peered":"2024-09-21T19:00:50.728180+0000","last_clean":"2024-09-21T19:00:50.728180+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T19:00:50.728180+0000","last_undegraded":"2024-09-21T19:00:50.728180+0000","last_fullsized":"2024-09-21T19:00:50.728180+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.816307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.735855+0000","last_change":"2024-09-21T19:00:25.249425+0000","last_active":"2024-09-21T19:00:50.735855+0000","last_peered":"2024-09-21T19:00:50.735855+0000","last_clean":"2024-09-21T19:00:50.735855+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T19:00:50.735855+0000","last_undegraded":"2024-09-21T19:00:50.735855+0000","last_fullsized":"2024-09-21T19:00:50.735855+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.249360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.249360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:32.354078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"214'10177","reported_seq":10272,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.528116+0000","last_change":"2024-09-21T18:58:44.183202+0000","last_active":"2024-09-21T19:00:51.528116+0000","last_peered":"2024-09-21T19:00:51.528116+0000","last_clean":"2024-09-21T19:00:51.528116+0000","last_became_active":"2024-09-21T18:58:44.182910+0000","last_became_peered":"2024-09-21T18:58:44.182910+0000","last_unstale":"2024-09-21T19:00:51.528116+0000","last_undegraded":"2024-09-21T19:00:51.528116+0000","last_fullsized":"2024-09-21T19:00:51.528116+0000","mapping_epoch":172,"log_start":"191'5200","ondisk_log_start":"191'5200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.830014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386203648,"num_objects":5893,"num_object_clones":0,"num_object_copies":17679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10177,"num_write_kb":514240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":107,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736279+0000","last_change":"2024-09-21T18:59:41.588294+0000","last_active":"2024-09-21T19:00:50.736279+0000","last_peered":"2024-09-21T19:00:50.736279+0000","last_clean":"2024-09-21T19:00:50.736279+0000","last_became_active":"2024-09-21T18:58:40.916397+0000","last_became_peered":"2024-09-21T18:58:40.916397+0000","last_unstale":"2024-09-21T19:00:50.736279+0000","last_undegraded":"2024-09-21T19:00:50.736279+0000","last_fullsized":"2024-09-21T19:00:50.736279+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:41.588231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:41.588231+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":479,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.734341+0000","last_change":"2024-09-21T18:59:44.360729+0000","last_active":"2024-09-21T19:00:50.734341+0000","last_peered":"2024-09-21T19:00:50.734341+0000","last_clean":"2024-09-21T19:00:50.734341+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:00:50.734341+0000","last_undegraded":"2024-09-21T19:00:50.734341+0000","last_fullsized":"2024-09-21T19:00:50.734341+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:59:44.360619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.360619+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.396002+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":364,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727113+0000","last_change":"2024-09-21T18:59:55.645493+0000","last_active":"2024-09-21T19:00:50.727113+0000","last_peered":"2024-09-21T19:00:50.727113+0000","last_clean":"2024-09-21T19:00:50.727113+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T19:00:50.727113+0000","last_undegraded":"2024-09-21T19:00:50.727113+0000","last_fullsized":"2024-09-21T19:00:50.727113+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:55.645419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.645419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:19.840924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":360,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.737101+0000","last_change":"2024-09-21T18:59:19.567802+0000","last_active":"2024-09-21T19:00:50.737101+0000","last_peered":"2024-09-21T19:00:50.737101+0000","last_clean":"2024-09-21T19:00:50.737101+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T19:00:50.737101+0000","last_undegraded":"2024-09-21T19:00:50.737101+0000","last_fullsized":"2024-09-21T19:00:50.737101+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T18:59:19.567649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.567649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.724299+0000","last_change":"2024-09-21T19:00:25.316275+0000","last_active":"2024-09-21T19:00:50.724299+0000","last_peered":"2024-09-21T19:00:50.724299+0000","last_clean":"2024-09-21T19:00:50.724299+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:00:50.724299+0000","last_undegraded":"2024-09-21T19:00:50.724299+0000","last_fullsized":"2024-09-21T19:00:50.724299+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.314718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.314718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.602505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":331,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.730430+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:00:50.730430+0000","last_peered":"2024-09-21T19:00:50.730430+0000","last_clean":"2024-09-21T19:00:50.730430+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:00:50.730430+0000","last_undegraded":"2024-09-21T19:00:50.730430+0000","last_fullsized":"2024-09-21T19:00:50.730430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:05.377423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"214'10699","reported_seq":10794,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:52.347121+0000","last_change":"2024-09-21T18:58:44.100975+0000","last_active":"2024-09-21T19:00:52.347121+0000","last_peered":"2024-09-21T19:00:52.347121+0000","last_clean":"2024-09-21T19:00:52.347121+0000","last_became_active":"2024-09-21T18:58:44.100676+0000","last_became_peered":"2024-09-21T18:58:44.100676+0000","last_unstale":"2024-09-21T19:00:52.347121+0000","last_undegraded":"2024-09-21T19:00:52.347121+0000","last_fullsized":"2024-09-21T19:00:52.347121+0000","mapping_epoch":172,"log_start":"194'6200","ondisk_log_start":"194'6200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.859120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392757248,"num_objects":5993,"num_object_clones":0,"num_object_copies":17979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10699,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":420,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744647+0000","last_change":"2024-09-21T18:59:49.845480+0000","last_active":"2024-09-21T19:00:50.744647+0000","last_peered":"2024-09-21T19:00:50.744647+0000","last_clean":"2024-09-21T19:00:50.744647+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:00:50.744647+0000","last_undegraded":"2024-09-21T19:00:50.744647+0000","last_fullsized":"2024-09-21T19:00:50.744647+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:49.845397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.845397+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:01.762366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":309,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725470+0000","last_change":"2024-09-21T18:59:48.817751+0000","last_active":"2024-09-21T19:00:50.725470+0000","last_peered":"2024-09-21T19:00:50.725470+0000","last_clean":"2024-09-21T19:00:50.725470+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:00:50.725470+0000","last_undegraded":"2024-09-21T19:00:50.725470+0000","last_fullsized":"2024-09-21T19:00:50.725470+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T18:59:48.817656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.817656+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:59.596604+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":348,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733721+0000","last_change":"2024-09-21T18:59:50.378784+0000","last_active":"2024-09-21T19:00:50.733721+0000","last_peered":"2024-09-21T19:00:50.733721+0000","last_clean":"2024-09-21T19:00:50.733721+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T19:00:50.733721+0000","last_undegraded":"2024-09-21T19:00:50.733721+0000","last_fullsized":"2024-09-21T19:00:50.733721+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:50.378581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.378581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:57.041830+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":389,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726207+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:00:50.726207+0000","last_peered":"2024-09-21T19:00:50.726207+0000","last_clean":"2024-09-21T19:00:50.726207+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:00:50.726207+0000","last_undegraded":"2024-09-21T19:00:50.726207+0000","last_fullsized":"2024-09-21T19:00:50.726207+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.686451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727561+0000","last_change":"2024-09-21T19:00:29.002389+0000","last_active":"2024-09-21T19:00:50.727561+0000","last_peered":"2024-09-21T19:00:50.727561+0000","last_clean":"2024-09-21T19:00:50.727561+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:00:50.727561+0000","last_undegraded":"2024-09-21T19:00:50.727561+0000","last_fullsized":"2024-09-21T19:00:50.727561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.002321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.002321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:56.121924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":333,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745430+0000","last_change":"2024-09-21T19:00:32.573881+0000","last_active":"2024-09-21T19:00:50.745430+0000","last_peered":"2024-09-21T19:00:50.745430+0000","last_clean":"2024-09-21T19:00:50.745430+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T19:00:50.745430+0000","last_undegraded":"2024-09-21T19:00:50.745430+0000","last_fullsized":"2024-09-21T19:00:50.745430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:00:32.573704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.573704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.007025+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"214'10465","reported_seq":10631,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.960557+0000","last_change":"2024-09-21T18:59:55.688614+0000","last_active":"2024-09-21T19:00:51.960557+0000","last_peered":"2024-09-21T19:00:51.960557+0000","last_clean":"2024-09-21T19:00:51.960557+0000","last_became_active":"2024-09-21T18:58:44.099439+0000","last_became_peered":"2024-09-21T18:58:44.099439+0000","last_unstale":"2024-09-21T19:00:51.960557+0000","last_undegraded":"2024-09-21T19:00:51.960557+0000","last_fullsized":"2024-09-21T19:00:51.960557+0000","mapping_epoch":172,"log_start":"191'5400","ondisk_log_start":"191'5400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'6308","last_scrub_stamp":"2024-09-21T18:59:55.688429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.688429+0000","objects_scrubbed":6231,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:56.502643+0000","scrub_duration":2287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387907584,"num_objects":5919,"num_object_clones":0,"num_object_copies":17757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10465,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":107,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.735022+0000","last_change":"2024-09-21T18:59:42.333738+0000","last_active":"2024-09-21T19:00:50.735022+0000","last_peered":"2024-09-21T19:00:50.735022+0000","last_clean":"2024-09-21T19:00:50.735022+0000","last_became_active":"2024-09-21T18:58:40.528545+0000","last_became_peered":"2024-09-21T18:58:40.528545+0000","last_unstale":"2024-09-21T19:00:50.735022+0000","last_undegraded":"2024-09-21T19:00:50.735022+0000","last_fullsized":"2024-09-21T19:00:50.735022+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:42.333677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.333677+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:55.111757+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":244,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725009+0000","last_change":"2024-09-21T18:59:53.225904+0000","last_active":"2024-09-21T19:00:50.725009+0000","last_peered":"2024-09-21T19:00:50.725009+0000","last_clean":"2024-09-21T19:00:50.725009+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:00:50.725009+0000","last_undegraded":"2024-09-21T19:00:50.725009+0000","last_fullsized":"2024-09-21T19:00:50.725009+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:53.225823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.225823+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:05.051041+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":230,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726713+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:00:50.726713+0000","last_peered":"2024-09-21T19:00:50.726713+0000","last_clean":"2024-09-21T19:00:50.726713+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:00:50.726713+0000","last_undegraded":"2024-09-21T19:00:50.726713+0000","last_fullsized":"2024-09-21T19:00:50.726713+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.948040+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":832,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744559+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T19:00:50.744559+0000","last_peered":"2024-09-21T19:00:50.744559+0000","last_clean":"2024-09-21T19:00:50.744559+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:00:50.744559+0000","last_undegraded":"2024-09-21T19:00:50.744559+0000","last_fullsized":"2024-09-21T19:00:50.744559+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.500330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":453,"num_read_kb":448,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":364,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.737024+0000","last_change":"2024-09-21T18:59:21.571613+0000","last_active":"2024-09-21T19:00:50.737024+0000","last_peered":"2024-09-21T19:00:50.737024+0000","last_clean":"2024-09-21T19:00:50.737024+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:00:50.737024+0000","last_undegraded":"2024-09-21T19:00:50.737024+0000","last_fullsized":"2024-09-21T19:00:50.737024+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:21.571551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.571551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733216+0000","last_change":"2024-09-21T19:00:33.443101+0000","last_active":"2024-09-21T19:00:50.733216+0000","last_peered":"2024-09-21T19:00:50.733216+0000","last_clean":"2024-09-21T19:00:50.733216+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:00:50.733216+0000","last_undegraded":"2024-09-21T19:00:50.733216+0000","last_fullsized":"2024-09-21T19:00:50.733216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:33.443040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.443040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.013096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"214'10707","reported_seq":10877,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.549785+0000","last_change":"2024-09-21T19:00:00.245865+0000","last_active":"2024-09-21T19:00:51.549785+0000","last_peered":"2024-09-21T19:00:51.549785+0000","last_clean":"2024-09-21T19:00:51.549785+0000","last_became_active":"2024-09-21T18:58:44.096509+0000","last_became_peered":"2024-09-21T18:58:44.096509+0000","last_unstale":"2024-09-21T19:00:51.549785+0000","last_undegraded":"2024-09-21T19:00:51.549785+0000","last_fullsized":"2024-09-21T19:00:51.549785+0000","mapping_epoch":172,"log_start":"190'5100","ondisk_log_start":"190'5100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'6787","last_scrub_stamp":"2024-09-21T19:00:00.245707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.245707+0000","objects_scrubbed":6706,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.145095+0000","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402456576,"num_objects":6141,"num_object_clones":0,"num_object_copies":18423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":452,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729155+0000","last_change":"2024-09-21T18:59:12.162986+0000","last_active":"2024-09-21T19:00:50.729155+0000","last_peered":"2024-09-21T19:00:50.729155+0000","last_clean":"2024-09-21T19:00:50.729155+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T19:00:50.729155+0000","last_undegraded":"2024-09-21T19:00:50.729155+0000","last_fullsized":"2024-09-21T19:00:50.729155+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:59:12.162725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.162725+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.317843+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":343,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728823+0000","last_change":"2024-09-21T18:59:50.993922+0000","last_active":"2024-09-21T19:00:50.728823+0000","last_peered":"2024-09-21T19:00:50.728823+0000","last_clean":"2024-09-21T19:00:50.728823+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T19:00:50.728823+0000","last_undegraded":"2024-09-21T19:00:50.728823+0000","last_fullsized":"2024-09-21T19:00:50.728823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.712596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":414,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744599+0000","last_change":"2024-09-21T18:59:44.783805+0000","last_active":"2024-09-21T19:00:50.744599+0000","last_peered":"2024-09-21T19:00:50.744599+0000","last_clean":"2024-09-21T19:00:50.744599+0000","last_became_active":"2024-09-21T18:58:39.058347+0000","last_became_peered":"2024-09-21T18:58:39.058347+0000","last_unstale":"2024-09-21T19:00:50.744599+0000","last_undegraded":"2024-09-21T19:00:50.744599+0000","last_fullsized":"2024-09-21T19:00:50.744599+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:44.783735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.783735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:56.772817+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":333,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745261+0000","last_change":"2024-09-21T19:00:00.889227+0000","last_active":"2024-09-21T19:00:50.745261+0000","last_peered":"2024-09-21T19:00:50.745261+0000","last_clean":"2024-09-21T19:00:50.745261+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:00:50.745261+0000","last_undegraded":"2024-09-21T19:00:50.745261+0000","last_fullsized":"2024-09-21T19:00:50.745261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.889156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.889156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:16.434600+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736963+0000","last_change":"2024-09-21T18:58:30.445025+0000","last_active":"2024-09-21T19:00:50.736963+0000","last_peered":"2024-09-21T19:00:50.736963+0000","last_clean":"2024-09-21T19:00:50.736963+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T19:00:50.736963+0000","last_undegraded":"2024-09-21T19:00:50.736963+0000","last_fullsized":"2024-09-21T19:00:50.736963+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:30.444954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:30.444954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.851328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"214'10539","reported_seq":10634,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.530705+0000","last_change":"2024-09-21T18:58:44.097275+0000","last_active":"2024-09-21T19:00:51.530705+0000","last_peered":"2024-09-21T19:00:51.530705+0000","last_clean":"2024-09-21T19:00:51.530705+0000","last_became_active":"2024-09-21T18:58:44.096986+0000","last_became_peered":"2024-09-21T18:58:44.096986+0000","last_unstale":"2024-09-21T19:00:51.530705+0000","last_undegraded":"2024-09-21T19:00:51.530705+0000","last_fullsized":"2024-09-21T19:00:51.530705+0000","mapping_epoch":172,"log_start":"192'5900","ondisk_log_start":"192'5900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.628491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394723328,"num_objects":6023,"num_object_clones":0,"num_object_copies":18069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":365,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726762+0000","last_change":"2024-09-21T19:00:01.693318+0000","last_active":"2024-09-21T19:00:50.726762+0000","last_peered":"2024-09-21T19:00:50.726762+0000","last_clean":"2024-09-21T19:00:50.726762+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T19:00:50.726762+0000","last_undegraded":"2024-09-21T19:00:50.726762+0000","last_fullsized":"2024-09-21T19:00:50.726762+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:01.693221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.693221+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.644648+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726264+0000","last_change":"2024-09-21T19:00:27.721451+0000","last_active":"2024-09-21T19:00:50.726264+0000","last_peered":"2024-09-21T19:00:50.726264+0000","last_clean":"2024-09-21T19:00:50.726264+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:00:50.726264+0000","last_undegraded":"2024-09-21T19:00:50.726264+0000","last_fullsized":"2024-09-21T19:00:50.726264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:27.721385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:27.721385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.383225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":350,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736868+0000","last_change":"2024-09-21T18:59:52.640497+0000","last_active":"2024-09-21T19:00:50.736868+0000","last_peered":"2024-09-21T19:00:50.736868+0000","last_clean":"2024-09-21T19:00:50.736868+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T19:00:50.736868+0000","last_undegraded":"2024-09-21T19:00:50.736868+0000","last_fullsized":"2024-09-21T19:00:50.736868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.640432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.640432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:53.254498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":315,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729626+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T19:00:50.729626+0000","last_peered":"2024-09-21T19:00:50.729626+0000","last_clean":"2024-09-21T19:00:50.729626+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T19:00:50.729626+0000","last_undegraded":"2024-09-21T19:00:50.729626+0000","last_fullsized":"2024-09-21T19:00:50.729626+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.918918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"214'10534","reported_seq":10718,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.952491+0000","last_change":"2024-09-21T19:00:19.801173+0000","last_active":"2024-09-21T19:00:51.952491+0000","last_peered":"2024-09-21T19:00:51.952491+0000","last_clean":"2024-09-21T19:00:51.952491+0000","last_became_active":"2024-09-21T18:58:44.097827+0000","last_became_peered":"2024-09-21T18:58:44.097827+0000","last_unstale":"2024-09-21T19:00:51.952491+0000","last_undegraded":"2024-09-21T19:00:51.952491+0000","last_fullsized":"2024-09-21T19:00:51.952491+0000","mapping_epoch":172,"log_start":"191'5500","ondisk_log_start":"191'5500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8051","last_scrub_stamp":"2024-09-21T19:00:19.800992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.800992+0000","objects_scrubbed":7979,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.556183+0000","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394395648,"num_objects":6018,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":529664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":114,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736538+0000","last_change":"2024-09-21T18:59:47.518847+0000","last_active":"2024-09-21T19:00:50.736538+0000","last_peered":"2024-09-21T19:00:50.736538+0000","last_clean":"2024-09-21T19:00:50.736538+0000","last_became_active":"2024-09-21T18:58:40.917793+0000","last_became_peered":"2024-09-21T18:58:40.917793+0000","last_unstale":"2024-09-21T19:00:50.736538+0000","last_undegraded":"2024-09-21T19:00:50.736538+0000","last_fullsized":"2024-09-21T19:00:50.736538+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.518777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.518777+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.628241+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":366,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.727051+0000","last_change":"2024-09-21T18:59:03.883391+0000","last_active":"2024-09-21T19:00:50.727051+0000","last_peered":"2024-09-21T19:00:50.727051+0000","last_clean":"2024-09-21T19:00:50.727051+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:00:50.727051+0000","last_undegraded":"2024-09-21T19:00:50.727051+0000","last_fullsized":"2024-09-21T19:00:50.727051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:59:03.883306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:03.883306+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.725718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":341,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728558+0000","last_change":"2024-09-21T18:59:06.194777+0000","last_active":"2024-09-21T19:00:50.728558+0000","last_peered":"2024-09-21T19:00:50.728558+0000","last_clean":"2024-09-21T19:00:50.728558+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T19:00:50.728558+0000","last_undegraded":"2024-09-21T19:00:50.728558+0000","last_fullsized":"2024-09-21T19:00:50.728558+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.194722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.194722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.287924+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":340,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733306+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T19:00:50.733306+0000","last_peered":"2024-09-21T19:00:50.733306+0000","last_clean":"2024-09-21T19:00:50.733306+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:00:50.733306+0000","last_undegraded":"2024-09-21T19:00:50.733306+0000","last_fullsized":"2024-09-21T19:00:50.733306+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.313883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725623+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:00:50.725623+0000","last_peered":"2024-09-21T19:00:50.725623+0000","last_clean":"2024-09-21T19:00:50.725623+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:00:50.725623+0000","last_undegraded":"2024-09-21T19:00:50.725623+0000","last_fullsized":"2024-09-21T19:00:50.725623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"214'10490","reported_seq":10586,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:52.341011+0000","last_change":"2024-09-21T18:58:44.096415+0000","last_active":"2024-09-21T19:00:52.341011+0000","last_peered":"2024-09-21T19:00:52.341011+0000","last_clean":"2024-09-21T19:00:52.341011+0000","last_became_active":"2024-09-21T18:58:44.096142+0000","last_became_peered":"2024-09-21T18:58:44.096142+0000","last_unstale":"2024-09-21T19:00:52.341011+0000","last_undegraded":"2024-09-21T19:00:52.341011+0000","last_fullsized":"2024-09-21T19:00:52.341011+0000","mapping_epoch":172,"log_start":"194'6000","ondisk_log_start":"194'6000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:07.897929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220608,"num_objects":5878,"num_object_clones":0,"num_object_copies":17634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10490,"num_write_kb":523776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":420,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744542+0000","last_change":"2024-09-21T18:59:50.870015+0000","last_active":"2024-09-21T19:00:50.744542+0000","last_peered":"2024-09-21T19:00:50.744542+0000","last_clean":"2024-09-21T19:00:50.744542+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:00:50.744542+0000","last_undegraded":"2024-09-21T19:00:50.744542+0000","last_fullsized":"2024-09-21T19:00:50.744542+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:50.869945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.869945+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:07.279297+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":353,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726545+0000","last_change":"2024-09-21T18:59:53.699627+0000","last_active":"2024-09-21T19:00:50.726545+0000","last_peered":"2024-09-21T19:00:50.726545+0000","last_clean":"2024-09-21T19:00:50.726545+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T19:00:50.726545+0000","last_undegraded":"2024-09-21T19:00:50.726545+0000","last_fullsized":"2024-09-21T19:00:50.726545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.929430+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":371,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729239+0000","last_change":"2024-09-21T18:58:30.050910+0000","last_active":"2024-09-21T19:00:50.729239+0000","last_peered":"2024-09-21T19:00:50.729239+0000","last_clean":"2024-09-21T19:00:50.729239+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:00:50.729239+0000","last_undegraded":"2024-09-21T19:00:50.729239+0000","last_fullsized":"2024-09-21T19:00:50.729239+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:58:30.050859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:30.050859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.904611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":451,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725162+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:00:50.725162+0000","last_peered":"2024-09-21T19:00:50.725162+0000","last_clean":"2024-09-21T19:00:50.725162+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:00:50.725162+0000","last_undegraded":"2024-09-21T19:00:50.725162+0000","last_fullsized":"2024-09-21T19:00:50.725162+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.071423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726471+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:00:50.726471+0000","last_peered":"2024-09-21T19:00:50.726471+0000","last_clean":"2024-09-21T19:00:50.726471+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:00:50.726471+0000","last_undegraded":"2024-09-21T19:00:50.726471+0000","last_fullsized":"2024-09-21T19:00:50.726471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.791557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"214'10574","reported_seq":10669,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.962546+0000","last_change":"2024-09-21T18:58:44.101043+0000","last_active":"2024-09-21T19:00:51.962546+0000","last_peered":"2024-09-21T19:00:51.962546+0000","last_clean":"2024-09-21T19:00:51.962546+0000","last_became_active":"2024-09-21T18:58:44.100559+0000","last_became_peered":"2024-09-21T18:58:44.100559+0000","last_unstale":"2024-09-21T19:00:51.962546+0000","last_undegraded":"2024-09-21T19:00:51.962546+0000","last_fullsized":"2024-09-21T19:00:51.962546+0000","mapping_epoch":172,"log_start":"191'5500","ondisk_log_start":"191'5500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402128896,"num_objects":6136,"num_object_clones":0,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10574,"num_write_kb":534720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":224,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723990+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T19:00:50.723990+0000","last_peered":"2024-09-21T19:00:50.723990+0000","last_clean":"2024-09-21T19:00:50.723990+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:00:50.723990+0000","last_undegraded":"2024-09-21T19:00:50.723990+0000","last_fullsized":"2024-09-21T19:00:50.723990+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.261047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":350,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.737257+0000","last_change":"2024-09-21T18:59:15.560811+0000","last_active":"2024-09-21T19:00:50.737257+0000","last_peered":"2024-09-21T19:00:50.737257+0000","last_clean":"2024-09-21T19:00:50.737257+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T19:00:50.737257+0000","last_undegraded":"2024-09-21T19:00:50.737257+0000","last_fullsized":"2024-09-21T19:00:50.737257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:15.560713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:15.560713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":334,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725606+0000","last_change":"2024-09-21T18:59:43.841286+0000","last_active":"2024-09-21T19:00:50.725606+0000","last_peered":"2024-09-21T19:00:50.725606+0000","last_clean":"2024-09-21T19:00:50.725606+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:00:50.725606+0000","last_undegraded":"2024-09-21T19:00:50.725606+0000","last_fullsized":"2024-09-21T19:00:50.725606+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:43.841222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:43.841222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726634+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:00:50.726634+0000","last_peered":"2024-09-21T19:00:50.726634+0000","last_clean":"2024-09-21T19:00:50.726634+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:00:50.726634+0000","last_undegraded":"2024-09-21T19:00:50.726634+0000","last_fullsized":"2024-09-21T19:00:50.726634+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.293410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"214'10478","reported_seq":10643,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.750871+0000","last_change":"2024-09-21T18:59:54.271364+0000","last_active":"2024-09-21T19:00:51.750871+0000","last_peered":"2024-09-21T19:00:51.750871+0000","last_clean":"2024-09-21T19:00:51.750871+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:00:51.750871+0000","last_undegraded":"2024-09-21T19:00:51.750871+0000","last_fullsized":"2024-09-21T19:00:51.750871+0000","mapping_epoch":172,"log_start":"190'5000","ondisk_log_start":"190'5000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'6188","last_scrub_stamp":"2024-09-21T18:59:54.271275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.271275+0000","objects_scrubbed":6120,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:55.967625+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391249920,"num_objects":5970,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10478,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":368,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728405+0000","last_change":"2024-09-21T19:00:23.328177+0000","last_active":"2024-09-21T19:00:50.728405+0000","last_peered":"2024-09-21T19:00:50.728405+0000","last_clean":"2024-09-21T19:00:50.728405+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T19:00:50.728405+0000","last_undegraded":"2024-09-21T19:00:50.728405+0000","last_fullsized":"2024-09-21T19:00:50.728405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.465067+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":341,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745636+0000","last_change":"2024-09-21T18:58:37.143097+0000","last_active":"2024-09-21T19:00:50.745636+0000","last_peered":"2024-09-21T19:00:50.745636+0000","last_clean":"2024-09-21T19:00:50.745636+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T19:00:50.745636+0000","last_undegraded":"2024-09-21T19:00:50.745636+0000","last_fullsized":"2024-09-21T19:00:50.745636+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:37.143050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:37.143050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.764335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":348,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723643+0000","last_change":"2024-09-21T19:00:30.996323+0000","last_active":"2024-09-21T19:00:50.723643+0000","last_peered":"2024-09-21T19:00:50.723643+0000","last_clean":"2024-09-21T19:00:50.723643+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:00:50.723643+0000","last_undegraded":"2024-09-21T19:00:50.723643+0000","last_fullsized":"2024-09-21T19:00:50.723643+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:30.996260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:30.996260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:34.407198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":329,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744505+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:00:50.744505+0000","last_peered":"2024-09-21T19:00:50.744505+0000","last_clean":"2024-09-21T19:00:50.744505+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:00:50.744505+0000","last_undegraded":"2024-09-21T19:00:50.744505+0000","last_fullsized":"2024-09-21T19:00:50.744505+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"214'10507","reported_seq":10672,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.531220+0000","last_change":"2024-09-21T18:59:52.633622+0000","last_active":"2024-09-21T19:00:51.531220+0000","last_peered":"2024-09-21T19:00:51.531220+0000","last_clean":"2024-09-21T19:00:51.531220+0000","last_became_active":"2024-09-21T18:58:44.184048+0000","last_became_peered":"2024-09-21T18:58:44.184048+0000","last_unstale":"2024-09-21T19:00:51.531220+0000","last_undegraded":"2024-09-21T19:00:51.531220+0000","last_fullsized":"2024-09-21T19:00:51.531220+0000","mapping_epoch":172,"log_start":"190'4900","ondisk_log_start":"190'4900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'6240","last_scrub_stamp":"2024-09-21T18:59:52.633386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.633386+0000","objects_scrubbed":6189,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:00.395113+0000","scrub_duration":1379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400228352,"num_objects":6107,"num_object_clones":0,"num_object_copies":18321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10507,"num_write_kb":531648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":380,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726339+0000","last_change":"2024-09-21T18:59:17.862166+0000","last_active":"2024-09-21T19:00:50.726339+0000","last_peered":"2024-09-21T19:00:50.726339+0000","last_clean":"2024-09-21T19:00:50.726339+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:00:50.726339+0000","last_undegraded":"2024-09-21T19:00:50.726339+0000","last_fullsized":"2024-09-21T19:00:50.726339+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:17.862105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.862105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.784245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":349,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.746067+0000","last_change":"2024-09-21T19:00:22.616468+0000","last_active":"2024-09-21T19:00:50.746067+0000","last_peered":"2024-09-21T19:00:50.746067+0000","last_clean":"2024-09-21T19:00:50.746067+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:00:50.746067+0000","last_undegraded":"2024-09-21T19:00:50.746067+0000","last_fullsized":"2024-09-21T19:00:50.746067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:22.616386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.616386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:52.426205+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":356,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.724372+0000","last_change":"2024-09-21T18:59:16.539031+0000","last_active":"2024-09-21T19:00:50.724372+0000","last_peered":"2024-09-21T19:00:50.724372+0000","last_clean":"2024-09-21T19:00:50.724372+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:00:50.724372+0000","last_undegraded":"2024-09-21T19:00:50.724372+0000","last_fullsized":"2024-09-21T19:00:50.724372+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.538853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.538853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.903925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":323,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728944+0000","last_change":"2024-09-21T19:00:34.960755+0000","last_active":"2024-09-21T19:00:50.728944+0000","last_peered":"2024-09-21T19:00:50.728944+0000","last_clean":"2024-09-21T19:00:50.728944+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:00:50.728944+0000","last_undegraded":"2024-09-21T19:00:50.728944+0000","last_fullsized":"2024-09-21T19:00:50.728944+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.960689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.960689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:57.667040+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"214'10390","reported_seq":10573,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:51.547948+0000","last_change":"2024-09-21T19:00:21.263588+0000","last_active":"2024-09-21T19:00:51.547948+0000","last_peered":"2024-09-21T19:00:51.547948+0000","last_clean":"2024-09-21T19:00:51.547948+0000","last_became_active":"2024-09-21T18:58:44.094697+0000","last_became_peered":"2024-09-21T18:58:44.094697+0000","last_unstale":"2024-09-21T19:00:51.547948+0000","last_undegraded":"2024-09-21T19:00:51.547948+0000","last_fullsized":"2024-09-21T19:00:51.547948+0000","mapping_epoch":172,"log_start":"191'5400","ondisk_log_start":"191'5400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8114","last_scrub_stamp":"2024-09-21T19:00:21.263245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.263245+0000","objects_scrubbed":8036,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.729679+0000","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393871360,"num_objects":6010,"num_object_clones":0,"num_object_copies":18030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":371,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736573+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T19:00:50.736573+0000","last_peered":"2024-09-21T19:00:50.736573+0000","last_clean":"2024-09-21T19:00:50.736573+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T19:00:50.736573+0000","last_undegraded":"2024-09-21T19:00:50.736573+0000","last_fullsized":"2024-09-21T19:00:50.736573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.863281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":371,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745676+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T19:00:50.745676+0000","last_peered":"2024-09-21T19:00:50.745676+0000","last_clean":"2024-09-21T19:00:50.745676+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:00:50.745676+0000","last_undegraded":"2024-09-21T19:00:50.745676+0000","last_fullsized":"2024-09-21T19:00:50.745676+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.427939+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":445,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744480+0000","last_change":"2024-09-21T18:59:56.831306+0000","last_active":"2024-09-21T19:00:50.744480+0000","last_peered":"2024-09-21T19:00:50.744480+0000","last_clean":"2024-09-21T19:00:50.744480+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T19:00:50.744480+0000","last_undegraded":"2024-09-21T19:00:50.744480+0000","last_fullsized":"2024-09-21T19:00:50.744480+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:59:56.831225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.831225+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:06.230252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":352,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736895+0000","last_change":"2024-09-21T18:59:59.767058+0000","last_active":"2024-09-21T19:00:50.736895+0000","last_peered":"2024-09-21T19:00:50.736895+0000","last_clean":"2024-09-21T19:00:50.736895+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T19:00:50.736895+0000","last_undegraded":"2024-09-21T19:00:50.736895+0000","last_fullsized":"2024-09-21T19:00:50.736895+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.766985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.766985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:00.540870+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":355,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728364+0000","last_change":"2024-09-21T18:59:14.180429+0000","last_active":"2024-09-21T19:00:50.728364+0000","last_peered":"2024-09-21T19:00:50.728364+0000","last_clean":"2024-09-21T19:00:50.728364+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T19:00:50.728364+0000","last_undegraded":"2024-09-21T19:00:50.728364+0000","last_fullsized":"2024-09-21T19:00:50.728364+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.387472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":348,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.730971+0000","last_change":"2024-09-21T18:59:59.958853+0000","last_active":"2024-09-21T19:00:50.730971+0000","last_peered":"2024-09-21T19:00:50.730971+0000","last_clean":"2024-09-21T19:00:50.730971+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:00:50.730971+0000","last_undegraded":"2024-09-21T19:00:50.730971+0000","last_fullsized":"2024-09-21T19:00:50.730971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T18:59:59.958609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.958609+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:01.002394+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":350,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736656+0000","last_change":"2024-09-21T19:00:29.955243+0000","last_active":"2024-09-21T19:00:50.736656+0000","last_peered":"2024-09-21T19:00:50.736656+0000","last_clean":"2024-09-21T19:00:50.736656+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T19:00:50.736656+0000","last_undegraded":"2024-09-21T19:00:50.736656+0000","last_fullsized":"2024-09-21T19:00:50.736656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.955184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.955184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.019605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":321,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745393+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T19:00:50.745393+0000","last_peered":"2024-09-21T19:00:50.745393+0000","last_clean":"2024-09-21T19:00:50.745393+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T19:00:50.745393+0000","last_undegraded":"2024-09-21T19:00:50.745393+0000","last_fullsized":"2024-09-21T19:00:50.745393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.437166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":232,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725038+0000","last_change":"2024-09-21T19:00:34.345923+0000","last_active":"2024-09-21T19:00:50.725038+0000","last_peered":"2024-09-21T19:00:50.725038+0000","last_clean":"2024-09-21T19:00:50.725038+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:00:50.725038+0000","last_undegraded":"2024-09-21T19:00:50.725038+0000","last_fullsized":"2024-09-21T19:00:50.725038+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:00:34.345792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.345792+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:58.340045+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":341,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744652+0000","last_change":"2024-09-21T19:00:29.586470+0000","last_active":"2024-09-21T19:00:50.744652+0000","last_peered":"2024-09-21T19:00:50.744652+0000","last_clean":"2024-09-21T19:00:50.744652+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T19:00:50.744652+0000","last_undegraded":"2024-09-21T19:00:50.744652+0000","last_fullsized":"2024-09-21T19:00:50.744652+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.586411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.586411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.139362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":335,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.730788+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:00:50.730788+0000","last_peered":"2024-09-21T19:00:50.730788+0000","last_clean":"2024-09-21T19:00:50.730788+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:00:50.730788+0000","last_undegraded":"2024-09-21T19:00:50.730788+0000","last_fullsized":"2024-09-21T19:00:50.730788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.620068+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723365+0000","last_change":"2024-09-21T18:59:47.825900+0000","last_active":"2024-09-21T19:00:50.723365+0000","last_peered":"2024-09-21T19:00:50.723365+0000","last_clean":"2024-09-21T19:00:50.723365+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:00:50.723365+0000","last_undegraded":"2024-09-21T19:00:50.723365+0000","last_fullsized":"2024-09-21T19:00:50.723365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:47.825835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.825835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.012332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":354,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736792+0000","last_change":"2024-09-21T18:58:58.524782+0000","last_active":"2024-09-21T19:00:50.736792+0000","last_peered":"2024-09-21T19:00:50.736792+0000","last_clean":"2024-09-21T19:00:50.736792+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T19:00:50.736792+0000","last_undegraded":"2024-09-21T19:00:50.736792+0000","last_fullsized":"2024-09-21T19:00:50.736792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:58.524708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.524708+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.871526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.724621+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T19:00:50.724621+0000","last_peered":"2024-09-21T19:00:50.724621+0000","last_clean":"2024-09-21T19:00:50.724621+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:00:50.724621+0000","last_undegraded":"2024-09-21T19:00:50.724621+0000","last_fullsized":"2024-09-21T19:00:50.724621+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.288280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":444,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729774+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:00:50.729774+0000","last_peered":"2024-09-21T19:00:50.729774+0000","last_clean":"2024-09-21T19:00:50.729774+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:00:50.729774+0000","last_undegraded":"2024-09-21T19:00:50.729774+0000","last_fullsized":"2024-09-21T19:00:50.729774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:22.257098+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726398+0000","last_change":"2024-09-21T19:00:32.753965+0000","last_active":"2024-09-21T19:00:50.726398+0000","last_peered":"2024-09-21T19:00:50.726398+0000","last_clean":"2024-09-21T19:00:50.726398+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:00:50.726398+0000","last_undegraded":"2024-09-21T19:00:50.726398+0000","last_fullsized":"2024-09-21T19:00:50.726398+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:32.753902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.753902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.288209+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":346,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.730016+0000","last_change":"2024-09-21T19:00:28.893623+0000","last_active":"2024-09-21T19:00:50.730016+0000","last_peered":"2024-09-21T19:00:50.730016+0000","last_clean":"2024-09-21T19:00:50.730016+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:00:50.730016+0000","last_undegraded":"2024-09-21T19:00:50.730016+0000","last_fullsized":"2024-09-21T19:00:50.730016+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:28.893403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.893403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.904866+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":341,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745061+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:00:50.745061+0000","last_peered":"2024-09-21T19:00:50.745061+0000","last_clean":"2024-09-21T19:00:50.745061+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:00:50.745061+0000","last_undegraded":"2024-09-21T19:00:50.745061+0000","last_fullsized":"2024-09-21T19:00:50.745061+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.507018+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":449,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736684+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T19:00:50.736684+0000","last_peered":"2024-09-21T19:00:50.736684+0000","last_clean":"2024-09-21T19:00:50.736684+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T19:00:50.736684+0000","last_undegraded":"2024-09-21T19:00:50.736684+0000","last_fullsized":"2024-09-21T19:00:50.736684+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.883299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725228+0000","last_change":"2024-09-21T18:59:45.212437+0000","last_active":"2024-09-21T19:00:50.725228+0000","last_peered":"2024-09-21T19:00:50.725228+0000","last_clean":"2024-09-21T19:00:50.725228+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:00:50.725228+0000","last_undegraded":"2024-09-21T19:00:50.725228+0000","last_fullsized":"2024-09-21T19:00:50.725228+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.212362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.212362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":376,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726730+0000","last_change":"2024-09-21T18:59:56.667414+0000","last_active":"2024-09-21T19:00:50.726730+0000","last_peered":"2024-09-21T19:00:50.726730+0000","last_clean":"2024-09-21T19:00:50.726730+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T19:00:50.726730+0000","last_undegraded":"2024-09-21T19:00:50.726730+0000","last_fullsized":"2024-09-21T19:00:50.726730+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:56.667316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.667316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:03.881734+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.726743+0000","last_change":"2024-09-21T19:00:34.746168+0000","last_active":"2024-09-21T19:00:50.726743+0000","last_peered":"2024-09-21T19:00:50.726743+0000","last_clean":"2024-09-21T19:00:50.726743+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T19:00:50.726743+0000","last_undegraded":"2024-09-21T19:00:50.726743+0000","last_fullsized":"2024-09-21T19:00:50.726743+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.746126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.746126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733725+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:00:50.733725+0000","last_peered":"2024-09-21T19:00:50.733725+0000","last_clean":"2024-09-21T19:00:50.733725+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:00:50.733725+0000","last_undegraded":"2024-09-21T19:00:50.733725+0000","last_fullsized":"2024-09-21T19:00:50.733725+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:00:59.880814+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":337,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728164+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T19:00:50.728164+0000","last_peered":"2024-09-21T19:00:50.728164+0000","last_clean":"2024-09-21T19:00:50.728164+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:00:50.728164+0000","last_undegraded":"2024-09-21T19:00:50.728164+0000","last_fullsized":"2024-09-21T19:00:50.728164+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.894710+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":362,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.734278+0000","last_change":"2024-09-21T18:58:59.379970+0000","last_active":"2024-09-21T19:00:50.734278+0000","last_peered":"2024-09-21T19:00:50.734278+0000","last_clean":"2024-09-21T19:00:50.734278+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T19:00:50.734278+0000","last_undegraded":"2024-09-21T19:00:50.734278+0000","last_fullsized":"2024-09-21T19:00:50.734278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:59.379849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:59.379849+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.273644+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":349,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.745552+0000","last_change":"2024-09-21T18:58:34.182624+0000","last_active":"2024-09-21T19:00:50.745552+0000","last_peered":"2024-09-21T19:00:50.745552+0000","last_clean":"2024-09-21T19:00:50.745552+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:00:50.745552+0000","last_undegraded":"2024-09-21T19:00:50.745552+0000","last_fullsized":"2024-09-21T19:00:50.745552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:34.182578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:34.182578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.687690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":349,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729031+0000","last_change":"2024-09-21T18:58:58.952448+0000","last_active":"2024-09-21T19:00:50.729031+0000","last_peered":"2024-09-21T19:00:50.729031+0000","last_clean":"2024-09-21T19:00:50.729031+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:00:50.729031+0000","last_undegraded":"2024-09-21T19:00:50.729031+0000","last_fullsized":"2024-09-21T19:00:50.729031+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:58.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.895529+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":230,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.723313+0000","last_change":"2024-09-21T18:59:42.223794+0000","last_active":"2024-09-21T19:00:50.723313+0000","last_peered":"2024-09-21T19:00:50.723313+0000","last_clean":"2024-09-21T19:00:50.723313+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:00:50.723313+0000","last_undegraded":"2024-09-21T19:00:50.723313+0000","last_fullsized":"2024-09-21T19:00:50.723313+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:42.223731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.223731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.039045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":358,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728349+0000","last_change":"2024-09-21T19:00:00.939564+0000","last_active":"2024-09-21T19:00:50.728349+0000","last_peered":"2024-09-21T19:00:50.728349+0000","last_clean":"2024-09-21T19:00:50.728349+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T19:00:50.728349+0000","last_undegraded":"2024-09-21T19:00:50.728349+0000","last_fullsized":"2024-09-21T19:00:50.728349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:20.583806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.736684+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:00:50.736684+0000","last_peered":"2024-09-21T19:00:50.736684+0000","last_clean":"2024-09-21T19:00:50.736684+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:00:50.736684+0000","last_undegraded":"2024-09-21T19:00:50.736684+0000","last_fullsized":"2024-09-21T19:00:50.736684+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.872793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":341,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744287+0000","last_change":"2024-09-21T19:00:16.651104+0000","last_active":"2024-09-21T19:00:50.744287+0000","last_peered":"2024-09-21T19:00:50.744287+0000","last_clean":"2024-09-21T19:00:50.744287+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T19:00:50.744287+0000","last_undegraded":"2024-09-21T19:00:50.744287+0000","last_fullsized":"2024-09-21T19:00:50.744287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:16.650906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:16.650906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.021875+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":345,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.729160+0000","last_change":"2024-09-21T18:59:58.945778+0000","last_active":"2024-09-21T19:00:50.729160+0000","last_peered":"2024-09-21T19:00:50.729160+0000","last_clean":"2024-09-21T19:00:50.729160+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T19:00:50.729160+0000","last_undegraded":"2024-09-21T19:00:50.729160+0000","last_fullsized":"2024-09-21T19:00:50.729160+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:58.945631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:58.945631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:10.452414+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.732108+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T19:00:50.732108+0000","last_peered":"2024-09-21T19:00:50.732108+0000","last_clean":"2024-09-21T19:00:50.732108+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T19:00:50.732108+0000","last_undegraded":"2024-09-21T19:00:50.732108+0000","last_fullsized":"2024-09-21T19:00:50.732108+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.954764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":222,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.724568+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T19:00:50.724568+0000","last_peered":"2024-09-21T19:00:50.724568+0000","last_clean":"2024-09-21T19:00:50.724568+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:00:50.724568+0000","last_undegraded":"2024-09-21T19:00:50.724568+0000","last_fullsized":"2024-09-21T19:00:50.724568+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.291673+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":333,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728251+0000","last_change":"2024-09-21T18:59:17.117569+0000","last_active":"2024-09-21T19:00:50.728251+0000","last_peered":"2024-09-21T19:00:50.728251+0000","last_clean":"2024-09-21T19:00:50.728251+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T19:00:50.728251+0000","last_undegraded":"2024-09-21T19:00:50.728251+0000","last_fullsized":"2024-09-21T19:00:50.728251+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.359379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":329,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.744739+0000","last_change":"2024-09-21T18:59:59.934497+0000","last_active":"2024-09-21T19:00:50.744739+0000","last_peered":"2024-09-21T19:00:50.744739+0000","last_clean":"2024-09-21T19:00:50.744739+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:00:50.744739+0000","last_undegraded":"2024-09-21T19:00:50.744739+0000","last_fullsized":"2024-09-21T19:00:50.744739+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.934416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.934416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:02.394867+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.733560+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:00:50.733560+0000","last_peered":"2024-09-21T19:00:50.733560+0000","last_clean":"2024-09-21T19:00:50.733560+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:00:50.733560+0000","last_undegraded":"2024-09-21T19:00:50.733560+0000","last_fullsized":"2024-09-21T19:00:50.733560+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:21.108434+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":365,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.732283+0000","last_change":"2024-09-21T19:00:17.839077+0000","last_active":"2024-09-21T19:00:50.732283+0000","last_peered":"2024-09-21T19:00:50.732283+0000","last_clean":"2024-09-21T19:00:50.732283+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:00:50.732283+0000","last_undegraded":"2024-09-21T19:00:50.732283+0000","last_fullsized":"2024-09-21T19:00:50.732283+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:17.838903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:17.838903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.006877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":356,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.728646+0000","last_change":"2024-09-21T18:59:18.108977+0000","last_active":"2024-09-21T19:00:50.728646+0000","last_peered":"2024-09-21T19:00:50.728646+0000","last_clean":"2024-09-21T19:00:50.728646+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T19:00:50.728646+0000","last_undegraded":"2024-09-21T19:00:50.728646+0000","last_fullsized":"2024-09-21T19:00:50.728646+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:18.108904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:18.108904+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.281496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":230,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-21T19:00:50.725195+0000","last_change":"2024-09-21T18:59:04.021340+0000","last_active":"2024-09-21T19:00:50.725195+0000","last_peered":"2024-09-21T19:00:50.725195+0000","last_clean":"2024-09-21T19:00:50.725195+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:00:50.725195+0000","last_undegraded":"2024-09-21T19:00:50.725195+0000","last_fullsized":"2024-09-21T19:00:50.725195+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:04.019934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:04.019934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.208611+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6299516952,"num_objects":96124,"num_object_clones":0,"num_object_copies":288372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168336,"num_write_kb":8462657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18922156032,"data_stored":18922143816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80936,"ondisk_log_size":80936,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":453,"num_read_kb":448,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266785,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2187608,"kb_used_data":1931392,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91557544,"statfs":{"total":95995035648,"available":93754925056,"internally_reserved":0,"allocated":1977745408,"data_stored":1977391190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":262207946},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528422,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2210380,"kb_used_data":1950004,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":91534772,"statfs":{"total":95995035648,"available":93731606528,"internally_reserved":0,"allocated":1996804096,"data_stored":1996461713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":266467782},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790057,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3380880,"kb_used_data":3096888,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":90364272,"statfs":{"total":95995035648,"available":92533014528,"internally_reserved":0,"allocated":3171213312,"data_stored":3170867311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018991,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2202644,"kb_used_data":1958908,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91542508,"statfs":{"total":95995035648,"available":93739528192,"internally_reserved":0,"allocated":2005921792,"data_stored":2005571926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":249428422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182516,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2681508,"kb_used_data":2307412,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91063644,"statfs":{"total":95995035648,"available":93249171456,"internally_reserved":0,"allocated":2362789888,"data_stored":2362416509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2603032,"kb_used_data":2332928,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91142120,"statfs":{"total":95995035648,"available":93329530880,"internally_reserved":0,"allocated":2388918272,"data_stored":2388564634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3050276,"kb_used_data":2703820,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90694876,"statfs":{"total":95995035648,"available":92871553024,"internally_reserved":0,"allocated":2768711680,"data_stored":2768345503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2728216,"kb_used_data":2335936,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91016936,"statfs":{"total":95995035648,"available":93201342464,"internally_reserved":0,"allocated":2391998464,"data_stored":2391644911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2375221248,"data_stored":2375221248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2745827328,"data_stored":2745827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2362445824,"data_stored":2362441752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2346188800,"data_stored":2346188800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1984167936,"data_stored":1984167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3143176192,"data_stored":3143172120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1988820992,"data_stored":1988820992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1976307712,"data_stored":1976303640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:00:55.703 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T19:00:55.703 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:00:55.703 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T19:00:55.703 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T19:00:55.703 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T19:00:55.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-21T19:00:55.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:55.771 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:55.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:55.932 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:55.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:55 smithi104 ceph-mon[32438]: from='client.25405 v1:172.21.15.81:0/1807569549' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:00:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:55 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1408510203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:00:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:55 smithi104 ceph-mon[32438]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T19:00:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:55 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T19:00:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:55 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T19:00:56.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[29384]: from='client.25405 v1:172.21.15.81:0/1807569549' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1408510203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[29384]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[39192]: from='client.25405 v1:172.21.15.81:0/1807569549' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:00:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1408510203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:00:56.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[39192]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T19:00:56.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T19:00:56.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:55 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T19:00:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.075 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:56.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:56.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:56.217 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:56.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:56.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:56.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:56.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:56.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:56.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:56.769 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:56.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:56.902 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:56.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:57.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:57.047 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:57.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:56 smithi081 ceph-mon[29384]: pgmap v365: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 805 op/s 2024-09-21T19:00:57.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:56 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T19:00:57.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:56 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T19:00:57.059 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:56 smithi081 ceph-mon[39192]: pgmap v365: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 805 op/s 2024-09-21T19:00:57.060 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:56 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T19:00:57.060 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:56 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T19:00:57.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:57.188 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:56 smithi104 ceph-mon[32438]: pgmap v365: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 805 op/s 2024-09-21T19:00:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:56 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T19:00:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:56 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T19:00:57.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:57.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:00:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:00:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:57.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:57.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:57.457 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:57.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:57.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:57.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:00:57.723 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:00:57.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:00:57.856 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:57.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:00:57.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:58.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[29384]: 3.4 scrub starts 2024-09-21T19:00:58.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[29384]: 3.4 scrub ok 2024-09-21T19:00:58.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T19:00:58.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T19:00:58.028 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[39192]: 3.4 scrub starts 2024-09-21T19:00:58.029 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[39192]: 3.4 scrub ok 2024-09-21T19:00:58.029 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T19:00:58.029 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:57 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T19:00:58.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:00:58.031 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:58.044 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:58.045 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:00:58.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:57 smithi104 ceph-mon[32438]: 3.4 scrub starts 2024-09-21T19:00:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:57 smithi104 ceph-mon[32438]: 3.4 scrub ok 2024-09-21T19:00:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:57 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T19:00:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:57 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T19:00:58.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:00:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:00:58.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:58.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:58.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:58.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:58.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:00:58.588 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:00:58.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:00:58.725 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:00:58.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:58.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[29384]: pgmap v366: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[29384]: 6.16 scrub starts 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[29384]: 6.16 scrub ok 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[39192]: pgmap v366: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:00:58.871 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:00:58.872 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:58.872 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[39192]: 6.16 scrub starts 2024-09-21T19:00:58.872 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:58 smithi081 ceph-mon[39192]: 6.16 scrub ok 2024-09-21T19:00:58.933 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:00:58.933 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T19:00:58.933 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:00:58.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:59.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:59.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:00:59.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.165 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:59.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:58 smithi104 ceph-mon[32438]: pgmap v366: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-21T19:00:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:58 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:00:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:58 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:00:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:00:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:58 smithi104 ceph-mon[32438]: 6.16 scrub starts 2024-09-21T19:00:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:58 smithi104 ceph-mon[32438]: 6.16 scrub ok 2024-09-21T19:00:59.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:00:59.287 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:00:59.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:00:59.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:00:59.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:59.560 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:59.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:00:59.697 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:00:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:00:59.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:59.830 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:00:59.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:00:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:00:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:00:59.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:00.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:59 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T19:01:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:00:59 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:59 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:00:59 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T19:01:00.088 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:00.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:00.220 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:00.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:59 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T19:01:00.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:00:59 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T19:01:00.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:00.356 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:00.415 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:00.415 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:00.415 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:00.415 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:00.416 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:00.421 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:00.455 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:00.456 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:00.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:00.489 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:00.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:00.621 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:00.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:00.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:00.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:00.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:00.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:00.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:01.020 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: pgmap v367: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1782598099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T19:01:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: pgmap v367: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1782598099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:00 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T19:01:01.122 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:01.155 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: pgmap v367: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1782598099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T19:01:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:00 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T19:01:01.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.293 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:01.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:01.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:01.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:01.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:01.563 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:01.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:01.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:01.825 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:01.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:01.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:02.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:01 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:01:02.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:02.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:02.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:02.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T19:01:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:01:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:01 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:01:02.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:02.337 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:02.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:02.472 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:02.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:02.605 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:02.706 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:02.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:02.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:02.880 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:02.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:02.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:03.017 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:03.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:03.150 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:03.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:03.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:02 smithi104 ceph-mon[32438]: pgmap v369: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s 2024-09-21T19:01:03.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:02 smithi104 ceph-mon[32438]: 8.5 scrub starts 2024-09-21T19:01:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:02 smithi104 ceph-mon[32438]: 8.6 scrub starts 2024-09-21T19:01:03.284 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:03.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:02 smithi081 ceph-mon[39192]: pgmap v369: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s 2024-09-21T19:01:03.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:02 smithi081 ceph-mon[39192]: 8.5 scrub starts 2024-09-21T19:01:03.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:02 smithi081 ceph-mon[39192]: 8.6 scrub starts 2024-09-21T19:01:03.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:02 smithi081 ceph-mon[29384]: pgmap v369: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s 2024-09-21T19:01:03.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:02 smithi081 ceph-mon[29384]: 8.5 scrub starts 2024-09-21T19:01:03.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:02 smithi081 ceph-mon[29384]: 8.6 scrub starts 2024-09-21T19:01:03.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:03.416 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:03.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:03.548 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:03.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:03.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:03.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:03.820 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:03.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:03.944 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:03.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:04.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:04.076 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:04.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.204 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:04.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:03 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:01:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:03 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:01:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:03 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T19:01:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:03 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T19:01:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:03 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T19:01:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:03 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T19:01:04.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T19:01:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:03 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T19:01:04.339 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:04.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:04.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:04.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:04.654 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:04.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:04.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:04.907 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:04.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:05.008 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:05.041 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:05.111 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T19:01:05.111 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T19:01:05.111 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 8.a scrub starts 2024-09-21T19:01:05.111 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: pgmap v370: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-21T19:01:05.111 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 8.5 scrub ok 2024-09-21T19:01:05.111 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 3.b scrub starts 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 3.b scrub ok 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[39192]: 8.6 scrub ok 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 8.a scrub starts 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: pgmap v370: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 8.5 scrub ok 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 3.b scrub starts 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 3.b scrub ok 2024-09-21T19:01:05.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:04 smithi081 ceph-mon[29384]: 8.6 scrub ok 2024-09-21T19:01:05.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.188 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:05.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 8.a scrub starts 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: pgmap v370: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 8.5 scrub ok 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 3.b scrub starts 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 3.b scrub ok 2024-09-21T19:01:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:04 smithi104 ceph-mon[32438]: 8.6 scrub ok 2024-09-21T19:01:05.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:05.321 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:05.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:05.458 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:05.547 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:05.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:05.576 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:05.594 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:05.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.733 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:05.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:05.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:05.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:05.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:05.995 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:05.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[39192]: 8.a scrub ok 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T19:01:06.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T19:01:06.048 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:05 smithi081 ceph-mon[29384]: 8.a scrub ok 2024-09-21T19:01:06.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:06.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:06.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:06.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:05 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T19:01:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:05 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T19:01:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:05 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T19:01:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:05 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T19:01:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:05 smithi104 ceph-mon[32438]: 8.a scrub ok 2024-09-21T19:01:06.279 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:06.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.429 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:06.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:06.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:06.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:06.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:06.747 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:06.748 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:06.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:06.816 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:06.862 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:01:06.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:06.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:07.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:07.083 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:07.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:01:07.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: pgmap v371: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: 1.2 scrub starts 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: 1.2 scrub ok 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/552478539' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[29384]: from='client.35516 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T19:01:07.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: pgmap v371: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: 1.2 scrub starts 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: 1.2 scrub ok 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/552478539' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:07.152 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:06 smithi081 ceph-mon[39192]: from='client.35516 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:07.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:07.214 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T19:01:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: pgmap v371: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: 1.2 scrub starts 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: 1.2 scrub ok 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/552478539' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:06 smithi104 ceph-mon[32438]: from='client.35516 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:07.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.365 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:07.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:07.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.510 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:07.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:07.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:01:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:01:07.611 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:07.645 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:07.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:07.770 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:07.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:07.906 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:07.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:08.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:08.046 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: 8.7 scrub starts 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: 6.3 scrub starts 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: 6.3 scrub ok 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: 4.17 scrub starts 2024-09-21T19:01:08.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[29384]: 4.17 scrub ok 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: 8.7 scrub starts 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: 6.3 scrub starts 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: 6.3 scrub ok 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: 4.17 scrub starts 2024-09-21T19:01:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:07 smithi081 ceph-mon[39192]: 4.17 scrub ok 2024-09-21T19:01:08.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:08.174 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:08.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: 8.7 scrub starts 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: 6.3 scrub starts 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: 6.3 scrub ok 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: 4.17 scrub starts 2024-09-21T19:01:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:07 smithi104 ceph-mon[32438]: 4.17 scrub ok 2024-09-21T19:01:08.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:08.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:01:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:08.399 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:08.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.463 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:08.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:08.564 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:08.626 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:08.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:08.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:08.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:08.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:08.984 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[39192]: pgmap v373: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[39192]: 3.7 scrub starts 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[39192]: 3.7 scrub ok 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:08.987 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T19:01:08.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T19:01:08.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[29384]: pgmap v373: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:01:08.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[29384]: 3.7 scrub starts 2024-09-21T19:01:08.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[29384]: 3.7 scrub ok 2024-09-21T19:01:08.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:09.008 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:09.058 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:09.071 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:09.071 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:09.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:09.120 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:09.121 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:09.156 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:09.156 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:09.194 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:09.194 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:09.225 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:09.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T19:01:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T19:01:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-mon[32438]: pgmap v373: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:01:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-mon[32438]: 3.7 scrub starts 2024-09-21T19:01:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-mon[32438]: 3.7 scrub ok 2024-09-21T19:01:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:09.259 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:09.259 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:09.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:09.290 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:09.290 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:09.324 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:09.324 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:09.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:09.393 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:09.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:09.530 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:09.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:09.665 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:09.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:09.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:01:09.799 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:211468 7.e 7:76afa725:::benchmark_data_smithi081_73391_object72469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.049251+0000", 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.80877144900001, 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.127376505, 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.865 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 211468 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049251+0000", 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049251+0000", 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049253+0000", 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.809e-06 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049259+0000", 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4999999999999999e-06 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049271+0000", 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2374e-05 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049278+0000", 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4070000000000004e-06 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049367+0000", 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8351999999999997e-05 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.049479+0000", 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111938 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.053479+0000", 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039997540000000003 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.057552+0000", 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040731259999999998 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.066084+0000", 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0085322060000000005 2024-09-21T19:01:09.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.066124+0000", 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9983e-05 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.176433+0000", 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.110309291 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.176507+0000", 2024-09-21T19:01:09.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3805000000000002e-05 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.176628+0000", 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012095999999999999 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:211482 7.e 7:7d19c424:::benchmark_data_smithi081_73391_object72483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.059878+0000", 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.79814489099999, 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.14083991500000001, 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.871 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 211482 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059878+0000", 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059878+0000", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059880+0000", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6899999999999999e-06 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059884+0000", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5419999999999999e-06 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059893+0000", 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5099999999999998e-06 2024-09-21T19:01:09.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059898+0000", 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0679999999999996e-06 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.059957+0000", 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9672000000000001e-05 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060077+0000", 2024-09-21T19:01:09.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119122 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060205+0000", 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012813399999999999 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.066132+0000", 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059272989999999996 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.068065+0000", 2024-09-21T19:01:09.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019327319999999999 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.172812+0000", 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.104747322 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200489+0000", 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.027677021999999999 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200601+0000", 2024-09-21T19:01:09.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011212 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200718+0000", 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011668200000000001 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:211484 7.e 7:766f76ed:::benchmark_data_smithi081_73391_object72485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.060345+0000", 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.79767802800001, 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.151649492, 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.877 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 211484 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060345+0000", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060345+0000", 2024-09-21T19:01:09.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060346+0000", 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.367e-06 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060351+0000", 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7140000000000001e-06 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060361+0000", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9019999999999994e-06 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060368+0000", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2540000000000003e-06 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.063268+0000", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029002329999999999 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.063370+0000", 2024-09-21T19:01:09.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010184 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.063509+0000", 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138756 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.066137+0000", 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002628198 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.068114+0000", 2024-09-21T19:01:09.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001976925 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.172880+0000", 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.10476640199999999 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.211831+0000", 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.038950767999999997 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.211888+0000", 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7318999999999998e-05 2024-09-21T19:01:09.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.211994+0000", 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105814 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242078 7.8 7:1513feb5:::benchmark_data_smithi081_73391_object103079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.501345+0000", 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.35667804600001, 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17642243699999999, 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.883 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242078 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501345+0000", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501345+0000", 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501347+0000", 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.832e-06 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501352+0000", 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3480000000000003e-06 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501362+0000", 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.005e-05 2024-09-21T19:01:09.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501368+0000", 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6080000000000003e-06 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501450+0000", 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2725000000000005e-05 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501604+0000", 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015366199999999999 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501725+0000", 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120918 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673364+0000", 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.171639186 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673504+0000", 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013990399999999999 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677670+0000", 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041661290000000002 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677723+0000", 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2386999999999998e-05 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677767+0000", 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4688000000000003e-05 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.888 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242073 7.3 e148/79 7:c6fda3ec:::benchmark_data_smithi081_73391_object103074:head v 148'15005, mlcod=148'15003)", 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.502659+0000", 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.355363501, 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17046399800000001, 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502659+0000", 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502659+0000", 2024-09-21T19:01:09.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502660+0000", 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3200000000000004e-07 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502666+0000", 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9630000000000003e-06 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502675+0000", 2024-09-21T19:01:09.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0990000000000005e-06 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502682+0000", 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4590000000000003e-06 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503232+0000", 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00054988699999999999 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503249+0000", 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7062999999999999e-05 2024-09-21T19:01:09.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511382+0000", 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0081333470000000008 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673123+0000", 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16174134800000001 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242075 7.2 e148/79 7:4d0ebf32:::benchmark_data_smithi081_73391_object103076:head v 148'15270, mlcod=148'15268)", 2024-09-21T19:01:09.893 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.502699+0000", 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.35532411, 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17067586200000001, 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502699+0000", 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502699+0000", 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502699+0000", 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6199999999999999e-07 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502703+0000", 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.216e-06 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502708+0000", 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7670000000000003e-06 2024-09-21T19:01:09.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502712+0000", 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7230000000000002e-06 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502921+0000", 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000209178 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502939+0000", 2024-09-21T19:01:09.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7954000000000001e-05 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511572+0000", 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008633267 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673375+0000", 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16180239499999999 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.897 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242076 7.a e148/79 7:5697a0ec:::benchmark_data_smithi081_73391_object103077:head v 148'14869, mlcod=148'14866)", 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.505580+0000", 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.35244301200001, 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16782707999999999, 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505580+0000", 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505580+0000", 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505581+0000", 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1299999999999998e-07 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505587+0000", 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6560000000000003e-06 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505595+0000", 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4399999999999999e-06 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505601+0000", 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5050000000000004e-06 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505781+0000", 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000179696 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505801+0000", 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0080000000000001e-05 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673378+0000", 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16757718899999999 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673407+0000", 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8600999999999999e-05 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.901 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242079 7.3 e148/79 7:cff90720:::benchmark_data_smithi081_73391_object103080:head v 148'15006, mlcod=148'15003)", 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.505621+0000", 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.35240212400001, 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16760440800000001, 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505621+0000", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505621+0000", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505621+0000", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2199999999999999e-07 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505625+0000", 2024-09-21T19:01:09.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8419999999999998e-06 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505630+0000", 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1309999999999997e-06 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505635+0000", 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6120000000000001e-06 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506368+0000", 2024-09-21T19:01:09.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00073327099999999997 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506385+0000", 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7022999999999999e-05 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673193+0000", 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16680836199999999 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673225+0000", 2024-09-21T19:01:09.905 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1745000000000002e-05 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:242078 7.8 e148/79 ondisk, result = 0)", 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.505908+0000", 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.35211448699999, 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.167463786, 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.906 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505908+0000", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505908+0000", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505909+0000", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6200000000000006e-07 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505913+0000", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1389999999999997e-06 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505923+0000", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7839999999999998e-06 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.505929+0000", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8069999999999996e-06 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673347+0000", 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16741841700000001 2024-09-21T19:01:09.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673357+0000", 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7380000000000006e-06 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673372+0000", 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4939e-05 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:242078 7.8 e149/79 ondisk, result = 0)", 2024-09-21T19:01:09.909 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.509490+0000", 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.34853241900001, 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16402091499999999, 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509490+0000", 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509490+0000", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509492+0000", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6199999999999999e-06 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509498+0000", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7060000000000001e-06 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509508+0000", 2024-09-21T19:01:09.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0334e-05 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509515+0000", 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9489999999999999e-06 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673492+0000", 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16397676 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673499+0000", 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2699999999999999e-06 2024-09-21T19:01:09.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673511+0000", 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2276000000000001e-05 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242085 7.8 7:1ea57226:::benchmark_data_smithi081_73391_object103086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512238+0000", 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 200.345784485, 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16951476600000001, 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.913 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242085 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512238+0000", 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512238+0000", 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512241+0000", 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3709999999999998e-06 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512246+0000", 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4149999999999998e-06 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673424+0000", 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16117761899999999 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673434+0000", 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9359999999999994e-06 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673542+0000", 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010833899999999999 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673650+0000", 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010826 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673759+0000", 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010894400000000001 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.681598+0000", 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078391940000000007 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.681608+0000", 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1190000000000003e-06 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.681640+0000", 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2898999999999998e-05 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.681685+0000", 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4128000000000002e-05 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.681753+0000", 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8541999999999995e-05 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.918 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242814 7.2 e149/79 7:4a8d833a:::benchmark_data_smithi081_73391_object103815:head v 149'15314, mlcod=149'15312)", 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.349306+0000", 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.508716362, 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13460330600000001, 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349306+0000", 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349306+0000", 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349308+0000", 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3149999999999999e-06 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349314+0000", 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9000000000000003e-06 2024-09-21T19:01:09.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349323+0000", 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5230000000000001e-06 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349330+0000", 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5239999999999997e-06 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351574+0000", 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002244696 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351593+0000", 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8134e-05 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.483868+0000", 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13227534799999999 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.483910+0000", 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1866000000000002e-05 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.922 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242819 7.8 7:128352a3:::benchmark_data_smithi081_73391_object103820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.350026+0000", 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50799686299999, 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.14931591499999999, 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242819 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350026+0000", 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350026+0000", 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350027+0000", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.133e-06 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350031+0000", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7780000000000001e-06 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350040+0000", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6479999999999992e-06 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350046+0000", 2024-09-21T19:01:09.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0780000000000001e-06 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351868+0000", 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001822782 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.352031+0000", 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000162406 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T19:01:09.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.352133+0000", 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102568 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.358925+0000", 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0067918199999999996 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.480171+0000", 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.121245924 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499082+0000", 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018910955 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499221+0000", 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013939399999999999 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499342+0000", 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120429 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.928 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:09.930 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242825 7.e 7:77125aca:::benchmark_data_smithi081_73391_object103826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:09.930 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.351940+0000", 2024-09-21T19:01:09.930 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50608309699999, 2024-09-21T19:01:09.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12909812500000001, 2024-09-21T19:01:09.930 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.930 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242825 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351940+0000", 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351940+0000", 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351941+0000", 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5260000000000001e-06 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351946+0000", 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0669999999999999e-06 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351957+0000", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0197000000000001e-05 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351973+0000", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6096e-05 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.352080+0000", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010726900000000001 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.352174+0000", 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3937000000000005e-05 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.352270+0000", 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6315000000000002e-05 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.360041+0000", 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0077709709999999998 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.360086+0000", 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4716999999999999e-05 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.360129+0000", 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2815999999999998e-05 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.360212+0000", 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3163000000000003e-05 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481038+0000", 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.120826051 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242826 7.e 7:76ea5f0c:::benchmark_data_smithi081_73391_object103827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.351986+0000", 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50603656999999, 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.129244105, 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.936 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242826 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351986+0000", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351986+0000", 2024-09-21T19:01:09.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351987+0000", 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7899999999999998e-07 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351990+0000", 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7530000000000002e-06 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351995+0000", 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6200000000000004e-06 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.352001+0000", 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4029999999999998e-06 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356265+0000", 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042642399999999999 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356377+0000", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011244 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356492+0000", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114787 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481088+0000", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.124595467 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481099+0000", 2024-09-21T19:01:09.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1416999999999999e-05 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481134+0000", 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4576999999999998e-05 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481188+0000", 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4120999999999999e-05 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481230+0000", 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2601000000000001e-05 2024-09-21T19:01:09.941 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242821 7.a e149/79 7:5657abd2:::benchmark_data_smithi081_73391_object103822:head v 149'14907, mlcod=149'14904)", 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.354587+0000", 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50343619200001, 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12954126299999999, 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354587+0000", 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354587+0000", 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354588+0000", 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5269999999999999e-06 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354594+0000", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.229e-06 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354604+0000", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1700000000000003e-06 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354610+0000", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5529999999999998e-06 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356919+0000", 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023085760000000001 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356935+0000", 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5954999999999999e-05 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.483915+0000", 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12698078099999999 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.484128+0000", 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021247200000000001 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242817 7.3 e149/79 7:c6315629:::benchmark_data_smithi081_73391_object103818:head v 149'15048, mlcod=149'15046)", 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.354626+0000", 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50339683499999, 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.126657879, 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354626+0000", 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354626+0000", 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354626+0000", 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8299999999999997e-07 2024-09-21T19:01:09.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354631+0000", 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0389999999999998e-06 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354636+0000", 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7950000000000004e-06 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354641+0000", 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4399999999999998e-06 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354688+0000", 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7524000000000001e-05 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354707+0000", 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8824000000000001e-05 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481237+0000", 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.126529583 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481284+0000", 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7191000000000003e-05 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242822 7.2 e149/79 7:4babd305:::benchmark_data_smithi081_73391_object103823:head v 149'15315, mlcod=149'15312)", 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.355558+0000", 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50246519000001, 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13575166699999999, 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.950 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.355558+0000", 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.355558+0000", 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.355558+0000", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6499999999999998e-07 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.355564+0000", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3920000000000002e-06 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.355572+0000", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2009999999999999e-06 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.355579+0000", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4479999999999999e-06 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357991+0000", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024123209999999998 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.358009+0000", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7884999999999999e-05 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491234+0000", 2024-09-21T19:01:09.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.133225554 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491309+0000", 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5000999999999996e-05 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242829 7.8 7:189584f5:::benchmark_data_smithi081_73391_object103830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.357177+0000", 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.50084627699999, 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.14242949699999999, 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242829 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357177+0000", 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357177+0000", 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357178+0000", 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.84e-06 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357183+0000", 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2289999999999997e-06 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357191+0000", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1480000000000006e-06 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.357197+0000", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3300000000000004e-06 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.358954+0000", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017569910000000001 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.359056+0000", 2024-09-21T19:01:09.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101458 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,7", 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.359163+0000", 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107289 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.486097+0000", 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12693417500000001 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.486287+0000", 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018988800000000001 2024-09-21T19:01:09.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499366+0000", 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013078889 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499494+0000", 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012856 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499606+0000", 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001117 2024-09-21T19:01:09.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.34742.0:242829 7.8 e149/79 ondisk, result = 0)", 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.362798+0000", 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 199.49522438899999, 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13681107300000001, 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.362798+0000", 2024-09-21T19:01:09.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.362798+0000", 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.362800+0000", 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2529999999999999e-06 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.362804+0000", 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2540000000000003e-06 2024-09-21T19:01:09.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.480766+0000", 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.117961996 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.480772+0000", 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3350000000000003e-06 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.486274+0000", 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055013509999999998 2024-09-21T19:01:09.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.486281+0000", 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7649999999999993e-06 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.499610+0000", 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013328118999999999 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:09.963 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:09.964 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:01:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:09.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:10.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:10.120 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:10.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:09 smithi104 ceph-mon[32438]: 8.7 scrub ok 2024-09-21T19:01:10.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1135172355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:10.158 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:09 smithi104 ceph-mon[32438]: 8.e scrub starts 2024-09-21T19:01:10.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.257 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:10.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:10.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:09 smithi081 ceph-mon[29384]: 8.7 scrub ok 2024-09-21T19:01:10.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1135172355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:10.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:09 smithi081 ceph-mon[29384]: 8.e scrub starts 2024-09-21T19:01:10.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:09 smithi081 ceph-mon[39192]: 8.7 scrub ok 2024-09-21T19:01:10.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1135172355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:10.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:09 smithi081 ceph-mon[39192]: 8.e scrub starts 2024-09-21T19:01:10.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:10.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:10.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:10.525 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T19:01:10.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:01:10.525 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T19:01:10.525 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T19:01:10.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:01:10.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:10.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:10.660 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:10.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:10.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:10.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:10.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:11.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:11.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:11.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:11.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:10 smithi104 ceph-mon[32438]: pgmap v374: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-21T19:01:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:10 smithi104 ceph-mon[32438]: 8.2 scrub starts 2024-09-21T19:01:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:10 smithi104 ceph-mon[32438]: 8.0 scrub starts 2024-09-21T19:01:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:10 smithi104 ceph-mon[32438]: 6.7 scrub starts 2024-09-21T19:01:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:10 smithi104 ceph-mon[32438]: 6.7 scrub ok 2024-09-21T19:01:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:10 smithi104 ceph-mon[32438]: 8.c scrub starts 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[39192]: pgmap v374: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[39192]: 8.2 scrub starts 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[39192]: 8.0 scrub starts 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[39192]: 6.7 scrub starts 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[39192]: 6.7 scrub ok 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[39192]: 8.c scrub starts 2024-09-21T19:01:11.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[29384]: pgmap v374: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-21T19:01:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[29384]: 8.2 scrub starts 2024-09-21T19:01:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[29384]: 8.0 scrub starts 2024-09-21T19:01:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[29384]: 6.7 scrub starts 2024-09-21T19:01:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[29384]: 6.7 scrub ok 2024-09-21T19:01:11.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:10 smithi081 ceph-mon[29384]: 8.c scrub starts 2024-09-21T19:01:11.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:11.360 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:11.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:11.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:11.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:11.616 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:11.717 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:11.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:11.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:11.876 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:11.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:11.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:12.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:12.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:12.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 3.d scrub starts 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 3.d scrub ok 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 8.2 scrub ok 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 8.e scrub ok 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 5.1e scrub starts 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 5.1e scrub ok 2024-09-21T19:01:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:11 smithi104 ceph-mon[32438]: 8.0 scrub ok 2024-09-21T19:01:12.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 3.d scrub starts 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 3.d scrub ok 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 8.2 scrub ok 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 8.e scrub ok 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 5.1e scrub starts 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 5.1e scrub ok 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[39192]: 8.0 scrub ok 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 3.d scrub starts 2024-09-21T19:01:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 3.d scrub ok 2024-09-21T19:01:12.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 8.2 scrub ok 2024-09-21T19:01:12.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 8.e scrub ok 2024-09-21T19:01:12.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 5.1e scrub starts 2024-09-21T19:01:12.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 5.1e scrub ok 2024-09-21T19:01:12.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:11 smithi081 ceph-mon[29384]: 8.0 scrub ok 2024-09-21T19:01:12.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:12.403 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:12.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:12.538 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:12.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.668 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:12.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:12.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:12.798 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:12.900 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:12.934 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:12.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:13.036 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:13.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:01:13.130 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:13.132 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:13.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:13.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:13.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:12 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T19:01:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:12 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T19:01:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:12 smithi104 ceph-mon[32438]: pgmap v375: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-09-21T19:01:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:12 smithi104 ceph-mon[32438]: 6.1b scrub starts 2024-09-21T19:01:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:12 smithi104 ceph-mon[32438]: 6.1b scrub ok 2024-09-21T19:01:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:12 smithi104 ceph-mon[32438]: 8.c scrub ok 2024-09-21T19:01:13.266 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T19:01:13.266 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T19:01:13.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[39192]: pgmap v375: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-09-21T19:01:13.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[39192]: 6.1b scrub starts 2024-09-21T19:01:13.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[39192]: 6.1b scrub ok 2024-09-21T19:01:13.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[39192]: 8.c scrub ok 2024-09-21T19:01:13.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T19:01:13.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T19:01:13.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[29384]: pgmap v375: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-09-21T19:01:13.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[29384]: 6.1b scrub starts 2024-09-21T19:01:13.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[29384]: 6.1b scrub ok 2024-09-21T19:01:13.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:12 smithi081 ceph-mon[29384]: 8.c scrub ok 2024-09-21T19:01:13.323 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:13.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:13.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:13.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:13.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:13.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:13.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:13.772 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:13.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:13.908 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:13.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:14.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:14.022 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:01:14.050 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:14.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:14.180 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:14.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:13 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T19:01:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:13 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T19:01:14.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:13 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T19:01:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T19:01:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:13 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T19:01:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T19:01:14.415 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:14.458 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:14.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:14.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:14.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.749 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:14.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:14.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:14.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:14.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:14.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:15.020 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:15.122 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:15.153 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:15.255 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:14 smithi104 ceph-mon[32438]: pgmap v376: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-21T19:01:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:14 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T19:01:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:14 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T19:01:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:14 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T19:01:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:14 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T19:01:15.286 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[39192]: pgmap v376: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[29384]: pgmap v376: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T19:01:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T19:01:15.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:14 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T19:01:15.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:15.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:15.476 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:01:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:01:15.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:15.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:15.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:15.714 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:15.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:15.850 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:15.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:15.983 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:15.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:16.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:16.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:16.156 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:16.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:16.242 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[29384]: 8.8 scrub starts 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[29384]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:15 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:15 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:15 smithi104 ceph-mon[32438]: 8.8 scrub starts 2024-09-21T19:01:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:15 smithi104 ceph-mon[32438]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T19:01:16.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:15 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:01:16.258 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:16.258 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:16.258 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[39192]: 8.8 scrub starts 2024-09-21T19:01:16.258 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[39192]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T19:01:16.258 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:15 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:01:16.344 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:16.380 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:16.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:16.510 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:16.611 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.652 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:16.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:16.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:16.787 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:16.862 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-21T19:01:14.946779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4737473075,"num_objects":72479,"num_object_clones":0,"num_object_copies":217437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72479,"num_whiteouts":0,"num_read":1191,"num_read_kb":943,"num_write":192616,"num_write_kb":8462702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":6230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82528,"ondisk_log_size":82528,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16454888,"kb_used_data":14022504,"kb_used_omap":12,"kb_used_meta":2431731,"kb_avail":733506328,"statfs":{"total":767960285184,"available":751110479872,"internally_reserved":0,"allocated":14359044096,"data_stored":14356077401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":2490093108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,3,7,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-733872128,"num_objects":-11198,"num_object_clones":0,"num_object_copies":-33594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11198,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11198,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.399449"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":236,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869863+0000","last_change":"2024-09-21T18:59:50.144510+0000","last_active":"2024-09-21T19:01:06.869863+0000","last_peered":"2024-09-21T19:01:06.869863+0000","last_clean":"2024-09-21T19:01:06.869863+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:01:06.869863+0000","last_undegraded":"2024-09-21T19:01:06.869863+0000","last_fullsized":"2024-09-21T19:01:06.869863+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.144434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.144434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:15.699799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":377,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859672+0000","last_change":"2024-09-21T18:58:35.867068+0000","last_active":"2024-09-21T19:01:06.859672+0000","last_peered":"2024-09-21T19:01:06.859672+0000","last_clean":"2024-09-21T19:01:06.859672+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T19:01:06.859672+0000","last_undegraded":"2024-09-21T19:01:06.859672+0000","last_fullsized":"2024-09-21T19:01:06.859672+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:35.867013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:35.867013+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.705270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867865+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:01:06.867865+0000","last_peered":"2024-09-21T19:01:06.867865+0000","last_clean":"2024-09-21T19:01:06.867865+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:01:06.867865+0000","last_undegraded":"2024-09-21T19:01:06.867865+0000","last_fullsized":"2024-09-21T19:01:06.867865+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":328,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859919+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T19:01:06.859919+0000","last_peered":"2024-09-21T19:01:06.859919+0000","last_clean":"2024-09-21T19:01:06.859919+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:01:06.859919+0000","last_undegraded":"2024-09-21T19:01:06.859919+0000","last_fullsized":"2024-09-21T19:01:06.859919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.765211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.832004+0000","last_change":"2024-09-21T19:01:11.832004+0000","last_active":"2024-09-21T19:01:11.832004+0000","last_peered":"2024-09-21T19:01:11.832004+0000","last_clean":"2024-09-21T19:01:11.832004+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T19:01:11.832004+0000","last_undegraded":"2024-09-21T19:01:11.832004+0000","last_fullsized":"2024-09-21T19:01:11.832004+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.831931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.831931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.385915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860100+0000","last_change":"2024-09-21T18:58:36.978527+0000","last_active":"2024-09-21T19:01:06.860100+0000","last_peered":"2024-09-21T19:01:06.860100+0000","last_clean":"2024-09-21T19:01:06.860100+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T19:01:06.860100+0000","last_undegraded":"2024-09-21T19:01:06.860100+0000","last_fullsized":"2024-09-21T19:01:06.860100+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:36.978471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:36.978471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.858664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":408,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.858584+0000","last_change":"2024-09-21T19:00:33.910621+0000","last_active":"2024-09-21T19:01:06.858584+0000","last_peered":"2024-09-21T19:01:06.858584+0000","last_clean":"2024-09-21T19:01:06.858584+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:01:06.858584+0000","last_undegraded":"2024-09-21T19:01:06.858584+0000","last_fullsized":"2024-09-21T19:01:06.858584+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:33.910535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.910535+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.545163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859662+0000","last_change":"2024-09-21T19:01:02.741840+0000","last_active":"2024-09-21T19:01:06.859662+0000","last_peered":"2024-09-21T19:01:06.859662+0000","last_clean":"2024-09-21T19:01:06.859662+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T19:01:06.859662+0000","last_undegraded":"2024-09-21T19:01:06.859662+0000","last_fullsized":"2024-09-21T19:01:06.859662+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.741768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.741768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:25.815288+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863681+0000","last_change":"2024-09-21T19:00:28.036188+0000","last_active":"2024-09-21T19:01:06.863681+0000","last_peered":"2024-09-21T19:01:06.863681+0000","last_clean":"2024-09-21T19:01:06.863681+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T19:01:06.863681+0000","last_undegraded":"2024-09-21T19:01:06.863681+0000","last_fullsized":"2024-09-21T19:01:06.863681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:28.036123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.036123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.098067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":402,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860549+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:01:06.860549+0000","last_peered":"2024-09-21T19:01:06.860549+0000","last_clean":"2024-09-21T19:01:06.860549+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:01:06.860549+0000","last_undegraded":"2024-09-21T19:01:06.860549+0000","last_fullsized":"2024-09-21T19:01:06.860549+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":362,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863390+0000","last_change":"2024-09-21T18:59:20.569305+0000","last_active":"2024-09-21T19:01:06.863390+0000","last_peered":"2024-09-21T19:01:06.863390+0000","last_clean":"2024-09-21T19:01:06.863390+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:01:06.863390+0000","last_undegraded":"2024-09-21T19:01:06.863390+0000","last_fullsized":"2024-09-21T19:01:06.863390+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:20.569235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.569235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.891681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865523+0000","last_change":"2024-09-21T18:59:51.189725+0000","last_active":"2024-09-21T19:01:06.865523+0000","last_peered":"2024-09-21T19:01:06.865523+0000","last_clean":"2024-09-21T19:01:06.865523+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:01:06.865523+0000","last_undegraded":"2024-09-21T19:01:06.865523+0000","last_fullsized":"2024-09-21T19:01:06.865523+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.189659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.189659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.202774+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863652+0000","last_change":"2024-09-21T18:59:48.531625+0000","last_active":"2024-09-21T19:01:06.863652+0000","last_peered":"2024-09-21T19:01:06.863652+0000","last_clean":"2024-09-21T19:01:06.863652+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T19:01:06.863652+0000","last_undegraded":"2024-09-21T19:01:06.863652+0000","last_fullsized":"2024-09-21T19:01:06.863652+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.531568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.531568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":354,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859866+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:01:06.859866+0000","last_peered":"2024-09-21T19:01:06.859866+0000","last_clean":"2024-09-21T19:01:06.859866+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:01:06.859866+0000","last_undegraded":"2024-09-21T19:01:06.859866+0000","last_fullsized":"2024-09-21T19:01:06.859866+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.764044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867447+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T19:01:06.867447+0000","last_peered":"2024-09-21T19:01:06.867447+0000","last_clean":"2024-09-21T19:01:06.867447+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:01:06.867447+0000","last_undegraded":"2024-09-21T19:01:06.867447+0000","last_fullsized":"2024-09-21T19:01:06.867447+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.462353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":352,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863282+0000","last_change":"2024-09-21T19:01:06.061093+0000","last_active":"2024-09-21T19:01:06.863282+0000","last_peered":"2024-09-21T19:01:06.863282+0000","last_clean":"2024-09-21T19:01:06.863282+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T19:01:06.863282+0000","last_undegraded":"2024-09-21T19:01:06.863282+0000","last_fullsized":"2024-09-21T19:01:06.863282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.061023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.061023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.606007+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860210+0000","last_change":"2024-09-21T18:58:32.943602+0000","last_active":"2024-09-21T19:01:06.860210+0000","last_peered":"2024-09-21T19:01:06.860210+0000","last_clean":"2024-09-21T19:01:06.860210+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T19:01:06.860210+0000","last_undegraded":"2024-09-21T19:01:06.860210+0000","last_fullsized":"2024-09-21T19:01:06.860210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:32.943549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:32.943549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:10.666685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866153+0000","last_change":"2024-09-21T18:59:46.218922+0000","last_active":"2024-09-21T19:01:06.866153+0000","last_peered":"2024-09-21T19:01:06.866153+0000","last_clean":"2024-09-21T19:01:06.866153+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:01:06.866153+0000","last_undegraded":"2024-09-21T19:01:06.866153+0000","last_fullsized":"2024-09-21T19:01:06.866153+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.218852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.218852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:08.131850+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":377,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870800+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:01:06.870800+0000","last_peered":"2024-09-21T19:01:06.870800+0000","last_clean":"2024-09-21T19:01:06.870800+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:01:06.870800+0000","last_undegraded":"2024-09-21T19:01:06.870800+0000","last_fullsized":"2024-09-21T19:01:06.870800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.239922+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":362,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.861763+0000","last_change":"2024-09-21T19:00:15.241392+0000","last_active":"2024-09-21T19:01:06.861763+0000","last_peered":"2024-09-21T19:01:06.861763+0000","last_clean":"2024-09-21T19:01:06.861763+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:01:06.861763+0000","last_undegraded":"2024-09-21T19:01:06.861763+0000","last_fullsized":"2024-09-21T19:01:06.861763+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:15.241320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:15.241320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.208748+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860279+0000","last_change":"2024-09-21T18:59:19.930405+0000","last_active":"2024-09-21T19:01:06.860279+0000","last_peered":"2024-09-21T19:01:06.860279+0000","last_clean":"2024-09-21T19:01:06.860279+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T19:01:06.860279+0000","last_undegraded":"2024-09-21T19:01:06.860279+0000","last_fullsized":"2024-09-21T19:01:06.860279+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.867337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860604+0000","last_change":"2024-09-21T19:00:00.649580+0000","last_active":"2024-09-21T19:01:06.860604+0000","last_peered":"2024-09-21T19:01:06.860604+0000","last_clean":"2024-09-21T19:01:06.860604+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:01:06.860604+0000","last_undegraded":"2024-09-21T19:01:06.860604+0000","last_fullsized":"2024-09-21T19:01:06.860604+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.649397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.649397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:24.015113+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":395,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859227+0000","last_change":"2024-09-21T19:00:18.857976+0000","last_active":"2024-09-21T19:01:06.859227+0000","last_peered":"2024-09-21T19:01:06.859227+0000","last_clean":"2024-09-21T19:01:06.859227+0000","last_became_active":"2024-09-21T18:58:23.993245+0000","last_became_peered":"2024-09-21T18:58:23.993245+0000","last_unstale":"2024-09-21T19:01:06.859227+0000","last_undegraded":"2024-09-21T19:01:06.859227+0000","last_fullsized":"2024-09-21T19:01:06.859227+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:00:18.857901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.857901+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.470039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":340,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859993+0000","last_change":"2024-09-21T18:58:28.912619+0000","last_active":"2024-09-21T19:01:06.859993+0000","last_peered":"2024-09-21T19:01:06.859993+0000","last_clean":"2024-09-21T19:01:06.859993+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:01:06.859993+0000","last_undegraded":"2024-09-21T19:01:06.859993+0000","last_fullsized":"2024-09-21T19:01:06.859993+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:28.912554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:28.912554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.868513+0000","last_change":"2024-09-21T18:58:38.110564+0000","last_active":"2024-09-21T19:01:06.868513+0000","last_peered":"2024-09-21T19:01:06.868513+0000","last_clean":"2024-09-21T19:01:06.868513+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:01:06.868513+0000","last_undegraded":"2024-09-21T19:01:06.868513+0000","last_fullsized":"2024-09-21T19:01:06.868513+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:58:38.110509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:38.110509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.724266+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":360,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867940+0000","last_change":"2024-09-21T19:00:23.223251+0000","last_active":"2024-09-21T19:01:06.867940+0000","last_peered":"2024-09-21T19:01:06.867940+0000","last_clean":"2024-09-21T19:01:06.867940+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:01:06.867940+0000","last_undegraded":"2024-09-21T19:01:06.867940+0000","last_fullsized":"2024-09-21T19:01:06.867940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:23.223202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.223202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:35.046166+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":248,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869584+0000","last_change":"2024-09-21T19:00:22.256583+0000","last_active":"2024-09-21T19:01:06.869584+0000","last_peered":"2024-09-21T19:01:06.869584+0000","last_clean":"2024-09-21T19:01:06.869584+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:01:06.869584+0000","last_undegraded":"2024-09-21T19:01:06.869584+0000","last_fullsized":"2024-09-21T19:01:06.869584+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:22.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.256498+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.316225+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":455,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869299+0000","last_change":"2024-09-21T19:00:35.363014+0000","last_active":"2024-09-21T19:01:06.869299+0000","last_peered":"2024-09-21T19:01:06.869299+0000","last_clean":"2024-09-21T19:01:06.869299+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T19:01:06.869299+0000","last_undegraded":"2024-09-21T19:01:06.869299+0000","last_fullsized":"2024-09-21T19:01:06.869299+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.428300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"217'11884","reported_seq":11986,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.981452+0000","last_change":"2024-09-21T18:58:44.098981+0000","last_active":"2024-09-21T19:01:11.981452+0000","last_peered":"2024-09-21T19:01:11.981452+0000","last_clean":"2024-09-21T19:01:11.981452+0000","last_became_active":"2024-09-21T18:58:44.098681+0000","last_became_peered":"2024-09-21T18:58:44.098681+0000","last_unstale":"2024-09-21T19:01:11.981452+0000","last_undegraded":"2024-09-21T19:01:11.981452+0000","last_fullsized":"2024-09-21T19:01:11.981452+0000","mapping_epoch":172,"log_start":"200'6800","ondisk_log_start":"200'6800","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.272947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291766272,"num_objects":4452,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11884,"num_write_kb":522752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":121,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863155+0000","last_change":"2024-09-21T19:01:01.036323+0000","last_active":"2024-09-21T19:01:06.863155+0000","last_peered":"2024-09-21T19:01:06.863155+0000","last_clean":"2024-09-21T19:01:06.863155+0000","last_became_active":"2024-09-21T18:58:40.917818+0000","last_became_peered":"2024-09-21T18:58:40.917818+0000","last_unstale":"2024-09-21T19:01:06.863155+0000","last_undegraded":"2024-09-21T19:01:06.863155+0000","last_fullsized":"2024-09-21T19:01:06.863155+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:01.036275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:01.036275+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:01.626107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":369,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.872476+0000","last_change":"2024-09-21T18:59:21.369411+0000","last_active":"2024-09-21T19:01:06.872476+0000","last_peered":"2024-09-21T19:01:06.872476+0000","last_clean":"2024-09-21T19:01:06.872476+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:01:06.872476+0000","last_undegraded":"2024-09-21T19:01:06.872476+0000","last_fullsized":"2024-09-21T19:01:06.872476+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:59:21.369328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.369328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.377767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":357,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865234+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:01:06.865234+0000","last_peered":"2024-09-21T19:01:06.865234+0000","last_clean":"2024-09-21T19:01:06.865234+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:01:06.865234+0000","last_undegraded":"2024-09-21T19:01:06.865234+0000","last_fullsized":"2024-09-21T19:01:06.865234+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:17.164534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":475,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860451+0000","last_change":"2024-09-21T19:00:58.742094+0000","last_active":"2024-09-21T19:01:06.860451+0000","last_peered":"2024-09-21T19:01:06.860451+0000","last_clean":"2024-09-21T19:01:06.860451+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:01:06.860451+0000","last_undegraded":"2024-09-21T19:01:06.860451+0000","last_fullsized":"2024-09-21T19:01:06.860451+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:00:58.742016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:58.742016+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.133618+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863399+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T19:01:06.863399+0000","last_peered":"2024-09-21T19:01:06.863399+0000","last_clean":"2024-09-21T19:01:06.863399+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T19:01:06.863399+0000","last_undegraded":"2024-09-21T19:01:06.863399+0000","last_fullsized":"2024-09-21T19:01:06.863399+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.873896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"217'12133","reported_seq":12351,"reported_epoch":217,"state":"active+clean+scrubbing","last_fresh":"2024-09-21T19:01:11.914892+0000","last_change":"2024-09-21T19:01:10.478450+0000","last_active":"2024-09-21T19:01:11.914892+0000","last_peered":"2024-09-21T19:01:11.914892+0000","last_clean":"2024-09-21T19:01:11.914892+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:01:11.914892+0000","last_undegraded":"2024-09-21T19:01:11.914892+0000","last_fullsized":"2024-09-21T19:01:11.914892+0000","mapping_epoch":172,"log_start":"196'6600","ondisk_log_start":"196'6600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'6593","last_scrub_stamp":"2024-09-21T18:59:57.437417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.437417+0000","objects_scrubbed":3774,"log_size":5533,"log_dups_size":0,"ondisk_log_size":5533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 1s","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299433984,"num_objects":4569,"num_object_clones":0,"num_object_copies":13707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12133,"num_write_kb":534464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":113,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860464+0000","last_change":"2024-09-21T18:59:47.823699+0000","last_active":"2024-09-21T19:01:06.860464+0000","last_peered":"2024-09-21T19:01:06.860464+0000","last_clean":"2024-09-21T19:01:06.860464+0000","last_became_active":"2024-09-21T18:58:40.916167+0000","last_became_peered":"2024-09-21T18:58:40.916167+0000","last_unstale":"2024-09-21T19:01:06.860464+0000","last_undegraded":"2024-09-21T19:01:06.860464+0000","last_fullsized":"2024-09-21T19:01:06.860464+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.823624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.823624+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:16.932400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":428,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865050+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:01:06.865050+0000","last_peered":"2024-09-21T19:01:06.865050+0000","last_clean":"2024-09-21T19:01:06.865050+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:01:06.865050+0000","last_undegraded":"2024-09-21T19:01:06.865050+0000","last_fullsized":"2024-09-21T19:01:06.865050+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":371,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:07.875861+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:01:07.875861+0000","last_peered":"2024-09-21T19:01:07.875861+0000","last_clean":"2024-09-21T19:01:07.875861+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:01:07.875861+0000","last_undegraded":"2024-09-21T19:01:07.875861+0000","last_fullsized":"2024-09-21T19:01:07.875861+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:28.008242+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":340,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859745+0000","last_change":"2024-09-21T18:59:06.911709+0000","last_active":"2024-09-21T19:01:06.859745+0000","last_peered":"2024-09-21T19:01:06.859745+0000","last_clean":"2024-09-21T19:01:06.859745+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:01:06.859745+0000","last_undegraded":"2024-09-21T19:01:06.859745+0000","last_fullsized":"2024-09-21T19:01:06.859745+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.911638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.911638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.735119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":328,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860451+0000","last_change":"2024-09-21T18:59:13.953988+0000","last_active":"2024-09-21T19:01:06.860451+0000","last_peered":"2024-09-21T19:01:06.860451+0000","last_clean":"2024-09-21T19:01:06.860451+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:01:06.860451+0000","last_undegraded":"2024-09-21T19:01:06.860451+0000","last_fullsized":"2024-09-21T19:01:06.860451+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:13.953912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:13.953912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.742766+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"217'12045","reported_seq":12235,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.575900+0000","last_change":"2024-09-21T19:00:18.040765+0000","last_active":"2024-09-21T19:01:11.575900+0000","last_peered":"2024-09-21T19:01:11.575900+0000","last_clean":"2024-09-21T19:01:11.575900+0000","last_became_active":"2024-09-21T18:58:44.184359+0000","last_became_peered":"2024-09-21T18:58:44.184359+0000","last_unstale":"2024-09-21T19:01:11.575900+0000","last_undegraded":"2024-09-21T19:01:11.575900+0000","last_fullsized":"2024-09-21T19:01:11.575900+0000","mapping_epoch":172,"log_start":"200'7100","ondisk_log_start":"200'7100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7943","last_scrub_stamp":"2024-09-21T19:00:18.040678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.040678+0000","objects_scrubbed":7868,"log_size":4945,"log_dups_size":0,"ondisk_log_size":4945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.698500+0000","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795608,"num_objects":4545,"num_object_clones":0,"num_object_copies":13635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12045,"num_write_kb":530817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":118,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.857688+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:01:06.857688+0000","last_peered":"2024-09-21T19:01:06.857688+0000","last_clean":"2024-09-21T19:01:06.857688+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:01:06.857688+0000","last_undegraded":"2024-09-21T19:01:06.857688+0000","last_fullsized":"2024-09-21T19:01:06.857688+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:49.951550+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":365,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859806+0000","last_change":"2024-09-21T18:59:20.963196+0000","last_active":"2024-09-21T19:01:06.859806+0000","last_peered":"2024-09-21T19:01:06.859806+0000","last_clean":"2024-09-21T19:01:06.859806+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T19:01:06.859806+0000","last_undegraded":"2024-09-21T19:01:06.859806+0000","last_fullsized":"2024-09-21T19:01:06.859806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:59:20.962753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.962753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.779332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":240,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869991+0000","last_change":"2024-09-21T18:59:54.225013+0000","last_active":"2024-09-21T19:01:06.869991+0000","last_peered":"2024-09-21T19:01:06.869991+0000","last_clean":"2024-09-21T19:01:06.869991+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:01:06.869991+0000","last_undegraded":"2024-09-21T19:01:06.869991+0000","last_fullsized":"2024-09-21T19:01:06.869991+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:54.224944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.224944+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.892818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":439,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:07.545048+0000","last_change":"2024-09-21T19:00:22.845024+0000","last_active":"2024-09-21T19:01:07.545048+0000","last_peered":"2024-09-21T19:01:07.545048+0000","last_clean":"2024-09-21T19:01:07.545048+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:01:07.545048+0000","last_undegraded":"2024-09-21T19:01:07.545048+0000","last_fullsized":"2024-09-21T19:01:07.545048+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:22.844953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.844953+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:26.214253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":335,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867747+0000","last_change":"2024-09-21T19:01:06.644894+0000","last_active":"2024-09-21T19:01:06.867747+0000","last_peered":"2024-09-21T19:01:06.867747+0000","last_clean":"2024-09-21T19:01:06.867747+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T19:01:06.867747+0000","last_undegraded":"2024-09-21T19:01:06.867747+0000","last_fullsized":"2024-09-21T19:01:06.867747+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.644848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.644848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.908909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"217'12101","reported_seq":12257,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.884762+0000","last_change":"2024-09-21T19:01:11.369577+0000","last_active":"2024-09-21T19:01:11.884762+0000","last_peered":"2024-09-21T19:01:11.884762+0000","last_clean":"2024-09-21T19:01:11.884762+0000","last_became_active":"2024-09-21T18:58:44.096773+0000","last_became_peered":"2024-09-21T18:58:44.096773+0000","last_unstale":"2024-09-21T19:01:11.884762+0000","last_undegraded":"2024-09-21T19:01:11.884762+0000","last_fullsized":"2024-09-21T19:01:11.884762+0000","mapping_epoch":172,"log_start":"197'6600","ondisk_log_start":"197'6600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12063","last_scrub_stamp":"2024-09-21T19:01:11.369324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.369324+0000","objects_scrubbed":4626,"log_size":5501,"log_dups_size":0,"ondisk_log_size":5501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:37.327409+0000","scrub_duration":1905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296419328,"num_objects":4523,"num_object_clones":0,"num_object_copies":13569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12101,"num_write_kb":531968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":197,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869055+0000","last_change":"2024-09-21T18:59:46.185760+0000","last_active":"2024-09-21T19:01:06.869055+0000","last_peered":"2024-09-21T19:01:06.869055+0000","last_clean":"2024-09-21T19:01:06.869055+0000","last_became_active":"2024-09-21T18:58:40.530131+0000","last_became_peered":"2024-09-21T18:58:40.530131+0000","last_unstale":"2024-09-21T19:01:06.869055+0000","last_undegraded":"2024-09-21T19:01:06.869055+0000","last_fullsized":"2024-09-21T19:01:06.869055+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.185693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.185693+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":351,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.862427+0000","last_change":"2024-09-21T18:59:17.346990+0000","last_active":"2024-09-21T19:01:06.862427+0000","last_peered":"2024-09-21T19:01:06.862427+0000","last_clean":"2024-09-21T19:01:06.862427+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:01:06.862427+0000","last_undegraded":"2024-09-21T19:01:06.862427+0000","last_fullsized":"2024-09-21T19:01:06.862427+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:17.346414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.346414+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.396410+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863745+0000","last_change":"2024-09-21T18:59:16.589986+0000","last_active":"2024-09-21T19:01:06.863745+0000","last_peered":"2024-09-21T19:01:06.863745+0000","last_clean":"2024-09-21T19:01:06.863745+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T19:01:06.863745+0000","last_undegraded":"2024-09-21T19:01:06.863745+0000","last_fullsized":"2024-09-21T19:01:06.863745+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.589912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.589912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.916495+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":288,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866112+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T19:01:06.866112+0000","last_peered":"2024-09-21T19:01:06.866112+0000","last_clean":"2024-09-21T19:01:06.866112+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T19:01:06.866112+0000","last_undegraded":"2024-09-21T19:01:06.866112+0000","last_fullsized":"2024-09-21T19:01:06.866112+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.258988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865646+0000","last_change":"2024-09-21T18:59:44.256590+0000","last_active":"2024-09-21T19:01:06.865646+0000","last_peered":"2024-09-21T19:01:06.865646+0000","last_clean":"2024-09-21T19:01:06.865646+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:01:06.865646+0000","last_undegraded":"2024-09-21T19:01:06.865646+0000","last_fullsized":"2024-09-21T19:01:06.865646+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:44.256513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.256513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.406650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"217'12034","reported_seq":12136,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:12.350020+0000","last_change":"2024-09-21T18:58:44.097045+0000","last_active":"2024-09-21T19:01:12.350020+0000","last_peered":"2024-09-21T19:01:12.350020+0000","last_clean":"2024-09-21T19:01:12.350020+0000","last_became_active":"2024-09-21T18:58:44.096634+0000","last_became_peered":"2024-09-21T18:58:44.096634+0000","last_unstale":"2024-09-21T19:01:12.350020+0000","last_undegraded":"2024-09-21T19:01:12.350020+0000","last_fullsized":"2024-09-21T19:01:12.350020+0000","mapping_epoch":172,"log_start":"202'7500","ondisk_log_start":"202'7500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.889384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296615936,"num_objects":4526,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12034,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":205,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869403+0000","last_change":"2024-09-21T19:01:00.420038+0000","last_active":"2024-09-21T19:01:06.869403+0000","last_peered":"2024-09-21T19:01:06.869403+0000","last_clean":"2024-09-21T19:01:06.869403+0000","last_became_active":"2024-09-21T18:58:40.528964+0000","last_became_peered":"2024-09-21T18:58:40.528964+0000","last_unstale":"2024-09-21T19:01:06.869403+0000","last_undegraded":"2024-09-21T19:01:06.869403+0000","last_fullsized":"2024-09-21T19:01:06.869403+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:00.419946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.419946+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.968029+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":262,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866222+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:01:06.866222+0000","last_peered":"2024-09-21T19:01:06.866222+0000","last_clean":"2024-09-21T19:01:06.866222+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:01:06.866222+0000","last_undegraded":"2024-09-21T19:01:06.866222+0000","last_fullsized":"2024-09-21T19:01:06.866222+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.035114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863274+0000","last_change":"2024-09-21T19:00:56.467643+0000","last_active":"2024-09-21T19:01:06.863274+0000","last_peered":"2024-09-21T19:01:06.863274+0000","last_clean":"2024-09-21T19:01:06.863274+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:01:06.863274+0000","last_undegraded":"2024-09-21T19:01:06.863274+0000","last_fullsized":"2024-09-21T19:01:06.863274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.462694+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":353,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867005+0000","last_change":"2024-09-21T19:00:34.409013+0000","last_active":"2024-09-21T19:01:06.867005+0000","last_peered":"2024-09-21T19:01:06.867005+0000","last_clean":"2024-09-21T19:01:06.867005+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T19:01:06.867005+0000","last_undegraded":"2024-09-21T19:01:06.867005+0000","last_fullsized":"2024-09-21T19:01:06.867005+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.816307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867781+0000","last_change":"2024-09-21T19:00:25.249425+0000","last_active":"2024-09-21T19:01:06.867781+0000","last_peered":"2024-09-21T19:01:06.867781+0000","last_clean":"2024-09-21T19:01:06.867781+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T19:01:06.867781+0000","last_undegraded":"2024-09-21T19:01:06.867781+0000","last_fullsized":"2024-09-21T19:01:06.867781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.249360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.249360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:32.354078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"217'11590","reported_seq":11692,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.545576+0000","last_change":"2024-09-21T18:58:44.183202+0000","last_active":"2024-09-21T19:01:11.545576+0000","last_peered":"2024-09-21T19:01:11.545576+0000","last_clean":"2024-09-21T19:01:11.545576+0000","last_became_active":"2024-09-21T18:58:44.182910+0000","last_became_peered":"2024-09-21T18:58:44.182910+0000","last_unstale":"2024-09-21T19:01:11.545576+0000","last_undegraded":"2024-09-21T19:01:11.545576+0000","last_fullsized":"2024-09-21T19:01:11.545576+0000","mapping_epoch":172,"log_start":"198'6600","ondisk_log_start":"198'6600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":514240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":121,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863106+0000","last_change":"2024-09-21T19:00:55.902634+0000","last_active":"2024-09-21T19:01:06.863106+0000","last_peered":"2024-09-21T19:01:06.863106+0000","last_clean":"2024-09-21T19:01:06.863106+0000","last_became_active":"2024-09-21T18:58:40.916397+0000","last_became_peered":"2024-09-21T18:58:40.916397+0000","last_unstale":"2024-09-21T19:01:06.863106+0000","last_undegraded":"2024-09-21T19:01:06.863106+0000","last_fullsized":"2024-09-21T19:01:06.863106+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:55.902117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:55.902117+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.038393+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":489,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863176+0000","last_change":"2024-09-21T18:59:44.360729+0000","last_active":"2024-09-21T19:01:06.863176+0000","last_peered":"2024-09-21T19:01:06.863176+0000","last_clean":"2024-09-21T19:01:06.863176+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:01:06.863176+0000","last_undegraded":"2024-09-21T19:01:06.863176+0000","last_fullsized":"2024-09-21T19:01:06.863176+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:59:44.360619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.360619+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.396002+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":370,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860399+0000","last_change":"2024-09-21T18:59:55.645493+0000","last_active":"2024-09-21T19:01:06.860399+0000","last_peered":"2024-09-21T19:01:06.860399+0000","last_clean":"2024-09-21T19:01:06.860399+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T19:01:06.860399+0000","last_undegraded":"2024-09-21T19:01:06.860399+0000","last_fullsized":"2024-09-21T19:01:06.860399+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:55.645419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.645419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:19.840924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":374,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863577+0000","last_change":"2024-09-21T19:00:54.906034+0000","last_active":"2024-09-21T19:01:06.863577+0000","last_peered":"2024-09-21T19:01:06.863577+0000","last_clean":"2024-09-21T19:01:06.863577+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T19:01:06.863577+0000","last_undegraded":"2024-09-21T19:01:06.863577+0000","last_fullsized":"2024-09-21T19:01:06.863577+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:00:54.905971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:54.905971+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:05.061588+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866535+0000","last_change":"2024-09-21T19:00:25.316275+0000","last_active":"2024-09-21T19:01:06.866535+0000","last_peered":"2024-09-21T19:01:06.866535+0000","last_clean":"2024-09-21T19:01:06.866535+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:01:06.866535+0000","last_undegraded":"2024-09-21T19:01:06.866535+0000","last_fullsized":"2024-09-21T19:01:06.866535+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.314718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.314718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.602505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859817+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:01:06.859817+0000","last_peered":"2024-09-21T19:01:06.859817+0000","last_clean":"2024-09-21T19:01:06.859817+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:01:06.859817+0000","last_undegraded":"2024-09-21T19:01:06.859817+0000","last_fullsized":"2024-09-21T19:01:06.859817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"217'12168","reported_seq":12270,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:12.361903+0000","last_change":"2024-09-21T18:58:44.100975+0000","last_active":"2024-09-21T19:01:12.361903+0000","last_peered":"2024-09-21T19:01:12.361903+0000","last_clean":"2024-09-21T19:01:12.361903+0000","last_became_active":"2024-09-21T18:58:44.100676+0000","last_became_peered":"2024-09-21T18:58:44.100676+0000","last_unstale":"2024-09-21T19:01:12.361903+0000","last_undegraded":"2024-09-21T19:01:12.361903+0000","last_fullsized":"2024-09-21T19:01:12.361903+0000","mapping_epoch":172,"log_start":"203'7600","ondisk_log_start":"203'7600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296484864,"num_objects":4524,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":434,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867238+0000","last_change":"2024-09-21T19:01:05.625849+0000","last_active":"2024-09-21T19:01:06.867238+0000","last_peered":"2024-09-21T19:01:06.867238+0000","last_clean":"2024-09-21T19:01:06.867238+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:01:06.867238+0000","last_undegraded":"2024-09-21T19:01:06.867238+0000","last_fullsized":"2024-09-21T19:01:06.867238+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:05.625798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.625798+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.456539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":324,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870518+0000","last_change":"2024-09-21T19:01:01.980676+0000","last_active":"2024-09-21T19:01:06.870518+0000","last_peered":"2024-09-21T19:01:06.870518+0000","last_clean":"2024-09-21T19:01:06.870518+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:01:06.870518+0000","last_undegraded":"2024-09-21T19:01:06.870518+0000","last_fullsized":"2024-09-21T19:01:06.870518+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T19:01:01.980397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:01.980397+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.229183+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":363,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:07.496557+0000","last_change":"2024-09-21T19:01:07.496476+0000","last_active":"2024-09-21T19:01:07.496557+0000","last_peered":"2024-09-21T19:01:07.496557+0000","last_clean":"2024-09-21T19:01:07.496557+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T19:01:07.496557+0000","last_undegraded":"2024-09-21T19:01:07.496557+0000","last_fullsized":"2024-09-21T19:01:07.496557+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.105065+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":395,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869311+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:01:06.869311+0000","last_peered":"2024-09-21T19:01:06.869311+0000","last_clean":"2024-09-21T19:01:06.869311+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:01:06.869311+0000","last_undegraded":"2024-09-21T19:01:06.869311+0000","last_fullsized":"2024-09-21T19:01:06.869311+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.686451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870997+0000","last_change":"2024-09-21T19:00:29.002389+0000","last_active":"2024-09-21T19:01:06.870997+0000","last_peered":"2024-09-21T19:01:06.870997+0000","last_clean":"2024-09-21T19:01:06.870997+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:01:06.870997+0000","last_undegraded":"2024-09-21T19:01:06.870997+0000","last_fullsized":"2024-09-21T19:01:06.870997+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.002321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.002321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:56.121924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869113+0000","last_change":"2024-09-21T19:00:32.573881+0000","last_active":"2024-09-21T19:01:06.869113+0000","last_peered":"2024-09-21T19:01:06.869113+0000","last_clean":"2024-09-21T19:01:06.869113+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T19:01:06.869113+0000","last_undegraded":"2024-09-21T19:01:06.869113+0000","last_fullsized":"2024-09-21T19:01:06.869113+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:00:32.573704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.573704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.007025+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"217'11949","reported_seq":12180,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.991649+0000","last_change":"2024-09-21T19:01:05.288396+0000","last_active":"2024-09-21T19:01:11.991649+0000","last_peered":"2024-09-21T19:01:11.991649+0000","last_clean":"2024-09-21T19:01:11.991649+0000","last_became_active":"2024-09-21T18:58:44.099439+0000","last_became_peered":"2024-09-21T18:58:44.099439+0000","last_unstale":"2024-09-21T19:01:11.991649+0000","last_undegraded":"2024-09-21T19:01:11.991649+0000","last_fullsized":"2024-09-21T19:01:11.991649+0000","mapping_epoch":172,"log_start":"199'6900","ondisk_log_start":"199'6900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11506","last_scrub_stamp":"2024-09-21T19:01:05.288327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.288327+0000","objects_scrubbed":4952,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:28.984314+0000","scrub_duration":1801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290652160,"num_objects":4435,"num_object_clones":0,"num_object_copies":13305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":121,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863633+0000","last_change":"2024-09-21T19:00:59.488920+0000","last_active":"2024-09-21T19:01:06.863633+0000","last_peered":"2024-09-21T19:01:06.863633+0000","last_clean":"2024-09-21T19:01:06.863633+0000","last_became_active":"2024-09-21T18:58:40.528545+0000","last_became_peered":"2024-09-21T18:58:40.528545+0000","last_unstale":"2024-09-21T19:01:06.863633+0000","last_undegraded":"2024-09-21T19:01:06.863633+0000","last_fullsized":"2024-09-21T19:01:06.863633+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:59.488847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:59.488847+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.549020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":250,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869798+0000","last_change":"2024-09-21T18:59:53.225904+0000","last_active":"2024-09-21T19:01:06.869798+0000","last_peered":"2024-09-21T19:01:06.869798+0000","last_clean":"2024-09-21T19:01:06.869798+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:01:06.869798+0000","last_undegraded":"2024-09-21T19:01:06.869798+0000","last_fullsized":"2024-09-21T19:01:06.869798+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:53.225823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.225823+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":236,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.872132+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:01:06.872132+0000","last_peered":"2024-09-21T19:01:06.872132+0000","last_clean":"2024-09-21T19:01:06.872132+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:01:06.872132+0000","last_undegraded":"2024-09-21T19:01:06.872132+0000","last_fullsized":"2024-09-21T19:01:06.872132+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.948040+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":859,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:10.952887+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T19:01:10.952887+0000","last_peered":"2024-09-21T19:01:10.952887+0000","last_clean":"2024-09-21T19:01:10.952887+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:01:10.952887+0000","last_undegraded":"2024-09-21T19:01:10.952887+0000","last_fullsized":"2024-09-21T19:01:10.952887+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.500330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":473,"num_read_kb":468,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":378,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863969+0000","last_change":"2024-09-21T19:00:56.950003+0000","last_active":"2024-09-21T19:01:06.863969+0000","last_peered":"2024-09-21T19:01:06.863969+0000","last_clean":"2024-09-21T19:01:06.863969+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:01:06.863969+0000","last_undegraded":"2024-09-21T19:01:06.863969+0000","last_fullsized":"2024-09-21T19:01:06.863969+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.949931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.949931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.427785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.862760+0000","last_change":"2024-09-21T19:00:33.443101+0000","last_active":"2024-09-21T19:01:06.862760+0000","last_peered":"2024-09-21T19:01:06.862760+0000","last_clean":"2024-09-21T19:01:06.862760+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:01:06.862760+0000","last_undegraded":"2024-09-21T19:01:06.862760+0000","last_fullsized":"2024-09-21T19:01:06.862760+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:33.443040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.443040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.013096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"217'12237","reported_seq":12414,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.537554+0000","last_change":"2024-09-21T19:00:00.245865+0000","last_active":"2024-09-21T19:01:11.537554+0000","last_peered":"2024-09-21T19:01:11.537554+0000","last_clean":"2024-09-21T19:01:11.537554+0000","last_became_active":"2024-09-21T18:58:44.096509+0000","last_became_peered":"2024-09-21T18:58:44.096509+0000","last_unstale":"2024-09-21T19:01:11.537554+0000","last_undegraded":"2024-09-21T19:01:11.537554+0000","last_fullsized":"2024-09-21T19:01:11.537554+0000","mapping_epoch":172,"log_start":"196'6600","ondisk_log_start":"196'6600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'6787","last_scrub_stamp":"2024-09-21T19:00:00.245707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.245707+0000","objects_scrubbed":6706,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.145095+0000","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302186496,"num_objects":4611,"num_object_clones":0,"num_object_copies":13833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":462,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867121+0000","last_change":"2024-09-21T18:59:12.162986+0000","last_active":"2024-09-21T19:01:06.867121+0000","last_peered":"2024-09-21T19:01:06.867121+0000","last_clean":"2024-09-21T19:01:06.867121+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T19:01:06.867121+0000","last_undegraded":"2024-09-21T19:01:06.867121+0000","last_fullsized":"2024-09-21T19:01:06.867121+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:59:12.162725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.162725+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.317843+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":349,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.857998+0000","last_change":"2024-09-21T18:59:50.993922+0000","last_active":"2024-09-21T19:01:06.857998+0000","last_peered":"2024-09-21T19:01:06.857998+0000","last_clean":"2024-09-21T19:01:06.857998+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T19:01:06.857998+0000","last_undegraded":"2024-09-21T19:01:06.857998+0000","last_fullsized":"2024-09-21T19:01:06.857998+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.712596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":429,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867540+0000","last_change":"2024-09-21T19:00:57.551189+0000","last_active":"2024-09-21T19:01:06.867540+0000","last_peered":"2024-09-21T19:01:06.867540+0000","last_clean":"2024-09-21T19:01:06.867540+0000","last_became_active":"2024-09-21T18:58:39.058347+0000","last_became_peered":"2024-09-21T18:58:39.058347+0000","last_unstale":"2024-09-21T19:01:06.867540+0000","last_undegraded":"2024-09-21T19:01:06.867540+0000","last_fullsized":"2024-09-21T19:01:06.867540+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:57.551119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:57.551119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.725028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.868633+0000","last_change":"2024-09-21T19:00:00.889227+0000","last_active":"2024-09-21T19:01:06.868633+0000","last_peered":"2024-09-21T19:01:06.868633+0000","last_clean":"2024-09-21T19:01:06.868633+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:01:06.868633+0000","last_undegraded":"2024-09-21T19:01:06.868633+0000","last_fullsized":"2024-09-21T19:01:06.868633+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.889156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.889156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:16.434600+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:10.038311+0000","last_change":"2024-09-21T19:01:10.038311+0000","last_active":"2024-09-21T19:01:10.038311+0000","last_peered":"2024-09-21T19:01:10.038311+0000","last_clean":"2024-09-21T19:01:10.038311+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T19:01:10.038311+0000","last_undegraded":"2024-09-21T19:01:10.038311+0000","last_fullsized":"2024-09-21T19:01:10.038311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:37.189738+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"217'12069","reported_seq":12225,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.541040+0000","last_change":"2024-09-21T19:01:10.721872+0000","last_active":"2024-09-21T19:01:11.541040+0000","last_peered":"2024-09-21T19:01:11.541040+0000","last_clean":"2024-09-21T19:01:11.541040+0000","last_became_active":"2024-09-21T18:58:44.096986+0000","last_became_peered":"2024-09-21T18:58:44.096986+0000","last_unstale":"2024-09-21T19:01:11.541040+0000","last_undegraded":"2024-09-21T19:01:11.541040+0000","last_fullsized":"2024-09-21T19:01:11.541040+0000","mapping_epoch":172,"log_start":"202'7400","ondisk_log_start":"202'7400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12018","last_scrub_stamp":"2024-09-21T19:01:10.721645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.721645+0000","objects_scrubbed":4572,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.013892+0000","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294453248,"num_objects":4493,"num_object_clones":0,"num_object_copies":13479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":371,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860308+0000","last_change":"2024-09-21T19:00:01.693318+0000","last_active":"2024-09-21T19:01:06.860308+0000","last_peered":"2024-09-21T19:01:06.860308+0000","last_clean":"2024-09-21T19:01:06.860308+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T19:01:06.860308+0000","last_undegraded":"2024-09-21T19:01:06.860308+0000","last_fullsized":"2024-09-21T19:01:06.860308+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:01.693221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.693221+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.644648+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859961+0000","last_change":"2024-09-21T19:00:27.721451+0000","last_active":"2024-09-21T19:01:06.859961+0000","last_peered":"2024-09-21T19:01:06.859961+0000","last_clean":"2024-09-21T19:01:06.859961+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:01:06.859961+0000","last_undegraded":"2024-09-21T19:01:06.859961+0000","last_fullsized":"2024-09-21T19:01:06.859961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:27.721385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:27.721385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.383225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863780+0000","last_change":"2024-09-21T19:01:02.080311+0000","last_active":"2024-09-21T19:01:06.863780+0000","last_peered":"2024-09-21T19:01:06.863780+0000","last_clean":"2024-09-21T19:01:06.863780+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T19:01:06.863780+0000","last_undegraded":"2024-09-21T19:01:06.863780+0000","last_fullsized":"2024-09-21T19:01:06.863780+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.080248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.080248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.771263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":321,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.857794+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T19:01:06.857794+0000","last_peered":"2024-09-21T19:01:06.857794+0000","last_clean":"2024-09-21T19:01:06.857794+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T19:01:06.857794+0000","last_undegraded":"2024-09-21T19:01:06.857794+0000","last_fullsized":"2024-09-21T19:01:06.857794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.918918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"217'12014","reported_seq":12204,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:12.014739+0000","last_change":"2024-09-21T19:00:19.801173+0000","last_active":"2024-09-21T19:01:12.014739+0000","last_peered":"2024-09-21T19:01:12.014739+0000","last_clean":"2024-09-21T19:01:12.014739+0000","last_became_active":"2024-09-21T18:58:44.097827+0000","last_became_peered":"2024-09-21T18:58:44.097827+0000","last_unstale":"2024-09-21T19:01:12.014739+0000","last_undegraded":"2024-09-21T19:01:12.014739+0000","last_fullsized":"2024-09-21T19:01:12.014739+0000","mapping_epoch":172,"log_start":"200'7000","ondisk_log_start":"200'7000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8051","last_scrub_stamp":"2024-09-21T19:00:19.800992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.800992+0000","objects_scrubbed":7979,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.556183+0000","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297402368,"num_objects":4538,"num_object_clones":0,"num_object_copies":13614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12014,"num_write_kb":529664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":128,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865367+0000","last_change":"2024-09-21T19:01:05.092005+0000","last_active":"2024-09-21T19:01:06.865367+0000","last_peered":"2024-09-21T19:01:06.865367+0000","last_clean":"2024-09-21T19:01:06.865367+0000","last_became_active":"2024-09-21T18:58:40.917793+0000","last_became_peered":"2024-09-21T18:58:40.917793+0000","last_unstale":"2024-09-21T19:01:06.865367+0000","last_undegraded":"2024-09-21T19:01:06.865367+0000","last_fullsized":"2024-09-21T19:01:06.865367+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:05.091927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.091927+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.822244+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":372,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860400+0000","last_change":"2024-09-21T18:59:03.883391+0000","last_active":"2024-09-21T19:01:06.860400+0000","last_peered":"2024-09-21T19:01:06.860400+0000","last_clean":"2024-09-21T19:01:06.860400+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:01:06.860400+0000","last_undegraded":"2024-09-21T19:01:06.860400+0000","last_fullsized":"2024-09-21T19:01:06.860400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:59:03.883306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:03.883306+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.725718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":347,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867054+0000","last_change":"2024-09-21T18:59:06.194777+0000","last_active":"2024-09-21T19:01:06.867054+0000","last_peered":"2024-09-21T19:01:06.867054+0000","last_clean":"2024-09-21T19:01:06.867054+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T19:01:06.867054+0000","last_undegraded":"2024-09-21T19:01:06.867054+0000","last_fullsized":"2024-09-21T19:01:06.867054+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.194722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.194722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:14.287924+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863003+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T19:01:06.863003+0000","last_peered":"2024-09-21T19:01:06.863003+0000","last_clean":"2024-09-21T19:01:06.863003+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:01:06.863003+0000","last_undegraded":"2024-09-21T19:01:06.863003+0000","last_fullsized":"2024-09-21T19:01:06.863003+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.313883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870660+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:01:06.870660+0000","last_peered":"2024-09-21T19:01:06.870660+0000","last_clean":"2024-09-21T19:01:06.870660+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:01:06.870660+0000","last_undegraded":"2024-09-21T19:01:06.870660+0000","last_fullsized":"2024-09-21T19:01:06.870660+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.899930+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"217'12019","reported_seq":12174,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:12.364996+0000","last_change":"2024-09-21T19:01:11.670563+0000","last_active":"2024-09-21T19:01:12.364996+0000","last_peered":"2024-09-21T19:01:12.364996+0000","last_clean":"2024-09-21T19:01:12.364996+0000","last_became_active":"2024-09-21T18:58:44.096142+0000","last_became_peered":"2024-09-21T18:58:44.096142+0000","last_unstale":"2024-09-21T19:01:12.364996+0000","last_undegraded":"2024-09-21T19:01:12.364996+0000","last_fullsized":"2024-09-21T19:01:12.364996+0000","mapping_epoch":172,"log_start":"203'7500","ondisk_log_start":"203'7500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11953","last_scrub_stamp":"2024-09-21T19:01:11.670275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.670275+0000","objects_scrubbed":4463,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:34.755111+0000","scrub_duration":1749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285016064,"num_objects":4349,"num_object_clones":0,"num_object_copies":13047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12019,"num_write_kb":523776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":426,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867688+0000","last_change":"2024-09-21T18:59:50.870015+0000","last_active":"2024-09-21T19:01:06.867688+0000","last_peered":"2024-09-21T19:01:06.867688+0000","last_clean":"2024-09-21T19:01:06.867688+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:01:06.867688+0000","last_undegraded":"2024-09-21T19:01:06.867688+0000","last_fullsized":"2024-09-21T19:01:06.867688+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:50.869945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.869945+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:07.279297+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":359,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860011+0000","last_change":"2024-09-21T18:59:53.699627+0000","last_active":"2024-09-21T19:01:06.860011+0000","last_peered":"2024-09-21T19:01:06.860011+0000","last_clean":"2024-09-21T19:01:06.860011+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T19:01:06.860011+0000","last_undegraded":"2024-09-21T19:01:06.860011+0000","last_fullsized":"2024-09-21T19:01:06.860011+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:09.929430+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":385,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.858732+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:01:06.858732+0000","last_peered":"2024-09-21T19:01:06.858732+0000","last_clean":"2024-09-21T19:01:06.858732+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:01:06.858732+0000","last_undegraded":"2024-09-21T19:01:06.858732+0000","last_fullsized":"2024-09-21T19:01:06.858732+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:26.395243+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":461,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870605+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:01:06.870605+0000","last_peered":"2024-09-21T19:01:06.870605+0000","last_clean":"2024-09-21T19:01:06.870605+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:01:06.870605+0000","last_undegraded":"2024-09-21T19:01:06.870605+0000","last_fullsized":"2024-09-21T19:01:06.870605+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.071423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860121+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:01:06.860121+0000","last_peered":"2024-09-21T19:01:06.860121+0000","last_clean":"2024-09-21T19:01:06.860121+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:01:06.860121+0000","last_undegraded":"2024-09-21T19:01:06.860121+0000","last_fullsized":"2024-09-21T19:01:06.860121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.791557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"217'12092","reported_seq":12250,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:12.016386+0000","last_change":"2024-09-21T19:01:08.501885+0000","last_active":"2024-09-21T19:01:12.016386+0000","last_peered":"2024-09-21T19:01:12.016386+0000","last_clean":"2024-09-21T19:01:12.016386+0000","last_became_active":"2024-09-21T18:58:44.100559+0000","last_became_peered":"2024-09-21T18:58:44.100559+0000","last_unstale":"2024-09-21T19:01:12.016386+0000","last_undegraded":"2024-09-21T19:01:12.016386+0000","last_fullsized":"2024-09-21T19:01:12.016386+0000","mapping_epoch":172,"log_start":"199'7000","ondisk_log_start":"199'7000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11887","last_scrub_stamp":"2024-09-21T19:01:08.501670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:08.501670+0000","objects_scrubbed":4884,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.153152+0000","scrub_duration":1984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302645248,"num_objects":4618,"num_object_clones":0,"num_object_copies":13854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12092,"num_write_kb":534720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":230,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866298+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T19:01:06.866298+0000","last_peered":"2024-09-21T19:01:06.866298+0000","last_clean":"2024-09-21T19:01:06.866298+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:01:06.866298+0000","last_undegraded":"2024-09-21T19:01:06.866298+0000","last_fullsized":"2024-09-21T19:01:06.866298+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.261047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863710+0000","last_change":"2024-09-21T18:59:15.560811+0000","last_active":"2024-09-21T19:01:06.863710+0000","last_peered":"2024-09-21T19:01:06.863710+0000","last_clean":"2024-09-21T19:01:06.863710+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T19:01:06.863710+0000","last_undegraded":"2024-09-21T19:01:06.863710+0000","last_fullsized":"2024-09-21T19:01:06.863710+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:15.560713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:15.560713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.858874+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":340,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869894+0000","last_change":"2024-09-21T18:59:43.841286+0000","last_active":"2024-09-21T19:01:06.869894+0000","last_peered":"2024-09-21T19:01:06.869894+0000","last_clean":"2024-09-21T19:01:06.869894+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:01:06.869894+0000","last_undegraded":"2024-09-21T19:01:06.869894+0000","last_fullsized":"2024-09-21T19:01:06.869894+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:43.841222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:43.841222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.957920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.872058+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:01:06.872058+0000","last_peered":"2024-09-21T19:01:06.872058+0000","last_clean":"2024-09-21T19:01:06.872058+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:01:06.872058+0000","last_undegraded":"2024-09-21T19:01:06.872058+0000","last_fullsized":"2024-09-21T19:01:06.872058+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.293410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"217'11968","reported_seq":12198,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.871581+0000","last_change":"2024-09-21T19:01:04.185407+0000","last_active":"2024-09-21T19:01:11.871581+0000","last_peered":"2024-09-21T19:01:11.871581+0000","last_clean":"2024-09-21T19:01:11.871581+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:01:11.871581+0000","last_undegraded":"2024-09-21T19:01:11.871581+0000","last_fullsized":"2024-09-21T19:01:11.871581+0000","mapping_epoch":172,"log_start":"196'6500","ondisk_log_start":"196'6500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11433","last_scrub_stamp":"2024-09-21T19:01:04.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.185285+0000","objects_scrubbed":5079,"log_size":5468,"log_dups_size":0,"ondisk_log_size":5468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.471224+0000","scrub_duration":1738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11968,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":374,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866788+0000","last_change":"2024-09-21T19:00:23.328177+0000","last_active":"2024-09-21T19:01:06.866788+0000","last_peered":"2024-09-21T19:01:06.866788+0000","last_clean":"2024-09-21T19:01:06.866788+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T19:01:06.866788+0000","last_undegraded":"2024-09-21T19:01:06.866788+0000","last_fullsized":"2024-09-21T19:01:06.866788+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.465067+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":355,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:10.619281+0000","last_change":"2024-09-21T19:01:10.619281+0000","last_active":"2024-09-21T19:01:10.619281+0000","last_peered":"2024-09-21T19:01:10.619281+0000","last_clean":"2024-09-21T19:01:10.619281+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T19:01:10.619281+0000","last_undegraded":"2024-09-21T19:01:10.619281+0000","last_fullsized":"2024-09-21T19:01:10.619281+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.619235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.619235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:12.039735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":354,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870272+0000","last_change":"2024-09-21T19:00:30.996323+0000","last_active":"2024-09-21T19:01:06.870272+0000","last_peered":"2024-09-21T19:01:06.870272+0000","last_clean":"2024-09-21T19:01:06.870272+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:01:06.870272+0000","last_undegraded":"2024-09-21T19:01:06.870272+0000","last_fullsized":"2024-09-21T19:01:06.870272+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:30.996260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:30.996260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:34.407198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":335,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867432+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:01:06.867432+0000","last_peered":"2024-09-21T19:01:06.867432+0000","last_clean":"2024-09-21T19:01:06.867432+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:01:06.867432+0000","last_undegraded":"2024-09-21T19:01:06.867432+0000","last_fullsized":"2024-09-21T19:01:06.867432+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"217'11982","reported_seq":12213,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.538537+0000","last_change":"2024-09-21T19:01:03.618355+0000","last_active":"2024-09-21T19:01:11.538537+0000","last_peered":"2024-09-21T19:01:11.538537+0000","last_clean":"2024-09-21T19:01:11.538537+0000","last_became_active":"2024-09-21T18:58:44.184048+0000","last_became_peered":"2024-09-21T18:58:44.184048+0000","last_unstale":"2024-09-21T19:01:11.538537+0000","last_undegraded":"2024-09-21T19:01:11.538537+0000","last_fullsized":"2024-09-21T19:01:11.538537+0000","mapping_epoch":172,"log_start":"196'6400","ondisk_log_start":"196'6400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11443","last_scrub_stamp":"2024-09-21T19:01:03.618148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.618148+0000","objects_scrubbed":5222,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:27.366618+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11982,"num_write_kb":531648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":386,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859913+0000","last_change":"2024-09-21T18:59:17.862166+0000","last_active":"2024-09-21T19:01:06.859913+0000","last_peered":"2024-09-21T19:01:06.859913+0000","last_clean":"2024-09-21T19:01:06.859913+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:01:06.859913+0000","last_undegraded":"2024-09-21T19:01:06.859913+0000","last_fullsized":"2024-09-21T19:01:06.859913+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:17.862105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.862105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.784245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":355,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.868453+0000","last_change":"2024-09-21T19:00:22.616468+0000","last_active":"2024-09-21T19:01:06.868453+0000","last_peered":"2024-09-21T19:01:06.868453+0000","last_clean":"2024-09-21T19:01:06.868453+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:01:06.868453+0000","last_undegraded":"2024-09-21T19:01:06.868453+0000","last_fullsized":"2024-09-21T19:01:06.868453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:22.616386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.616386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:52.426205+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":362,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.868672+0000","last_change":"2024-09-21T18:59:16.539031+0000","last_active":"2024-09-21T19:01:06.868672+0000","last_peered":"2024-09-21T19:01:06.868672+0000","last_clean":"2024-09-21T19:01:06.868672+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:01:06.868672+0000","last_undegraded":"2024-09-21T19:01:06.868672+0000","last_fullsized":"2024-09-21T19:01:06.868672+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.538853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.538853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.903925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.858423+0000","last_change":"2024-09-21T19:00:34.960755+0000","last_active":"2024-09-21T19:01:06.858423+0000","last_peered":"2024-09-21T19:01:06.858423+0000","last_clean":"2024-09-21T19:01:06.858423+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:01:06.858423+0000","last_undegraded":"2024-09-21T19:01:06.858423+0000","last_fullsized":"2024-09-21T19:01:06.858423+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.960689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.960689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:57.667040+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"217'11895","reported_seq":12085,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.556459+0000","last_change":"2024-09-21T19:00:21.263588+0000","last_active":"2024-09-21T19:01:11.556459+0000","last_peered":"2024-09-21T19:01:11.556459+0000","last_clean":"2024-09-21T19:01:11.556459+0000","last_became_active":"2024-09-21T18:58:44.094697+0000","last_became_peered":"2024-09-21T18:58:44.094697+0000","last_unstale":"2024-09-21T19:01:11.556459+0000","last_undegraded":"2024-09-21T19:01:11.556459+0000","last_fullsized":"2024-09-21T19:01:11.556459+0000","mapping_epoch":172,"log_start":"199'6900","ondisk_log_start":"199'6900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8114","last_scrub_stamp":"2024-09-21T19:00:21.263245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.263245+0000","objects_scrubbed":8036,"log_size":4995,"log_dups_size":0,"ondisk_log_size":4995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.729679+0000","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295239680,"num_objects":4505,"num_object_clones":0,"num_object_copies":13515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11895,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":377,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865429+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T19:01:06.865429+0000","last_peered":"2024-09-21T19:01:06.865429+0000","last_clean":"2024-09-21T19:01:06.865429+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T19:01:06.865429+0000","last_undegraded":"2024-09-21T19:01:06.865429+0000","last_fullsized":"2024-09-21T19:01:06.865429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.863281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":377,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.868265+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T19:01:06.868265+0000","last_peered":"2024-09-21T19:01:06.868265+0000","last_clean":"2024-09-21T19:01:06.868265+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:01:06.868265+0000","last_undegraded":"2024-09-21T19:01:06.868265+0000","last_fullsized":"2024-09-21T19:01:06.868265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.427939+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":455,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867636+0000","last_change":"2024-09-21T18:59:56.831306+0000","last_active":"2024-09-21T19:01:06.867636+0000","last_peered":"2024-09-21T19:01:06.867636+0000","last_clean":"2024-09-21T19:01:06.867636+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T19:01:06.867636+0000","last_undegraded":"2024-09-21T19:01:06.867636+0000","last_fullsized":"2024-09-21T19:01:06.867636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:59:56.831225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.831225+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":366,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863296+0000","last_change":"2024-09-21T19:01:04.083677+0000","last_active":"2024-09-21T19:01:06.863296+0000","last_peered":"2024-09-21T19:01:06.863296+0000","last_clean":"2024-09-21T19:01:06.863296+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T19:01:06.863296+0000","last_undegraded":"2024-09-21T19:01:06.863296+0000","last_fullsized":"2024-09-21T19:01:06.863296+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:04.083562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.083562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:22.889131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":361,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866316+0000","last_change":"2024-09-21T18:59:14.180429+0000","last_active":"2024-09-21T19:01:06.866316+0000","last_peered":"2024-09-21T19:01:06.866316+0000","last_clean":"2024-09-21T19:01:06.866316+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T19:01:06.866316+0000","last_undegraded":"2024-09-21T19:01:06.866316+0000","last_fullsized":"2024-09-21T19:01:06.866316+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.387472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":362,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.925416+0000","last_change":"2024-09-21T19:01:06.925416+0000","last_active":"2024-09-21T19:01:06.925416+0000","last_peered":"2024-09-21T19:01:06.925416+0000","last_clean":"2024-09-21T19:01:06.925416+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:01:06.925416+0000","last_undegraded":"2024-09-21T19:01:06.925416+0000","last_fullsized":"2024-09-21T19:01:06.925416+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T19:01:06.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.925267+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.186913+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863793+0000","last_change":"2024-09-21T19:00:29.955243+0000","last_active":"2024-09-21T19:01:06.863793+0000","last_peered":"2024-09-21T19:01:06.863793+0000","last_clean":"2024-09-21T19:01:06.863793+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T19:01:06.863793+0000","last_undegraded":"2024-09-21T19:01:06.863793+0000","last_fullsized":"2024-09-21T19:01:06.863793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.955184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.955184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.019605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":327,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869064+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T19:01:06.869064+0000","last_peered":"2024-09-21T19:01:06.869064+0000","last_clean":"2024-09-21T19:01:06.869064+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T19:01:06.869064+0000","last_undegraded":"2024-09-21T19:01:06.869064+0000","last_fullsized":"2024-09-21T19:01:06.869064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.437166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":238,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869904+0000","last_change":"2024-09-21T19:00:34.345923+0000","last_active":"2024-09-21T19:01:06.869904+0000","last_peered":"2024-09-21T19:01:06.869904+0000","last_clean":"2024-09-21T19:01:06.869904+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:01:06.869904+0000","last_undegraded":"2024-09-21T19:01:06.869904+0000","last_fullsized":"2024-09-21T19:01:06.869904+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:00:34.345792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.345792+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:58.340045+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":347,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867872+0000","last_change":"2024-09-21T19:00:29.586470+0000","last_active":"2024-09-21T19:01:06.867872+0000","last_peered":"2024-09-21T19:01:06.867872+0000","last_clean":"2024-09-21T19:01:06.867872+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T19:01:06.867872+0000","last_undegraded":"2024-09-21T19:01:06.867872+0000","last_fullsized":"2024-09-21T19:01:06.867872+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.586411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.586411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.139362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":341,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860349+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:01:06.860349+0000","last_peered":"2024-09-21T19:01:06.860349+0000","last_clean":"2024-09-21T19:01:06.860349+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:01:06.860349+0000","last_undegraded":"2024-09-21T19:01:06.860349+0000","last_fullsized":"2024-09-21T19:01:06.860349+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.620068+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870399+0000","last_change":"2024-09-21T19:01:02.984598+0000","last_active":"2024-09-21T19:01:06.870399+0000","last_peered":"2024-09-21T19:01:06.870399+0000","last_clean":"2024-09-21T19:01:06.870399+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:01:06.870399+0000","last_undegraded":"2024-09-21T19:01:06.870399+0000","last_fullsized":"2024-09-21T19:01:06.870399+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.984535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.984535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.393677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":360,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863649+0000","last_change":"2024-09-21T18:58:58.524782+0000","last_active":"2024-09-21T19:01:06.863649+0000","last_peered":"2024-09-21T19:01:06.863649+0000","last_clean":"2024-09-21T19:01:06.863649+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T19:01:06.863649+0000","last_undegraded":"2024-09-21T19:01:06.863649+0000","last_fullsized":"2024-09-21T19:01:06.863649+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:58.524708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.524708+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.871526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.871937+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T19:01:06.871937+0000","last_peered":"2024-09-21T19:01:06.871937+0000","last_clean":"2024-09-21T19:01:06.871937+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:01:06.871937+0000","last_undegraded":"2024-09-21T19:01:06.871937+0000","last_fullsized":"2024-09-21T19:01:06.871937+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.288280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":454,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859714+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:01:06.859714+0000","last_peered":"2024-09-21T19:01:06.859714+0000","last_clean":"2024-09-21T19:01:06.859714+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:01:06.859714+0000","last_undegraded":"2024-09-21T19:01:06.859714+0000","last_fullsized":"2024-09-21T19:01:06.859714+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:22.257098+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860221+0000","last_change":"2024-09-21T19:00:32.753965+0000","last_active":"2024-09-21T19:01:06.860221+0000","last_peered":"2024-09-21T19:01:06.860221+0000","last_clean":"2024-09-21T19:01:06.860221+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:01:06.860221+0000","last_undegraded":"2024-09-21T19:01:06.860221+0000","last_fullsized":"2024-09-21T19:01:06.860221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:32.753902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.753902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.288209+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":352,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859899+0000","last_change":"2024-09-21T19:00:28.893623+0000","last_active":"2024-09-21T19:01:06.859899+0000","last_peered":"2024-09-21T19:01:06.859899+0000","last_clean":"2024-09-21T19:01:06.859899+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:01:06.859899+0000","last_undegraded":"2024-09-21T19:01:06.859899+0000","last_fullsized":"2024-09-21T19:01:06.859899+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:28.893403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.893403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.904866+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":347,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.868457+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:01:06.868457+0000","last_peered":"2024-09-21T19:01:06.868457+0000","last_clean":"2024-09-21T19:01:06.868457+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:01:06.868457+0000","last_undegraded":"2024-09-21T19:01:06.868457+0000","last_fullsized":"2024-09-21T19:01:06.868457+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.507018+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":459,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863546+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T19:01:06.863546+0000","last_peered":"2024-09-21T19:01:06.863546+0000","last_clean":"2024-09-21T19:01:06.863546+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T19:01:06.863546+0000","last_undegraded":"2024-09-21T19:01:06.863546+0000","last_fullsized":"2024-09-21T19:01:06.863546+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.883299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":236,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.870882+0000","last_change":"2024-09-21T19:00:58.413991+0000","last_active":"2024-09-21T19:01:06.870882+0000","last_peered":"2024-09-21T19:01:06.870882+0000","last_clean":"2024-09-21T19:01:06.870882+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:01:06.870882+0000","last_undegraded":"2024-09-21T19:01:06.870882+0000","last_fullsized":"2024-09-21T19:01:06.870882+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:58.413866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:58.413866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.192707+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":390,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.859971+0000","last_change":"2024-09-21T19:01:04.724924+0000","last_active":"2024-09-21T19:01:06.859971+0000","last_peered":"2024-09-21T19:01:06.859971+0000","last_clean":"2024-09-21T19:01:06.859971+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T19:01:06.859971+0000","last_undegraded":"2024-09-21T19:01:06.859971+0000","last_fullsized":"2024-09-21T19:01:06.859971+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:01:04.724868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.724868+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:24.749814+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.860362+0000","last_change":"2024-09-21T19:00:34.746168+0000","last_active":"2024-09-21T19:01:06.860362+0000","last_peered":"2024-09-21T19:01:06.860362+0000","last_clean":"2024-09-21T19:01:06.860362+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T19:01:06.860362+0000","last_undegraded":"2024-09-21T19:01:06.860362+0000","last_fullsized":"2024-09-21T19:01:06.860362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.746126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.746126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.862057+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:01:06.862057+0000","last_peered":"2024-09-21T19:01:06.862057+0000","last_clean":"2024-09-21T19:01:06.862057+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:01:06.862057+0000","last_undegraded":"2024-09-21T19:01:06.862057+0000","last_fullsized":"2024-09-21T19:01:06.862057+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":343,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.857682+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T19:01:06.857682+0000","last_peered":"2024-09-21T19:01:06.857682+0000","last_clean":"2024-09-21T19:01:06.857682+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:01:06.857682+0000","last_undegraded":"2024-09-21T19:01:06.857682+0000","last_fullsized":"2024-09-21T19:01:06.857682+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.894710+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":376,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.541442+0000","last_change":"2024-09-21T19:01:11.541442+0000","last_active":"2024-09-21T19:01:11.541442+0000","last_peered":"2024-09-21T19:01:11.541442+0000","last_clean":"2024-09-21T19:01:11.541442+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T19:01:11.541442+0000","last_undegraded":"2024-09-21T19:01:11.541442+0000","last_fullsized":"2024-09-21T19:01:11.541442+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.015757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":355,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869220+0000","last_change":"2024-09-21T18:58:34.182624+0000","last_active":"2024-09-21T19:01:06.869220+0000","last_peered":"2024-09-21T19:01:06.869220+0000","last_clean":"2024-09-21T19:01:06.869220+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:01:06.869220+0000","last_undegraded":"2024-09-21T19:01:06.869220+0000","last_fullsized":"2024-09-21T19:01:06.869220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:34.182578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:34.182578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:12.687690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":355,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.858520+0000","last_change":"2024-09-21T18:58:58.952448+0000","last_active":"2024-09-21T19:01:06.858520+0000","last_peered":"2024-09-21T19:01:06.858520+0000","last_clean":"2024-09-21T19:01:06.858520+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:01:06.858520+0000","last_undegraded":"2024-09-21T19:01:06.858520+0000","last_fullsized":"2024-09-21T19:01:06.858520+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:58.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.895529+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":236,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865719+0000","last_change":"2024-09-21T18:59:42.223794+0000","last_active":"2024-09-21T19:01:06.865719+0000","last_peered":"2024-09-21T19:01:06.865719+0000","last_clean":"2024-09-21T19:01:06.865719+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:01:06.865719+0000","last_undegraded":"2024-09-21T19:01:06.865719+0000","last_fullsized":"2024-09-21T19:01:06.865719+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:42.223731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.223731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:11.039045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.858224+0000","last_change":"2024-09-21T19:00:00.939564+0000","last_active":"2024-09-21T19:01:06.858224+0000","last_peered":"2024-09-21T19:01:06.858224+0000","last_clean":"2024-09-21T19:01:06.858224+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T19:01:06.858224+0000","last_undegraded":"2024-09-21T19:01:06.858224+0000","last_fullsized":"2024-09-21T19:01:06.858224+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:20.583806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.863888+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:01:06.863888+0000","last_peered":"2024-09-21T19:01:06.863888+0000","last_clean":"2024-09-21T19:01:06.863888+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:01:06.863888+0000","last_undegraded":"2024-09-21T19:01:06.863888+0000","last_fullsized":"2024-09-21T19:01:06.863888+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.872793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":347,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867171+0000","last_change":"2024-09-21T19:00:16.651104+0000","last_active":"2024-09-21T19:01:06.867171+0000","last_peered":"2024-09-21T19:01:06.867171+0000","last_clean":"2024-09-21T19:01:06.867171+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T19:01:06.867171+0000","last_undegraded":"2024-09-21T19:01:06.867171+0000","last_fullsized":"2024-09-21T19:01:06.867171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:16.650906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:16.650906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.021875+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":351,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.858652+0000","last_change":"2024-09-21T18:59:58.945778+0000","last_active":"2024-09-21T19:01:06.858652+0000","last_peered":"2024-09-21T19:01:06.858652+0000","last_clean":"2024-09-21T19:01:06.858652+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T19:01:06.858652+0000","last_undegraded":"2024-09-21T19:01:06.858652+0000","last_fullsized":"2024-09-21T19:01:06.858652+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:58.945631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:58.945631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:10.452414+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":348,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.864441+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T19:01:06.864441+0000","last_peered":"2024-09-21T19:01:06.864441+0000","last_clean":"2024-09-21T19:01:06.864441+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T19:01:06.864441+0000","last_undegraded":"2024-09-21T19:01:06.864441+0000","last_fullsized":"2024-09-21T19:01:06.864441+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.954764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":228,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869627+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T19:01:06.869627+0000","last_peered":"2024-09-21T19:01:06.869627+0000","last_clean":"2024-09-21T19:01:06.869627+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:01:06.869627+0000","last_undegraded":"2024-09-21T19:01:06.869627+0000","last_fullsized":"2024-09-21T19:01:06.869627+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.291673+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.866616+0000","last_change":"2024-09-21T18:59:17.117569+0000","last_active":"2024-09-21T19:01:06.866616+0000","last_peered":"2024-09-21T19:01:06.866616+0000","last_clean":"2024-09-21T19:01:06.866616+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T19:01:06.866616+0000","last_undegraded":"2024-09-21T19:01:06.866616+0000","last_fullsized":"2024-09-21T19:01:06.866616+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.359379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":335,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.867346+0000","last_change":"2024-09-21T18:59:59.934497+0000","last_active":"2024-09-21T19:01:06.867346+0000","last_peered":"2024-09-21T19:01:06.867346+0000","last_clean":"2024-09-21T19:01:06.867346+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:01:06.867346+0000","last_undegraded":"2024-09-21T19:01:06.867346+0000","last_fullsized":"2024-09-21T19:01:06.867346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.934416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.934416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.861525+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:01:06.861525+0000","last_peered":"2024-09-21T19:01:06.861525+0000","last_clean":"2024-09-21T19:01:06.861525+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:01:06.861525+0000","last_undegraded":"2024-09-21T19:01:06.861525+0000","last_fullsized":"2024-09-21T19:01:06.861525+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:21.108434+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":371,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.865151+0000","last_change":"2024-09-21T19:00:17.839077+0000","last_active":"2024-09-21T19:01:06.865151+0000","last_peered":"2024-09-21T19:01:06.865151+0000","last_clean":"2024-09-21T19:01:06.865151+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:01:06.865151+0000","last_undegraded":"2024-09-21T19:01:06.865151+0000","last_fullsized":"2024-09-21T19:01:06.865151+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:17.838903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:17.838903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.006877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":362,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:06.869351+0000","last_change":"2024-09-21T18:59:18.108977+0000","last_active":"2024-09-21T19:01:06.869351+0000","last_peered":"2024-09-21T19:01:06.869351+0000","last_clean":"2024-09-21T19:01:06.869351+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T19:01:06.869351+0000","last_undegraded":"2024-09-21T19:01:06.869351+0000","last_fullsized":"2024-09-21T19:01:06.869351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:18.108904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:18.108904+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.281496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":244,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-21T19:01:11.531925+0000","last_change":"2024-09-21T19:01:11.531925+0000","last_active":"2024-09-21T19:01:11.531925+0000","last_peered":"2024-09-21T19:01:11.531925+0000","last_clean":"2024-09-21T19:01:11.531925+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:01:11.531925+0000","last_undegraded":"2024-09-21T19:01:11.531925+0000","last_fullsized":"2024-09-21T19:01:11.531925+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:40.641599+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4736876568,"num_objects":72280,"num_object_clones":0,"num_object_copies":216840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192180,"num_write_kb":8462657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14231744512,"data_stored":14231732296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81180,"ondisk_log_size":81180,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":473,"num_read_kb":468,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266789,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1722484,"kb_used_data":1474724,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92022668,"statfs":{"total":95995035648,"available":94231212032,"internally_reserved":0,"allocated":1510117376,"data_stored":1509747223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528426,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1724776,"kb_used_data":1468696,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92020376,"statfs":{"total":95995035648,"available":94228865024,"internally_reserved":0,"allocated":1503944704,"data_stored":1503586386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790061,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2602924,"kb_used_data":2323228,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91142228,"statfs":{"total":95995035648,"available":93329641472,"internally_reserved":0,"allocated":2378985472,"data_stored":2378623536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018995,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734640,"kb_used_data":1474400,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":92010512,"statfs":{"total":95995035648,"available":94218764288,"internally_reserved":0,"allocated":1509785600,"data_stored":1509419799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":266467782},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182520,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2115848,"kb_used_data":1741688,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91629304,"statfs":{"total":95995035648,"available":93828407296,"internally_reserved":0,"allocated":1783488512,"data_stored":1783099198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411454,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2024692,"kb_used_data":1754660,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91720460,"statfs":{"total":95995035648,"available":93921751040,"internally_reserved":0,"allocated":1796771840,"data_stored":1796402267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574978,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2368512,"kb_used_data":2034416,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91376640,"statfs":{"total":95995035648,"available":93569679360,"internally_reserved":0,"allocated":2083241984,"data_stored":2082859872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738504,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2161012,"kb_used_data":1750692,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91584140,"statfs":{"total":95995035648,"available":93782159360,"internally_reserved":0,"allocated":1792708608,"data_stored":1792339120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1780613120,"data_stored":1780613120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2065825792,"data_stored":2065825792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1778126848,"data_stored":1778122776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1764163584,"data_stored":1764163584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1492647936,"data_stored":1492647936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2358448128,"data_stored":2358444056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1500053504,"data_stored":1500053504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1491865600,"data_stored":1491861528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:01:16.910 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-21T19:01:16.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:16.952 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:16.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:17.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:17.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:17.249 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: from='client.15639 v1:172.21.15.81:0/3201942195' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: 8.3 scrub starts 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: pgmap v378: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T19:01:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T19:01:17.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3329278834' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:17.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:16 smithi104 ceph-mon[32438]: from='client.35531 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: from='client.15639 v1:172.21.15.81:0/3201942195' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: 8.3 scrub starts 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: pgmap v378: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:01:17.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3329278834' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[39192]: from='client.35531 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: from='client.15639 v1:172.21.15.81:0/3201942195' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: 8.3 scrub starts 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: pgmap v378: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T19:01:17.285 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3329278834' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:16 smithi081 ceph-mon[29384]: from='client.35531 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:17.286 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:01:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:01:17.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:17.381 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:17.459 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:17.459 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:17.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:17.501 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:17.501 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:17.540 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:17.540 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:17.568 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:17.568 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:17.599 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:17.600 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:17.617 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:01:17.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:17.629 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:17.630 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:17.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:17.663 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:17.663 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:17.695 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:17.695 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:17.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:17.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:17.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:17.919 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:17.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:18.021 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:18.052 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:18.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:18.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:18.183 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[39192]: 8.8 scrub ok 2024-09-21T19:01:18.183 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[39192]: 8.3 scrub ok 2024-09-21T19:01:18.183 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:18.183 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[39192]: from='client.35531 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[39192]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3621653588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[29384]: 8.8 scrub ok 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[29384]: 8.3 scrub ok 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[29384]: from='client.35531 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[29384]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T19:01:18.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3621653588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:18.185 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:18.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:01:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:01:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:17 smithi104 ceph-mon[32438]: 8.8 scrub ok 2024-09-21T19:01:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:17 smithi104 ceph-mon[32438]: 8.3 scrub ok 2024-09-21T19:01:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:17 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:17 smithi104 ceph-mon[32438]: from='client.35531 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:01:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:17 smithi104 ceph-mon[32438]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T19:01:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3621653588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:18.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:18.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:18.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:18.461 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:18.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:18.600 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:18.607 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:18.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:18.736 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:18.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:18.877 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:18.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:18.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:19.012 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:19.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:19.146 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:19.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:18 smithi104 ceph-mon[32438]: pgmap v380: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T19:01:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:19.278 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:18 smithi081 ceph-mon[39192]: pgmap v380: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T19:01:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:18 smithi081 ceph-mon[29384]: pgmap v380: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T19:01:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:19.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:19.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:19.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:19.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:19.694 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:19.694 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:19.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:19.835 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:19.936 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:19.972 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:19.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:20.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:20.109 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:20.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:20.240 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:20.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:20.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:20.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:01:20.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:20.536 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:20.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:20.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:01:20.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:20.777 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:20.902 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:20.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:20.938 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:20.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:21.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:21.066 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:21.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:01:21.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:21.217 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:20 smithi104 ceph-mon[32438]: pgmap v381: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T19:01:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:20 smithi081 ceph-mon[29384]: pgmap v381: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T19:01:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:20 smithi081 ceph-mon[39192]: pgmap v381: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T19:01:21.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:21.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:21.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:21.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:21.598 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:21.633 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:21.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:21.775 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:21.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:21.909 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:21.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:21.911 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T19:01:21.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:01:21.911 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-09-21T19:01:21.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:01:22.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:22.039 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:22.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:22.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:22.167 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:22.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:22.298 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:22.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:22.431 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:22.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:22.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:22.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:22.692 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:22.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:22.827 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:22.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:22.968 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:22.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:23.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:23.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:23.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:23.236 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:23.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:23 smithi104 ceph-mon[32438]: pgmap v382: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 730 op/s 2024-09-21T19:01:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:23 smithi081 ceph-mon[29384]: pgmap v382: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 730 op/s 2024-09-21T19:01:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:23 smithi081 ceph-mon[39192]: pgmap v382: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 730 op/s 2024-09-21T19:01:23.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:23.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:23.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:23.509 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:23.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:23.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:23.678 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:23.678 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:23.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:23.810 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:23.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:23.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:23.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:23.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:24.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:24.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:24.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:24.217 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:24.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:24.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:24.386 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:01:24.453 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:24.455 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:24.478 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:24.534 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:24.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:24.606 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:24.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:24.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:24.846 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:24.874 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:24.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:24.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:25.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:25.042 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:25.144 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:25.187 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:25 smithi081 ceph-mon[29384]: pgmap v383: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-21T19:01:25.188 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:25 smithi081 ceph-mon[39192]: pgmap v383: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-21T19:01:25.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:25.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:25.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:25.452 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:25.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:25 smithi104 ceph-mon[32438]: pgmap v383: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-21T19:01:25.554 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:25.581 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:25.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:25.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:25.798 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:25.824 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:01:25.831 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:01:25.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:25.860 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:25.962 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:25.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:26.024 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:01:26.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1227623215' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:26.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:26 smithi081 ceph-mon[39192]: from='client.25441 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:26.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1227623215' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:26.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:26 smithi081 ceph-mon[29384]: from='client.25441 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:26.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:26.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:26.137 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:26.137 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:26.175 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:26.211 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:26.211 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:26.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:26.242 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:26.242 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:26.257 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:26.276 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:26.309 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:26.341 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:26.341 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:26.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:26.375 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:26.375 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:26.389 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:26.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:26.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1227623215' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:26 smithi104 ceph-mon[32438]: from='client.25441 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:26.632 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:26.662 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:26.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:26.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:26.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:26.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:26.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:27.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:27.079 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:27.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:27.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[39192]: pgmap v384: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[39192]: from='client.15654 v1:172.21.15.81:0/1606858814' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[39192]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[39192]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2540117596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[29384]: pgmap v384: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[29384]: from='client.15654 v1:172.21.15.81:0/1606858814' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:27.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[29384]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:27.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[29384]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T19:01:27.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2540117596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:27.221 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:27.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:27.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:27.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:27.499 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:27 smithi104 ceph-mon[32438]: pgmap v384: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T19:01:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:27 smithi104 ceph-mon[32438]: from='client.15654 v1:172.21.15.81:0/1606858814' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:27 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:27 smithi104 ceph-mon[32438]: from='client.25441 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:27 smithi104 ceph-mon[32438]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T19:01:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:27 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2540117596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:27.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:01:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:01:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:01:27.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:27.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:27.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:27.761 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:27.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:27.898 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:27.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:28.000 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:28.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.034 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-21T19:01:25.552845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3932690995,"num_objects":60199,"num_object_clones":0,"num_object_copies":180597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60199,"num_whiteouts":0,"num_read":1201,"num_read_kb":953,"num_write":204896,"num_write_kb":8462702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":6230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82308,"ondisk_log_size":82308,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14146408,"kb_used_data":11692200,"kb_used_omap":12,"kb_used_meta":2453939,"kb_avail":735814808,"statfs":{"total":767960285184,"available":753474363392,"internally_reserved":0,"allocated":11972812800,"data_stored":11969761385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":2512834100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-804782080,"num_objects":-12280,"num_object_clones":0,"num_object_copies":-36840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12280,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12280,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002764"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":240,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160995+0000","last_change":"2024-09-21T18:59:50.144510+0000","last_active":"2024-09-21T19:01:17.160995+0000","last_peered":"2024-09-21T19:01:17.160995+0000","last_clean":"2024-09-21T19:01:17.160995+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:01:17.160995+0000","last_undegraded":"2024-09-21T19:01:17.160995+0000","last_fullsized":"2024-09-21T19:01:17.160995+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.144434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.144434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":389,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.145739+0000","last_change":"2024-09-21T19:01:15.751481+0000","last_active":"2024-09-21T19:01:17.145739+0000","last_peered":"2024-09-21T19:01:17.145739+0000","last_clean":"2024-09-21T19:01:17.145739+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T19:01:17.145739+0000","last_undegraded":"2024-09-21T19:01:17.145739+0000","last_fullsized":"2024-09-21T19:01:17.145739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:15.751424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:15.751424+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:17.420109+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150890+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:01:17.150890+0000","last_peered":"2024-09-21T19:01:17.150890+0000","last_clean":"2024-09-21T19:01:17.150890+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:01:17.150890+0000","last_undegraded":"2024-09-21T19:01:17.150890+0000","last_fullsized":"2024-09-21T19:01:17.150890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":332,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.145917+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T19:01:17.145917+0000","last_peered":"2024-09-21T19:01:17.145917+0000","last_clean":"2024-09-21T19:01:17.145917+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:01:17.145917+0000","last_undegraded":"2024-09-21T19:01:17.145917+0000","last_fullsized":"2024-09-21T19:01:17.145917+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.765211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":349,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153069+0000","last_change":"2024-09-21T19:01:11.832004+0000","last_active":"2024-09-21T19:01:17.153069+0000","last_peered":"2024-09-21T19:01:17.153069+0000","last_clean":"2024-09-21T19:01:17.153069+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T19:01:17.153069+0000","last_undegraded":"2024-09-21T19:01:17.153069+0000","last_fullsized":"2024-09-21T19:01:17.153069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.831931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.831931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.385915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":354,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.152507+0000","last_change":"2024-09-21T18:58:36.978527+0000","last_active":"2024-09-21T19:01:17.152507+0000","last_peered":"2024-09-21T19:01:17.152507+0000","last_clean":"2024-09-21T19:01:17.152507+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T19:01:17.152507+0000","last_undegraded":"2024-09-21T19:01:17.152507+0000","last_fullsized":"2024-09-21T19:01:17.152507+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:36.978471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:36.978471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":412,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151075+0000","last_change":"2024-09-21T19:00:33.910621+0000","last_active":"2024-09-21T19:01:17.151075+0000","last_peered":"2024-09-21T19:01:17.151075+0000","last_clean":"2024-09-21T19:01:17.151075+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:01:17.151075+0000","last_undegraded":"2024-09-21T19:01:17.151075+0000","last_fullsized":"2024-09-21T19:01:17.151075+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:33.910535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.910535+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.545163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.145714+0000","last_change":"2024-09-21T19:01:02.741840+0000","last_active":"2024-09-21T19:01:17.145714+0000","last_peered":"2024-09-21T19:01:17.145714+0000","last_clean":"2024-09-21T19:01:17.145714+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T19:01:17.145714+0000","last_undegraded":"2024-09-21T19:01:17.145714+0000","last_fullsized":"2024-09-21T19:01:17.145714+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.741768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.741768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:25.815288+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156689+0000","last_change":"2024-09-21T19:00:28.036188+0000","last_active":"2024-09-21T19:01:17.156689+0000","last_peered":"2024-09-21T19:01:17.156689+0000","last_clean":"2024-09-21T19:01:17.156689+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T19:01:17.156689+0000","last_undegraded":"2024-09-21T19:01:17.156689+0000","last_fullsized":"2024-09-21T19:01:17.156689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:28.036123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.036123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.098067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":406,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146086+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:01:17.146086+0000","last_peered":"2024-09-21T19:01:17.146086+0000","last_clean":"2024-09-21T19:01:17.146086+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:01:17.146086+0000","last_undegraded":"2024-09-21T19:01:17.146086+0000","last_fullsized":"2024-09-21T19:01:17.146086+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":366,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158270+0000","last_change":"2024-09-21T18:59:20.569305+0000","last_active":"2024-09-21T19:01:17.158270+0000","last_peered":"2024-09-21T19:01:17.158270+0000","last_clean":"2024-09-21T19:01:17.158270+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:01:17.158270+0000","last_undegraded":"2024-09-21T19:01:17.158270+0000","last_fullsized":"2024-09-21T19:01:17.158270+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:20.569235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.569235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.891681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158853+0000","last_change":"2024-09-21T18:59:51.189725+0000","last_active":"2024-09-21T19:01:17.158853+0000","last_peered":"2024-09-21T19:01:17.158853+0000","last_clean":"2024-09-21T19:01:17.158853+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:01:17.158853+0000","last_undegraded":"2024-09-21T19:01:17.158853+0000","last_fullsized":"2024-09-21T19:01:17.158853+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.189659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.189659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156201+0000","last_change":"2024-09-21T18:59:48.531625+0000","last_active":"2024-09-21T19:01:17.156201+0000","last_peered":"2024-09-21T19:01:17.156201+0000","last_clean":"2024-09-21T19:01:17.156201+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T19:01:17.156201+0000","last_undegraded":"2024-09-21T19:01:17.156201+0000","last_fullsized":"2024-09-21T19:01:17.156201+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.531568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.531568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":358,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146276+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:01:17.146276+0000","last_peered":"2024-09-21T19:01:17.146276+0000","last_clean":"2024-09-21T19:01:17.146276+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:01:17.146276+0000","last_undegraded":"2024-09-21T19:01:17.146276+0000","last_fullsized":"2024-09-21T19:01:17.146276+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.764044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":343,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158483+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T19:01:17.158483+0000","last_peered":"2024-09-21T19:01:17.158483+0000","last_clean":"2024-09-21T19:01:17.158483+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:01:17.158483+0000","last_undegraded":"2024-09-21T19:01:17.158483+0000","last_fullsized":"2024-09-21T19:01:17.158483+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.462353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":356,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155708+0000","last_change":"2024-09-21T19:01:06.061093+0000","last_active":"2024-09-21T19:01:17.155708+0000","last_peered":"2024-09-21T19:01:17.155708+0000","last_clean":"2024-09-21T19:01:17.155708+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T19:01:17.155708+0000","last_undegraded":"2024-09-21T19:01:17.155708+0000","last_fullsized":"2024-09-21T19:01:17.155708+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.061023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.061023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.606007+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":349,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146316+0000","last_change":"2024-09-21T19:01:13.714975+0000","last_active":"2024-09-21T19:01:17.146316+0000","last_peered":"2024-09-21T19:01:17.146316+0000","last_clean":"2024-09-21T19:01:17.146316+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T19:01:17.146316+0000","last_undegraded":"2024-09-21T19:01:17.146316+0000","last_fullsized":"2024-09-21T19:01:17.146316+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:13.714855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.714855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.348742+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":240,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.159963+0000","last_change":"2024-09-21T19:01:16.593955+0000","last_active":"2024-09-21T19:01:17.159963+0000","last_peered":"2024-09-21T19:01:17.159963+0000","last_clean":"2024-09-21T19:01:17.159963+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:01:17.159963+0000","last_undegraded":"2024-09-21T19:01:17.159963+0000","last_fullsized":"2024-09-21T19:01:17.159963+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:16.593895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.593895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.530636+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":381,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156212+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:01:17.156212+0000","last_peered":"2024-09-21T19:01:17.156212+0000","last_clean":"2024-09-21T19:01:17.156212+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:01:17.156212+0000","last_undegraded":"2024-09-21T19:01:17.156212+0000","last_fullsized":"2024-09-21T19:01:17.156212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.239922+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":366,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151334+0000","last_change":"2024-09-21T19:00:15.241392+0000","last_active":"2024-09-21T19:01:17.151334+0000","last_peered":"2024-09-21T19:01:17.151334+0000","last_clean":"2024-09-21T19:01:17.151334+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:01:17.151334+0000","last_undegraded":"2024-09-21T19:01:17.151334+0000","last_fullsized":"2024-09-21T19:01:17.151334+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:15.241320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:15.241320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.208748+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":333,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157756+0000","last_change":"2024-09-21T18:59:19.930405+0000","last_active":"2024-09-21T19:01:17.157756+0000","last_peered":"2024-09-21T19:01:17.157756+0000","last_clean":"2024-09-21T19:01:17.157756+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T19:01:17.157756+0000","last_undegraded":"2024-09-21T19:01:17.157756+0000","last_fullsized":"2024-09-21T19:01:17.157756+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.867337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":360,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146187+0000","last_change":"2024-09-21T19:00:00.649580+0000","last_active":"2024-09-21T19:01:17.146187+0000","last_peered":"2024-09-21T19:01:17.146187+0000","last_clean":"2024-09-21T19:01:17.146187+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:01:17.146187+0000","last_undegraded":"2024-09-21T19:01:17.146187+0000","last_fullsized":"2024-09-21T19:01:17.146187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.649397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.649397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:24.015113+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":399,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151960+0000","last_change":"2024-09-21T19:00:18.857976+0000","last_active":"2024-09-21T19:01:17.151960+0000","last_peered":"2024-09-21T19:01:17.151960+0000","last_clean":"2024-09-21T19:01:17.151960+0000","last_became_active":"2024-09-21T18:58:23.993245+0000","last_became_peered":"2024-09-21T18:58:23.993245+0000","last_unstale":"2024-09-21T19:01:17.151960+0000","last_undegraded":"2024-09-21T19:01:17.151960+0000","last_fullsized":"2024-09-21T19:01:17.151960+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:00:18.857901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.857901+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.470039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":344,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146823+0000","last_change":"2024-09-21T18:58:28.912619+0000","last_active":"2024-09-21T19:01:17.146823+0000","last_peered":"2024-09-21T19:01:17.146823+0000","last_clean":"2024-09-21T19:01:17.146823+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:01:17.146823+0000","last_undegraded":"2024-09-21T19:01:17.146823+0000","last_fullsized":"2024-09-21T19:01:17.146823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:28.912554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:28.912554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":348,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158069+0000","last_change":"2024-09-21T18:58:38.110564+0000","last_active":"2024-09-21T19:01:17.158069+0000","last_peered":"2024-09-21T19:01:17.158069+0000","last_clean":"2024-09-21T19:01:17.158069+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:01:17.158069+0000","last_undegraded":"2024-09-21T19:01:17.158069+0000","last_fullsized":"2024-09-21T19:01:17.158069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:58:38.110509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:38.110509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":364,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151071+0000","last_change":"2024-09-21T19:00:23.223251+0000","last_active":"2024-09-21T19:01:17.151071+0000","last_peered":"2024-09-21T19:01:17.151071+0000","last_clean":"2024-09-21T19:01:17.151071+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:01:17.151071+0000","last_undegraded":"2024-09-21T19:01:17.151071+0000","last_fullsized":"2024-09-21T19:01:17.151071+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:23.223202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.223202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:35.046166+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":252,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160273+0000","last_change":"2024-09-21T19:00:22.256583+0000","last_active":"2024-09-21T19:01:17.160273+0000","last_peered":"2024-09-21T19:01:17.160273+0000","last_clean":"2024-09-21T19:01:17.160273+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:01:17.160273+0000","last_undegraded":"2024-09-21T19:01:17.160273+0000","last_fullsized":"2024-09-21T19:01:17.160273+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:22.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.256498+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.316225+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":461,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153509+0000","last_change":"2024-09-21T19:00:35.363014+0000","last_active":"2024-09-21T19:01:17.153509+0000","last_peered":"2024-09-21T19:01:17.153509+0000","last_clean":"2024-09-21T19:01:17.153509+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T19:01:17.153509+0000","last_undegraded":"2024-09-21T19:01:17.153509+0000","last_fullsized":"2024-09-21T19:01:17.153509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.428300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"219'12631","reported_seq":12786,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.023426+0000","last_change":"2024-09-21T19:01:16.860194+0000","last_active":"2024-09-21T19:01:22.023426+0000","last_peered":"2024-09-21T19:01:22.023426+0000","last_clean":"2024-09-21T19:01:22.023426+0000","last_became_active":"2024-09-21T18:58:44.098681+0000","last_became_peered":"2024-09-21T18:58:44.098681+0000","last_unstale":"2024-09-21T19:01:22.023426+0000","last_undegraded":"2024-09-21T19:01:22.023426+0000","last_fullsized":"2024-09-21T19:01:22.023426+0000","mapping_epoch":172,"log_start":"204'7600","ondisk_log_start":"204'7600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'12221","last_scrub_stamp":"2024-09-21T19:01:16.859976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.859976+0000","objects_scrubbed":4162,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:33.448541+0000","scrub_duration":1345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242810880,"num_objects":3705,"num_object_clones":0,"num_object_copies":11115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12631,"num_write_kb":522752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":125,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155824+0000","last_change":"2024-09-21T19:01:01.036323+0000","last_active":"2024-09-21T19:01:17.155824+0000","last_peered":"2024-09-21T19:01:17.155824+0000","last_clean":"2024-09-21T19:01:17.155824+0000","last_became_active":"2024-09-21T18:58:40.917818+0000","last_became_peered":"2024-09-21T18:58:40.917818+0000","last_unstale":"2024-09-21T19:01:17.155824+0000","last_undegraded":"2024-09-21T19:01:17.155824+0000","last_fullsized":"2024-09-21T19:01:17.155824+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:01.036275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:01.036275+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:01.626107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":373,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151528+0000","last_change":"2024-09-21T18:59:21.369411+0000","last_active":"2024-09-21T19:01:17.151528+0000","last_peered":"2024-09-21T19:01:17.151528+0000","last_clean":"2024-09-21T19:01:17.151528+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:01:17.151528+0000","last_undegraded":"2024-09-21T19:01:17.151528+0000","last_fullsized":"2024-09-21T19:01:17.151528+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:59:21.369328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.369328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.377767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":361,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151195+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:01:17.151195+0000","last_peered":"2024-09-21T19:01:17.151195+0000","last_clean":"2024-09-21T19:01:17.151195+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:01:17.151195+0000","last_undegraded":"2024-09-21T19:01:17.151195+0000","last_fullsized":"2024-09-21T19:01:17.151195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:17.164534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":481,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146453+0000","last_change":"2024-09-21T19:00:58.742094+0000","last_active":"2024-09-21T19:01:17.146453+0000","last_peered":"2024-09-21T19:01:17.146453+0000","last_clean":"2024-09-21T19:01:17.146453+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:01:17.146453+0000","last_undegraded":"2024-09-21T19:01:17.146453+0000","last_fullsized":"2024-09-21T19:01:17.146453+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:00:58.742016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:58.742016+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.133618+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":362,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156172+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T19:01:17.156172+0000","last_peered":"2024-09-21T19:01:17.156172+0000","last_clean":"2024-09-21T19:01:17.156172+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T19:01:17.156172+0000","last_undegraded":"2024-09-21T19:01:17.156172+0000","last_fullsized":"2024-09-21T19:01:17.156172+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.873896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"219'12912","reported_seq":13148,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.910596+0000","last_change":"2024-09-21T19:01:12.177132+0000","last_active":"2024-09-21T19:01:21.910596+0000","last_peered":"2024-09-21T19:01:21.910596+0000","last_clean":"2024-09-21T19:01:21.910596+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:01:21.910596+0000","last_undegraded":"2024-09-21T19:01:21.910596+0000","last_fullsized":"2024-09-21T19:01:21.910596+0000","mapping_epoch":172,"log_start":"202'7400","ondisk_log_start":"202'7400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12149","last_scrub_stamp":"2024-09-21T19:01:12.176892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:12.176892+0000","objects_scrubbed":4607,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.880107+0000","scrub_duration":1697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248381440,"num_objects":3790,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":534464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":117,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146710+0000","last_change":"2024-09-21T18:59:47.823699+0000","last_active":"2024-09-21T19:01:17.146710+0000","last_peered":"2024-09-21T19:01:17.146710+0000","last_clean":"2024-09-21T19:01:17.146710+0000","last_became_active":"2024-09-21T18:58:40.916167+0000","last_became_peered":"2024-09-21T18:58:40.916167+0000","last_unstale":"2024-09-21T19:01:17.146710+0000","last_undegraded":"2024-09-21T19:01:17.146710+0000","last_fullsized":"2024-09-21T19:01:17.146710+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.823624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.823624+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":432,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150965+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:01:17.150965+0000","last_peered":"2024-09-21T19:01:17.150965+0000","last_clean":"2024-09-21T19:01:17.150965+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:01:17.150965+0000","last_undegraded":"2024-09-21T19:01:17.150965+0000","last_fullsized":"2024-09-21T19:01:17.150965+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":375,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158302+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:01:17.158302+0000","last_peered":"2024-09-21T19:01:17.158302+0000","last_clean":"2024-09-21T19:01:17.158302+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:01:17.158302+0000","last_undegraded":"2024-09-21T19:01:17.158302+0000","last_fullsized":"2024-09-21T19:01:17.158302+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:28.008242+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":344,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.145850+0000","last_change":"2024-09-21T18:59:06.911709+0000","last_active":"2024-09-21T19:01:17.145850+0000","last_peered":"2024-09-21T19:01:17.145850+0000","last_clean":"2024-09-21T19:01:17.145850+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:01:17.145850+0000","last_undegraded":"2024-09-21T19:01:17.145850+0000","last_fullsized":"2024-09-21T19:01:17.145850+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.911638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.911638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":332,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146050+0000","last_change":"2024-09-21T18:59:13.953988+0000","last_active":"2024-09-21T19:01:17.146050+0000","last_peered":"2024-09-21T19:01:17.146050+0000","last_clean":"2024-09-21T19:01:17.146050+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:01:17.146050+0000","last_undegraded":"2024-09-21T19:01:17.146050+0000","last_fullsized":"2024-09-21T19:01:17.146050+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:13.953912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:13.953912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.742766+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"219'12838","reported_seq":13032,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.581511+0000","last_change":"2024-09-21T19:00:18.040765+0000","last_active":"2024-09-21T19:01:21.581511+0000","last_peered":"2024-09-21T19:01:21.581511+0000","last_clean":"2024-09-21T19:01:21.581511+0000","last_became_active":"2024-09-21T18:58:44.184359+0000","last_became_peered":"2024-09-21T18:58:44.184359+0000","last_unstale":"2024-09-21T19:01:21.581511+0000","last_undegraded":"2024-09-21T19:01:21.581511+0000","last_fullsized":"2024-09-21T19:01:21.581511+0000","mapping_epoch":172,"log_start":"204'7900","ondisk_log_start":"204'7900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'7943","last_scrub_stamp":"2024-09-21T19:00:18.040678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.040678+0000","objects_scrubbed":7868,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.698500+0000","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245825560,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12838,"num_write_kb":530817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":122,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.152602+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:01:17.152602+0000","last_peered":"2024-09-21T19:01:17.152602+0000","last_clean":"2024-09-21T19:01:17.152602+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:01:17.152602+0000","last_undegraded":"2024-09-21T19:01:17.152602+0000","last_fullsized":"2024-09-21T19:01:17.152602+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:49.951550+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":369,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.145949+0000","last_change":"2024-09-21T18:59:20.963196+0000","last_active":"2024-09-21T19:01:17.145949+0000","last_peered":"2024-09-21T19:01:17.145949+0000","last_clean":"2024-09-21T19:01:17.145949+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T19:01:17.145949+0000","last_undegraded":"2024-09-21T19:01:17.145949+0000","last_fullsized":"2024-09-21T19:01:17.145949+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:59:20.962753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.962753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.779332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":244,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.166911+0000","last_change":"2024-09-21T18:59:54.225013+0000","last_active":"2024-09-21T19:01:17.166911+0000","last_peered":"2024-09-21T19:01:17.166911+0000","last_clean":"2024-09-21T19:01:17.166911+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:01:17.166911+0000","last_undegraded":"2024-09-21T19:01:17.166911+0000","last_fullsized":"2024-09-21T19:01:17.166911+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:54.224944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.224944+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":445,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150818+0000","last_change":"2024-09-21T19:00:22.845024+0000","last_active":"2024-09-21T19:01:17.150818+0000","last_peered":"2024-09-21T19:01:17.150818+0000","last_clean":"2024-09-21T19:01:17.150818+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:01:17.150818+0000","last_undegraded":"2024-09-21T19:01:17.150818+0000","last_fullsized":"2024-09-21T19:01:17.150818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:22.844953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.844953+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:26.214253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":339,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157789+0000","last_change":"2024-09-21T19:01:06.644894+0000","last_active":"2024-09-21T19:01:17.157789+0000","last_peered":"2024-09-21T19:01:17.157789+0000","last_clean":"2024-09-21T19:01:17.157789+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T19:01:17.157789+0000","last_undegraded":"2024-09-21T19:01:17.157789+0000","last_fullsized":"2024-09-21T19:01:17.157789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.644848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.644848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.908909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"219'12850","reported_seq":13010,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.915289+0000","last_change":"2024-09-21T19:01:11.369577+0000","last_active":"2024-09-21T19:01:21.915289+0000","last_peered":"2024-09-21T19:01:21.915289+0000","last_clean":"2024-09-21T19:01:21.915289+0000","last_became_active":"2024-09-21T18:58:44.096773+0000","last_became_peered":"2024-09-21T18:58:44.096773+0000","last_unstale":"2024-09-21T19:01:21.915289+0000","last_undegraded":"2024-09-21T19:01:21.915289+0000","last_fullsized":"2024-09-21T19:01:21.915289+0000","mapping_epoch":172,"log_start":"202'7300","ondisk_log_start":"202'7300","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12063","last_scrub_stamp":"2024-09-21T19:01:11.369324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.369324+0000","objects_scrubbed":4626,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:37.327409+0000","scrub_duration":1905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247332864,"num_objects":3774,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12850,"num_write_kb":531968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":201,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153321+0000","last_change":"2024-09-21T18:59:46.185760+0000","last_active":"2024-09-21T19:01:17.153321+0000","last_peered":"2024-09-21T19:01:17.153321+0000","last_clean":"2024-09-21T19:01:17.153321+0000","last_became_active":"2024-09-21T18:58:40.530131+0000","last_became_peered":"2024-09-21T18:58:40.530131+0000","last_unstale":"2024-09-21T19:01:17.153321+0000","last_undegraded":"2024-09-21T19:01:17.153321+0000","last_fullsized":"2024-09-21T19:01:17.153321+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.185693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.185693+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":355,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150617+0000","last_change":"2024-09-21T18:59:17.346990+0000","last_active":"2024-09-21T19:01:17.150617+0000","last_peered":"2024-09-21T19:01:17.150617+0000","last_clean":"2024-09-21T19:01:17.150617+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:01:17.150617+0000","last_undegraded":"2024-09-21T19:01:17.150617+0000","last_fullsized":"2024-09-21T19:01:17.150617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:17.346414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.346414+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.396410+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":360,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156794+0000","last_change":"2024-09-21T18:59:16.589986+0000","last_active":"2024-09-21T19:01:17.156794+0000","last_peered":"2024-09-21T19:01:17.156794+0000","last_clean":"2024-09-21T19:01:17.156794+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T19:01:17.156794+0000","last_undegraded":"2024-09-21T19:01:17.156794+0000","last_fullsized":"2024-09-21T19:01:17.156794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.589912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.589912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.916495+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":294,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.159920+0000","last_change":"2024-09-21T18:55:51.541778+0000","last_active":"2024-09-21T19:01:17.159920+0000","last_peered":"2024-09-21T19:01:17.159920+0000","last_clean":"2024-09-21T19:01:17.159920+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T19:01:17.159920+0000","last_undegraded":"2024-09-21T19:01:17.159920+0000","last_fullsized":"2024-09-21T19:01:17.159920+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'9","last_scrub_stamp":"2024-09-21T18:54:41.272584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:54:41.272584+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.258988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.159008+0000","last_change":"2024-09-21T18:59:44.256590+0000","last_active":"2024-09-21T19:01:17.159008+0000","last_peered":"2024-09-21T19:01:17.159008+0000","last_clean":"2024-09-21T19:01:17.159008+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:01:17.159008+0000","last_undegraded":"2024-09-21T19:01:17.159008+0000","last_fullsized":"2024-09-21T19:01:17.159008+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:44.256513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.256513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.406650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"219'12822","reported_seq":12925,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.409571+0000","last_change":"2024-09-21T18:58:44.097045+0000","last_active":"2024-09-21T19:01:22.409571+0000","last_peered":"2024-09-21T19:01:22.409571+0000","last_clean":"2024-09-21T19:01:22.409571+0000","last_became_active":"2024-09-21T18:58:44.096634+0000","last_became_peered":"2024-09-21T18:58:44.096634+0000","last_unstale":"2024-09-21T19:01:22.409571+0000","last_undegraded":"2024-09-21T19:01:22.409571+0000","last_fullsized":"2024-09-21T19:01:22.409571+0000","mapping_epoch":172,"log_start":"205'8300","ondisk_log_start":"205'8300","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.889384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244973568,"num_objects":3738,"num_object_clones":0,"num_object_copies":11214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12822,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":209,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155904+0000","last_change":"2024-09-21T19:01:00.420038+0000","last_active":"2024-09-21T19:01:17.155904+0000","last_peered":"2024-09-21T19:01:17.155904+0000","last_clean":"2024-09-21T19:01:17.155904+0000","last_became_active":"2024-09-21T18:58:40.528964+0000","last_became_peered":"2024-09-21T18:58:40.528964+0000","last_unstale":"2024-09-21T19:01:17.155904+0000","last_undegraded":"2024-09-21T19:01:17.155904+0000","last_fullsized":"2024-09-21T19:01:17.155904+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:00.419946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.419946+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.968029+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":266,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160041+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:01:17.160041+0000","last_peered":"2024-09-21T19:01:17.160041+0000","last_clean":"2024-09-21T19:01:17.160041+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:01:17.160041+0000","last_undegraded":"2024-09-21T19:01:17.160041+0000","last_fullsized":"2024-09-21T19:01:17.160041+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.035114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":360,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150844+0000","last_change":"2024-09-21T19:00:56.467643+0000","last_active":"2024-09-21T19:01:17.150844+0000","last_peered":"2024-09-21T19:01:17.150844+0000","last_clean":"2024-09-21T19:01:17.150844+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:01:17.150844+0000","last_undegraded":"2024-09-21T19:01:17.150844+0000","last_fullsized":"2024-09-21T19:01:17.150844+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.462694+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":357,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.152982+0000","last_change":"2024-09-21T19:00:34.409013+0000","last_active":"2024-09-21T19:01:17.152982+0000","last_peered":"2024-09-21T19:01:17.152982+0000","last_clean":"2024-09-21T19:01:17.152982+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T19:01:17.152982+0000","last_undegraded":"2024-09-21T19:01:17.152982+0000","last_fullsized":"2024-09-21T19:01:17.152982+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.816307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150797+0000","last_change":"2024-09-21T19:00:25.249425+0000","last_active":"2024-09-21T19:01:17.150797+0000","last_peered":"2024-09-21T19:01:17.150797+0000","last_clean":"2024-09-21T19:01:17.150797+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T19:01:17.150797+0000","last_undegraded":"2024-09-21T19:01:17.150797+0000","last_fullsized":"2024-09-21T19:01:17.150797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.249360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.249360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:32.354078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"219'12374","reported_seq":12531,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.563090+0000","last_change":"2024-09-21T19:01:16.186191+0000","last_active":"2024-09-21T19:01:21.563090+0000","last_peered":"2024-09-21T19:01:21.563090+0000","last_clean":"2024-09-21T19:01:21.563090+0000","last_became_active":"2024-09-21T18:58:44.182910+0000","last_became_peered":"2024-09-21T18:58:44.182910+0000","last_unstale":"2024-09-21T19:01:21.563090+0000","last_undegraded":"2024-09-21T19:01:21.563090+0000","last_fullsized":"2024-09-21T19:01:21.563090+0000","mapping_epoch":172,"log_start":"203'7400","ondisk_log_start":"203'7400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'11951","last_scrub_stamp":"2024-09-21T19:01:16.186043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.186043+0000","objects_scrubbed":4160,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:16.214720+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242221056,"num_objects":3696,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12374,"num_write_kb":514240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":125,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156657+0000","last_change":"2024-09-21T19:00:55.902634+0000","last_active":"2024-09-21T19:01:17.156657+0000","last_peered":"2024-09-21T19:01:17.156657+0000","last_clean":"2024-09-21T19:01:17.156657+0000","last_became_active":"2024-09-21T18:58:40.916397+0000","last_became_peered":"2024-09-21T18:58:40.916397+0000","last_unstale":"2024-09-21T19:01:17.156657+0000","last_undegraded":"2024-09-21T19:01:17.156657+0000","last_fullsized":"2024-09-21T19:01:17.156657+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:55.902117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:55.902117+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.038393+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":495,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150737+0000","last_change":"2024-09-21T18:59:44.360729+0000","last_active":"2024-09-21T19:01:17.150737+0000","last_peered":"2024-09-21T19:01:17.150737+0000","last_clean":"2024-09-21T19:01:17.150737+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:01:17.150737+0000","last_undegraded":"2024-09-21T19:01:17.150737+0000","last_fullsized":"2024-09-21T19:01:17.150737+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:59:44.360619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.360619+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.396002+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":374,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146476+0000","last_change":"2024-09-21T18:59:55.645493+0000","last_active":"2024-09-21T19:01:17.146476+0000","last_peered":"2024-09-21T19:01:17.146476+0000","last_clean":"2024-09-21T19:01:17.146476+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T19:01:17.146476+0000","last_undegraded":"2024-09-21T19:01:17.146476+0000","last_fullsized":"2024-09-21T19:01:17.146476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:55.645419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.645419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:19.840924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":378,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156566+0000","last_change":"2024-09-21T19:00:54.906034+0000","last_active":"2024-09-21T19:01:17.156566+0000","last_peered":"2024-09-21T19:01:17.156566+0000","last_clean":"2024-09-21T19:01:17.156566+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T19:01:17.156566+0000","last_undegraded":"2024-09-21T19:01:17.156566+0000","last_fullsized":"2024-09-21T19:01:17.156566+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:00:54.905971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:54.905971+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:05.061588+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160632+0000","last_change":"2024-09-21T19:00:25.316275+0000","last_active":"2024-09-21T19:01:17.160632+0000","last_peered":"2024-09-21T19:01:17.160632+0000","last_clean":"2024-09-21T19:01:17.160632+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:01:17.160632+0000","last_undegraded":"2024-09-21T19:01:17.160632+0000","last_fullsized":"2024-09-21T19:01:17.160632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.314718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.314718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.602505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":341,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.152284+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:01:17.152284+0000","last_peered":"2024-09-21T19:01:17.152284+0000","last_clean":"2024-09-21T19:01:17.152284+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:01:17.152284+0000","last_undegraded":"2024-09-21T19:01:17.152284+0000","last_fullsized":"2024-09-21T19:01:17.152284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"219'12932","reported_seq":13038,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.377982+0000","last_change":"2024-09-21T18:58:44.100975+0000","last_active":"2024-09-21T19:01:22.377982+0000","last_peered":"2024-09-21T19:01:22.377982+0000","last_clean":"2024-09-21T19:01:22.377982+0000","last_became_active":"2024-09-21T18:58:44.100676+0000","last_became_peered":"2024-09-21T18:58:44.100676+0000","last_unstale":"2024-09-21T19:01:22.377982+0000","last_undegraded":"2024-09-21T19:01:22.377982+0000","last_fullsized":"2024-09-21T19:01:22.377982+0000","mapping_epoch":172,"log_start":"205'8400","ondisk_log_start":"205'8400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:17.750120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246415360,"num_objects":3760,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12932,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":438,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158307+0000","last_change":"2024-09-21T19:01:05.625849+0000","last_active":"2024-09-21T19:01:17.158307+0000","last_peered":"2024-09-21T19:01:17.158307+0000","last_clean":"2024-09-21T19:01:17.158307+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:01:17.158307+0000","last_undegraded":"2024-09-21T19:01:17.158307+0000","last_fullsized":"2024-09-21T19:01:17.158307+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:05.625798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.625798+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.456539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":328,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157233+0000","last_change":"2024-09-21T19:01:01.980676+0000","last_active":"2024-09-21T19:01:17.157233+0000","last_peered":"2024-09-21T19:01:17.157233+0000","last_clean":"2024-09-21T19:01:17.157233+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:01:17.157233+0000","last_undegraded":"2024-09-21T19:01:17.157233+0000","last_fullsized":"2024-09-21T19:01:17.157233+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T19:01:01.980397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:01.980397+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.229183+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":367,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151658+0000","last_change":"2024-09-21T19:01:07.496476+0000","last_active":"2024-09-21T19:01:17.151658+0000","last_peered":"2024-09-21T19:01:17.151658+0000","last_clean":"2024-09-21T19:01:17.151658+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T19:01:17.151658+0000","last_undegraded":"2024-09-21T19:01:17.151658+0000","last_fullsized":"2024-09-21T19:01:17.151658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.105065+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":399,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157934+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:01:17.157934+0000","last_peered":"2024-09-21T19:01:17.157934+0000","last_clean":"2024-09-21T19:01:17.157934+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:01:17.157934+0000","last_undegraded":"2024-09-21T19:01:17.157934+0000","last_fullsized":"2024-09-21T19:01:17.157934+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.686451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157197+0000","last_change":"2024-09-21T19:00:29.002389+0000","last_active":"2024-09-21T19:01:17.157197+0000","last_peered":"2024-09-21T19:01:17.157197+0000","last_clean":"2024-09-21T19:01:17.157197+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:01:17.157197+0000","last_undegraded":"2024-09-21T19:01:17.157197+0000","last_fullsized":"2024-09-21T19:01:17.157197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.002321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.002321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:56.121924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":343,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160190+0000","last_change":"2024-09-21T19:00:32.573881+0000","last_active":"2024-09-21T19:01:17.160190+0000","last_peered":"2024-09-21T19:01:17.160190+0000","last_clean":"2024-09-21T19:01:17.160190+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T19:01:17.160190+0000","last_undegraded":"2024-09-21T19:01:17.160190+0000","last_fullsized":"2024-09-21T19:01:17.160190+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:00:32.573704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.573704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.007025+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"219'12708","reported_seq":12943,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.040423+0000","last_change":"2024-09-21T19:01:05.288396+0000","last_active":"2024-09-21T19:01:22.040423+0000","last_peered":"2024-09-21T19:01:22.040423+0000","last_clean":"2024-09-21T19:01:22.040423+0000","last_became_active":"2024-09-21T18:58:44.099439+0000","last_became_peered":"2024-09-21T18:58:44.099439+0000","last_unstale":"2024-09-21T19:01:22.040423+0000","last_undegraded":"2024-09-21T19:01:22.040423+0000","last_fullsized":"2024-09-21T19:01:22.040423+0000","mapping_epoch":172,"log_start":"204'7700","ondisk_log_start":"204'7700","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11506","last_scrub_stamp":"2024-09-21T19:01:05.288327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.288327+0000","objects_scrubbed":4952,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:28.984314+0000","scrub_duration":1801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240910336,"num_objects":3676,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12708,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":125,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151283+0000","last_change":"2024-09-21T19:00:59.488920+0000","last_active":"2024-09-21T19:01:17.151283+0000","last_peered":"2024-09-21T19:01:17.151283+0000","last_clean":"2024-09-21T19:01:17.151283+0000","last_became_active":"2024-09-21T18:58:40.528545+0000","last_became_peered":"2024-09-21T18:58:40.528545+0000","last_unstale":"2024-09-21T19:01:17.151283+0000","last_undegraded":"2024-09-21T19:01:17.151283+0000","last_fullsized":"2024-09-21T19:01:17.151283+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:59.488847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:59.488847+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.549020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":262,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.166766+0000","last_change":"2024-09-21T19:01:13.569543+0000","last_active":"2024-09-21T19:01:17.166766+0000","last_peered":"2024-09-21T19:01:17.166766+0000","last_clean":"2024-09-21T19:01:17.166766+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:01:17.166766+0000","last_undegraded":"2024-09-21T19:01:17.166766+0000","last_fullsized":"2024-09-21T19:01:17.166766+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:01:13.569464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.569464+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:19.751336+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":240,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.163306+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:01:17.163306+0000","last_peered":"2024-09-21T19:01:17.163306+0000","last_clean":"2024-09-21T19:01:17.163306+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:01:17.163306+0000","last_undegraded":"2024-09-21T19:01:17.163306+0000","last_fullsized":"2024-09-21T19:01:17.163306+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":873,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:19.965840+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T19:01:19.965840+0000","last_peered":"2024-09-21T19:01:19.965840+0000","last_clean":"2024-09-21T19:01:19.965840+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:01:19.965840+0000","last_undegraded":"2024-09-21T19:01:19.965840+0000","last_fullsized":"2024-09-21T19:01:19.965840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.500330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":483,"num_read_kb":478,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":382,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156445+0000","last_change":"2024-09-21T19:00:56.950003+0000","last_active":"2024-09-21T19:01:17.156445+0000","last_peered":"2024-09-21T19:01:17.156445+0000","last_clean":"2024-09-21T19:01:17.156445+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:01:17.156445+0000","last_undegraded":"2024-09-21T19:01:17.156445+0000","last_fullsized":"2024-09-21T19:01:17.156445+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.949931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.949931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.427785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150375+0000","last_change":"2024-09-21T19:00:33.443101+0000","last_active":"2024-09-21T19:01:17.150375+0000","last_peered":"2024-09-21T19:01:17.150375+0000","last_clean":"2024-09-21T19:01:17.150375+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:01:17.150375+0000","last_undegraded":"2024-09-21T19:01:17.150375+0000","last_fullsized":"2024-09-21T19:01:17.150375+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:33.443040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.443040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.013096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"219'13013","reported_seq":13194,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.566592+0000","last_change":"2024-09-21T19:00:00.245865+0000","last_active":"2024-09-21T19:01:21.566592+0000","last_peered":"2024-09-21T19:01:21.566592+0000","last_clean":"2024-09-21T19:01:21.566592+0000","last_became_active":"2024-09-21T18:58:44.096509+0000","last_became_peered":"2024-09-21T18:58:44.096509+0000","last_unstale":"2024-09-21T19:01:21.566592+0000","last_undegraded":"2024-09-21T19:01:21.566592+0000","last_fullsized":"2024-09-21T19:01:21.566592+0000","mapping_epoch":172,"log_start":"202'7400","ondisk_log_start":"202'7400","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'6787","last_scrub_stamp":"2024-09-21T19:00:00.245707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.245707+0000","objects_scrubbed":6706,"log_size":5613,"log_dups_size":0,"ondisk_log_size":5613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:29.145095+0000","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251330560,"num_objects":3835,"num_object_clones":0,"num_object_copies":11505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13013,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":468,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.154848+0000","last_change":"2024-09-21T18:59:12.162986+0000","last_active":"2024-09-21T19:01:17.154848+0000","last_peered":"2024-09-21T19:01:17.154848+0000","last_clean":"2024-09-21T19:01:17.154848+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T19:01:17.154848+0000","last_undegraded":"2024-09-21T19:01:17.154848+0000","last_fullsized":"2024-09-21T19:01:17.154848+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:59:12.162725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.162725+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.317843+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":353,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151500+0000","last_change":"2024-09-21T18:59:50.993922+0000","last_active":"2024-09-21T19:01:17.151500+0000","last_peered":"2024-09-21T19:01:17.151500+0000","last_clean":"2024-09-21T19:01:17.151500+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T19:01:17.151500+0000","last_undegraded":"2024-09-21T19:01:17.151500+0000","last_fullsized":"2024-09-21T19:01:17.151500+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":433,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158413+0000","last_change":"2024-09-21T19:00:57.551189+0000","last_active":"2024-09-21T19:01:17.158413+0000","last_peered":"2024-09-21T19:01:17.158413+0000","last_clean":"2024-09-21T19:01:17.158413+0000","last_became_active":"2024-09-21T18:58:39.058347+0000","last_became_peered":"2024-09-21T18:58:39.058347+0000","last_unstale":"2024-09-21T19:01:17.158413+0000","last_undegraded":"2024-09-21T19:01:17.158413+0000","last_fullsized":"2024-09-21T19:01:17.158413+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:57.551119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:57.551119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.725028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":343,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158763+0000","last_change":"2024-09-21T19:00:00.889227+0000","last_active":"2024-09-21T19:01:17.158763+0000","last_peered":"2024-09-21T19:01:17.158763+0000","last_clean":"2024-09-21T19:01:17.158763+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:01:17.158763+0000","last_undegraded":"2024-09-21T19:01:17.158763+0000","last_fullsized":"2024-09-21T19:01:17.158763+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.889156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.889156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":348,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156316+0000","last_change":"2024-09-21T19:01:10.038311+0000","last_active":"2024-09-21T19:01:17.156316+0000","last_peered":"2024-09-21T19:01:17.156316+0000","last_clean":"2024-09-21T19:01:17.156316+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T19:01:17.156316+0000","last_undegraded":"2024-09-21T19:01:17.156316+0000","last_fullsized":"2024-09-21T19:01:17.156316+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:37.189738+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"219'12817","reported_seq":12977,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.524455+0000","last_change":"2024-09-21T19:01:10.721872+0000","last_active":"2024-09-21T19:01:21.524455+0000","last_peered":"2024-09-21T19:01:21.524455+0000","last_clean":"2024-09-21T19:01:21.524455+0000","last_became_active":"2024-09-21T18:58:44.096986+0000","last_became_peered":"2024-09-21T18:58:44.096986+0000","last_unstale":"2024-09-21T19:01:21.524455+0000","last_undegraded":"2024-09-21T19:01:21.524455+0000","last_fullsized":"2024-09-21T19:01:21.524455+0000","mapping_epoch":172,"log_start":"204'8200","ondisk_log_start":"204'8200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12018","last_scrub_stamp":"2024-09-21T19:01:10.721645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.721645+0000","objects_scrubbed":4572,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.013892+0000","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245432320,"num_objects":3745,"num_object_clones":0,"num_object_copies":11235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12817,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":375,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146506+0000","last_change":"2024-09-21T19:00:01.693318+0000","last_active":"2024-09-21T19:01:17.146506+0000","last_peered":"2024-09-21T19:01:17.146506+0000","last_clean":"2024-09-21T19:01:17.146506+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T19:01:17.146506+0000","last_undegraded":"2024-09-21T19:01:17.146506+0000","last_fullsized":"2024-09-21T19:01:17.146506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:01.693221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.693221+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.644648+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146077+0000","last_change":"2024-09-21T19:00:27.721451+0000","last_active":"2024-09-21T19:01:17.146077+0000","last_peered":"2024-09-21T19:01:17.146077+0000","last_clean":"2024-09-21T19:01:17.146077+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:01:17.146077+0000","last_undegraded":"2024-09-21T19:01:17.146077+0000","last_fullsized":"2024-09-21T19:01:17.146077+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:27.721385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:27.721385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.383225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":368,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156179+0000","last_change":"2024-09-21T19:01:02.080311+0000","last_active":"2024-09-21T19:01:17.156179+0000","last_peered":"2024-09-21T19:01:17.156179+0000","last_clean":"2024-09-21T19:01:17.156179+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T19:01:17.156179+0000","last_undegraded":"2024-09-21T19:01:17.156179+0000","last_fullsized":"2024-09-21T19:01:17.156179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.080248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.080248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.771263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":325,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.152682+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T19:01:17.152682+0000","last_peered":"2024-09-21T19:01:17.152682+0000","last_clean":"2024-09-21T19:01:17.152682+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T19:01:17.152682+0000","last_undegraded":"2024-09-21T19:01:17.152682+0000","last_fullsized":"2024-09-21T19:01:17.152682+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.918918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"219'12775","reported_seq":12969,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.024376+0000","last_change":"2024-09-21T19:00:19.801173+0000","last_active":"2024-09-21T19:01:22.024376+0000","last_peered":"2024-09-21T19:01:22.024376+0000","last_clean":"2024-09-21T19:01:22.024376+0000","last_became_active":"2024-09-21T18:58:44.097827+0000","last_became_peered":"2024-09-21T18:58:44.097827+0000","last_unstale":"2024-09-21T19:01:22.024376+0000","last_undegraded":"2024-09-21T19:01:22.024376+0000","last_fullsized":"2024-09-21T19:01:22.024376+0000","mapping_epoch":172,"log_start":"203'7700","ondisk_log_start":"203'7700","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8051","last_scrub_stamp":"2024-09-21T19:00:19.800992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.800992+0000","objects_scrubbed":7979,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.556183+0000","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247529472,"num_objects":3777,"num_object_clones":0,"num_object_copies":11331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":529664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":132,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155688+0000","last_change":"2024-09-21T19:01:05.092005+0000","last_active":"2024-09-21T19:01:17.155688+0000","last_peered":"2024-09-21T19:01:17.155688+0000","last_clean":"2024-09-21T19:01:17.155688+0000","last_became_active":"2024-09-21T18:58:40.917793+0000","last_became_peered":"2024-09-21T18:58:40.917793+0000","last_unstale":"2024-09-21T19:01:17.155688+0000","last_undegraded":"2024-09-21T19:01:17.155688+0000","last_fullsized":"2024-09-21T19:01:17.155688+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:05.091927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.091927+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.822244+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":376,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146598+0000","last_change":"2024-09-21T18:59:03.883391+0000","last_active":"2024-09-21T19:01:17.146598+0000","last_peered":"2024-09-21T19:01:17.146598+0000","last_clean":"2024-09-21T19:01:17.146598+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:01:17.146598+0000","last_undegraded":"2024-09-21T19:01:17.146598+0000","last_fullsized":"2024-09-21T19:01:17.146598+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:59:03.883306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:03.883306+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.725718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":359,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153280+0000","last_change":"2024-09-21T19:01:14.376753+0000","last_active":"2024-09-21T19:01:17.153280+0000","last_peered":"2024-09-21T19:01:17.153280+0000","last_clean":"2024-09-21T19:01:17.153280+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T19:01:17.153280+0000","last_undegraded":"2024-09-21T19:01:17.153280+0000","last_fullsized":"2024-09-21T19:01:17.153280+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:14.376671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:14.376671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:34.966862+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":350,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150492+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T19:01:17.150492+0000","last_peered":"2024-09-21T19:01:17.150492+0000","last_clean":"2024-09-21T19:01:17.150492+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:01:17.150492+0000","last_undegraded":"2024-09-21T19:01:17.150492+0000","last_fullsized":"2024-09-21T19:01:17.150492+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.313883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157157+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:01:17.157157+0000","last_peered":"2024-09-21T19:01:17.157157+0000","last_clean":"2024-09-21T19:01:17.157157+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:01:17.157157+0000","last_undegraded":"2024-09-21T19:01:17.157157+0000","last_fullsized":"2024-09-21T19:01:17.157157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.899930+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"219'12771","reported_seq":12929,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.380736+0000","last_change":"2024-09-21T19:01:11.670563+0000","last_active":"2024-09-21T19:01:22.380736+0000","last_peered":"2024-09-21T19:01:22.380736+0000","last_clean":"2024-09-21T19:01:22.380736+0000","last_became_active":"2024-09-21T18:58:44.096142+0000","last_became_peered":"2024-09-21T18:58:44.096142+0000","last_unstale":"2024-09-21T19:01:22.380736+0000","last_undegraded":"2024-09-21T19:01:22.380736+0000","last_fullsized":"2024-09-21T19:01:22.380736+0000","mapping_epoch":172,"log_start":"205'8200","ondisk_log_start":"205'8200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11953","last_scrub_stamp":"2024-09-21T19:01:11.670275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.670275+0000","objects_scrubbed":4463,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:34.755111+0000","scrub_duration":1749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235732992,"num_objects":3597,"num_object_clones":0,"num_object_copies":10791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12771,"num_write_kb":523776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":430,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158005+0000","last_change":"2024-09-21T18:59:50.870015+0000","last_active":"2024-09-21T19:01:17.158005+0000","last_peered":"2024-09-21T19:01:17.158005+0000","last_clean":"2024-09-21T19:01:17.158005+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:01:17.158005+0000","last_undegraded":"2024-09-21T19:01:17.158005+0000","last_fullsized":"2024-09-21T19:01:17.158005+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:50.869945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.869945+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":363,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146874+0000","last_change":"2024-09-21T18:59:53.699627+0000","last_active":"2024-09-21T19:01:17.146874+0000","last_peered":"2024-09-21T19:01:17.146874+0000","last_clean":"2024-09-21T19:01:17.146874+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T19:01:17.146874+0000","last_undegraded":"2024-09-21T19:01:17.146874+0000","last_fullsized":"2024-09-21T19:01:17.146874+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":389,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.154710+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:01:17.154710+0000","last_peered":"2024-09-21T19:01:17.154710+0000","last_clean":"2024-09-21T19:01:17.154710+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:01:17.154710+0000","last_undegraded":"2024-09-21T19:01:17.154710+0000","last_fullsized":"2024-09-21T19:01:17.154710+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:26.395243+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":467,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157155+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:01:17.157155+0000","last_peered":"2024-09-21T19:01:17.157155+0000","last_clean":"2024-09-21T19:01:17.157155+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:01:17.157155+0000","last_undegraded":"2024-09-21T19:01:17.157155+0000","last_fullsized":"2024-09-21T19:01:17.157155+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.071423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146455+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:01:17.146455+0000","last_peered":"2024-09-21T19:01:17.146455+0000","last_clean":"2024-09-21T19:01:17.146455+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:01:17.146455+0000","last_undegraded":"2024-09-21T19:01:17.146455+0000","last_fullsized":"2024-09-21T19:01:17.146455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.791557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"219'12906","reported_seq":13071,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:22.046022+0000","last_change":"2024-09-21T19:01:08.501885+0000","last_active":"2024-09-21T19:01:22.046022+0000","last_peered":"2024-09-21T19:01:22.046022+0000","last_clean":"2024-09-21T19:01:22.046022+0000","last_became_active":"2024-09-21T18:58:44.100559+0000","last_became_peered":"2024-09-21T18:58:44.100559+0000","last_unstale":"2024-09-21T19:01:22.046022+0000","last_undegraded":"2024-09-21T19:01:22.046022+0000","last_fullsized":"2024-09-21T19:01:22.046022+0000","mapping_epoch":172,"log_start":"204'7900","ondisk_log_start":"204'7900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11887","last_scrub_stamp":"2024-09-21T19:01:08.501670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:08.501670+0000","objects_scrubbed":4884,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.153152+0000","scrub_duration":1984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249298944,"num_objects":3804,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12906,"num_write_kb":534720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":234,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160121+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T19:01:17.160121+0000","last_peered":"2024-09-21T19:01:17.160121+0000","last_clean":"2024-09-21T19:01:17.160121+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:01:17.160121+0000","last_undegraded":"2024-09-21T19:01:17.160121+0000","last_fullsized":"2024-09-21T19:01:17.160121+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.261047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":360,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156322+0000","last_change":"2024-09-21T18:59:15.560811+0000","last_active":"2024-09-21T19:01:17.156322+0000","last_peered":"2024-09-21T19:01:17.156322+0000","last_clean":"2024-09-21T19:01:17.156322+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T19:01:17.156322+0000","last_undegraded":"2024-09-21T19:01:17.156322+0000","last_fullsized":"2024-09-21T19:01:17.156322+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:15.560713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:15.560713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.858874+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":344,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155356+0000","last_change":"2024-09-21T18:59:43.841286+0000","last_active":"2024-09-21T19:01:17.155356+0000","last_peered":"2024-09-21T19:01:17.155356+0000","last_clean":"2024-09-21T19:01:17.155356+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:01:17.155356+0000","last_undegraded":"2024-09-21T19:01:17.155356+0000","last_fullsized":"2024-09-21T19:01:17.155356+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:43.841222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:43.841222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.957920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.163223+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:01:17.163223+0000","last_peered":"2024-09-21T19:01:17.163223+0000","last_clean":"2024-09-21T19:01:17.163223+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:01:17.163223+0000","last_undegraded":"2024-09-21T19:01:17.163223+0000","last_fullsized":"2024-09-21T19:01:17.163223+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.293410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"219'12707","reported_seq":12941,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.883885+0000","last_change":"2024-09-21T19:01:04.185407+0000","last_active":"2024-09-21T19:01:21.883885+0000","last_peered":"2024-09-21T19:01:21.883885+0000","last_clean":"2024-09-21T19:01:21.883885+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:01:21.883885+0000","last_undegraded":"2024-09-21T19:01:21.883885+0000","last_fullsized":"2024-09-21T19:01:21.883885+0000","mapping_epoch":172,"log_start":"202'7200","ondisk_log_start":"202'7200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11433","last_scrub_stamp":"2024-09-21T19:01:04.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.185285+0000","objects_scrubbed":5079,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.471224+0000","scrub_duration":1738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245170176,"num_objects":3741,"num_object_clones":0,"num_object_copies":11223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12707,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":378,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153080+0000","last_change":"2024-09-21T19:00:23.328177+0000","last_active":"2024-09-21T19:01:17.153080+0000","last_peered":"2024-09-21T19:01:17.153080+0000","last_clean":"2024-09-21T19:01:17.153080+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T19:01:17.153080+0000","last_undegraded":"2024-09-21T19:01:17.153080+0000","last_fullsized":"2024-09-21T19:01:17.153080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.465067+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":359,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160438+0000","last_change":"2024-09-21T19:01:10.619281+0000","last_active":"2024-09-21T19:01:17.160438+0000","last_peered":"2024-09-21T19:01:17.160438+0000","last_clean":"2024-09-21T19:01:17.160438+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T19:01:17.160438+0000","last_undegraded":"2024-09-21T19:01:17.160438+0000","last_fullsized":"2024-09-21T19:01:17.160438+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.619235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.619235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:12.039735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":358,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155201+0000","last_change":"2024-09-21T19:00:30.996323+0000","last_active":"2024-09-21T19:01:17.155201+0000","last_peered":"2024-09-21T19:01:17.155201+0000","last_clean":"2024-09-21T19:01:17.155201+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:01:17.155201+0000","last_undegraded":"2024-09-21T19:01:17.155201+0000","last_fullsized":"2024-09-21T19:01:17.155201+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:30.996260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:30.996260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:34.407198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":339,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158331+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:01:17.158331+0000","last_peered":"2024-09-21T19:01:17.158331+0000","last_clean":"2024-09-21T19:01:17.158331+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:01:17.158331+0000","last_undegraded":"2024-09-21T19:01:17.158331+0000","last_fullsized":"2024-09-21T19:01:17.158331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"219'12762","reported_seq":12997,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.524616+0000","last_change":"2024-09-21T19:01:03.618355+0000","last_active":"2024-09-21T19:01:21.524616+0000","last_peered":"2024-09-21T19:01:21.524616+0000","last_clean":"2024-09-21T19:01:21.524616+0000","last_became_active":"2024-09-21T18:58:44.184048+0000","last_became_peered":"2024-09-21T18:58:44.184048+0000","last_unstale":"2024-09-21T19:01:21.524616+0000","last_undegraded":"2024-09-21T19:01:21.524616+0000","last_fullsized":"2024-09-21T19:01:21.524616+0000","mapping_epoch":172,"log_start":"201'7200","ondisk_log_start":"201'7200","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11443","last_scrub_stamp":"2024-09-21T19:01:03.618148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.618148+0000","objects_scrubbed":5222,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:27.366618+0000","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252444672,"num_objects":3852,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12762,"num_write_kb":531648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":390,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.145986+0000","last_change":"2024-09-21T18:59:17.862166+0000","last_active":"2024-09-21T19:01:17.145986+0000","last_peered":"2024-09-21T19:01:17.145986+0000","last_clean":"2024-09-21T19:01:17.145986+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:01:17.145986+0000","last_undegraded":"2024-09-21T19:01:17.145986+0000","last_fullsized":"2024-09-21T19:01:17.145986+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:17.862105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.862105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.784245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":359,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.162033+0000","last_change":"2024-09-21T19:00:22.616468+0000","last_active":"2024-09-21T19:01:17.162033+0000","last_peered":"2024-09-21T19:01:17.162033+0000","last_clean":"2024-09-21T19:01:17.162033+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:01:17.162033+0000","last_undegraded":"2024-09-21T19:01:17.162033+0000","last_fullsized":"2024-09-21T19:01:17.162033+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:22.616386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.616386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:52.426205+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":366,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155443+0000","last_change":"2024-09-21T18:59:16.539031+0000","last_active":"2024-09-21T19:01:17.155443+0000","last_peered":"2024-09-21T19:01:17.155443+0000","last_clean":"2024-09-21T19:01:17.155443+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:01:17.155443+0000","last_undegraded":"2024-09-21T19:01:17.155443+0000","last_fullsized":"2024-09-21T19:01:17.155443+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.538853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.538853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.903925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":333,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153150+0000","last_change":"2024-09-21T19:00:34.960755+0000","last_active":"2024-09-21T19:01:17.153150+0000","last_peered":"2024-09-21T19:01:17.153150+0000","last_clean":"2024-09-21T19:01:17.153150+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:01:17.153150+0000","last_undegraded":"2024-09-21T19:01:17.153150+0000","last_fullsized":"2024-09-21T19:01:17.153150+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.960689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.960689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:57.667040+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"219'12642","reported_seq":12836,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:21.567248+0000","last_change":"2024-09-21T19:00:21.263588+0000","last_active":"2024-09-21T19:01:21.567248+0000","last_peered":"2024-09-21T19:01:21.567248+0000","last_clean":"2024-09-21T19:01:21.567248+0000","last_became_active":"2024-09-21T18:58:44.094697+0000","last_became_peered":"2024-09-21T18:58:44.094697+0000","last_unstale":"2024-09-21T19:01:21.567248+0000","last_undegraded":"2024-09-21T19:01:21.567248+0000","last_fullsized":"2024-09-21T19:01:21.567248+0000","mapping_epoch":172,"log_start":"204'7700","ondisk_log_start":"204'7700","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8114","last_scrub_stamp":"2024-09-21T19:00:21.263245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.263245+0000","objects_scrubbed":8036,"log_size":4942,"log_dups_size":0,"ondisk_log_size":4942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.729679+0000","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246284288,"num_objects":3758,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12642,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":381,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155838+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T19:01:17.155838+0000","last_peered":"2024-09-21T19:01:17.155838+0000","last_clean":"2024-09-21T19:01:17.155838+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T19:01:17.155838+0000","last_undegraded":"2024-09-21T19:01:17.155838+0000","last_fullsized":"2024-09-21T19:01:17.155838+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.863281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":381,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.159082+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T19:01:17.159082+0000","last_peered":"2024-09-21T19:01:17.159082+0000","last_clean":"2024-09-21T19:01:17.159082+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:01:17.159082+0000","last_undegraded":"2024-09-21T19:01:17.159082+0000","last_fullsized":"2024-09-21T19:01:17.159082+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.427939+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":461,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157956+0000","last_change":"2024-09-21T18:59:56.831306+0000","last_active":"2024-09-21T19:01:17.157956+0000","last_peered":"2024-09-21T19:01:17.157956+0000","last_clean":"2024-09-21T19:01:17.157956+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T19:01:17.157956+0000","last_undegraded":"2024-09-21T19:01:17.157956+0000","last_fullsized":"2024-09-21T19:01:17.157956+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:59:56.831225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.831225+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":370,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156337+0000","last_change":"2024-09-21T19:01:04.083677+0000","last_active":"2024-09-21T19:01:17.156337+0000","last_peered":"2024-09-21T19:01:17.156337+0000","last_clean":"2024-09-21T19:01:17.156337+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T19:01:17.156337+0000","last_undegraded":"2024-09-21T19:01:17.156337+0000","last_fullsized":"2024-09-21T19:01:17.156337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:04.083562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.083562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:22.889131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":365,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.154354+0000","last_change":"2024-09-21T18:59:14.180429+0000","last_active":"2024-09-21T19:01:17.154354+0000","last_peered":"2024-09-21T19:01:17.154354+0000","last_clean":"2024-09-21T19:01:17.154354+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T19:01:17.154354+0000","last_undegraded":"2024-09-21T19:01:17.154354+0000","last_fullsized":"2024-09-21T19:01:17.154354+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.387472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":366,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158095+0000","last_change":"2024-09-21T19:01:06.925416+0000","last_active":"2024-09-21T19:01:17.158095+0000","last_peered":"2024-09-21T19:01:17.158095+0000","last_clean":"2024-09-21T19:01:17.158095+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:01:17.158095+0000","last_undegraded":"2024-09-21T19:01:17.158095+0000","last_fullsized":"2024-09-21T19:01:17.158095+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T19:01:06.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.925267+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.186913+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":360,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155923+0000","last_change":"2024-09-21T19:00:29.955243+0000","last_active":"2024-09-21T19:01:17.155923+0000","last_peered":"2024-09-21T19:01:17.155923+0000","last_clean":"2024-09-21T19:01:17.155923+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T19:01:17.155923+0000","last_undegraded":"2024-09-21T19:01:17.155923+0000","last_fullsized":"2024-09-21T19:01:17.155923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.955184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.955184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.019605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":331,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160181+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T19:01:17.160181+0000","last_peered":"2024-09-21T19:01:17.160181+0000","last_clean":"2024-09-21T19:01:17.160181+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T19:01:17.160181+0000","last_undegraded":"2024-09-21T19:01:17.160181+0000","last_fullsized":"2024-09-21T19:01:17.160181+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.437166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":242,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.166842+0000","last_change":"2024-09-21T19:00:34.345923+0000","last_active":"2024-09-21T19:01:17.166842+0000","last_peered":"2024-09-21T19:01:17.166842+0000","last_clean":"2024-09-21T19:01:17.166842+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:01:17.166842+0000","last_undegraded":"2024-09-21T19:01:17.166842+0000","last_fullsized":"2024-09-21T19:01:17.166842+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:00:34.345792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.345792+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:58.340045+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":351,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158205+0000","last_change":"2024-09-21T19:00:29.586470+0000","last_active":"2024-09-21T19:01:17.158205+0000","last_peered":"2024-09-21T19:01:17.158205+0000","last_clean":"2024-09-21T19:01:17.158205+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T19:01:17.158205+0000","last_undegraded":"2024-09-21T19:01:17.158205+0000","last_fullsized":"2024-09-21T19:01:17.158205+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.586411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.586411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.139362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":345,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157907+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:01:17.157907+0000","last_peered":"2024-09-21T19:01:17.157907+0000","last_clean":"2024-09-21T19:01:17.157907+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:01:17.157907+0000","last_undegraded":"2024-09-21T19:01:17.157907+0000","last_fullsized":"2024-09-21T19:01:17.157907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":348,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155553+0000","last_change":"2024-09-21T19:01:02.984598+0000","last_active":"2024-09-21T19:01:17.155553+0000","last_peered":"2024-09-21T19:01:17.155553+0000","last_clean":"2024-09-21T19:01:17.155553+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:01:17.155553+0000","last_undegraded":"2024-09-21T19:01:17.155553+0000","last_fullsized":"2024-09-21T19:01:17.155553+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.984535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.984535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.393677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":364,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156577+0000","last_change":"2024-09-21T18:58:58.524782+0000","last_active":"2024-09-21T19:01:17.156577+0000","last_peered":"2024-09-21T19:01:17.156577+0000","last_clean":"2024-09-21T19:01:17.156577+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T19:01:17.156577+0000","last_undegraded":"2024-09-21T19:01:17.156577+0000","last_fullsized":"2024-09-21T19:01:17.156577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:58.524708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.524708+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.871526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.159503+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T19:01:17.159503+0000","last_peered":"2024-09-21T19:01:17.159503+0000","last_clean":"2024-09-21T19:01:17.159503+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:01:17.159503+0000","last_undegraded":"2024-09-21T19:01:17.159503+0000","last_fullsized":"2024-09-21T19:01:17.159503+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.288280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":460,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156844+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:01:17.156844+0000","last_peered":"2024-09-21T19:01:17.156844+0000","last_clean":"2024-09-21T19:01:17.156844+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:01:17.156844+0000","last_undegraded":"2024-09-21T19:01:17.156844+0000","last_fullsized":"2024-09-21T19:01:17.156844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:22.257098+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146495+0000","last_change":"2024-09-21T19:00:32.753965+0000","last_active":"2024-09-21T19:01:17.146495+0000","last_peered":"2024-09-21T19:01:17.146495+0000","last_clean":"2024-09-21T19:01:17.146495+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:01:17.146495+0000","last_undegraded":"2024-09-21T19:01:17.146495+0000","last_fullsized":"2024-09-21T19:01:17.146495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:32.753902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.753902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.288209+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":356,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157202+0000","last_change":"2024-09-21T19:00:28.893623+0000","last_active":"2024-09-21T19:01:17.157202+0000","last_peered":"2024-09-21T19:01:17.157202+0000","last_clean":"2024-09-21T19:01:17.157202+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:01:17.157202+0000","last_undegraded":"2024-09-21T19:01:17.157202+0000","last_fullsized":"2024-09-21T19:01:17.157202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:28.893403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.893403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:28.904866+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":351,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158152+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:01:17.158152+0000","last_peered":"2024-09-21T19:01:17.158152+0000","last_clean":"2024-09-21T19:01:17.158152+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:01:17.158152+0000","last_undegraded":"2024-09-21T19:01:17.158152+0000","last_fullsized":"2024-09-21T19:01:17.158152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.507018+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":465,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156430+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T19:01:17.156430+0000","last_peered":"2024-09-21T19:01:17.156430+0000","last_clean":"2024-09-21T19:01:17.156430+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T19:01:17.156430+0000","last_undegraded":"2024-09-21T19:01:17.156430+0000","last_fullsized":"2024-09-21T19:01:17.156430+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.883299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":240,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.167669+0000","last_change":"2024-09-21T19:00:58.413991+0000","last_active":"2024-09-21T19:01:17.167669+0000","last_peered":"2024-09-21T19:01:17.167669+0000","last_clean":"2024-09-21T19:01:17.167669+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:01:17.167669+0000","last_undegraded":"2024-09-21T19:01:17.167669+0000","last_fullsized":"2024-09-21T19:01:17.167669+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:58.413866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:58.413866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.192707+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":394,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.146280+0000","last_change":"2024-09-21T19:01:04.724924+0000","last_active":"2024-09-21T19:01:17.146280+0000","last_peered":"2024-09-21T19:01:17.146280+0000","last_clean":"2024-09-21T19:01:17.146280+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T19:01:17.146280+0000","last_undegraded":"2024-09-21T19:01:17.146280+0000","last_fullsized":"2024-09-21T19:01:17.146280+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:01:04.724868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.724868+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:24.749814+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.147003+0000","last_change":"2024-09-21T19:00:34.746168+0000","last_active":"2024-09-21T19:01:17.147003+0000","last_peered":"2024-09-21T19:01:17.147003+0000","last_clean":"2024-09-21T19:01:17.147003+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T19:01:17.147003+0000","last_undegraded":"2024-09-21T19:01:17.147003+0000","last_fullsized":"2024-09-21T19:01:17.147003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.746126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.746126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150419+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:01:17.150419+0000","last_peered":"2024-09-21T19:01:17.150419+0000","last_clean":"2024-09-21T19:01:17.150419+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:01:17.150419+0000","last_undegraded":"2024-09-21T19:01:17.150419+0000","last_fullsized":"2024-09-21T19:01:17.150419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":347,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150616+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T19:01:17.150616+0000","last_peered":"2024-09-21T19:01:17.150616+0000","last_clean":"2024-09-21T19:01:17.150616+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:01:17.150616+0000","last_undegraded":"2024-09-21T19:01:17.150616+0000","last_fullsized":"2024-09-21T19:01:17.150616+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.894710+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":380,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150764+0000","last_change":"2024-09-21T19:01:11.541442+0000","last_active":"2024-09-21T19:01:17.150764+0000","last_peered":"2024-09-21T19:01:17.150764+0000","last_clean":"2024-09-21T19:01:17.150764+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T19:01:17.150764+0000","last_undegraded":"2024-09-21T19:01:17.150764+0000","last_fullsized":"2024-09-21T19:01:17.150764+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.015757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":359,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160344+0000","last_change":"2024-09-21T18:58:34.182624+0000","last_active":"2024-09-21T19:01:17.160344+0000","last_peered":"2024-09-21T19:01:17.160344+0000","last_clean":"2024-09-21T19:01:17.160344+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:01:17.160344+0000","last_undegraded":"2024-09-21T19:01:17.160344+0000","last_fullsized":"2024-09-21T19:01:17.160344+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:34.182578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:34.182578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":359,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153222+0000","last_change":"2024-09-21T18:58:58.952448+0000","last_active":"2024-09-21T19:01:17.153222+0000","last_peered":"2024-09-21T19:01:17.153222+0000","last_clean":"2024-09-21T19:01:17.153222+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:01:17.153222+0000","last_undegraded":"2024-09-21T19:01:17.153222+0000","last_fullsized":"2024-09-21T19:01:17.153222+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:58.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.895529+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":240,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.159083+0000","last_change":"2024-09-21T18:59:42.223794+0000","last_active":"2024-09-21T19:01:17.159083+0000","last_peered":"2024-09-21T19:01:17.159083+0000","last_clean":"2024-09-21T19:01:17.159083+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:01:17.159083+0000","last_undegraded":"2024-09-21T19:01:17.159083+0000","last_fullsized":"2024-09-21T19:01:17.159083+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:42.223731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.223731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":368,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150813+0000","last_change":"2024-09-21T19:00:00.939564+0000","last_active":"2024-09-21T19:01:17.150813+0000","last_peered":"2024-09-21T19:01:17.150813+0000","last_clean":"2024-09-21T19:01:17.150813+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T19:01:17.150813+0000","last_undegraded":"2024-09-21T19:01:17.150813+0000","last_fullsized":"2024-09-21T19:01:17.150813+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:20.583806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.156002+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:01:17.156002+0000","last_peered":"2024-09-21T19:01:17.156002+0000","last_clean":"2024-09-21T19:01:17.156002+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:01:17.156002+0000","last_undegraded":"2024-09-21T19:01:17.156002+0000","last_fullsized":"2024-09-21T19:01:17.156002+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.872793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":351,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.157636+0000","last_change":"2024-09-21T19:00:16.651104+0000","last_active":"2024-09-21T19:01:17.157636+0000","last_peered":"2024-09-21T19:01:17.157636+0000","last_clean":"2024-09-21T19:01:17.157636+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T19:01:17.157636+0000","last_undegraded":"2024-09-21T19:01:17.157636+0000","last_fullsized":"2024-09-21T19:01:17.157636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:16.650906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:16.650906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:27.021875+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":355,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.155799+0000","last_change":"2024-09-21T18:59:58.945778+0000","last_active":"2024-09-21T19:01:17.155799+0000","last_peered":"2024-09-21T19:01:17.155799+0000","last_clean":"2024-09-21T19:01:17.155799+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T19:01:17.155799+0000","last_undegraded":"2024-09-21T19:01:17.155799+0000","last_fullsized":"2024-09-21T19:01:17.155799+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:58.945631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:58.945631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":352,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.150868+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T19:01:17.150868+0000","last_peered":"2024-09-21T19:01:17.150868+0000","last_clean":"2024-09-21T19:01:17.150868+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T19:01:17.150868+0000","last_undegraded":"2024-09-21T19:01:17.150868+0000","last_fullsized":"2024-09-21T19:01:17.150868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.954764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":232,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.160873+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T19:01:17.160873+0000","last_peered":"2024-09-21T19:01:17.160873+0000","last_clean":"2024-09-21T19:01:17.160873+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:01:17.160873+0000","last_undegraded":"2024-09-21T19:01:17.160873+0000","last_fullsized":"2024-09-21T19:01:17.160873+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.291673+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":343,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.152961+0000","last_change":"2024-09-21T18:59:17.117569+0000","last_active":"2024-09-21T19:01:17.152961+0000","last_peered":"2024-09-21T19:01:17.152961+0000","last_clean":"2024-09-21T19:01:17.152961+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T19:01:17.152961+0000","last_undegraded":"2024-09-21T19:01:17.152961+0000","last_fullsized":"2024-09-21T19:01:17.152961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.359379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":339,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.158397+0000","last_change":"2024-09-21T18:59:59.934497+0000","last_active":"2024-09-21T19:01:17.158397+0000","last_peered":"2024-09-21T19:01:17.158397+0000","last_clean":"2024-09-21T19:01:17.158397+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:01:17.158397+0000","last_undegraded":"2024-09-21T19:01:17.158397+0000","last_fullsized":"2024-09-21T19:01:17.158397+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.934416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.934416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":340,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151078+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:01:17.151078+0000","last_peered":"2024-09-21T19:01:17.151078+0000","last_clean":"2024-09-21T19:01:17.151078+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:01:17.151078+0000","last_undegraded":"2024-09-21T19:01:17.151078+0000","last_fullsized":"2024-09-21T19:01:17.151078+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:21.108434+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":375,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.151113+0000","last_change":"2024-09-21T19:00:17.839077+0000","last_active":"2024-09-21T19:01:17.151113+0000","last_peered":"2024-09-21T19:01:17.151113+0000","last_clean":"2024-09-21T19:01:17.151113+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:01:17.151113+0000","last_undegraded":"2024-09-21T19:01:17.151113+0000","last_fullsized":"2024-09-21T19:01:17.151113+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:17.838903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:17.838903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.006877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":366,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.153559+0000","last_change":"2024-09-21T18:59:18.108977+0000","last_active":"2024-09-21T19:01:17.153559+0000","last_peered":"2024-09-21T19:01:17.153559+0000","last_clean":"2024-09-21T19:01:17.153559+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T19:01:17.153559+0000","last_undegraded":"2024-09-21T19:01:17.153559+0000","last_fullsized":"2024-09-21T19:01:17.153559+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:18.108904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:18.108904+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.281496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":248,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T19:01:17.167550+0000","last_change":"2024-09-21T19:01:11.531925+0000","last_active":"2024-09-21T19:01:17.167550+0000","last_peered":"2024-09-21T19:01:17.167550+0000","last_clean":"2024-09-21T19:01:17.167550+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:01:17.167550+0000","last_undegraded":"2024-09-21T19:01:17.167550+0000","last_fullsized":"2024-09-21T19:01:17.167550+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:40.641599+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3932094488,"num_objects":60000,"num_object_clones":0,"num_object_copies":180000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204460,"num_write_kb":8462657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11820937216,"data_stored":11820925000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80960,"ondisk_log_size":80960,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":483,"num_read_kb":478,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266791,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1477964,"kb_used_data":1230204,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92267188,"statfs":{"total":95995035648,"available":94481600512,"internally_reserved":0,"allocated":1259728896,"data_stored":1259348153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528428,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472768,"kb_used_data":1220848,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92272384,"statfs":{"total":95995035648,"available":94486921216,"internally_reserved":0,"allocated":1250148352,"data_stored":1249779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790063,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2229636,"kb_used_data":1941556,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91515516,"statfs":{"total":95995035648,"available":93711888384,"internally_reserved":0,"allocated":1988153344,"data_stored":1987780818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018997,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491848,"kb_used_data":1235768,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92253304,"statfs":{"total":95995035648,"available":94467383296,"internally_reserved":0,"allocated":1265426432,"data_stored":1265050041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182522,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1820512,"kb_used_data":1446352,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91924640,"statfs":{"total":95995035648,"available":94130831360,"internally_reserved":0,"allocated":1481064448,"data_stored":1480664544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411456,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734668,"kb_used_data":1464700,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92010484,"statfs":{"total":95995035648,"available":94218735616,"internally_reserved":0,"allocated":1499852800,"data_stored":1499472637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574980,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2032152,"kb_used_data":1694152,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91713000,"statfs":{"total":95995035648,"available":93914112000,"internally_reserved":0,"allocated":1734811648,"data_stored":1734418946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738506,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1886860,"kb_used_data":1458620,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91858292,"statfs":{"total":95995035648,"available":94062891008,"internally_reserved":0,"allocated":1493626880,"data_stored":1493246802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1481506816,"data_stored":1481506816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1713307648,"data_stored":1713307648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1477120000,"data_stored":1477115928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1457586176,"data_stored":1457586176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1245642752,"data_stored":1245642752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1959333888,"data_stored":1959329816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1247805440,"data_stored":1247805440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1238634496,"data_stored":1238630424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:01:28.040 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T19:01:28.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_1 pg_num 26 2024-09-21T19:01:28.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:28.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:28.176 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:28.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:28.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.464 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:28.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:28.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:01:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:01:28.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:28.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:28.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:28.753 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:28.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:28.898 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:28.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:28.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:01:29.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:29.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:29.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:29.164 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:29.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:29.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:29.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:29 smithi081 ceph-mon[29384]: pgmap v386: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:29.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:29.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:29 smithi081 ceph-mon[39192]: pgmap v386: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:29.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:29.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:29.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:29.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:29 smithi104 ceph-mon[32438]: pgmap v386: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:29.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:29.548 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:29.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:29.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:29.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:29.774 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:01:29.774 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:01:29.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:01:29.774 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:01:29.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.774 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:145372 7.f 7:f3db53f7:::benchmark_data_smithi081_73391_object6373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:59.333907+0000", 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 330.43506383099998, 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.132201821, 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 145372 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333907+0000", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333907+0000", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333908+0000", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1289999999999999e-06 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333912+0000", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7720000000000001e-06 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333918+0000", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6059999999999999e-06 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.333923+0000", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1220000000000003e-06 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431117+0000", 2024-09-21T19:01:29.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097193494000000005 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431239+0000", 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012223400000000001 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.431352+0000", 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011341900000000001 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462090+0000", 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030737417 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462135+0000", 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5228999999999997e-05 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.462174+0000", 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9218000000000003e-05 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.465927+0000", 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037523449999999998 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.465992+0000", 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5376999999999994e-05 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:59.466108+0000", 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116459 2024-09-21T19:01:29.780 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:211482 7.e e138/114 7:7d19c424:::benchmark_data_smithi081_73391_object72483:head v 138'13046, mlcod=138'13043)", 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.060608+0000", 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 255.708362198, 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13969134899999999, 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060608+0000", 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060608+0000", 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060609+0000", 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6200000000000006e-07 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060615+0000", 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4049999999999999e-06 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060622+0000", 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.712e-06 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060628+0000", 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0120000000000002e-06 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.175684+0000", 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.115055169 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.175703+0000", 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9955000000000001e-05 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.199415+0000", 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.023711368 2024-09-21T19:01:29.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200300+0000", 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00088476600000000005 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:211485 7.f 7:f98a38d8:::benchmark_data_smithi081_73391_object72486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:14.060874+0000", 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 255.70809603199999, 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.143099278, 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.785 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 211485 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060874+0000", 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060874+0000", 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060876+0000", 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2270000000000001e-06 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.060880+0000", 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6120000000000001e-06 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.173490+0000", 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.1126096 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.173500+0000", 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0583e-05 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.179620+0000", 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006119691 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.179760+0000", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013987099999999999 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.179898+0000", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013791699999999999 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.184545+0000", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046469789999999999 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.197397+0000", 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012852542999999999 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.200323+0000", 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002925108 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.203537+0000", 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032148229999999999 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.203599+0000", 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1186000000000007e-05 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:14.203974+0000", 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00037513800000000001 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242073 7.3 7:c6fda3ec:::benchmark_data_smithi081_73391_object103074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:01:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.501575+0000", 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.26739540099999, 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17268383800000001, 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242073 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501575+0000", 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501575+0000", 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501577+0000", 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1289999999999999e-06 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501583+0000", 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4500000000000003e-06 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501595+0000", 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2077e-05 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501603+0000", 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3399999999999998e-06 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501722+0000", 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118658 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501830+0000", 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010876 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501939+0000", 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108435 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511744+0000", 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0098048200000000005 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511797+0000", 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3733999999999997e-05 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674170+0000", 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16237236399999999 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674221+0000", 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1059e-05 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674259+0000", 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8012000000000001e-05 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242076 7.a 7:5697a0ec:::benchmark_data_smithi081_73391_object103077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.501662+0000", 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.26730846300001, 2024-09-21T19:01:29.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.173717652, 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242076 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501662+0000", 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501662+0000", 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501662+0000", 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8999999999999997e-07 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501665+0000", 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.424e-06 2024-09-21T19:01:29.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501670+0000", 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5949999999999998e-06 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.501681+0000", 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0934000000000001e-05 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502591+0000", 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000909408 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502695+0000", 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104381 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502808+0000", 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011254799999999999 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511271+0000", 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084630829999999997 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512006+0000", 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00073563299999999997 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674954+0000", 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16294760899999999 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.675342+0000", 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038748999999999999 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.675380+0000", 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8056999999999998e-05 2024-09-21T19:01:29.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242074 7.0 e148/79 7:061b77dd:::benchmark_data_smithi081_73391_object103075:head v 148'15370, mlcod=148'15368)", 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.502435+0000", 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.26653572999999, 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17104414200000001, 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502435+0000", 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502435+0000", 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502435+0000", 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8100000000000002e-07 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502441+0000", 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.31e-06 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502446+0000", 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5219999999999998e-06 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.502450+0000", 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2490000000000003e-06 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509857+0000", 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074064150000000004 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509877+0000", 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0219000000000001e-05 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673418+0000", 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16354082 2024-09-21T19:01:29.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673479+0000", 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0925999999999999e-05 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242079 7.3 7:cff90720:::benchmark_data_smithi081_73391_object103080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.503647+0000", 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.26532340399999, 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.170710009, 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.808 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242079 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503647+0000", 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503647+0000", 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503649+0000", 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8729999999999999e-06 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503654+0000", 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.541e-06 2024-09-21T19:01:29.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503666+0000", 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1511e-05 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.503674+0000", 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2670000000000006e-06 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.504624+0000", 2024-09-21T19:01:29.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00095014000000000003 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.504730+0000", 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105939 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.504835+0000", 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010462 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511778+0000", 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0069427839999999996 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512126+0000", 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034867699999999998 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674292+0000", 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16216533999999999 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674329+0000", 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6977999999999997e-05 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674357+0000", 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8339000000000001e-05 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242080 7.1 e149/79 7:892cd020:::benchmark_data_smithi081_73391_object103081:head v 149'15044, mlcod=148'15042)", 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.506111+0000", 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.26285923899999, 2024-09-21T19:01:29.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.166020997, 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506111+0000", 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506111+0000", 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506112+0000", 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0559999999999999e-06 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506118+0000", 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5999999999999997e-06 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506127+0000", 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.465e-06 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.506134+0000", 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7360000000000004e-06 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.507428+0000", 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012939819999999999 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.507447+0000", 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9091999999999999e-05 2024-09-21T19:01:29.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672081+0000", 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.164633475 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672132+0000", 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1591000000000001e-05 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.818 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242081 7.7 e149/143 7:eb350bf9:::benchmark_data_smithi081_73391_object103082:head v 149'15160, mlcod=148'15158)", 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.509291+0000", 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.25967966799999, 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16315574999999999, 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509291+0000", 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509291+0000", 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509291+0000", 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7800000000000001e-07 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509294+0000", 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.258e-06 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509299+0000", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.198e-06 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509304+0000", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5120000000000002e-06 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509344+0000", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0198999999999999e-05 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.509360+0000", 2024-09-21T19:01:29.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5909e-05 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672405+0000", 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.163044511 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672447+0000", 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1984999999999999e-05 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242082 7.1 e149/79 7:83ef7caf:::benchmark_data_smithi081_73391_object103083:head v 149'15045, mlcod=148'15042)", 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.510875+0000", 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.25809591000001, 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16130624199999999, 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510875+0000", 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510875+0000", 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510875+0000", 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4900000000000005e-07 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510881+0000", 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8490000000000002e-06 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510888+0000", 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1060000000000002e-06 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.510893+0000", 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9400000000000001e-06 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511398+0000", 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00050453499999999997 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.511417+0000", 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9273e-05 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672136+0000", 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16071918600000001 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672181+0000", 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4604000000000002e-05 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.826 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242087 7.d e149/114 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head v 149'15301, mlcod=148'15299)", 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.513200+0000", 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.25577031700001, 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20102450699999999, 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513200+0000", 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513200+0000", 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513202+0000", 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.403e-06 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513208+0000", 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5769999999999999e-06 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672834+0000", 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.15962562899999999 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672843+0000", 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3409999999999992e-06 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702934+0000", 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030091353000000001 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702958+0000", 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4054e-05 2024-09-21T19:01:29.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.714194+0000", 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011235121000000001 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.714225+0000", 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1028999999999997e-05 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242089 7.2 7:47cc2c63:::benchmark_data_smithi081_73391_object103090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.513246+0000", 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.25572456099999, 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19053052500000001, 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242089 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.832 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513246+0000", 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513246+0000", 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513248+0000", 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.759e-06 2024-09-21T19:01:29.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.513252+0000", 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8230000000000002e-06 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673414+0000", 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16016180499999999 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673424+0000", 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.893e-06 2024-09-21T19:01:29.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673481+0000", 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6764000000000001e-05 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673610+0000", 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012902599999999999 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673721+0000", 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110931 2024-09-21T19:01:29.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.699219+0000", 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025498257 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702051+0000", 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028315860000000001 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.702478+0000", 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00042716299999999999 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.703734+0000", 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012565180000000001 2024-09-21T19:01:29.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.703776+0000", 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1999999999999998e-05 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242083 7.1 e149/79 7:8f0bc0f0:::benchmark_data_smithi081_73391_object103084:head v 149'15046, mlcod=148'15042)", 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.514317+0000", 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 220.25465356399999, 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.18463781100000001, 2024-09-21T19:01:29.838 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514317+0000", 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514317+0000", 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514318+0000", 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2699999999999999e-06 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514325+0000", 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8820000000000003e-06 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.671916+0000", 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.15759063000000001 2024-09-21T19:01:29.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.671926+0000", 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0431000000000001e-05 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.695786+0000", 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.023859405 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.695808+0000", 2024-09-21T19:01:29.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2393000000000001e-05 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.698910+0000", 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031024149999999999 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.698955+0000", 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4385000000000003e-05 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242810 7.a 7:50694f8c:::benchmark_data_smithi081_73391_object103811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.341722+0000", 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.42724829100001, 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136267994, 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242810 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341722+0000", 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341722+0000", 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341724+0000", 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.609e-06 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341729+0000", 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3870000000000003e-06 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341739+0000", 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0122000000000001e-05 2024-09-21T19:01:29.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.341746+0000", 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.55e-06 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.343863+0000", 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021173540000000001 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.343965+0000", 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101864 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.344070+0000", 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105202 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.347901+0000", 2024-09-21T19:01:29.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038303790000000001 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348980+0000", 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010796 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477914+0000", 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.128933728 2024-09-21T19:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477948+0000", 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3618000000000002e-05 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477990+0000", 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2580999999999998e-05 2024-09-21T19:01:29.849 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242814 7.2 7:4a8d833a:::benchmark_data_smithi081_73391_object103815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.348817+0000", 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.420153868, 2024-09-21T19:01:29.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13874188600000001, 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242814 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348817+0000", 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348817+0000", 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348818+0000", 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3400000000000001e-06 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348822+0000", 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8419999999999998e-06 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348828+0000", 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6109999999999999e-06 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348834+0000", 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4940000000000003e-06 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349009+0000", 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017474600000000001 2024-09-21T19:01:29.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349111+0000", 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102105 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.349211+0000", 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010063799999999999 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351733+0000", 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025221409999999999 2024-09-21T19:01:29.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477886+0000", 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12615274400000001 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487304+0000", 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0094181649999999992 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487511+0000", 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020704799999999999 2024-09-21T19:01:29.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487558+0000", 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7012000000000003e-05 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242817 7.3 7:c6315629:::benchmark_data_smithi081_73391_object103818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.348843+0000", 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.420127046, 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13292791100000001, 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.857 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.858 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:29.858 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242817 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348843+0000", 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348843+0000", 2024-09-21T19:01:29.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348844+0000", 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8699999999999995e-07 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348846+0000", 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.181e-06 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348851+0000", 2024-09-21T19:01:29.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6129999999999997e-06 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348855+0000", 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3529999999999997e-06 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351592+0000", 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027369949999999999 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351697+0000", 2024-09-21T19:01:29.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010456800000000001 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4", 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.351802+0000", 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010498800000000001 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354541+0000", 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027390589999999999 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.473798+0000", 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.119257029 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481514+0000", 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0077163730000000003 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481725+0000", 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000210762 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.481771+0000", 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6502999999999998e-05 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242821 7.a 7:5657abd2:::benchmark_data_smithi081_73391_object103822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.348867+0000", 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.420103482, 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.138406739, 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.864 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242821 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348867+0000", 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348867+0000", 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348867+0000", 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0999999999999999e-07 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348870+0000", 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1859999999999999e-06 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348874+0000", 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6469999999999997e-06 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.348878+0000", 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1869999999999999e-06 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350332+0000", 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001453338 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350437+0000", 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010491499999999999 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.350555+0000", 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118657 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353037+0000", 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024819899999999999 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475426+0000", 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12238887900000001 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487175+0000", 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011749177 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487232+0000", 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6196999999999997e-05 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487274+0000", 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2156000000000001e-05 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.869 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242822 7.2 7:4babd305:::benchmark_data_smithi081_73391_object103823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.353427+0000", 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.41554354600001, 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13814759800000001, 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242822 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.870 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353427+0000", 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353427+0000", 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353429+0000", 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8250000000000001e-06 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353434+0000", 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5609999999999998e-06 2024-09-21T19:01:29.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353445+0000", 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0652000000000001e-05 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353452+0000", 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2969999999999998e-06 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354086+0000", 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063416299999999998 2024-09-21T19:01:29.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354193+0000", 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106693 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354303+0000", 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010956800000000001 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477935+0000", 2024-09-21T19:01:29.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12363234200000001 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477947+0000", 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2431e-05 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491475+0000", 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013527656000000001 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491527+0000", 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1622999999999997e-05 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491575+0000", 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7787000000000002e-05 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242823 7.2 7:4c0cbab2:::benchmark_data_smithi081_73391_object103824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.353478+0000", 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.41549207, 2024-09-21T19:01:29.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.138776869, 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242823 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353478+0000", 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353478+0000", 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353479+0000", 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6199999999999997e-07 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353482+0000", 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8140000000000001e-06 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353487+0000", 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6019999999999999e-06 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.353493+0000", 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1730000000000003e-06 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356641+0000", 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031481109999999999 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356732+0000", 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1620999999999996e-05 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.356830+0000", 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7289999999999999e-05 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.477952+0000", 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12112197299999999 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487050+0000", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009098531 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491616+0000", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045655089999999997 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.491894+0000", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027829200000000003 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492255+0000", 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036129100000000002 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242827 7.2 7:42b56fec:::benchmark_data_smithi081_73391_object103828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:50.354737+0000", 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.41423300599999, 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.137678105, 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242827 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354737+0000", 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354737+0000", 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354738+0000", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.006e-06 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354743+0000", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3970000000000004e-06 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354752+0000", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8899999999999996e-06 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:29.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.354758+0000", 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6769999999999998e-06 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475439+0000", 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12068037199999999 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475564+0000", 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012529800000000001 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:29.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.475681+0000", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011681200000000001 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.482300+0000", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066191899999999996 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.487092+0000", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047915659999999997 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492297+0000", 2024-09-21T19:01:29.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005205093 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492382+0000", 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5304999999999997e-05 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:50.492416+0000", 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3498999999999999e-05 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:29.887 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:29.888 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:01:29.938 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:29.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:30.039 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:30.068 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:30.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:30.197 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:30.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:30.337 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:30.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:30.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:30.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:01:30.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:30.599 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:30.701 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:30.831 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:30.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:30.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:30.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.011 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:31.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:31.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:31.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:31.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:31.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:31.296 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:31 smithi081 ceph-mon[39192]: pgmap v387: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:31.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:31 smithi081 ceph-mon[29384]: pgmap v387: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:31.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:01:31.376 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:31 smithi104 ceph-mon[32438]: pgmap v387: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:31.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.420 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:31.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:31.521 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:31.569 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:31.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:31.748 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:31.823 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:31.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:31.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.018 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:32.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:32.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.161 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:32.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:32.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:32.365 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:01:32.365 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:01:32.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.365 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:217511 8.2 8:489b68ce:::benchmark_data_smithi081_105538_object85279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-21T19:01:32.365 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:32.353851+0000", 2024-09-21T19:01:32.365 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0073080649999999999, 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073378230000000003, 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 217511 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:32.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.353851+0000", 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.353851+0000", 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:32.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.353853+0000", 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9819999999999998e-06 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.353872+0000", 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9303000000000001e-05 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.353887+0000", 2024-09-21T19:01:32.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5251e-05 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.353894+0000", 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2830000000000004e-06 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.357261+0000", 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033665560000000001 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.357568+0000", 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030667500000000001 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.357959+0000", 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039118699999999998 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.35258.0:217514 8.e e220/172 8:7024bead:::benchmark_data_smithi081_105538_object85282:head v 220'13689, mlcod=220'13687)", 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:32.354745+0000", 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0064141429999999998, 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0065179410000000002, 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.354745+0000", 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.354745+0000", 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:32.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.354746+0000", 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.268e-06 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.354752+0000", 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1210000000000003e-06 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.354763+0000", 2024-09-21T19:01:32.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1374e-05 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.354769+0000", 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7699999999999998e-06 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.360878+0000", 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061084629999999997 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.360922+0000", 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4597e-05 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:217513 8.2 8.92c71622 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:32.356842+0000", 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0043167250000000004, 2024-09-21T19:01:32.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044739920000000004, 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 217513 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.356842+0000", 2024-09-21T19:01:32.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.356842+0000", 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.356844+0000", 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6470000000000001e-06 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.356849+0000", 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6380000000000001e-06 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.356860+0000", 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1009000000000001e-05 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:32.356868+0000", 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7419999999999997e-06 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.378 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:32.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:32.379 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:01:32.379 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 3 2024-09-21T19:01:32.379 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:01:32.404 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:32.437 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:32.451 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:32.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:32.591 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:32.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:32.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:32.837 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:32.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:32.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:33.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 8 pg_num to 26 2024-09-21T19:01:33.081 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:01:33.106 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:33.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:33.147 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:33.147 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:33.179 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:33.179 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[29384]: pgmap v388: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4094852176' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/306032499' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[29384]: from='client.25462 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[39192]: pgmap v388: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4094852176' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/306032499' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:01:33.180 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:33 smithi081 ceph-mon[39192]: from='client.25462 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:01:33.215 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:33.215 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:33.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:33.247 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:33.247 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:33 smithi104 ceph-mon[32438]: pgmap v388: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T19:01:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4094852176' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T19:01:33.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/306032499' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:01:33.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:33 smithi104 ceph-mon[32438]: from='client.25462 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:01:33.275 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:33.284 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:33.284 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:33.316 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:33.316 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:33.349 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:33.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:33.419 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:33.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:01:33.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:33.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:33.655 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:33.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:33.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:33.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:33.926 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:33.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:33.974 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:33.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:34.076 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:34.164 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:34.265 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1972952711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4094852176' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: from='client.25462 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:34.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:34 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1972952711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4094852176' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: from='client.25462 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1972952711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4094852176' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-21T19:01:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: from='client.25462 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:01:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T19:01:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:34 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:34.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:34.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:34.440 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:34.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:34.544 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:34.545 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:34.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:34.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:34.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:34.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:34.808 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:34.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:34.931 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:34.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:35.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:35.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:35.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:01:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:35.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[29384]: 3.1a scrub starts 2024-09-21T19:01:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[29384]: 3.1a scrub ok 2024-09-21T19:01:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[29384]: pgmap v390: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-21T19:01:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T19:01:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[29384]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T19:01:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[39192]: 3.1a scrub starts 2024-09-21T19:01:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[39192]: 3.1a scrub ok 2024-09-21T19:01:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[39192]: pgmap v390: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-21T19:01:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T19:01:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:35 smithi081 ceph-mon[39192]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T19:01:35.328 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:35.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:35 smithi104 ceph-mon[32438]: 3.1a scrub starts 2024-09-21T19:01:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:35 smithi104 ceph-mon[32438]: 3.1a scrub ok 2024-09-21T19:01:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:35 smithi104 ceph-mon[32438]: pgmap v390: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-21T19:01:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:35 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T19:01:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:35 smithi104 ceph-mon[32438]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T19:01:35.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:35.585 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:35.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.739 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:35.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:35.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:35.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:35.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:35.976 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:36.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:36.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:01:36.152 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:36.152 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:36.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:36.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:36.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:36.378 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:36.385 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:36.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: 8.d scrub starts 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:36.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:36 smithi104 ceph-mon[32438]: 8.6 scrub starts 2024-09-21T19:01:36.501 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: 8.d scrub starts 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:36.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[39192]: 8.6 scrub starts 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: 8.d scrub starts 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:01:36.507 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:36 smithi081 ceph-mon[29384]: 8.6 scrub starts 2024-09-21T19:01:36.603 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:36.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:36.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:36.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:36.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:36.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:36.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:36.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:01:36.994 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:36.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:37.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:37.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:37.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:37.236 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:37.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 8.d scrub ok 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: pgmap v393: 168 pgs: 10 unknown, 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 8.2 scrub starts 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 8.2 scrub ok 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 8.6 scrub ok 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 5.3 scrub starts 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 5.3 scrub ok 2024-09-21T19:01:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:37 smithi104 ceph-mon[32438]: 8.5 scrub starts 2024-09-21T19:01:37.259 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:01:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 8.d scrub ok 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: pgmap v393: 168 pgs: 10 unknown, 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 8.2 scrub starts 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 8.2 scrub ok 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 8.6 scrub ok 2024-09-21T19:01:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 5.3 scrub starts 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 5.3 scrub ok 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[29384]: 8.5 scrub starts 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 8.d scrub ok 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: pgmap v393: 168 pgs: 10 unknown, 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 8.2 scrub starts 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 8.2 scrub ok 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 8.6 scrub ok 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T19:01:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 5.3 scrub starts 2024-09-21T19:01:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 5.3 scrub ok 2024-09-21T19:01:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:37 smithi081 ceph-mon[39192]: 8.5 scrub starts 2024-09-21T19:01:37.360 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:37.381 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:37.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:37.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:37.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:01:37.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:37.628 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:37.695 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:37.695 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:37.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:37.751 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:37.794 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:37.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:37.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:37.976 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:37.997 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:37.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:38.099 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:01:38.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 8.5 scrub ok 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 8.4 scrub starts 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 8.4 scrub ok 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T19:01:38.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/610646752' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[29384]: from='client.35564 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 8.5 scrub ok 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 8.4 scrub starts 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 8.4 scrub ok 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/610646752' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:38.193 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:38 smithi081 ceph-mon[39192]: from='client.35564 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:38.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:38.263 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:38.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:01:38.345 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 8.5 scrub ok 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 8.4 scrub starts 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 8.4 scrub ok 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/610646752' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:38.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:38 smithi104 ceph-mon[32438]: from='client.35564 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:38.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:38.395 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:38.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:38.520 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:38.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:38.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:38.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:38.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:38.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:01:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:38.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:39.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:39.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:39.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:39.215 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: pgmap v396: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:01:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: from='client.35564 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 223'13378 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 223'13664 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 223'13676 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13591 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 223'13823 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 223'14017 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 223'13917 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13676 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13917 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 223'13591 2024-09-21T19:01:39.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/120075564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:39.316 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:39.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:39.356 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:39.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.494 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:39.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: pgmap v396: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:01:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: from='client.35564 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 223'13378 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 223'13664 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 223'13676 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13591 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 223'13823 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 223'14017 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 223'13917 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13676 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13917 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 223'13591 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/120075564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:39.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: pgmap v396: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: from='client.35564 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 223'13378 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 223'13664 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 223'13676 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13591 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 223'13823 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 223'14017 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 223'13917 2024-09-21T19:01:39.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13676 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13829 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 223'13688 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13917 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 223'13754 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 223'13591 2024-09-21T19:01:39.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/120075564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:39.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:39.633 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:39.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:39.760 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:39.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:39.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:39.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:39.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:39.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:39.850 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:39.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:39.893 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:39.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:39.924 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T19:01:39.924 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:01:39.925 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-09-21T19:01:39.925 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:01:39.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.025 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:40.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:40.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:40.158 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:40.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:40.305 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:40.332 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:01:40.332 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:01:40.332 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:01:40.333 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:01:40.333 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:01:40.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:40.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:40.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:40.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:40.714 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:40.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:40.855 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:40.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:40.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:41.000 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:41.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:01:41.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:41.139 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:41.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:41.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:41 smithi104 ceph-mon[32438]: pgmap v398: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:01:41.282 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:41.361 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:41 smithi081 ceph-mon[39192]: pgmap v398: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:01:41.361 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:41 smithi081 ceph-mon[29384]: pgmap v398: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:01:41.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:41.431 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:41.523 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:41.558 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:41.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:41.689 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:41.792 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:41.870 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:41.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:41.922 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:41.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:42.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:42.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:42.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:42.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:42.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.415 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:42.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:42.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:42.553 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:42.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.690 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:42.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:42.757 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:01:42.757 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114537 7.0 7:04929e7d:::benchmark_data_smithi081_73391_object114536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723398+0000", 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 371.75816032400002, 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105603195, 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.758 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114537 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723398+0000", 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723400+0000", 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.119e-06 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723433+0000", 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3302000000000001e-05 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723444+0000", 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0523000000000001e-05 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723452+0000", 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8199999999999997e-06 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727483+0000", 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040313839999999998 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727571+0000", 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7770999999999997e-05 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727655+0000", 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4322999999999999e-05 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730033+0000", 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002377948 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730045+0000", 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2091e-05 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828840+0000", 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098794333999999998 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.828919+0000", 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9752000000000005e-05 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.829001+0000", 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1828000000000003e-05 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.763 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114544 7.0 7:016dd1d4:::benchmark_data_smithi081_73391_object114543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.730922+0000", 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 371.75063652799997, 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101948043, 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114544 2024-09-21T19:01:42.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730922+0000", 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730924+0000", 2024-09-21T19:01:42.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1280000000000002e-06 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730973+0000", 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8687000000000003e-05 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730987+0000", 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.448e-05 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730996+0000", 2024-09-21T19:01:42.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9719999999999998e-06 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731081+0000", 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5292999999999998e-05 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731162+0000", 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0603000000000001e-05 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.731260+0000", 2024-09-21T19:01:42.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8259999999999998e-05 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.733069+0000", 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018090840000000001 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.780503+0000", 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.047434142999999998 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832639+0000", 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052135212 2024-09-21T19:01:42.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832711+0000", 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2327000000000001e-05 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.832870+0000", 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015885399999999999 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145381 7.0 7:07a4e617:::benchmark_data_smithi081_73391_object6382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T19:01:42.769 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.342438+0000", 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 343.13912070600003, 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11277905000000001, 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145381 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342438+0000", 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342438+0000", 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342439+0000", 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7179999999999999e-06 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342444+0000", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342456+0000", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2302e-05 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342464+0000", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.447e-06 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342510+0000", 2024-09-21T19:01:42.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5977999999999999e-05 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342606+0000", 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6277e-05 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.342712+0000", 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106092 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.348117+0000", 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005405371 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.349145+0000", 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010274399999999999 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455138+0000", 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105993714 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455174+0000", 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5976999999999998e-05 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455217+0000", 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2103e-05 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145387 7.0 7:0b525978:::benchmark_data_smithi081_73391_object6388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T19:01:42.775 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.343230+0000", 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 343.13832792400001, 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.112080033, 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145387 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343230+0000", 2024-09-21T19:01:42.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343230+0000", 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343232+0000", 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.421e-06 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343237+0000", 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3650000000000003e-06 2024-09-21T19:01:42.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343252+0000", 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4705000000000001e-05 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.343260+0000", 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3629999999999994e-06 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345358+0000", 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020982319999999998 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345479+0000", 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012078699999999999 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.345612+0000", 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013247700000000001 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.348281+0000", 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002669186 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.349239+0000", 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095836999999999995 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455251+0000", 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10601202899999999 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455284+0000", 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3090000000000003e-05 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.455310+0000", 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6007999999999999e-05 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:169006 7.0 7:0bdc217b:::benchmark_data_smithi081_73391_object30007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.638054+0000", 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 317.84350412999999, 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101027187, 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.781 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 169006 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638054+0000", 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638054+0000", 2024-09-21T19:01:42.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638056+0000", 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.057e-06 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638062+0000", 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1940000000000003e-06 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638079+0000", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6223999999999999e-05 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638086+0000", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6199999999999999e-06 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638134+0000", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7886000000000002e-05 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638238+0000", 2024-09-21T19:01:42.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104114 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.638337+0000", 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8924000000000004e-05 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.641212+0000", 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028744550000000002 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.646975+0000", 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057632689999999997 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.738982+0000", 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092007007000000002 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739038+0000", 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6113999999999998e-05 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739081+0000", 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3322999999999998e-05 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:169009 7.0 7:01cb78f3:::benchmark_data_smithi081_73391_object30010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:56:24.644717+0000", 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 317.83684169899999, 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106867806, 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.787 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 169009 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644717+0000", 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644717+0000", 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644718+0000", 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7910000000000001e-06 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644726+0000", 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6960000000000005e-06 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644736+0000", 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0375e-05 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644743+0000", 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3679999999999998e-06 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644795+0000", 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2070999999999997e-05 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644903+0000", 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108062 2024-09-21T19:01:42.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.644986+0000", 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3425999999999997e-05 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.647116+0000", 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021293950000000001 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.739125+0000", 2024-09-21T19:01:42.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092009274000000002 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751496+0000", 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012371312000000001 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751539+0000", 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2358999999999999e-05 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:56:24.751584+0000", 2024-09-21T19:01:42.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5677000000000003e-05 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211274 7.0 7:0d096775:::benchmark_data_smithi081_73391_object72275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.770245+0000", 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 268.71131340400001, 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.111625435, 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.793 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211274 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770245+0000", 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770245+0000", 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770246+0000", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7999999999999995e-07 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770248+0000", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770254+0000", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1460000000000004e-06 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.770258+0000", 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8149999999999996e-06 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776110+0000", 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005852037 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776217+0000", 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106234 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776328+0000", 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011097 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782087+0000", 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057596749999999997 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.786947+0000", 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048601130000000001 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881796+0000", 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094848779999999994 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881834+0000", 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7861999999999998e-05 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881870+0000", 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6266999999999997e-05 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.798 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211279 7.0 7:05526e4b:::benchmark_data_smithi081_73391_object72280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.776231+0000", 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 268.70532684699998, 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.105721686, 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211279 2024-09-21T19:01:42.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776231+0000", 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776231+0000", 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776234+0000", 2024-09-21T19:01:42.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2989999999999999e-06 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776239+0000", 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5840000000000002e-06 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776250+0000", 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1207e-05 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.776258+0000", 2024-09-21T19:01:42.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5469999999999999e-06 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.779375+0000", 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031171200000000001 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.779473+0000", 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8119e-05 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.779587+0000", 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011357 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.782093+0000", 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025064290000000001 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787025+0000", 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049313739999999997 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881902+0000", 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094877192999999999 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881930+0000", 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8305000000000001e-05 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.881953+0000", 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2938999999999998e-05 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211292 7.0 7:0d9ec215:::benchmark_data_smithi081_73391_object72293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:13.787634+0000", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 268.69392472800001, 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107733332, 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211292 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787634+0000", 2024-09-21T19:01:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787634+0000", 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787635+0000", 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9190000000000002e-06 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787641+0000", 2024-09-21T19:01:42.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8139999999999997e-06 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787654+0000", 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2692e-05 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787662+0000", 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7989999999999994e-06 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787713+0000", 2024-09-21T19:01:42.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1008999999999999e-05 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787831+0000", 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117778 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.787907+0000", 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6227e-05 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.790256+0000", 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023488369999999999 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.791100+0000", 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00084454799999999998 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895304+0000", 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104203948 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895338+0000", 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4042999999999999e-05 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:13.895367+0000", 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8717999999999998e-05 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242074 7.0 7:061b77dd:::benchmark_data_smithi081_73391_object103075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:01:42.810 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.497997+0000", 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 232.983561579, 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.175851656, 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242074 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.497997+0000", 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.497997+0000", 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.497999+0000", 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0600000000000002e-06 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498005+0000", 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.908e-06 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498020+0000", 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5089000000000001e-05 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498029+0000", 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1970000000000002e-06 2024-09-21T19:01:42.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498086+0000", 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6614999999999999e-05 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498200+0000", 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011442099999999999 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.498280+0000", 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0018999999999994e-05 2024-09-21T19:01:42.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.501049+0000", 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027687020000000001 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511643+0000", 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010594643000000001 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.673781+0000", 2024-09-21T19:01:42.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.162137751 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.673817+0000", 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5805e-05 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.673848+0000", 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1445999999999997e-05 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.816 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242084 7.0 7:0c19e5c0:::benchmark_data_smithi081_73391_object103085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.512045+0000", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 232.96951290199999, 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19629996199999999, 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242084 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512045+0000", 2024-09-21T19:01:42.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512045+0000", 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512047+0000", 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.804e-06 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512052+0000", 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9400000000000001e-06 2024-09-21T19:01:42.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512061+0000", 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.251e-06 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512067+0000", 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512105+0000", 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7907000000000001e-05 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512108+0000", 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8970000000000001e-06 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.513850+0000", 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001741289 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.513953+0000", 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103705 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.514153+0000", 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019997999999999999 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516849+0000", 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026958249999999998 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.678567+0000", 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16171744499999999 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.708270+0000", 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029703475999999999 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.708314+0000", 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3520999999999999e-05 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.708345+0000", 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1631999999999998e-05 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.822 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242824 7.0 7:0ae306ee:::benchmark_data_smithi081_73391_object103825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:50.350874+0000", 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 232.13068384100001, 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12322082400000001, 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:42.823 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242824 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350874+0000", 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350874+0000", 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350877+0000", 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.446e-06 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350884+0000", 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4039999999999998e-06 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350901+0000", 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6529e-05 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350909+0000", 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6300000000000004e-06 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.350971+0000", 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1586000000000003e-05 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.351093+0000", 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012243900000000001 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.351207+0000", 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113706 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.354027+0000", 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028203239999999999 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.359333+0000", 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00530541 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.474025+0000", 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.114692632 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.474063+0000", 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7687e-05 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.474095+0000", 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2030999999999999e-05 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:76380 8.8 8:110b1bc1:::benchmark_data_smithi081_105538_object76379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:59:34.165431+0000", 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 128.316127706, 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101021844, 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 76380 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.829 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165431+0000", 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165431+0000", 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165432+0000", 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.595e-06 2024-09-21T19:01:42.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165465+0000", 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2530000000000002e-05 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165478+0000", 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2945999999999999e-05 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165485+0000", 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9190000000000002e-06 2024-09-21T19:01:42.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.166989+0000", 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001504785 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.167077+0000", 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7427000000000002e-05 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.167234+0000", 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015755899999999999 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.172324+0000", 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050901380000000001 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.172337+0000", 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2218000000000001e-05 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.266365+0000", 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094028016000000006 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.266414+0000", 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9805000000000002e-05 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.266452+0000", 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7905999999999999e-05 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:76381 8.8 8:1ed98622:::benchmark_data_smithi081_105538_object76380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:59:34.165494+0000", 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 128.316064694, 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.101100236, 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 76381 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165494+0000", 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165494+0000", 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165494+0000", 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6199999999999998e-07 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165515+0000", 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0550000000000001e-05 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165520+0000", 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4140000000000002e-06 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.165524+0000", 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5549999999999998e-06 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.169605+0000", 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040808820000000001 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.169746+0000", 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000141464 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.169906+0000", 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015968499999999999 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.172343+0000", 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024376789999999999 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.172631+0000", 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000287242 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.266498+0000", 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.093867807999999997 2024-09-21T19:01:42.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.266555+0000", 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7003000000000003e-05 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.266594+0000", 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8392000000000001e-05 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.840 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:76675 8.d 8:b4ba6698:::benchmark_data_smithi081_105538_object76674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:59:34.541447+0000", 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 127.940111573, 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.124185263, 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 76675 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541447+0000", 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541447+0000", 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541450+0000", 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8490000000000002e-06 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541499+0000", 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9357999999999997e-05 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541515+0000", 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5828000000000002e-05 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541522+0000", 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3019999999999997e-06 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541576+0000", 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4496999999999997e-05 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541697+0000", 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120837 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,7", 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.541816+0000", 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011846200000000001 2024-09-21T19:01:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.544401+0000", 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00258515 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.544798+0000", 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039676000000000002 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.665526+0000", 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12072843699999999 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.665579+0000", 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3306000000000001e-05 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.665632+0000", 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2476999999999999e-05 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:76692 8.4 8:20d494a0:::benchmark_data_smithi081_105538_object76691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:59:34.556264+0000", 2024-09-21T19:01:42.846 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 127.92529409300001, 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.115772497, 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 76692 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556264+0000", 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556264+0000", 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556266+0000", 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9750000000000001e-06 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556330+0000", 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3676000000000005e-05 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556341+0000", 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0818e-05 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556354+0000", 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3833e-05 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556406+0000", 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1916000000000002e-05 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556523+0000", 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116344 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,4", 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.556623+0000", 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010079 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.562284+0000", 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005660221 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.562335+0000", 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1415999999999997e-05 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.671969+0000", 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10963397699999999 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.672005+0000", 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5918999999999999e-05 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.672037+0000", 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1612000000000002e-05 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:76693 8.d 8:b3d08976:::benchmark_data_smithi081_105538_object76692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:59:34.557707+0000", 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 127.92385102599999, 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.111275834, 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 76693 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557707+0000", 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557707+0000", 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557709+0000", 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.928e-06 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557772+0000", 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3319999999999997e-05 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557790+0000", 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7875000000000001e-05 2024-09-21T19:01:42.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557798+0000", 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4259999999999997e-06 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.557899+0000", 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000101096 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558039+0000", 2024-09-21T19:01:42.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000140109 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,7", 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558194+0000", 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154695 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.560878+0000", 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026846050000000001 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.561126+0000", 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024791800000000002 2024-09-21T19:01:42.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.668875+0000", 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10774871 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.668935+0000", 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0254000000000003e-05 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.668983+0000", 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7898000000000001e-05 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:76694 8.4 8:2c91f862:::benchmark_data_smithi081_105538_object76693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:59:34.558643+0000", 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 127.92291486800001, 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.113533602, 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 76694 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558643+0000", 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558643+0000", 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558645+0000", 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3039999999999999e-06 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558702+0000", 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6975000000000003e-05 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558712+0000", 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.01e-05 2024-09-21T19:01:42.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558721+0000", 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3349999999999996e-06 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.558865+0000", 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000144029 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.559066+0000", 2024-09-21T19:01:42.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020122400000000001 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,4", 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.559200+0000", 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013318 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.562375+0000", 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031752239999999999 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.562403+0000", 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8079e-05 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.672083+0000", 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109679949 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.672138+0000", 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5305e-05 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:59:34.672177+0000", 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8897999999999999e-05 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:113170 8.8 8:1b33352e:::benchmark_data_smithi081_105538_object113169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:05.604659+0000", 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 96.876899503999994, 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11574250899999999, 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:42.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 113170 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.604659+0000", 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.604659+0000", 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.604661+0000", 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3140000000000002e-06 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.604692+0000", 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1031999999999997e-05 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.604706+0000", 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4038e-05 2024-09-21T19:01:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.604713+0000", 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3810000000000001e-06 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.606644+0000", 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001930525 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.606756+0000", 2024-09-21T19:01:42.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000112259 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.606884+0000", 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128193 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.609696+0000", 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002811568 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.610190+0000", 2024-09-21T19:01:42.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049412900000000005 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.720295+0000", 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11010487400000001 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.720349+0000", 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4353e-05 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:05.720401+0000", 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1842999999999998e-05 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.35258.0:116030 8.2 e202/172 8:4bedd4fe:::benchmark_data_smithi081_105538_object116029:head v 202'7261, mlcod=202'7259)", 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:00:08.244902+0000", 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 94.236656573000005, 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104468778, 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.244902+0000", 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.244902+0000", 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.244903+0000", 2024-09-21T19:01:42.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2920000000000001e-06 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.244965+0000", 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1981000000000002e-05 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.247677+0000", 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027121409999999999 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.247684+0000", 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5080000000000002e-06 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.331627+0000", 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.083943332999999995 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.331648+0000", 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1111000000000002e-05 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.349319+0000", 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017670826000000001 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:42.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:00:08.349370+0000", 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1585999999999997e-05 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:42.874 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:01:42.874 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:42.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:42.957 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:01:42.960 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:01:43.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:43.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:43.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:43.164 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:43.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:43.305 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[29384]: pgmap v399: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[29384]: 8.9 scrub starts 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[39192]: pgmap v399: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[39192]: 8.9 scrub starts 2024-09-21T19:01:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:43.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:43 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:43.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:01:43.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:43.446 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:43 smithi104 ceph-mon[32438]: pgmap v399: 168 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 9 unknown, 157 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:01:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:43 smithi104 ceph-mon[32438]: 8.9 scrub starts 2024-09-21T19:01:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:43 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:43 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:43.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:43.580 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:43.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:43.706 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:43.747 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:01:43.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:43.839 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:43.911 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-21T19:01:41.559065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2314934835,"num_objects":35513,"num_object_clones":0,"num_object_copies":106539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35513,"num_whiteouts":0,"num_read":1216,"num_read_kb":968,"num_write":166372,"num_write_kb":6350926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":6230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101978,"ondisk_log_size":101978,"up":498,"acting":498,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":501,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10956884,"kb_used_data":8372260,"kb_used_omap":12,"kb_used_meta":2584435,"kb_avail":739004332,"statfs":{"total":767960285184,"available":756740435968,"internally_reserved":0,"allocated":8573194240,"data_stored":8569952648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":2646462004},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-793182208,"num_objects":-12104,"num_object_clones":0,"num_object_copies":-36312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12104,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-51106,"num_write_kb":-2111776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001748"},"pg_stats":[{"pgid":"8.14","version":"223'13664","reported_seq":13874,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.099233+0000","last_change":"2024-09-21T19:01:36.570926+0000","last_active":"2024-09-21T19:01:36.099233+0000","last_peered":"2024-09-21T19:01:36.030717+0000","last_clean":"2024-09-21T19:01:36.030717+0000","last_became_active":"2024-09-21T19:01:35.119359+0000","last_became_peered":"2024-09-21T19:01:35.119359+0000","last_unstale":"2024-09-21T19:01:36.099233+0000","last_undegraded":"2024-09-21T19:01:36.099233+0000","last_fullsized":"2024-09-21T19:01:36.099233+0000","mapping_epoch":224,"log_start":"206'8600","ondisk_log_start":"206'8600","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"204'8114","last_scrub_stamp":"2024-09-21T19:00:21.263245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.263245+0000","objects_scrubbed":8036,"log_size":5064,"log_dups_size":0,"ondisk_log_size":5064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88145920,"num_objects":1345,"num_object_clones":0,"num_object_copies":4035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6855,"num_write_kb":262400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":250,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104645+0000","last_change":"2024-09-21T18:59:50.144510+0000","last_active":"2024-09-21T19:01:36.104645+0000","last_peered":"2024-09-21T19:01:36.104645+0000","last_clean":"2024-09-21T19:01:36.104645+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:01:36.104645+0000","last_undegraded":"2024-09-21T19:01:36.104645+0000","last_fullsized":"2024-09-21T19:01:36.104645+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.144434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.144434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:24.670179+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":399,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.097923+0000","last_change":"2024-09-21T19:01:15.751481+0000","last_active":"2024-09-21T19:01:36.097923+0000","last_peered":"2024-09-21T19:01:36.097923+0000","last_clean":"2024-09-21T19:01:36.097923+0000","last_became_active":"2024-09-21T18:54:36.757929+0000","last_became_peered":"2024-09-21T18:54:36.757929+0000","last_unstale":"2024-09-21T19:01:36.097923+0000","last_undegraded":"2024-09-21T19:01:36.097923+0000","last_fullsized":"2024-09-21T19:01:36.097923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:15.751424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:15.751424+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:17.420109+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.107514+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:01:36.107514+0000","last_peered":"2024-09-21T19:01:36.107514+0000","last_clean":"2024-09-21T19:01:36.107514+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:01:36.107514+0000","last_undegraded":"2024-09-21T19:01:36.107514+0000","last_fullsized":"2024-09-21T19:01:36.107514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:21.503742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":342,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.097284+0000","last_change":"2024-09-21T18:58:02.908124+0000","last_active":"2024-09-21T19:01:36.097284+0000","last_peered":"2024-09-21T19:01:36.097284+0000","last_clean":"2024-09-21T19:01:36.097284+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:01:36.097284+0000","last_undegraded":"2024-09-21T19:01:36.097284+0000","last_fullsized":"2024-09-21T19:01:36.097284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:02.908057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:02.908057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.765211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.15","version":"223'13823","reported_seq":14075,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.095579+0000","last_change":"2024-09-21T19:01:36.568007+0000","last_active":"2024-09-21T19:01:36.095579+0000","last_peered":"2024-09-21T19:01:36.055552+0000","last_clean":"2024-09-21T19:01:36.055552+0000","last_became_active":"2024-09-21T19:01:35.121296+0000","last_became_peered":"2024-09-21T19:01:35.121296+0000","last_unstale":"2024-09-21T19:01:36.095579+0000","last_undegraded":"2024-09-21T19:01:36.095579+0000","last_fullsized":"2024-09-21T19:01:36.095579+0000","mapping_epoch":224,"log_start":"204'8200","ondisk_log_start":"204'8200","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'11443","last_scrub_stamp":"2024-09-21T19:01:03.618148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.618148+0000","objects_scrubbed":5222,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89980928,"num_objects":1373,"num_object_clones":0,"num_object_copies":4119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":265824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":361,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.097824+0000","last_change":"2024-09-21T19:01:11.832004+0000","last_active":"2024-09-21T19:01:37.097824+0000","last_peered":"2024-09-21T19:01:37.097824+0000","last_clean":"2024-09-21T19:01:37.097824+0000","last_became_active":"2024-09-21T18:53:13.658328+0000","last_became_peered":"2024-09-21T18:53:13.658328+0000","last_unstale":"2024-09-21T19:01:37.097824+0000","last_undegraded":"2024-09-21T19:01:37.097824+0000","last_fullsized":"2024-09-21T19:01:37.097824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.831931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.831931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.385915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":366,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.099794+0000","last_change":"2024-09-21T18:58:36.978527+0000","last_active":"2024-09-21T19:01:37.099794+0000","last_peered":"2024-09-21T19:01:37.099794+0000","last_clean":"2024-09-21T19:01:37.099794+0000","last_became_active":"2024-09-21T18:53:07.390593+0000","last_became_peered":"2024-09-21T18:53:07.390593+0000","last_unstale":"2024-09-21T19:01:37.099794+0000","last_undegraded":"2024-09-21T19:01:37.099794+0000","last_fullsized":"2024-09-21T19:01:37.099794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:36.978471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:36.978471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.735886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":424,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093617+0000","last_change":"2024-09-21T19:00:33.910621+0000","last_active":"2024-09-21T19:01:37.093617+0000","last_peered":"2024-09-21T19:01:37.093617+0000","last_clean":"2024-09-21T19:01:37.093617+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:01:37.093617+0000","last_undegraded":"2024-09-21T19:01:37.093617+0000","last_fullsized":"2024-09-21T19:01:37.093617+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:33.910535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.910535+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.545163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.097846+0000","last_change":"2024-09-21T19:01:02.741840+0000","last_active":"2024-09-21T19:01:36.097846+0000","last_peered":"2024-09-21T19:01:36.097846+0000","last_clean":"2024-09-21T19:01:36.097846+0000","last_became_active":"2024-09-21T18:53:11.410199+0000","last_became_peered":"2024-09-21T18:53:11.410199+0000","last_unstale":"2024-09-21T19:01:36.097846+0000","last_undegraded":"2024-09-21T19:01:36.097846+0000","last_fullsized":"2024-09-21T19:01:36.097846+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.741768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.741768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:25.815288+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:01:34.061366+0000","last_change":"2024-09-21T19:01:34.061366+0000","last_active":"2024-09-21T19:01:34.061366+0000","last_peered":"2024-09-21T19:01:34.061366+0000","last_clean":"2024-09-21T19:01:34.061366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:01:34.061366+0000","last_undegraded":"2024-09-21T19:01:34.061366+0000","last_fullsized":"2024-09-21T19:01:34.061366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:34.061366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:01:34.061366+0000","last_clean_scrub_stamp":"2024-09-21T19:01:34.061366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104552+0000","last_change":"2024-09-21T19:00:28.036188+0000","last_active":"2024-09-21T19:01:36.104552+0000","last_peered":"2024-09-21T19:01:36.104552+0000","last_clean":"2024-09-21T19:01:36.104552+0000","last_became_active":"2024-09-21T18:53:07.400366+0000","last_became_peered":"2024-09-21T18:53:07.400366+0000","last_unstale":"2024-09-21T19:01:36.104552+0000","last_undegraded":"2024-09-21T19:01:36.104552+0000","last_fullsized":"2024-09-21T19:01:36.104552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:28.036123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.036123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.098067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":416,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.098205+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:01:36.098205+0000","last_peered":"2024-09-21T19:01:36.098205+0000","last_clean":"2024-09-21T19:01:36.098205+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:01:36.098205+0000","last_undegraded":"2024-09-21T19:01:36.098205+0000","last_fullsized":"2024-09-21T19:01:36.098205+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.756474+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":376,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102553+0000","last_change":"2024-09-21T18:59:20.569305+0000","last_active":"2024-09-21T19:01:36.102553+0000","last_peered":"2024-09-21T19:01:36.102553+0000","last_clean":"2024-09-21T19:01:36.102553+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:01:36.102553+0000","last_undegraded":"2024-09-21T19:01:36.102553+0000","last_fullsized":"2024-09-21T19:01:36.102553+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:20.569235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.569235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.891681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":242,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103954+0000","last_change":"2024-09-21T18:59:51.189725+0000","last_active":"2024-09-21T19:01:36.103954+0000","last_peered":"2024-09-21T19:01:36.103954+0000","last_clean":"2024-09-21T19:01:36.103954+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:01:36.103954+0000","last_undegraded":"2024-09-21T19:01:36.103954+0000","last_fullsized":"2024-09-21T19:01:36.103954+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.189659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.189659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:22.636508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.103019+0000","last_change":"2024-09-21T19:01:36.568032+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:01:36.103019+0000","last_undegraded":"2024-09-21T19:01:36.103019+0000","last_fullsized":"2024-09-21T19:01:36.103019+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11887","last_scrub_stamp":"2024-09-21T19:01:08.501670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:08.501670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103693+0000","last_change":"2024-09-21T18:59:48.531625+0000","last_active":"2024-09-21T19:01:36.103693+0000","last_peered":"2024-09-21T19:01:36.103693+0000","last_clean":"2024-09-21T19:01:36.103693+0000","last_became_active":"2024-09-21T18:53:07.401442+0000","last_became_peered":"2024-09-21T18:53:07.401442+0000","last_unstale":"2024-09-21T19:01:36.103693+0000","last_undegraded":"2024-09-21T19:01:36.103693+0000","last_fullsized":"2024-09-21T19:01:36.103693+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:48.531568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:48.531568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:24.978822+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":368,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.098885+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:01:36.098885+0000","last_peered":"2024-09-21T19:01:36.098885+0000","last_clean":"2024-09-21T19:01:36.098885+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:01:36.098885+0000","last_undegraded":"2024-09-21T19:01:36.098885+0000","last_fullsized":"2024-09-21T19:01:36.098885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.764044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":353,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093152+0000","last_change":"2024-09-21T18:57:13.008471+0000","last_active":"2024-09-21T19:01:36.093152+0000","last_peered":"2024-09-21T19:01:36.093152+0000","last_clean":"2024-09-21T19:01:36.093152+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:01:36.093152+0000","last_undegraded":"2024-09-21T19:01:36.093152+0000","last_fullsized":"2024-09-21T19:01:36.093152+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:13.008392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:13.008392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.462353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":366,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.106418+0000","last_change":"2024-09-21T19:01:06.061093+0000","last_active":"2024-09-21T19:01:36.106418+0000","last_peered":"2024-09-21T19:01:36.106418+0000","last_clean":"2024-09-21T19:01:36.106418+0000","last_became_active":"2024-09-21T18:53:13.427240+0000","last_became_peered":"2024-09-21T18:53:13.427240+0000","last_unstale":"2024-09-21T19:01:36.106418+0000","last_undegraded":"2024-09-21T19:01:36.106418+0000","last_fullsized":"2024-09-21T19:01:36.106418+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.061023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.061023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.606007+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.105077+0000","last_change":"2024-09-21T19:01:36.567952+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:01:36.105077+0000","last_undegraded":"2024-09-21T19:01:36.105077+0000","last_fullsized":"2024-09-21T19:01:36.105077+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11953","last_scrub_stamp":"2024-09-21T19:01:11.670275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.670275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":359,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099407+0000","last_change":"2024-09-21T19:01:13.714975+0000","last_active":"2024-09-21T19:01:36.099407+0000","last_peered":"2024-09-21T19:01:36.099407+0000","last_clean":"2024-09-21T19:01:36.099407+0000","last_became_active":"2024-09-21T18:53:13.657334+0000","last_became_peered":"2024-09-21T18:53:13.657334+0000","last_unstale":"2024-09-21T19:01:36.099407+0000","last_undegraded":"2024-09-21T19:01:36.099407+0000","last_fullsized":"2024-09-21T19:01:36.099407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:13.714855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.714855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.348742+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":250,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104104+0000","last_change":"2024-09-21T19:01:16.593955+0000","last_active":"2024-09-21T19:01:36.104104+0000","last_peered":"2024-09-21T19:01:36.104104+0000","last_clean":"2024-09-21T19:01:36.104104+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:01:36.104104+0000","last_undegraded":"2024-09-21T19:01:36.104104+0000","last_fullsized":"2024-09-21T19:01:36.104104+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:16.593895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.593895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.530636+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":391,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.100739+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:01:36.100739+0000","last_peered":"2024-09-21T19:01:36.100739+0000","last_clean":"2024-09-21T19:01:36.100739+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:01:36.100739+0000","last_undegraded":"2024-09-21T19:01:36.100739+0000","last_fullsized":"2024-09-21T19:01:36.100739+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.239922+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":376,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103732+0000","last_change":"2024-09-21T19:00:15.241392+0000","last_active":"2024-09-21T19:01:36.103732+0000","last_peered":"2024-09-21T19:01:36.103732+0000","last_clean":"2024-09-21T19:01:36.103732+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:01:36.103732+0000","last_undegraded":"2024-09-21T19:01:36.103732+0000","last_fullsized":"2024-09-21T19:01:36.103732+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:15.241320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:15.241320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.208748+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.101181+0000","last_change":"2024-09-21T19:01:36.598301+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:01:36.101181+0000","last_undegraded":"2024-09-21T19:01:36.101181+0000","last_fullsized":"2024-09-21T19:01:36.101181+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8051","last_scrub_stamp":"2024-09-21T19:00:19.800992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.800992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":346,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.100292+0000","last_change":"2024-09-21T18:59:19.930405+0000","last_active":"2024-09-21T19:01:37.100292+0000","last_peered":"2024-09-21T19:01:37.100292+0000","last_clean":"2024-09-21T19:01:37.100292+0000","last_became_active":"2024-09-21T18:53:13.657002+0000","last_became_peered":"2024-09-21T18:53:13.657002+0000","last_unstale":"2024-09-21T19:01:37.100292+0000","last_undegraded":"2024-09-21T19:01:37.100292+0000","last_fullsized":"2024-09-21T19:01:37.100292+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.930342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.930342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.867337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":378,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.098336+0000","last_change":"2024-09-21T19:01:33.521063+0000","last_active":"2024-09-21T19:01:36.098336+0000","last_peered":"2024-09-21T19:01:36.098336+0000","last_clean":"2024-09-21T19:01:36.098336+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:01:36.098336+0000","last_undegraded":"2024-09-21T19:01:36.098336+0000","last_fullsized":"2024-09-21T19:01:36.098336+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:33.520989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:33.520989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:03:02.224039+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":411,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.099171+0000","last_change":"2024-09-21T19:00:18.857976+0000","last_active":"2024-09-21T19:01:37.099171+0000","last_peered":"2024-09-21T19:01:37.099171+0000","last_clean":"2024-09-21T19:01:37.099171+0000","last_became_active":"2024-09-21T18:58:23.993245+0000","last_became_peered":"2024-09-21T18:58:23.993245+0000","last_unstale":"2024-09-21T19:01:37.099171+0000","last_undegraded":"2024-09-21T19:01:37.099171+0000","last_fullsized":"2024-09-21T19:01:37.099171+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:00:18.857901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:18.857901+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099060+0000","last_change":"2024-09-21T18:58:28.912619+0000","last_active":"2024-09-21T19:01:36.099060+0000","last_peered":"2024-09-21T19:01:36.099060+0000","last_clean":"2024-09-21T19:01:36.099060+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:01:36.099060+0000","last_undegraded":"2024-09-21T19:01:36.099060+0000","last_fullsized":"2024-09-21T19:01:36.099060+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:28.912554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:28.912554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.101995+0000","last_change":"2024-09-21T19:01:36.599622+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:01:36.101995+0000","last_undegraded":"2024-09-21T19:01:36.101995+0000","last_fullsized":"2024-09-21T19:01:36.101995+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12018","last_scrub_stamp":"2024-09-21T19:01:10.721645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.721645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":358,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.094197+0000","last_change":"2024-09-21T18:58:38.110564+0000","last_active":"2024-09-21T19:01:36.094197+0000","last_peered":"2024-09-21T19:01:36.094197+0000","last_clean":"2024-09-21T19:01:36.094197+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:01:36.094197+0000","last_undegraded":"2024-09-21T19:01:36.094197+0000","last_fullsized":"2024-09-21T19:01:36.094197+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T18:58:38.110509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:38.110509+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.751720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":374,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.107591+0000","last_change":"2024-09-21T19:00:23.223251+0000","last_active":"2024-09-21T19:01:36.107591+0000","last_peered":"2024-09-21T19:01:36.107591+0000","last_clean":"2024-09-21T19:01:36.107591+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:01:36.107591+0000","last_undegraded":"2024-09-21T19:01:36.107591+0000","last_fullsized":"2024-09-21T19:01:36.107591+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:23.223202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.223202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":262,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111065+0000","last_change":"2024-09-21T19:00:22.256583+0000","last_active":"2024-09-21T19:01:36.111065+0000","last_peered":"2024-09-21T19:01:36.111065+0000","last_clean":"2024-09-21T19:01:36.111065+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:01:36.111065+0000","last_undegraded":"2024-09-21T19:01:36.111065+0000","last_fullsized":"2024-09-21T19:01:36.111065+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:22.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.256498+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.316225+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":475,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104537+0000","last_change":"2024-09-21T19:00:35.363014+0000","last_active":"2024-09-21T19:01:36.104537+0000","last_peered":"2024-09-21T19:01:36.104537+0000","last_clean":"2024-09-21T19:01:36.104537+0000","last_became_active":"2024-09-21T18:53:11.407816+0000","last_became_peered":"2024-09-21T18:53:11.407816+0000","last_unstale":"2024-09-21T19:01:36.104537+0000","last_undegraded":"2024-09-21T19:01:36.104537+0000","last_fullsized":"2024-09-21T19:01:36.104537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.428300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"224'13591","reported_seq":13766,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103160+0000","last_change":"2024-09-21T19:01:35.111771+0000","last_active":"2024-09-21T19:01:36.103160+0000","last_peered":"2024-09-21T19:01:36.103160+0000","last_clean":"2024-09-21T19:01:36.103160+0000","last_became_active":"2024-09-21T19:01:35.091099+0000","last_became_peered":"2024-09-21T19:01:35.091099+0000","last_unstale":"2024-09-21T19:01:36.103160+0000","last_undegraded":"2024-09-21T19:01:36.103160+0000","last_fullsized":"2024-09-21T19:01:36.103160+0000","mapping_epoch":172,"log_start":"206'8500","ondisk_log_start":"206'8500","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'12221","last_scrub_stamp":"2024-09-21T19:01:16.859976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.859976+0000","objects_scrubbed":4162,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88408064,"num_objects":1349,"num_object_clones":0,"num_object_copies":4047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":261376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":135,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102612+0000","last_change":"2024-09-21T19:01:01.036323+0000","last_active":"2024-09-21T19:01:36.102612+0000","last_peered":"2024-09-21T19:01:36.102612+0000","last_clean":"2024-09-21T19:01:36.102612+0000","last_became_active":"2024-09-21T18:58:40.917818+0000","last_became_peered":"2024-09-21T18:58:40.917818+0000","last_unstale":"2024-09-21T19:01:36.102612+0000","last_undegraded":"2024-09-21T19:01:36.102612+0000","last_fullsized":"2024-09-21T19:01:36.102612+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:01.036275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:01.036275+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:01.626107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":383,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.105772+0000","last_change":"2024-09-21T18:59:21.369411+0000","last_active":"2024-09-21T19:01:36.105772+0000","last_peered":"2024-09-21T19:01:36.105772+0000","last_clean":"2024-09-21T19:01:36.105772+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:01:36.105772+0000","last_undegraded":"2024-09-21T19:01:36.105772+0000","last_fullsized":"2024-09-21T19:01:36.105772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:59:21.369328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:21.369328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.377767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":373,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.094051+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:01:37.094051+0000","last_peered":"2024-09-21T19:01:37.094051+0000","last_clean":"2024-09-21T19:01:37.094051+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:01:37.094051+0000","last_undegraded":"2024-09-21T19:01:37.094051+0000","last_fullsized":"2024-09-21T19:01:37.094051+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":494,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099507+0000","last_change":"2024-09-21T19:00:58.742094+0000","last_active":"2024-09-21T19:01:36.099507+0000","last_peered":"2024-09-21T19:01:36.099507+0000","last_clean":"2024-09-21T19:01:36.099507+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:01:36.099507+0000","last_undegraded":"2024-09-21T19:01:36.099507+0000","last_fullsized":"2024-09-21T19:01:36.099507+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:00:58.742016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:58.742016+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.133618+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":372,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104124+0000","last_change":"2024-09-21T18:58:00.690099+0000","last_active":"2024-09-21T19:01:36.104124+0000","last_peered":"2024-09-21T19:01:36.104124+0000","last_clean":"2024-09-21T19:01:36.104124+0000","last_became_active":"2024-09-21T18:55:51.524084+0000","last_became_peered":"2024-09-21T18:55:51.524084+0000","last_unstale":"2024-09-21T19:01:36.104124+0000","last_undegraded":"2024-09-21T19:01:36.104124+0000","last_fullsized":"2024-09-21T19:01:36.104124+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:00.690027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:00.690027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.873896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"224'13985","reported_seq":14234,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.109218+0000","last_change":"2024-09-21T19:01:12.177132+0000","last_active":"2024-09-21T19:01:36.109218+0000","last_peered":"2024-09-21T19:01:36.109218+0000","last_clean":"2024-09-21T19:01:36.109218+0000","last_became_active":"2024-09-21T18:58:44.182948+0000","last_became_peered":"2024-09-21T18:58:44.182948+0000","last_unstale":"2024-09-21T19:01:36.109218+0000","last_undegraded":"2024-09-21T19:01:36.109218+0000","last_fullsized":"2024-09-21T19:01:36.109218+0000","mapping_epoch":172,"log_start":"206'8805","ondisk_log_start":"206'8805","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12149","last_scrub_stamp":"2024-09-21T19:01:12.176892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:12.176892+0000","objects_scrubbed":4607,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:26.468219+0000","scrub_duration":1697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178061312,"num_objects":2717,"num_object_clones":0,"num_object_copies":8151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13985,"num_write_kb":534464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":127,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.100078+0000","last_change":"2024-09-21T18:59:47.823699+0000","last_active":"2024-09-21T19:01:36.100078+0000","last_peered":"2024-09-21T19:01:36.100078+0000","last_clean":"2024-09-21T19:01:36.100078+0000","last_became_active":"2024-09-21T18:58:40.916167+0000","last_became_peered":"2024-09-21T18:58:40.916167+0000","last_unstale":"2024-09-21T19:01:36.100078+0000","last_undegraded":"2024-09-21T19:01:36.100078+0000","last_fullsized":"2024-09-21T19:01:36.100078+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:47.823624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:47.823624+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:25.607138+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":444,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093673+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:01:37.093673+0000","last_peered":"2024-09-21T19:01:37.093673+0000","last_clean":"2024-09-21T19:01:37.093673+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:01:37.093673+0000","last_undegraded":"2024-09-21T19:01:37.093673+0000","last_fullsized":"2024-09-21T19:01:37.093673+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.772022+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":387,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.100936+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:01:37.100936+0000","last_peered":"2024-09-21T19:01:37.100936+0000","last_clean":"2024-09-21T19:01:37.100936+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:01:37.100936+0000","last_undegraded":"2024-09-21T19:01:37.100936+0000","last_fullsized":"2024-09-21T19:01:37.100936+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:28.008242+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":355,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.096692+0000","last_change":"2024-09-21T18:59:06.911709+0000","last_active":"2024-09-21T19:01:36.096692+0000","last_peered":"2024-09-21T19:01:36.096692+0000","last_clean":"2024-09-21T19:01:36.096692+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:01:36.096692+0000","last_undegraded":"2024-09-21T19:01:36.096692+0000","last_fullsized":"2024-09-21T19:01:36.096692+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:06.911638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:06.911638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:17.777898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":343,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.098115+0000","last_change":"2024-09-21T18:59:13.953988+0000","last_active":"2024-09-21T19:01:36.098115+0000","last_peered":"2024-09-21T19:01:36.098115+0000","last_clean":"2024-09-21T19:01:36.098115+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:01:36.098115+0000","last_undegraded":"2024-09-21T19:01:36.098115+0000","last_fullsized":"2024-09-21T19:01:36.098115+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:13.953912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:13.953912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.742766+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"224'13908","reported_seq":14150,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.127354+0000","last_change":"2024-09-21T19:01:35.455972+0000","last_active":"2024-09-21T19:01:36.127354+0000","last_peered":"2024-09-21T19:01:36.127354+0000","last_clean":"2024-09-21T19:01:36.127354+0000","last_became_active":"2024-09-21T18:58:44.184359+0000","last_became_peered":"2024-09-21T18:58:44.184359+0000","last_unstale":"2024-09-21T19:01:36.127354+0000","last_undegraded":"2024-09-21T19:01:36.127354+0000","last_fullsized":"2024-09-21T19:01:36.127354+0000","mapping_epoch":172,"log_start":"208'9495","ondisk_log_start":"208'9495","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'13846","last_scrub_stamp":"2024-09-21T19:01:35.455755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:35.455755+0000","objects_scrubbed":2775,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:44.163410+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702040,"num_objects":2682,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13908,"num_write_kb":530817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":134,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.098015+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:01:37.098015+0000","last_peered":"2024-09-21T19:01:37.098015+0000","last_clean":"2024-09-21T19:01:37.098015+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:01:37.098015+0000","last_undegraded":"2024-09-21T19:01:37.098015+0000","last_fullsized":"2024-09-21T19:01:37.098015+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:49.951550+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":380,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.096811+0000","last_change":"2024-09-21T18:59:20.963196+0000","last_active":"2024-09-21T19:01:36.096811+0000","last_peered":"2024-09-21T19:01:36.096811+0000","last_clean":"2024-09-21T19:01:36.096811+0000","last_became_active":"2024-09-21T18:53:09.400288+0000","last_became_peered":"2024-09-21T18:53:09.400288+0000","last_unstale":"2024-09-21T19:01:36.096811+0000","last_undegraded":"2024-09-21T19:01:36.096811+0000","last_fullsized":"2024-09-21T19:01:36.096811+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T18:59:20.962753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:20.962753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.779332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":254,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111307+0000","last_change":"2024-09-21T18:59:54.225013+0000","last_active":"2024-09-21T19:01:36.111307+0000","last_peered":"2024-09-21T19:01:36.111307+0000","last_clean":"2024-09-21T19:01:36.111307+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:01:36.111307+0000","last_undegraded":"2024-09-21T19:01:36.111307+0000","last_fullsized":"2024-09-21T19:01:36.111307+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T18:59:54.224944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:54.224944+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:21.623967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":460,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093439+0000","last_change":"2024-09-21T19:00:22.845024+0000","last_active":"2024-09-21T19:01:37.093439+0000","last_peered":"2024-09-21T19:01:37.093439+0000","last_clean":"2024-09-21T19:01:37.093439+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:01:37.093439+0000","last_undegraded":"2024-09-21T19:01:37.093439+0000","last_fullsized":"2024-09-21T19:01:37.093439+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:22.844953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.844953+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":350,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093524+0000","last_change":"2024-09-21T19:01:06.644894+0000","last_active":"2024-09-21T19:01:36.093524+0000","last_peered":"2024-09-21T19:01:36.093524+0000","last_clean":"2024-09-21T19:01:36.093524+0000","last_became_active":"2024-09-21T18:53:13.658017+0000","last_became_peered":"2024-09-21T18:53:13.658017+0000","last_unstale":"2024-09-21T19:01:36.093524+0000","last_undegraded":"2024-09-21T19:01:36.093524+0000","last_fullsized":"2024-09-21T19:01:36.093524+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.644848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.644848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:06.908909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"224'13892","reported_seq":14065,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.128106+0000","last_change":"2024-09-21T19:01:11.369577+0000","last_active":"2024-09-21T19:01:36.128106+0000","last_peered":"2024-09-21T19:01:36.128106+0000","last_clean":"2024-09-21T19:01:36.128106+0000","last_became_active":"2024-09-21T18:58:44.096773+0000","last_became_peered":"2024-09-21T18:58:44.096773+0000","last_unstale":"2024-09-21T19:01:36.128106+0000","last_undegraded":"2024-09-21T19:01:36.128106+0000","last_fullsized":"2024-09-21T19:01:36.128106+0000","mapping_epoch":172,"log_start":"206'8716","ondisk_log_start":"206'8716","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12063","last_scrub_stamp":"2024-09-21T19:01:11.369324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.369324+0000","objects_scrubbed":4626,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:30.595821+0000","scrub_duration":1905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179044352,"num_objects":2732,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13892,"num_write_kb":531968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":211,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104329+0000","last_change":"2024-09-21T18:59:46.185760+0000","last_active":"2024-09-21T19:01:36.104329+0000","last_peered":"2024-09-21T19:01:36.104329+0000","last_clean":"2024-09-21T19:01:36.104329+0000","last_became_active":"2024-09-21T18:58:40.530131+0000","last_became_peered":"2024-09-21T18:58:40.530131+0000","last_unstale":"2024-09-21T19:01:36.104329+0000","last_undegraded":"2024-09-21T19:01:36.104329+0000","last_fullsized":"2024-09-21T19:01:36.104329+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.185693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.185693+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.406096+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":366,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.105534+0000","last_change":"2024-09-21T18:59:17.346990+0000","last_active":"2024-09-21T19:01:36.105534+0000","last_peered":"2024-09-21T19:01:36.105534+0000","last_clean":"2024-09-21T19:01:36.105534+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:01:36.105534+0000","last_undegraded":"2024-09-21T19:01:36.105534+0000","last_fullsized":"2024-09-21T19:01:36.105534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:17.346414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.346414+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.396410+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":371,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104634+0000","last_change":"2024-09-21T18:59:16.589986+0000","last_active":"2024-09-21T19:01:36.104634+0000","last_peered":"2024-09-21T19:01:36.104634+0000","last_clean":"2024-09-21T19:01:36.104634+0000","last_became_active":"2024-09-21T18:53:07.403228+0000","last_became_peered":"2024-09-21T18:53:07.403228+0000","last_unstale":"2024-09-21T19:01:36.104634+0000","last_undegraded":"2024-09-21T19:01:36.104634+0000","last_fullsized":"2024-09-21T19:01:36.104634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.589912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.589912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.916495+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"114'10","reported_seq":315,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.539362+0000","last_change":"2024-09-21T19:01:36.539362+0000","last_active":"2024-09-21T19:01:36.539362+0000","last_peered":"2024-09-21T19:01:36.539362+0000","last_clean":"2024-09-21T19:01:36.539362+0000","last_became_active":"2024-09-21T18:55:51.519486+0000","last_became_peered":"2024-09-21T18:55:51.519486+0000","last_unstale":"2024-09-21T19:01:36.539362+0000","last_undegraded":"2024-09-21T19:01:36.539362+0000","last_fullsized":"2024-09-21T19:01:36.539362+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:01:36.539314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:36.539314+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:57.372618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":242,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104101+0000","last_change":"2024-09-21T18:59:44.256590+0000","last_active":"2024-09-21T19:01:36.104101+0000","last_peered":"2024-09-21T19:01:36.104101+0000","last_clean":"2024-09-21T19:01:36.104101+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:01:36.104101+0000","last_undegraded":"2024-09-21T19:01:36.104101+0000","last_fullsized":"2024-09-21T19:01:36.104101+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:44.256513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.256513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.406650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"224'13865","reported_seq":13986,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.100786+0000","last_change":"2024-09-21T18:58:44.097045+0000","last_active":"2024-09-21T19:01:37.100786+0000","last_peered":"2024-09-21T19:01:37.100786+0000","last_clean":"2024-09-21T19:01:37.100786+0000","last_became_active":"2024-09-21T18:58:44.096634+0000","last_became_peered":"2024-09-21T18:58:44.096634+0000","last_unstale":"2024-09-21T19:01:37.100786+0000","last_undegraded":"2024-09-21T19:01:37.100786+0000","last_fullsized":"2024-09-21T19:01:37.100786+0000","mapping_epoch":172,"log_start":"210'9698","ondisk_log_start":"210'9698","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176619520,"num_objects":2695,"num_object_clones":0,"num_object_copies":8085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13865,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":219,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103448+0000","last_change":"2024-09-21T19:01:00.420038+0000","last_active":"2024-09-21T19:01:36.103448+0000","last_peered":"2024-09-21T19:01:36.103448+0000","last_clean":"2024-09-21T19:01:36.103448+0000","last_became_active":"2024-09-21T18:58:40.528964+0000","last_became_peered":"2024-09-21T18:58:40.528964+0000","last_unstale":"2024-09-21T19:01:36.103448+0000","last_undegraded":"2024-09-21T19:01:36.103448+0000","last_fullsized":"2024-09-21T19:01:36.103448+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:00.419946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.419946+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.968029+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":276,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104190+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:01:36.104190+0000","last_peered":"2024-09-21T19:01:36.104190+0000","last_clean":"2024-09-21T19:01:36.104190+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:01:36.104190+0000","last_undegraded":"2024-09-21T19:01:36.104190+0000","last_fullsized":"2024-09-21T19:01:36.104190+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.035114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":371,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102786+0000","last_change":"2024-09-21T19:00:56.467643+0000","last_active":"2024-09-21T19:01:36.102786+0000","last_peered":"2024-09-21T19:01:36.102786+0000","last_clean":"2024-09-21T19:01:36.102786+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:01:36.102786+0000","last_undegraded":"2024-09-21T19:01:36.102786+0000","last_fullsized":"2024-09-21T19:01:36.102786+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.462694+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":368,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102388+0000","last_change":"2024-09-21T19:00:34.409013+0000","last_active":"2024-09-21T19:01:36.102388+0000","last_peered":"2024-09-21T19:01:36.102388+0000","last_clean":"2024-09-21T19:01:36.102388+0000","last_became_active":"2024-09-21T18:55:51.513921+0000","last_became_peered":"2024-09-21T18:55:51.513921+0000","last_unstale":"2024-09-21T19:01:36.102388+0000","last_undegraded":"2024-09-21T19:01:36.102388+0000","last_fullsized":"2024-09-21T19:01:36.102388+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.816307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":351,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.107457+0000","last_change":"2024-09-21T19:00:25.249425+0000","last_active":"2024-09-21T19:01:36.107457+0000","last_peered":"2024-09-21T19:01:36.107457+0000","last_clean":"2024-09-21T19:01:36.107457+0000","last_became_active":"2024-09-21T18:53:13.656895+0000","last_became_peered":"2024-09-21T18:53:13.656895+0000","last_unstale":"2024-09-21T19:01:36.107457+0000","last_undegraded":"2024-09-21T19:01:36.107457+0000","last_fullsized":"2024-09-21T19:01:36.107457+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.249360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.249360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:32.546697+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"223'13372","reported_seq":13549,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104526+0000","last_change":"2024-09-21T19:01:35.136659+0000","last_active":"2024-09-21T19:01:36.104526+0000","last_peered":"2024-09-21T19:01:36.104526+0000","last_clean":"2024-09-21T19:01:36.104526+0000","last_became_active":"2024-09-21T19:01:35.105654+0000","last_became_peered":"2024-09-21T19:01:35.105654+0000","last_unstale":"2024-09-21T19:01:36.104526+0000","last_undegraded":"2024-09-21T19:01:36.104526+0000","last_fullsized":"2024-09-21T19:01:36.104526+0000","mapping_epoch":172,"log_start":"206'8400","ondisk_log_start":"206'8400","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'11951","last_scrub_stamp":"2024-09-21T19:01:16.186043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.186043+0000","objects_scrubbed":4160,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6709,"num_write_kb":257120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":135,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.106342+0000","last_change":"2024-09-21T19:00:55.902634+0000","last_active":"2024-09-21T19:01:36.106342+0000","last_peered":"2024-09-21T19:01:36.106342+0000","last_clean":"2024-09-21T19:01:36.106342+0000","last_became_active":"2024-09-21T18:58:40.916397+0000","last_became_peered":"2024-09-21T18:58:40.916397+0000","last_unstale":"2024-09-21T19:01:36.106342+0000","last_undegraded":"2024-09-21T19:01:36.106342+0000","last_fullsized":"2024-09-21T19:01:36.106342+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:55.902117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:55.902117+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.038393+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":509,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102631+0000","last_change":"2024-09-21T18:59:44.360729+0000","last_active":"2024-09-21T19:01:36.102631+0000","last_peered":"2024-09-21T19:01:36.102631+0000","last_clean":"2024-09-21T19:01:36.102631+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:01:36.102631+0000","last_undegraded":"2024-09-21T19:01:36.102631+0000","last_fullsized":"2024-09-21T19:01:36.102631+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-21T18:59:44.360619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:44.360619+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.396002+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":385,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099897+0000","last_change":"2024-09-21T18:59:55.645493+0000","last_active":"2024-09-21T19:01:36.099897+0000","last_peered":"2024-09-21T19:01:36.099897+0000","last_clean":"2024-09-21T19:01:36.099897+0000","last_became_active":"2024-09-21T18:55:51.521850+0000","last_became_peered":"2024-09-21T18:55:51.521850+0000","last_unstale":"2024-09-21T19:01:36.099897+0000","last_undegraded":"2024-09-21T19:01:36.099897+0000","last_fullsized":"2024-09-21T19:01:36.099897+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:55.645419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:55.645419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:26.584415+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":389,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.105346+0000","last_change":"2024-09-21T19:00:54.906034+0000","last_active":"2024-09-21T19:01:36.105346+0000","last_peered":"2024-09-21T19:01:36.105346+0000","last_clean":"2024-09-21T19:01:36.105346+0000","last_became_active":"2024-09-21T18:53:06.394375+0000","last_became_peered":"2024-09-21T18:53:06.394375+0000","last_unstale":"2024-09-21T19:01:36.105346+0000","last_undegraded":"2024-09-21T19:01:36.105346+0000","last_fullsized":"2024-09-21T19:01:36.105346+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:00:54.905971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:54.905971+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:05.061588+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0014819169999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":242,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103886+0000","last_change":"2024-09-21T19:00:25.316275+0000","last_active":"2024-09-21T19:01:36.103886+0000","last_peered":"2024-09-21T19:01:36.103886+0000","last_clean":"2024-09-21T19:01:36.103886+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:01:36.103886+0000","last_undegraded":"2024-09-21T19:01:36.103886+0000","last_fullsized":"2024-09-21T19:01:36.103886+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:25.314718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:25.314718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.477738+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":353,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.099538+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:01:37.099538+0000","last_peered":"2024-09-21T19:01:37.099538+0000","last_clean":"2024-09-21T19:01:37.099538+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:01:37.099538+0000","last_undegraded":"2024-09-21T19:01:37.099538+0000","last_fullsized":"2024-09-21T19:01:37.099538+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:25.689922+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"224'13923","reported_seq":14044,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093892+0000","last_change":"2024-09-21T18:58:44.100975+0000","last_active":"2024-09-21T19:01:37.093892+0000","last_peered":"2024-09-21T19:01:37.093892+0000","last_clean":"2024-09-21T19:01:37.093892+0000","last_became_active":"2024-09-21T18:58:44.100676+0000","last_became_peered":"2024-09-21T18:58:44.100676+0000","last_unstale":"2024-09-21T19:01:37.093892+0000","last_undegraded":"2024-09-21T19:01:37.093892+0000","last_fullsized":"2024-09-21T19:01:37.093892+0000","mapping_epoch":172,"log_start":"210'9755","ondisk_log_start":"210'9755","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T18:58:43.073980+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181469184,"num_objects":2769,"num_object_clones":0,"num_object_copies":8307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13923,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":448,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.092982+0000","last_change":"2024-09-21T19:01:05.625849+0000","last_active":"2024-09-21T19:01:36.092982+0000","last_peered":"2024-09-21T19:01:36.092982+0000","last_clean":"2024-09-21T19:01:36.092982+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:01:36.092982+0000","last_undegraded":"2024-09-21T19:01:36.092982+0000","last_fullsized":"2024-09-21T19:01:36.092982+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:05.625798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.625798+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.456539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":338,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.101800+0000","last_change":"2024-09-21T19:01:01.980676+0000","last_active":"2024-09-21T19:01:36.101800+0000","last_peered":"2024-09-21T19:01:36.101800+0000","last_clean":"2024-09-21T19:01:36.101800+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:01:36.101800+0000","last_undegraded":"2024-09-21T19:01:36.101800+0000","last_fullsized":"2024-09-21T19:01:36.101800+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-21T19:01:01.980397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:01.980397+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:31.229183+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":377,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103766+0000","last_change":"2024-09-21T19:01:07.496476+0000","last_active":"2024-09-21T19:01:36.103766+0000","last_peered":"2024-09-21T19:01:36.103766+0000","last_clean":"2024-09-21T19:01:36.103766+0000","last_became_active":"2024-09-21T18:53:07.398694+0000","last_became_peered":"2024-09-21T18:53:07.398694+0000","last_unstale":"2024-09-21T19:01:36.103766+0000","last_undegraded":"2024-09-21T19:01:36.103766+0000","last_fullsized":"2024-09-21T19:01:36.103766+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:36.105065+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":410,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099275+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:01:36.099275+0000","last_peered":"2024-09-21T19:01:36.099275+0000","last_clean":"2024-09-21T19:01:36.099275+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:01:36.099275+0000","last_undegraded":"2024-09-21T19:01:36.099275+0000","last_fullsized":"2024-09-21T19:01:36.099275+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.686451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099976+0000","last_change":"2024-09-21T19:00:29.002389+0000","last_active":"2024-09-21T19:01:36.099976+0000","last_peered":"2024-09-21T19:01:36.099976+0000","last_clean":"2024-09-21T19:01:36.099976+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:01:36.099976+0000","last_undegraded":"2024-09-21T19:01:36.099976+0000","last_fullsized":"2024-09-21T19:01:36.099976+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.002321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.002321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:56.121924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.095830+0000","last_change":"2024-09-21T19:00:32.573881+0000","last_active":"2024-09-21T19:01:36.095830+0000","last_peered":"2024-09-21T19:01:36.095830+0000","last_clean":"2024-09-21T19:01:36.095830+0000","last_became_active":"2024-09-21T18:53:13.657321+0000","last_became_peered":"2024-09-21T18:53:13.657321+0000","last_unstale":"2024-09-21T19:01:36.095830+0000","last_undegraded":"2024-09-21T19:01:36.095830+0000","last_fullsized":"2024-09-21T19:01:36.095830+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:00:32.573704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.573704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:54.007025+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"224'13669","reported_seq":13917,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.119170+0000","last_change":"2024-09-21T19:01:05.288396+0000","last_active":"2024-09-21T19:01:36.119170+0000","last_peered":"2024-09-21T19:01:36.119170+0000","last_clean":"2024-09-21T19:01:36.119170+0000","last_became_active":"2024-09-21T18:58:44.099439+0000","last_became_peered":"2024-09-21T18:58:44.099439+0000","last_unstale":"2024-09-21T19:01:36.119170+0000","last_undegraded":"2024-09-21T19:01:36.119170+0000","last_fullsized":"2024-09-21T19:01:36.119170+0000","mapping_epoch":172,"log_start":"207'8886","ondisk_log_start":"207'8886","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11506","last_scrub_stamp":"2024-09-21T19:01:05.288327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.288327+0000","objects_scrubbed":4952,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:22.889578+0000","scrub_duration":1801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177930240,"num_objects":2715,"num_object_clones":0,"num_object_copies":8145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13669,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":135,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104306+0000","last_change":"2024-09-21T19:00:59.488920+0000","last_active":"2024-09-21T19:01:36.104306+0000","last_peered":"2024-09-21T19:01:36.104306+0000","last_clean":"2024-09-21T19:01:36.104306+0000","last_became_active":"2024-09-21T18:58:40.528545+0000","last_became_peered":"2024-09-21T18:58:40.528545+0000","last_unstale":"2024-09-21T19:01:36.104306+0000","last_undegraded":"2024-09-21T19:01:36.104306+0000","last_fullsized":"2024-09-21T19:01:36.104306+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:59.488847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:59.488847+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.549020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":272,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111146+0000","last_change":"2024-09-21T19:01:13.569543+0000","last_active":"2024-09-21T19:01:36.111146+0000","last_peered":"2024-09-21T19:01:36.111146+0000","last_clean":"2024-09-21T19:01:36.111146+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:01:36.111146+0000","last_undegraded":"2024-09-21T19:01:36.111146+0000","last_fullsized":"2024-09-21T19:01:36.111146+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:01:13.569464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.569464+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:19.751336+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":250,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.110611+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:01:36.110611+0000","last_peered":"2024-09-21T19:01:36.110611+0000","last_clean":"2024-09-21T19:01:36.110611+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:01:36.110611+0000","last_undegraded":"2024-09-21T19:01:36.110611+0000","last_fullsized":"2024-09-21T19:01:36.110611+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:23.667709+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":898,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093824+0000","last_change":"2024-09-21T18:57:15.070581+0000","last_active":"2024-09-21T19:01:36.093824+0000","last_peered":"2024-09-21T19:01:36.093824+0000","last_clean":"2024-09-21T19:01:36.093824+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:01:36.093824+0000","last_undegraded":"2024-09-21T19:01:36.093824+0000","last_fullsized":"2024-09-21T19:01:36.093824+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:57:15.070510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T18:57:15.070510+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.500330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":498,"num_read_kb":493,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":392,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103313+0000","last_change":"2024-09-21T19:00:56.950003+0000","last_active":"2024-09-21T19:01:36.103313+0000","last_peered":"2024-09-21T19:01:36.103313+0000","last_clean":"2024-09-21T19:01:36.103313+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:01:36.103313+0000","last_undegraded":"2024-09-21T19:01:36.103313+0000","last_fullsized":"2024-09-21T19:01:36.103313+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.949931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.949931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.427785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":351,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102374+0000","last_change":"2024-09-21T19:00:33.443101+0000","last_active":"2024-09-21T19:01:36.102374+0000","last_peered":"2024-09-21T19:01:36.102374+0000","last_clean":"2024-09-21T19:01:36.102374+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:01:36.102374+0000","last_undegraded":"2024-09-21T19:01:36.102374+0000","last_fullsized":"2024-09-21T19:01:36.102374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:33.443040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:33.443040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.013096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"224'14009","reported_seq":14210,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.124117+0000","last_change":"2024-09-21T19:01:35.128827+0000","last_active":"2024-09-21T19:01:36.124117+0000","last_peered":"2024-09-21T19:01:36.124117+0000","last_clean":"2024-09-21T19:01:36.124117+0000","last_became_active":"2024-09-21T19:01:35.105754+0000","last_became_peered":"2024-09-21T19:01:35.105754+0000","last_unstale":"2024-09-21T19:01:36.124117+0000","last_undegraded":"2024-09-21T19:01:36.124117+0000","last_fullsized":"2024-09-21T19:01:36.124117+0000","mapping_epoch":172,"log_start":"205'8400","ondisk_log_start":"205'8400","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'6787","last_scrub_stamp":"2024-09-21T19:00:00.245707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.245707+0000","objects_scrubbed":6706,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7028,"num_write_kb":269568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'16","reported_seq":482,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.101820+0000","last_change":"2024-09-21T18:59:12.162986+0000","last_active":"2024-09-21T19:01:36.101820+0000","last_peered":"2024-09-21T19:01:36.101820+0000","last_clean":"2024-09-21T19:01:36.101820+0000","last_became_active":"2024-09-21T18:53:09.719083+0000","last_became_peered":"2024-09-21T18:53:09.719083+0000","last_unstale":"2024-09-21T19:01:36.101820+0000","last_undegraded":"2024-09-21T19:01:36.101820+0000","last_fullsized":"2024-09-21T19:01:36.101820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2024-09-21T18:59:12.162725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.162725+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.317843+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":366,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.097570+0000","last_change":"2024-09-21T18:59:50.993922+0000","last_active":"2024-09-21T19:01:37.097570+0000","last_peered":"2024-09-21T19:01:37.097570+0000","last_clean":"2024-09-21T19:01:37.097570+0000","last_became_active":"2024-09-21T18:53:07.398722+0000","last_became_peered":"2024-09-21T18:53:07.398722+0000","last_unstale":"2024-09-21T19:01:37.097570+0000","last_undegraded":"2024-09-21T19:01:37.097570+0000","last_fullsized":"2024-09-21T19:01:37.097570+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":443,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.094008+0000","last_change":"2024-09-21T19:00:57.551189+0000","last_active":"2024-09-21T19:01:36.094008+0000","last_peered":"2024-09-21T19:01:36.094008+0000","last_clean":"2024-09-21T19:01:36.094008+0000","last_became_active":"2024-09-21T18:58:39.058347+0000","last_became_peered":"2024-09-21T18:58:39.058347+0000","last_unstale":"2024-09-21T19:01:36.094008+0000","last_undegraded":"2024-09-21T19:01:36.094008+0000","last_fullsized":"2024-09-21T19:01:36.094008+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:57.551119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:57.551119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:08.725028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.094462+0000","last_change":"2024-09-21T19:00:00.889227+0000","last_active":"2024-09-21T19:01:36.094462+0000","last_peered":"2024-09-21T19:01:36.094462+0000","last_clean":"2024-09-21T19:01:36.094462+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:01:36.094462+0000","last_undegraded":"2024-09-21T19:01:36.094462+0000","last_fullsized":"2024-09-21T19:01:36.094462+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.889156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.889156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:28.885642+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":358,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103166+0000","last_change":"2024-09-21T19:01:10.038311+0000","last_active":"2024-09-21T19:01:36.103166+0000","last_peered":"2024-09-21T19:01:36.103166+0000","last_clean":"2024-09-21T19:01:36.103166+0000","last_became_active":"2024-09-21T18:53:13.429965+0000","last_became_peered":"2024-09-21T18:53:13.429965+0000","last_unstale":"2024-09-21T19:01:36.103166+0000","last_undegraded":"2024-09-21T19:01:36.103166+0000","last_fullsized":"2024-09-21T19:01:36.103166+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:37.189738+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"224'13836","reported_seq":14039,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.097108+0000","last_change":"2024-09-21T19:01:35.902303+0000","last_active":"2024-09-21T19:01:36.097108+0000","last_peered":"2024-09-21T19:01:36.097108+0000","last_clean":"2024-09-21T19:01:36.097108+0000","last_became_active":"2024-09-21T19:01:35.091345+0000","last_became_peered":"2024-09-21T19:01:35.091345+0000","last_unstale":"2024-09-21T19:01:36.097108+0000","last_undegraded":"2024-09-21T19:01:36.097108+0000","last_fullsized":"2024-09-21T19:01:36.097108+0000","mapping_epoch":172,"log_start":"207'9100","ondisk_log_start":"207'9100","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'13822","last_scrub_stamp":"2024-09-21T19:01:35.902211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:35.902211+0000","objects_scrubbed":1427,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:03:04.816025+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92143616,"num_objects":1406,"num_object_clones":0,"num_object_copies":4218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":385,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099619+0000","last_change":"2024-09-21T19:00:01.693318+0000","last_active":"2024-09-21T19:01:36.099619+0000","last_peered":"2024-09-21T19:01:36.099619+0000","last_clean":"2024-09-21T19:01:36.099619+0000","last_became_active":"2024-09-21T18:53:09.406687+0000","last_became_peered":"2024-09-21T18:53:09.406687+0000","last_unstale":"2024-09-21T19:01:36.099619+0000","last_undegraded":"2024-09-21T19:01:36.099619+0000","last_fullsized":"2024-09-21T19:01:36.099619+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:01.693221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.693221+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":370,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.097060+0000","last_change":"2024-09-21T19:01:34.545925+0000","last_active":"2024-09-21T19:01:36.097060+0000","last_peered":"2024-09-21T19:01:36.097060+0000","last_clean":"2024-09-21T19:01:36.097060+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:01:36.097060+0000","last_undegraded":"2024-09-21T19:01:36.097060+0000","last_fullsized":"2024-09-21T19:01:36.097060+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:34.545879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:34.545879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:59.182821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":378,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103025+0000","last_change":"2024-09-21T19:01:02.080311+0000","last_active":"2024-09-21T19:01:36.103025+0000","last_peered":"2024-09-21T19:01:36.103025+0000","last_clean":"2024-09-21T19:01:36.103025+0000","last_became_active":"2024-09-21T18:53:11.407620+0000","last_became_peered":"2024-09-21T18:53:11.407620+0000","last_unstale":"2024-09-21T19:01:36.103025+0000","last_undegraded":"2024-09-21T19:01:36.103025+0000","last_fullsized":"2024-09-21T19:01:36.103025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.080248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.080248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.771263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.097907+0000","last_change":"2024-09-21T18:53:13.657700+0000","last_active":"2024-09-21T19:01:37.097907+0000","last_peered":"2024-09-21T19:01:37.097907+0000","last_clean":"2024-09-21T19:01:37.097907+0000","last_became_active":"2024-09-21T18:53:13.656898+0000","last_became_peered":"2024-09-21T18:53:13.656898+0000","last_unstale":"2024-09-21T19:01:37.097907+0000","last_undegraded":"2024-09-21T19:01:37.097907+0000","last_fullsized":"2024-09-21T19:01:37.097907+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:53:12.406226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.918918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"223'13769","reported_seq":13983,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.106971+0000","last_change":"2024-09-21T19:01:35.118438+0000","last_active":"2024-09-21T19:01:36.106971+0000","last_peered":"2024-09-21T19:01:36.106971+0000","last_clean":"2024-09-21T19:01:36.106971+0000","last_became_active":"2024-09-21T19:01:35.086598+0000","last_became_peered":"2024-09-21T19:01:35.086598+0000","last_unstale":"2024-09-21T19:01:36.106971+0000","last_undegraded":"2024-09-21T19:01:36.106971+0000","last_fullsized":"2024-09-21T19:01:36.106971+0000","mapping_epoch":172,"log_start":"206'8700","ondisk_log_start":"206'8700","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8051","last_scrub_stamp":"2024-09-21T19:00:19.800992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.800992+0000","objects_scrubbed":7979,"log_size":5069,"log_dups_size":0,"ondisk_log_size":5069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89358336,"num_objects":1363,"num_object_clones":0,"num_object_copies":4089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":264832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":142,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104745+0000","last_change":"2024-09-21T19:01:05.092005+0000","last_active":"2024-09-21T19:01:36.104745+0000","last_peered":"2024-09-21T19:01:36.104745+0000","last_clean":"2024-09-21T19:01:36.104745+0000","last_became_active":"2024-09-21T18:58:40.917793+0000","last_became_peered":"2024-09-21T18:58:40.917793+0000","last_unstale":"2024-09-21T19:01:36.104745+0000","last_undegraded":"2024-09-21T19:01:36.104745+0000","last_fullsized":"2024-09-21T19:01:36.104745+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:01:05.091927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:01:05.091927+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:14.822244+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":386,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.100013+0000","last_change":"2024-09-21T18:59:03.883391+0000","last_active":"2024-09-21T19:01:36.100013+0000","last_peered":"2024-09-21T19:01:36.100013+0000","last_clean":"2024-09-21T19:01:36.100013+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:01:36.100013+0000","last_undegraded":"2024-09-21T19:01:36.100013+0000","last_fullsized":"2024-09-21T19:01:36.100013+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T18:59:03.883306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:03.883306+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:48.725718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":369,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103222+0000","last_change":"2024-09-21T19:01:14.376753+0000","last_active":"2024-09-21T19:01:36.103222+0000","last_peered":"2024-09-21T19:01:36.103222+0000","last_clean":"2024-09-21T19:01:36.103222+0000","last_became_active":"2024-09-21T18:53:07.400237+0000","last_became_peered":"2024-09-21T18:53:07.400237+0000","last_unstale":"2024-09-21T19:01:36.103222+0000","last_undegraded":"2024-09-21T19:01:36.103222+0000","last_fullsized":"2024-09-21T19:01:36.103222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:14.376671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:14.376671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:34.966862+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":360,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102426+0000","last_change":"2024-09-21T18:58:04.470636+0000","last_active":"2024-09-21T19:01:36.102426+0000","last_peered":"2024-09-21T19:01:36.102426+0000","last_clean":"2024-09-21T19:01:36.102426+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:01:36.102426+0000","last_undegraded":"2024-09-21T19:01:36.102426+0000","last_fullsized":"2024-09-21T19:01:36.102426+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:04.470576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:04.470576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.313883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.101941+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:01:36.101941+0000","last_peered":"2024-09-21T19:01:36.101941+0000","last_clean":"2024-09-21T19:01:36.101941+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:01:36.101941+0000","last_undegraded":"2024-09-21T19:01:36.101941+0000","last_fullsized":"2024-09-21T19:01:36.101941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:53.899930+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"224'13687","reported_seq":13868,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093216+0000","last_change":"2024-09-21T19:01:35.109686+0000","last_active":"2024-09-21T19:01:37.093216+0000","last_peered":"2024-09-21T19:01:37.093216+0000","last_clean":"2024-09-21T19:01:37.093216+0000","last_became_active":"2024-09-21T19:01:35.091187+0000","last_became_peered":"2024-09-21T19:01:35.091187+0000","last_unstale":"2024-09-21T19:01:37.093216+0000","last_undegraded":"2024-09-21T19:01:37.093216+0000","last_fullsized":"2024-09-21T19:01:37.093216+0000","mapping_epoch":172,"log_start":"207'9100","ondisk_log_start":"207'9100","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11953","last_scrub_stamp":"2024-09-21T19:01:11.670275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.670275+0000","objects_scrubbed":4463,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85884928,"num_objects":1310,"num_object_clones":0,"num_object_copies":3930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6873,"num_write_kb":261888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":440,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093140+0000","last_change":"2024-09-21T18:59:50.870015+0000","last_active":"2024-09-21T19:01:36.093140+0000","last_peered":"2024-09-21T19:01:36.093140+0000","last_clean":"2024-09-21T19:01:36.093140+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:01:36.093140+0000","last_undegraded":"2024-09-21T19:01:36.093140+0000","last_fullsized":"2024-09-21T19:01:36.093140+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:50.869945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.869945+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:25.810413+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":373,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099146+0000","last_change":"2024-09-21T18:59:53.699627+0000","last_active":"2024-09-21T19:01:36.099146+0000","last_peered":"2024-09-21T19:01:36.099146+0000","last_clean":"2024-09-21T19:01:36.099146+0000","last_became_active":"2024-09-21T18:53:09.401340+0000","last_became_peered":"2024-09-21T18:53:09.401340+0000","last_unstale":"2024-09-21T19:01:36.099146+0000","last_undegraded":"2024-09-21T19:01:36.099146+0000","last_fullsized":"2024-09-21T19:01:36.099146+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:20.717257+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":401,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.098264+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:01:37.098264+0000","last_peered":"2024-09-21T19:01:37.098264+0000","last_clean":"2024-09-21T19:01:37.098264+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:01:37.098264+0000","last_undegraded":"2024-09-21T19:01:37.098264+0000","last_fullsized":"2024-09-21T19:01:37.098264+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:26.395243+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":480,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.100988+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:01:36.100988+0000","last_peered":"2024-09-21T19:01:36.100988+0000","last_clean":"2024-09-21T19:01:36.100988+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:01:36.100988+0000","last_undegraded":"2024-09-21T19:01:36.100988+0000","last_fullsized":"2024-09-21T19:01:36.100988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.071423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":350,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099364+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:01:36.099364+0000","last_peered":"2024-09-21T19:01:36.099364+0000","last_clean":"2024-09-21T19:01:36.099364+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:01:36.099364+0000","last_undegraded":"2024-09-21T19:01:36.099364+0000","last_fullsized":"2024-09-21T19:01:36.099364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:37.791557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"224'13926","reported_seq":14111,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104793+0000","last_change":"2024-09-21T19:01:35.117976+0000","last_active":"2024-09-21T19:01:36.104793+0000","last_peered":"2024-09-21T19:01:36.104793+0000","last_clean":"2024-09-21T19:01:36.104793+0000","last_became_active":"2024-09-21T19:01:35.092761+0000","last_became_peered":"2024-09-21T19:01:35.092761+0000","last_unstale":"2024-09-21T19:01:36.104793+0000","last_undegraded":"2024-09-21T19:01:36.104793+0000","last_fullsized":"2024-09-21T19:01:36.104793+0000","mapping_epoch":172,"log_start":"206'8800","ondisk_log_start":"206'8800","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11887","last_scrub_stamp":"2024-09-21T19:01:08.501670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:08.501670+0000","objects_scrubbed":4884,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":4095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6990,"num_write_kb":267360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":244,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104286+0000","last_change":"2024-09-21T18:55:51.560577+0000","last_active":"2024-09-21T19:01:36.104286+0000","last_peered":"2024-09-21T19:01:36.104286+0000","last_clean":"2024-09-21T19:01:36.104286+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:01:36.104286+0000","last_undegraded":"2024-09-21T19:01:36.104286+0000","last_fullsized":"2024-09-21T19:01:36.104286+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:54:52.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:54:52.386065+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:38.261047+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":370,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103816+0000","last_change":"2024-09-21T18:59:15.560811+0000","last_active":"2024-09-21T19:01:36.103816+0000","last_peered":"2024-09-21T19:01:36.103816+0000","last_clean":"2024-09-21T19:01:36.103816+0000","last_became_active":"2024-09-21T18:53:07.399731+0000","last_became_peered":"2024-09-21T18:53:07.399731+0000","last_unstale":"2024-09-21T19:01:36.103816+0000","last_undegraded":"2024-09-21T19:01:36.103816+0000","last_fullsized":"2024-09-21T19:01:36.103816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:15.560713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:15.560713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.858874+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099193+0000","last_change":"2024-09-21T18:59:43.841286+0000","last_active":"2024-09-21T19:01:36.099193+0000","last_peered":"2024-09-21T19:01:36.099193+0000","last_clean":"2024-09-21T19:01:36.099193+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:01:36.099193+0000","last_undegraded":"2024-09-21T19:01:36.099193+0000","last_fullsized":"2024-09-21T19:01:36.099193+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:43.841222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:43.841222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:50.957920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":242,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.110521+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:01:36.110521+0000","last_peered":"2024-09-21T19:01:36.110521+0000","last_clean":"2024-09-21T19:01:36.110521+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:01:36.110521+0000","last_undegraded":"2024-09-21T19:01:36.110521+0000","last_fullsized":"2024-09-21T19:01:36.110521+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.293410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"223'13686","reported_seq":13962,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111465+0000","last_change":"2024-09-21T19:01:36.016598+0000","last_active":"2024-09-21T19:01:36.111465+0000","last_peered":"2024-09-21T19:01:36.111465+0000","last_clean":"2024-09-21T19:01:36.111465+0000","last_became_active":"2024-09-21T19:01:35.108657+0000","last_became_peered":"2024-09-21T19:01:35.108657+0000","last_unstale":"2024-09-21T19:01:36.111465+0000","last_undegraded":"2024-09-21T19:01:36.111465+0000","last_fullsized":"2024-09-21T19:01:36.111465+0000","mapping_epoch":172,"log_start":"204'8100","ondisk_log_start":"204'8100","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'13683","last_scrub_stamp":"2024-09-21T19:01:36.016523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:36.016523+0000","objects_scrubbed":1358,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:58.549424+0000","scrub_duration":461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":388,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102477+0000","last_change":"2024-09-21T19:00:23.328177+0000","last_active":"2024-09-21T19:01:36.102477+0000","last_peered":"2024-09-21T19:01:36.102477+0000","last_clean":"2024-09-21T19:01:36.102477+0000","last_became_active":"2024-09-21T18:53:09.717651+0000","last_became_peered":"2024-09-21T18:53:09.717651+0000","last_unstale":"2024-09-21T19:01:36.102477+0000","last_undegraded":"2024-09-21T19:01:36.102477+0000","last_fullsized":"2024-09-21T19:01:36.102477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:51.465067+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":369,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.096147+0000","last_change":"2024-09-21T19:01:10.619281+0000","last_active":"2024-09-21T19:01:36.096147+0000","last_peered":"2024-09-21T19:01:36.096147+0000","last_clean":"2024-09-21T19:01:36.096147+0000","last_became_active":"2024-09-21T18:53:07.406112+0000","last_became_peered":"2024-09-21T18:53:07.406112+0000","last_unstale":"2024-09-21T19:01:36.096147+0000","last_undegraded":"2024-09-21T19:01:36.096147+0000","last_fullsized":"2024-09-21T19:01:36.096147+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.619235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.619235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:12.039735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":376,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099602+0000","last_change":"2024-09-21T19:01:35.242203+0000","last_active":"2024-09-21T19:01:36.099602+0000","last_peered":"2024-09-21T19:01:36.099602+0000","last_clean":"2024-09-21T19:01:36.099602+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:01:36.099602+0000","last_undegraded":"2024-09-21T19:01:36.099602+0000","last_fullsized":"2024-09-21T19:01:36.099602+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:35.242133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:35.242133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:47.195192+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":349,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093915+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:01:36.093915+0000","last_peered":"2024-09-21T19:01:36.093915+0000","last_clean":"2024-09-21T19:01:36.093915+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:01:36.093915+0000","last_undegraded":"2024-09-21T19:01:36.093915+0000","last_fullsized":"2024-09-21T19:01:36.093915+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"224'13832","reported_seq":14095,"reported_epoch":224,"state":"active+clean+scrubbing","last_fresh":"2024-09-21T19:01:36.593301+0000","last_change":"2024-09-21T19:01:36.559398+0000","last_active":"2024-09-21T19:01:36.593301+0000","last_peered":"2024-09-21T19:01:36.593301+0000","last_clean":"2024-09-21T19:01:36.593301+0000","last_became_active":"2024-09-21T19:01:35.090378+0000","last_became_peered":"2024-09-21T19:01:35.090378+0000","last_unstale":"2024-09-21T19:01:36.593301+0000","last_undegraded":"2024-09-21T19:01:36.593301+0000","last_fullsized":"2024-09-21T19:01:36.593301+0000","mapping_epoch":172,"log_start":"204'8200","ondisk_log_start":"204'8200","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'11443","last_scrub_stamp":"2024-09-21T19:01:03.618148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.618148+0000","objects_scrubbed":399,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":265824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":400,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.096933+0000","last_change":"2024-09-21T18:59:17.862166+0000","last_active":"2024-09-21T19:01:36.096933+0000","last_peered":"2024-09-21T19:01:36.096933+0000","last_clean":"2024-09-21T19:01:36.096933+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:01:36.096933+0000","last_undegraded":"2024-09-21T19:01:36.096933+0000","last_fullsized":"2024-09-21T19:01:36.096933+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T18:59:17.862105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.862105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.784245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":369,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.094694+0000","last_change":"2024-09-21T19:00:22.616468+0000","last_active":"2024-09-21T19:01:36.094694+0000","last_peered":"2024-09-21T19:01:36.094694+0000","last_clean":"2024-09-21T19:01:36.094694+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:01:36.094694+0000","last_undegraded":"2024-09-21T19:01:36.094694+0000","last_fullsized":"2024-09-21T19:01:36.094694+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:22.616386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.616386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:52.426205+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":376,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.098746+0000","last_change":"2024-09-21T18:59:16.539031+0000","last_active":"2024-09-21T19:01:36.098746+0000","last_peered":"2024-09-21T19:01:36.098746+0000","last_clean":"2024-09-21T19:01:36.098746+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:01:36.098746+0000","last_undegraded":"2024-09-21T19:01:36.098746+0000","last_fullsized":"2024-09-21T19:01:36.098746+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:16.538853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:16.538853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.903925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.097932+0000","last_change":"2024-09-21T19:00:34.960755+0000","last_active":"2024-09-21T19:01:37.097932+0000","last_peered":"2024-09-21T19:01:37.097932+0000","last_clean":"2024-09-21T19:01:37.097932+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:01:37.097932+0000","last_undegraded":"2024-09-21T19:01:37.097932+0000","last_fullsized":"2024-09-21T19:01:37.097932+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.960689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.960689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:57.667040+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"224'13654","reported_seq":13868,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103561+0000","last_change":"2024-09-21T19:01:35.164779+0000","last_active":"2024-09-21T19:01:36.103561+0000","last_peered":"2024-09-21T19:01:36.103561+0000","last_clean":"2024-09-21T19:01:36.103561+0000","last_became_active":"2024-09-21T19:01:35.117161+0000","last_became_peered":"2024-09-21T19:01:35.117161+0000","last_unstale":"2024-09-21T19:01:36.103561+0000","last_undegraded":"2024-09-21T19:01:36.103561+0000","last_fullsized":"2024-09-21T19:01:36.103561+0000","mapping_epoch":172,"log_start":"206'8600","ondisk_log_start":"206'8600","created":172,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'8114","last_scrub_stamp":"2024-09-21T19:00:21.263245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.263245+0000","objects_scrubbed":8036,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88801280,"num_objects":1355,"num_object_clones":0,"num_object_copies":4065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6845,"num_write_kb":262400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":391,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104891+0000","last_change":"2024-09-21T18:57:53.780960+0000","last_active":"2024-09-21T19:01:36.104891+0000","last_peered":"2024-09-21T19:01:36.104891+0000","last_clean":"2024-09-21T19:01:36.104891+0000","last_became_active":"2024-09-21T18:53:09.717469+0000","last_became_peered":"2024-09-21T18:53:09.717469+0000","last_unstale":"2024-09-21T19:01:36.104891+0000","last_undegraded":"2024-09-21T19:01:36.104891+0000","last_fullsized":"2024-09-21T19:01:36.104891+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T18:57:53.780909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.780909+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:36.863281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":391,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.094136+0000","last_change":"2024-09-21T18:57:58.230772+0000","last_active":"2024-09-21T19:01:36.094136+0000","last_peered":"2024-09-21T19:01:36.094136+0000","last_clean":"2024-09-21T19:01:36.094136+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:01:36.094136+0000","last_undegraded":"2024-09-21T19:01:36.094136+0000","last_fullsized":"2024-09-21T19:01:36.094136+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T18:57:58.230696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:58.230696+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.427939+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":474,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093051+0000","last_change":"2024-09-21T18:59:56.831306+0000","last_active":"2024-09-21T19:01:36.093051+0000","last_peered":"2024-09-21T19:01:36.093051+0000","last_clean":"2024-09-21T19:01:36.093051+0000","last_became_active":"2024-09-21T18:53:11.413552+0000","last_became_peered":"2024-09-21T18:53:11.413552+0000","last_unstale":"2024-09-21T19:01:36.093051+0000","last_undegraded":"2024-09-21T19:01:36.093051+0000","last_fullsized":"2024-09-21T19:01:36.093051+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T18:59:56.831225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.831225+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:24.794694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":380,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.102845+0000","last_change":"2024-09-21T19:01:04.083677+0000","last_active":"2024-09-21T19:01:36.102845+0000","last_peered":"2024-09-21T19:01:36.102845+0000","last_clean":"2024-09-21T19:01:36.102845+0000","last_became_active":"2024-09-21T18:55:51.524702+0000","last_became_peered":"2024-09-21T18:55:51.524702+0000","last_unstale":"2024-09-21T19:01:36.102845+0000","last_undegraded":"2024-09-21T19:01:36.102845+0000","last_fullsized":"2024-09-21T19:01:36.102845+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:04.083562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.083562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:22.889131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":375,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.101356+0000","last_change":"2024-09-21T18:59:14.180429+0000","last_active":"2024-09-21T19:01:36.101356+0000","last_peered":"2024-09-21T19:01:36.101356+0000","last_clean":"2024-09-21T19:01:36.101356+0000","last_became_active":"2024-09-21T18:55:51.513914+0000","last_became_peered":"2024-09-21T18:55:51.513914+0000","last_unstale":"2024-09-21T19:01:36.101356+0000","last_undegraded":"2024-09-21T19:01:36.101356+0000","last_fullsized":"2024-09-21T19:01:36.101356+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:46.387472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":378,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.101010+0000","last_change":"2024-09-21T19:01:06.925416+0000","last_active":"2024-09-21T19:01:37.101010+0000","last_peered":"2024-09-21T19:01:37.101010+0000","last_clean":"2024-09-21T19:01:37.101010+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:01:37.101010+0000","last_undegraded":"2024-09-21T19:01:37.101010+0000","last_fullsized":"2024-09-21T19:01:37.101010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T19:01:06.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.925267+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.186913+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":370,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103674+0000","last_change":"2024-09-21T19:00:29.955243+0000","last_active":"2024-09-21T19:01:36.103674+0000","last_peered":"2024-09-21T19:01:36.103674+0000","last_clean":"2024-09-21T19:01:36.103674+0000","last_became_active":"2024-09-21T18:53:11.407687+0000","last_became_peered":"2024-09-21T18:53:11.407687+0000","last_unstale":"2024-09-21T19:01:36.103674+0000","last_undegraded":"2024-09-21T19:01:36.103674+0000","last_fullsized":"2024-09-21T19:01:36.103674+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.955184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.955184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:40.019605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":341,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.094933+0000","last_change":"2024-09-21T18:58:03.140037+0000","last_active":"2024-09-21T19:01:36.094933+0000","last_peered":"2024-09-21T19:01:36.094933+0000","last_clean":"2024-09-21T19:01:36.094933+0000","last_became_active":"2024-09-21T18:53:13.657054+0000","last_became_peered":"2024-09-21T18:53:13.657054+0000","last_unstale":"2024-09-21T19:01:36.094933+0000","last_undegraded":"2024-09-21T19:01:36.094933+0000","last_fullsized":"2024-09-21T19:01:36.094933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:03.139963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:58:03.139963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.437166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":252,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111224+0000","last_change":"2024-09-21T19:00:34.345923+0000","last_active":"2024-09-21T19:01:36.111224+0000","last_peered":"2024-09-21T19:01:36.111224+0000","last_clean":"2024-09-21T19:01:36.111224+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:01:36.111224+0000","last_undegraded":"2024-09-21T19:01:36.111224+0000","last_fullsized":"2024-09-21T19:01:36.111224+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:00:34.345792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.345792+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:58.340045+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":361,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093332+0000","last_change":"2024-09-21T19:00:29.586470+0000","last_active":"2024-09-21T19:01:36.093332+0000","last_peered":"2024-09-21T19:01:36.093332+0000","last_clean":"2024-09-21T19:01:36.093332+0000","last_became_active":"2024-09-21T18:53:07.406688+0000","last_became_peered":"2024-09-21T18:53:07.406688+0000","last_unstale":"2024-09-21T19:01:36.093332+0000","last_undegraded":"2024-09-21T19:01:36.093332+0000","last_fullsized":"2024-09-21T19:01:36.093332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:29.586411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:29.586411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.139362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":357,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.100399+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:01:37.100399+0000","last_peered":"2024-09-21T19:01:37.100399+0000","last_clean":"2024-09-21T19:01:37.100399+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:01:37.100399+0000","last_undegraded":"2024-09-21T19:01:37.100399+0000","last_fullsized":"2024-09-21T19:01:37.100399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":358,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.100262+0000","last_change":"2024-09-21T19:01:02.984598+0000","last_active":"2024-09-21T19:01:36.100262+0000","last_peered":"2024-09-21T19:01:36.100262+0000","last_clean":"2024-09-21T19:01:36.100262+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:01:36.100262+0000","last_undegraded":"2024-09-21T19:01:36.100262+0000","last_fullsized":"2024-09-21T19:01:36.100262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:02.984535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:02.984535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:09.393677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"223'14016","reported_seq":14213,"reported_epoch":224,"state":"unknown","last_fresh":"2024-09-21T19:01:36.102388+0000","last_change":"2024-09-21T19:01:36.598316+0000","last_active":"2024-09-21T19:01:36.102388+0000","last_peered":"2024-09-21T19:01:36.086436+0000","last_clean":"2024-09-21T19:01:36.086436+0000","last_became_active":"2024-09-21T19:01:35.129866+0000","last_became_peered":"2024-09-21T19:01:35.129866+0000","last_unstale":"2024-09-21T19:01:36.102388+0000","last_undegraded":"2024-09-21T19:01:36.102388+0000","last_fullsized":"2024-09-21T19:01:36.102388+0000","mapping_epoch":224,"log_start":"205'8400","ondisk_log_start":"205'8400","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"198'6787","last_scrub_stamp":"2024-09-21T19:00:00.245707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.245707+0000","objects_scrubbed":6706,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91029504,"num_objects":1389,"num_object_clones":0,"num_object_copies":4167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7035,"num_write_kb":269568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":374,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104447+0000","last_change":"2024-09-21T18:58:58.524782+0000","last_active":"2024-09-21T19:01:36.104447+0000","last_peered":"2024-09-21T19:01:36.104447+0000","last_clean":"2024-09-21T19:01:36.104447+0000","last_became_active":"2024-09-21T18:53:09.719707+0000","last_became_peered":"2024-09-21T18:53:09.719707+0000","last_unstale":"2024-09-21T19:01:36.104447+0000","last_undegraded":"2024-09-21T19:01:36.104447+0000","last_fullsized":"2024-09-21T19:01:36.104447+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:58:58.524708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.524708+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.871526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":242,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.108453+0000","last_change":"2024-09-21T18:57:51.985119+0000","last_active":"2024-09-21T19:01:36.108453+0000","last_peered":"2024-09-21T19:01:36.108453+0000","last_clean":"2024-09-21T19:01:36.108453+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:01:36.108453+0000","last_undegraded":"2024-09-21T19:01:36.108453+0000","last_fullsized":"2024-09-21T19:01:36.108453+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:51.985062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:51.985062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.288280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":475,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.098251+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:01:37.098251+0000","last_peered":"2024-09-21T19:01:37.098251+0000","last_clean":"2024-09-21T19:01:37.098251+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:01:37.098251+0000","last_undegraded":"2024-09-21T19:01:37.098251+0000","last_fullsized":"2024-09-21T19:01:37.098251+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":350,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099683+0000","last_change":"2024-09-21T19:00:32.753965+0000","last_active":"2024-09-21T19:01:36.099683+0000","last_peered":"2024-09-21T19:01:36.099683+0000","last_clean":"2024-09-21T19:01:36.099683+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:01:36.099683+0000","last_undegraded":"2024-09-21T19:01:36.099683+0000","last_fullsized":"2024-09-21T19:01:36.099683+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:32.753902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:32.753902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.288209+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.18","version":"223'13378","reported_seq":13552,"reported_epoch":225,"state":"remapped+peering","last_fresh":"2024-09-21T19:01:37.093513+0000","last_change":"2024-09-21T19:01:37.093513+0000","last_active":"2024-09-21T19:01:37.093513+0000","last_peered":"2024-09-21T19:01:36.065371+0000","last_clean":"2024-09-21T19:01:36.065371+0000","last_became_active":"2024-09-21T19:01:35.138910+0000","last_became_peered":"2024-09-21T19:01:35.138910+0000","last_unstale":"2024-09-21T19:01:37.093513+0000","last_undegraded":"2024-09-21T19:01:37.093513+0000","last_fullsized":"2024-09-21T19:01:37.093513+0000","mapping_epoch":225,"log_start":"206'8400","ondisk_log_start":"206'8400","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'11951","last_scrub_stamp":"2024-09-21T19:01:16.186043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:58:43.073980+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.186043+0000","objects_scrubbed":4160,"log_size":4978,"log_dups_size":0,"ondisk_log_size":4978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86507520,"num_objects":1320,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":257120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":368,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.098681+0000","last_change":"2024-09-21T19:00:28.893623+0000","last_active":"2024-09-21T19:01:37.098681+0000","last_peered":"2024-09-21T19:01:37.098681+0000","last_clean":"2024-09-21T19:01:37.098681+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:01:37.098681+0000","last_undegraded":"2024-09-21T19:01:37.098681+0000","last_fullsized":"2024-09-21T19:01:37.098681+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:28.893403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.893403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":361,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093773+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:01:36.093773+0000","last_peered":"2024-09-21T19:01:36.093773+0000","last_clean":"2024-09-21T19:01:36.093773+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:01:36.093773+0000","last_undegraded":"2024-09-21T19:01:36.093773+0000","last_fullsized":"2024-09-21T19:01:36.093773+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.507018+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"114'10","reported_seq":478,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104263+0000","last_change":"2024-09-21T18:57:54.770497+0000","last_active":"2024-09-21T19:01:36.104263+0000","last_peered":"2024-09-21T19:01:36.104263+0000","last_clean":"2024-09-21T19:01:36.104263+0000","last_became_active":"2024-09-21T18:55:51.524088+0000","last_became_peered":"2024-09-21T18:55:51.524088+0000","last_unstale":"2024-09-21T19:01:36.104263+0000","last_undegraded":"2024-09-21T19:01:36.104263+0000","last_fullsized":"2024-09-21T19:01:36.104263+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T18:57:54.770431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:57:54.770431+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:41.883299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":250,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111662+0000","last_change":"2024-09-21T19:00:58.413991+0000","last_active":"2024-09-21T19:01:36.111662+0000","last_peered":"2024-09-21T19:01:36.111662+0000","last_clean":"2024-09-21T19:01:36.111662+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:01:36.111662+0000","last_undegraded":"2024-09-21T19:01:36.111662+0000","last_fullsized":"2024-09-21T19:01:36.111662+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:58.413866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:58.413866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:02.192707+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":404,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.098977+0000","last_change":"2024-09-21T19:01:04.724924+0000","last_active":"2024-09-21T19:01:36.098977+0000","last_peered":"2024-09-21T19:01:36.098977+0000","last_clean":"2024-09-21T19:01:36.098977+0000","last_became_active":"2024-09-21T18:54:36.757033+0000","last_became_peered":"2024-09-21T18:54:36.757033+0000","last_unstale":"2024-09-21T19:01:36.098977+0000","last_undegraded":"2024-09-21T19:01:36.098977+0000","last_fullsized":"2024-09-21T19:01:36.098977+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:01:04.724868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.724868+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:24.749814+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.099411+0000","last_change":"2024-09-21T19:00:34.746168+0000","last_active":"2024-09-21T19:01:36.099411+0000","last_peered":"2024-09-21T19:01:36.099411+0000","last_clean":"2024-09-21T19:01:36.099411+0000","last_became_active":"2024-09-21T18:53:07.407286+0000","last_became_peered":"2024-09-21T18:53:07.407286+0000","last_unstale":"2024-09-21T19:01:36.099411+0000","last_undegraded":"2024-09-21T19:01:36.099411+0000","last_fullsized":"2024-09-21T19:01:36.099411+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.746126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.746126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.927082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.105432+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:01:36.105432+0000","last_peered":"2024-09-21T19:01:36.105432+0000","last_clean":"2024-09-21T19:01:36.105432+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:01:36.105432+0000","last_undegraded":"2024-09-21T19:01:36.105432+0000","last_fullsized":"2024-09-21T19:01:36.105432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:18.551674+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":359,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093235+0000","last_change":"2024-09-21T18:57:53.746960+0000","last_active":"2024-09-21T19:01:37.093235+0000","last_peered":"2024-09-21T19:01:37.093235+0000","last_clean":"2024-09-21T19:01:37.093235+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:01:37.093235+0000","last_undegraded":"2024-09-21T19:01:37.093235+0000","last_fullsized":"2024-09-21T19:01:37.093235+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.746839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.746839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:42.894710+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":390,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.105691+0000","last_change":"2024-09-21T19:01:11.541442+0000","last_active":"2024-09-21T19:01:36.105691+0000","last_peered":"2024-09-21T19:01:36.105691+0000","last_clean":"2024-09-21T19:01:36.105691+0000","last_became_active":"2024-09-21T18:53:09.405818+0000","last_became_peered":"2024-09-21T18:53:09.405818+0000","last_unstale":"2024-09-21T19:01:36.105691+0000","last_undegraded":"2024-09-21T19:01:36.105691+0000","last_fullsized":"2024-09-21T19:01:36.105691+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.015757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":369,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.096009+0000","last_change":"2024-09-21T18:58:34.182624+0000","last_active":"2024-09-21T19:01:36.096009+0000","last_peered":"2024-09-21T19:01:36.096009+0000","last_clean":"2024-09-21T19:01:36.096009+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:01:36.096009+0000","last_undegraded":"2024-09-21T19:01:36.096009+0000","last_fullsized":"2024-09-21T19:01:36.096009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:34.182578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:58:34.182578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:19.748561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":371,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.098019+0000","last_change":"2024-09-21T18:58:58.952448+0000","last_active":"2024-09-21T19:01:37.098019+0000","last_peered":"2024-09-21T19:01:37.098019+0000","last_clean":"2024-09-21T19:01:37.098019+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:01:37.098019+0000","last_undegraded":"2024-09-21T19:01:37.098019+0000","last_fullsized":"2024-09-21T19:01:37.098019+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:58:58.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:58:58.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:43.895529+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":250,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104183+0000","last_change":"2024-09-21T18:59:42.223794+0000","last_active":"2024-09-21T19:01:36.104183+0000","last_peered":"2024-09-21T19:01:36.104183+0000","last_clean":"2024-09-21T19:01:36.104183+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:01:36.104183+0000","last_undegraded":"2024-09-21T19:01:36.104183+0000","last_fullsized":"2024-09-21T19:01:36.104183+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:42.223731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:42.223731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:17.606032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":380,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093366+0000","last_change":"2024-09-21T19:00:00.939564+0000","last_active":"2024-09-21T19:01:37.093366+0000","last_peered":"2024-09-21T19:01:37.093366+0000","last_clean":"2024-09-21T19:01:37.093366+0000","last_became_active":"2024-09-21T18:53:09.717536+0000","last_became_peered":"2024-09-21T18:53:09.717536+0000","last_unstale":"2024-09-21T19:01:37.093366+0000","last_undegraded":"2024-09-21T19:01:37.093366+0000","last_fullsized":"2024-09-21T19:01:37.093366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103758+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:01:36.103758+0000","last_peered":"2024-09-21T19:01:36.103758+0000","last_clean":"2024-09-21T19:01:36.103758+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:01:36.103758+0000","last_undegraded":"2024-09-21T19:01:36.103758+0000","last_fullsized":"2024-09-21T19:01:36.103758+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:47.872793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":361,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093400+0000","last_change":"2024-09-21T19:00:16.651104+0000","last_active":"2024-09-21T19:01:36.093400+0000","last_peered":"2024-09-21T19:01:36.093400+0000","last_clean":"2024-09-21T19:01:36.093400+0000","last_became_active":"2024-09-21T18:53:11.412679+0000","last_became_peered":"2024-09-21T18:53:11.412679+0000","last_unstale":"2024-09-21T19:01:36.093400+0000","last_undegraded":"2024-09-21T19:01:36.093400+0000","last_fullsized":"2024-09-21T19:01:36.093400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:16.650906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:16.650906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:29.919398+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":367,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.098173+0000","last_change":"2024-09-21T18:59:58.945778+0000","last_active":"2024-09-21T19:01:37.098173+0000","last_peered":"2024-09-21T19:01:37.098173+0000","last_clean":"2024-09-21T19:01:37.098173+0000","last_became_active":"2024-09-21T18:55:51.529498+0000","last_became_peered":"2024-09-21T18:55:51.529498+0000","last_unstale":"2024-09-21T19:01:37.098173+0000","last_undegraded":"2024-09-21T19:01:37.098173+0000","last_fullsized":"2024-09-21T19:01:37.098173+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:58.945631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:58.945631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":364,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093563+0000","last_change":"2024-09-21T18:54:36.780780+0000","last_active":"2024-09-21T19:01:37.093563+0000","last_peered":"2024-09-21T19:01:37.093563+0000","last_clean":"2024-09-21T19:01:37.093563+0000","last_became_active":"2024-09-21T18:54:36.755664+0000","last_became_peered":"2024-09-21T18:54:36.755664+0000","last_unstale":"2024-09-21T19:01:37.093563+0000","last_undegraded":"2024-09-21T19:01:37.093563+0000","last_fullsized":"2024-09-21T19:01:37.093563+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:53:08.386251+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":242,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103946+0000","last_change":"2024-09-21T18:57:53.023188+0000","last_active":"2024-09-21T19:01:36.103946+0000","last_peered":"2024-09-21T19:01:36.103946+0000","last_clean":"2024-09-21T19:01:36.103946+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:01:36.103946+0000","last_undegraded":"2024-09-21T19:01:36.103946+0000","last_fullsized":"2024-09-21T19:01:36.103946+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:53.023116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:53.023116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.291673+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":353,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103842+0000","last_change":"2024-09-21T18:59:17.117569+0000","last_active":"2024-09-21T19:01:36.103842+0000","last_peered":"2024-09-21T19:01:36.103842+0000","last_clean":"2024-09-21T19:01:36.103842+0000","last_became_active":"2024-09-21T18:53:11.412904+0000","last_became_peered":"2024-09-21T18:53:11.412904+0000","last_unstale":"2024-09-21T19:01:36.103842+0000","last_undegraded":"2024-09-21T19:01:36.103842+0000","last_fullsized":"2024-09-21T19:01:36.103842+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:44.359379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":349,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.093073+0000","last_change":"2024-09-21T18:59:59.934497+0000","last_active":"2024-09-21T19:01:36.093073+0000","last_peered":"2024-09-21T19:01:36.093073+0000","last_clean":"2024-09-21T19:01:36.093073+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:01:36.093073+0000","last_undegraded":"2024-09-21T19:01:36.093073+0000","last_fullsized":"2024-09-21T19:01:36.093073+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:59.934416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:59.934416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:22.757303+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:01:34.061366+0000","last_change":"2024-09-21T19:01:34.061366+0000","last_active":"2024-09-21T19:01:34.061366+0000","last_peered":"2024-09-21T19:01:34.061366+0000","last_clean":"2024-09-21T19:01:34.061366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:01:34.061366+0000","last_undegraded":"2024-09-21T19:01:34.061366+0000","last_fullsized":"2024-09-21T19:01:34.061366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:34.061366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:01:34.061366+0000","last_clean_scrub_stamp":"2024-09-21T19:01:34.061366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":350,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.103465+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:01:36.103465+0000","last_peered":"2024-09-21T19:01:36.103465+0000","last_clean":"2024-09-21T19:01:36.103465+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:01:36.103465+0000","last_undegraded":"2024-09-21T19:01:36.103465+0000","last_fullsized":"2024-09-21T19:01:36.103465+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:23.444545+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":387,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-21T19:01:37.093810+0000","last_change":"2024-09-21T19:00:17.839077+0000","last_active":"2024-09-21T19:01:37.093810+0000","last_peered":"2024-09-21T19:01:37.093810+0000","last_clean":"2024-09-21T19:01:37.093810+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:01:37.093810+0000","last_undegraded":"2024-09-21T19:01:37.093810+0000","last_fullsized":"2024-09-21T19:01:37.093810+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:17.838903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:17.838903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:45.006877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":377,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.104586+0000","last_change":"2024-09-21T18:59:18.108977+0000","last_active":"2024-09-21T19:01:36.104586+0000","last_peered":"2024-09-21T19:01:36.104586+0000","last_clean":"2024-09-21T19:01:36.104586+0000","last_became_active":"2024-09-21T18:53:09.410878+0000","last_became_peered":"2024-09-21T18:53:09.410878+0000","last_unstale":"2024-09-21T19:01:36.104586+0000","last_undegraded":"2024-09-21T19:01:36.104586+0000","last_fullsized":"2024-09-21T19:01:36.104586+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T18:59:18.108904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:18.108904+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:01:39.281496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":258,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-21T19:01:36.111591+0000","last_change":"2024-09-21T19:01:11.531925+0000","last_active":"2024-09-21T19:01:36.111591+0000","last_peered":"2024-09-21T19:01:36.111591+0000","last_clean":"2024-09-21T19:01:36.111591+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:01:36.111591+0000","last_undegraded":"2024-09-21T19:01:36.111591+0000","last_fullsized":"2024-09-21T19:01:36.111591+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:02:40.641599+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":2314338328,"num_objects":35314,"num_object_clones":0,"num_object_copies":105942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165936,"num_write_kb":6350881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8529063936,"data_stored":8529051720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100630,"ondisk_log_size":100630,"up":72,"acting":72,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":498,"num_read_kb":493,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266794,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1142020,"kb_used_data":872044,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92603132,"statfs":{"total":95995035648,"available":94825607168,"internally_reserved":0,"allocated":892973056,"data_stored":892571669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":276429258},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528431,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1150080,"kb_used_data":875948,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92595072,"statfs":{"total":95995035648,"available":94817353728,"internally_reserved":0,"allocated":896970752,"data_stored":896576195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790066,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1692860,"kb_used_data":1390948,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92052292,"statfs":{"total":95995035648,"available":94261547008,"internally_reserved":0,"allocated":1424330752,"data_stored":1423937582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019000,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1150140,"kb_used_data":888488,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92595012,"statfs":{"total":95995035648,"available":94817292288,"internally_reserved":0,"allocated":909811712,"data_stored":909414677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182525,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434408,"kb_used_data":1038092,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92310744,"statfs":{"total":95995035648,"available":94526201856,"internally_reserved":0,"allocated":1063006208,"data_stored":1062580655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":405797322},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1321804,"kb_used_data":1042104,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92423348,"statfs":{"total":95995035648,"available":94641508352,"internally_reserved":0,"allocated":1067114496,"data_stored":1066708684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1580568,"kb_used_data":1216196,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92164584,"statfs":{"total":95995035648,"available":94376534016,"internally_reserved":0,"allocated":1245384704,"data_stored":1244966353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1485004,"kb_used_data":1048440,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92260148,"statfs":{"total":95995035648,"available":94474391552,"internally_reserved":0,"allocated":1073602560,"data_stored":1073196833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1072758784,"data_stored":1072758784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1244528640,"data_stored":1244528640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1066012672,"data_stored":1066008600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1061552128,"data_stored":1061552128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":892272640,"data_stored":892272640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1411256320,"data_stored":1411252248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":896139264,"data_stored":896139264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":884543488,"data_stored":884539416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:01:43.917 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T19:01:43.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_1 pg_num 36 2024-09-21T19:01:43.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:43.962 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:43.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:44.064 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:44.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:44.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:01:44.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.232 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:44.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:44.333 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:44.368 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:44 smithi081 ceph-mon[29384]: 8.9 scrub ok 2024-09-21T19:01:44.368 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:44 smithi081 ceph-mon[29384]: from='client.35576 v1:172.21.15.81:0/3032336227' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:44.368 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:44 smithi081 ceph-mon[29384]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T19:01:44.372 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:44 smithi081 ceph-mon[39192]: 8.9 scrub ok 2024-09-21T19:01:44.372 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:44 smithi081 ceph-mon[39192]: from='client.35576 v1:172.21.15.81:0/3032336227' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:44.372 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:44 smithi081 ceph-mon[39192]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T19:01:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.380 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:44.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:44.481 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:44.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:44 smithi104 ceph-mon[32438]: 8.9 scrub ok 2024-09-21T19:01:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:44 smithi104 ceph-mon[32438]: from='client.35576 v1:172.21.15.81:0/3032336227' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:01:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:44 smithi104 ceph-mon[32438]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T19:01:44.514 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:44.517 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:44.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:44.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:44.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:44.800 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:44.902 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:44.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:45.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:45.054 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:45.054 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:45.054 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:45.054 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:45.054 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:45.055 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:45.056 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:45.056 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:45.056 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:45.056 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:45.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:45.120 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:45.121 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:45.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:45.199 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: pgmap v401: 168 pgs: 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 791 B/s rd, 159 MiB/s wr, 4.97k op/s; 19602/118722 objects misplaced (16.511%); 28 MiB/s, 451 objects/s recovering 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T19:01:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:45 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2099137675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:45.301 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: pgmap v401: 168 pgs: 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 791 B/s rd, 159 MiB/s wr, 4.97k op/s; 19602/118722 objects misplaced (16.511%); 28 MiB/s, 451 objects/s recovering 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T19:01:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2099137675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: pgmap v401: 168 pgs: 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 791 B/s rd, 159 MiB/s wr, 4.97k op/s; 19602/118722 objects misplaced (16.511%); 28 MiB/s, 451 objects/s recovering 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T19:01:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:45 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2099137675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.332 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:45.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:45.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:45.465 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:45.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:45.598 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:45.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:45.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:45.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:01:45.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:45.903 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:45.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:46.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:46.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:46.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:46.299 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:46.384 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:46.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:46.431 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:46.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:46.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:46.563 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:46.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:46.688 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:46.688 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:46.697 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:46.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:46.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[29384]: pgmap v404: 168 pgs: 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 172 MiB/s wr, 5.27k op/s; 19602/118722 objects misplaced (16.511%); 30 MiB/s, 486 objects/s recovering 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[29384]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[39192]: pgmap v404: 168 pgs: 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 172 MiB/s wr, 5.27k op/s; 19602/118722 objects misplaced (16.511%); 30 MiB/s, 486 objects/s recovering 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[39192]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T19:01:46.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:46 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T19:01:46.826 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:46.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:46.953 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:46.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:47.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:46 smithi104 ceph-mon[32438]: pgmap v404: 168 pgs: 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 172 MiB/s wr, 5.27k op/s; 19602/118722 objects misplaced (16.511%); 30 MiB/s, 486 objects/s recovering 2024-09-21T19:01:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:46 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T19:01:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:46 smithi104 ceph-mon[32438]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T19:01:47.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:46 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T19:01:47.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:47.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:47.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:47.221 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:47.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:47.350 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:47.374 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:01:47.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:47.477 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:47.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:01:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:01:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:01:47.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:47.602 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:47.637 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:47.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:47.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:47.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:47.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:47.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:47 smithi104 ceph-mon[32438]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T19:01:47.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/138996346' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T19:01:47.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:47.986 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:47.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:47 smithi081 ceph-mon[29384]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T19:01:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/138996346' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T19:01:48.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:47 smithi081 ceph-mon[39192]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T19:01:48.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/138996346' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T19:01:48.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:48.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:48.158 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:48.158 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:48.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:01:48.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:48.226 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 8 pg_num to 36 2024-09-21T19:01:48.232 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:48.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:01:48.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:48.389 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:48.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:48.513 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:48.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:48.639 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:48.741 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.769 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:48.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:48.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:48.901 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:48.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:49.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: pgmap v407: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11186/106890 objects misplaced (10.465%); 61 MiB/s, 981 objects/s recovering 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/138996346' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]': finished 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: 8.18 scrub starts 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: 8.18 scrub ok 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: 8.15 scrub starts 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: 8.8 scrub starts 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: 8.15 scrub ok 2024-09-21T19:01:49.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:49.029 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: pgmap v407: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11186/106890 objects misplaced (10.465%); 61 MiB/s, 981 objects/s recovering 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/138996346' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]': finished 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: 8.18 scrub starts 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: 8.18 scrub ok 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: 8.15 scrub starts 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: 8.8 scrub starts 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: 8.15 scrub ok 2024-09-21T19:01:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: pgmap v407: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11186/106890 objects misplaced (10.465%); 61 MiB/s, 981 objects/s recovering 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/138996346' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "36"}]': finished 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: 8.18 scrub starts 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: 8.18 scrub ok 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: 8.15 scrub starts 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: 8.8 scrub starts 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: 8.15 scrub ok 2024-09-21T19:01:49.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:49.131 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:49.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:49.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:49.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:49.291 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:49.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:49.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:01:49.398 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:49.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:49.520 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:49.577 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:01:49.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:49.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:49.675 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:49.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:49.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:49.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:49.932 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:49.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:49.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 8.8 scrub ok 2024-09-21T19:01:49.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T19:01:49.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 6.1a scrub starts 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 6.1a scrub ok 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 8.17 scrub starts 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 8.19 scrub starts 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: 8.17 scrub ok 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/877973515' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/877973515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:01:49.965 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:49 smithi104 ceph-mon[32438]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T19:01:50.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 8.8 scrub ok 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 6.1a scrub starts 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 6.1a scrub ok 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 8.17 scrub starts 2024-09-21T19:01:50.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 8.19 scrub starts 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: 8.17 scrub ok 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/877973515' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/877973515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[29384]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 8.8 scrub ok 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T19:01:50.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 6.1a scrub starts 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 6.1a scrub ok 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 8.17 scrub starts 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 8.19 scrub starts 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: 8.17 scrub ok 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/877973515' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/877973515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:01:50.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:49 smithi081 ceph-mon[39192]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T19:01:50.067 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:50.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:50.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:50.280 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:01:50.280 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:01:50.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:50.338 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:50.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:50.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:50.581 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:50.610 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:50.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:50.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:50.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:50.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:50.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:51.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:50 smithi104 ceph-mon[32438]: 8.19 scrub ok 2024-09-21T19:01:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:50 smithi104 ceph-mon[32438]: pgmap v410: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11186/106890 objects misplaced (10.465%); 61 MiB/s, 981 objects/s recovering 2024-09-21T19:01:51.013 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:51.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:01:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:50 smithi081 ceph-mon[29384]: 8.19 scrub ok 2024-09-21T19:01:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:50 smithi081 ceph-mon[29384]: pgmap v410: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11186/106890 objects misplaced (10.465%); 61 MiB/s, 981 objects/s recovering 2024-09-21T19:01:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:50 smithi081 ceph-mon[39192]: 8.19 scrub ok 2024-09-21T19:01:51.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:50 smithi081 ceph-mon[39192]: pgmap v410: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11186/106890 objects misplaced (10.465%); 61 MiB/s, 981 objects/s recovering 2024-09-21T19:01:51.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:51.158 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:51.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:51.293 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:51.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:51.428 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:51.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:51.459 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:51.530 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:51.564 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:51.666 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:51.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:51.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:51.915 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:51.915 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:51.915 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:01:51.915 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:51.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:51.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:51.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:01:51.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:51.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:51.965 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:51.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:51 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:51 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:51 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:01:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:01:52.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:52.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:52.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:52.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:52.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:52.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:52.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:01:52.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.522 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:52.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:52.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.670 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:52.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:52.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:01:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:01:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[29384]: pgmap v412: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 11186/106890 objects misplaced (10.465%); 32 MiB/s, 505 objects/s recovering 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[29384]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1575953686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[39192]: pgmap v412: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 11186/106890 objects misplaced (10.465%); 32 MiB/s, 505 objects/s recovering 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[39192]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T19:01:52.777 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1575953686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:52.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:52.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:52.933 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:52 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:01:52.934 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:52 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:01:52.934 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:52 smithi104 ceph-mon[32438]: pgmap v412: 168 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 159 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 11186/106890 objects misplaced (10.465%); 32 MiB/s, 505 objects/s recovering 2024-09-21T19:01:52.934 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:52 smithi104 ceph-mon[32438]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T19:01:52.934 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1575953686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:01:52.946 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:52.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:52.986 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:01:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:01:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:01:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:01:52.987 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:01:53.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:53.073 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:53.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:53.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.217 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:53.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:53.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.356 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:53.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:53.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:53.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:53.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.639 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:53.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:53.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:53.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:53.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:53 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:53.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:53 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:53.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:53.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:53 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:53.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:01:53.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:53.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:53.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:53.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:54.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:54.050 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:54.111 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:01:54.146 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:54.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:54.175 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:54.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:54.299 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:54.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:01:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:54.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:01:54.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:54.546 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:54.622 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:01:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54921 7.5 7:a314aa6c:::benchmark_data_smithi081_73391_object54920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.457130+0000", 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 428.16114857600002, 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.074546457999999996, 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54921 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457130+0000", 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6599999999999996e-07 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457166+0000", 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5052000000000003e-05 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457172+0000", 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8170000000000001e-06 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.457178+0000", 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9259999999999997e-06 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459247+0000", 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020685429999999999 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459371+0000", 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012465100000000001 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.459488+0000", 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011696000000000001 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461191+0000", 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001702924 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.502032+0000", 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.040840333999999999 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531576+0000", 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.029544002999999999 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531634+0000", 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8527999999999999e-05 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.531676+0000", 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2054000000000001e-05 2024-09-21T19:01:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:54929 7.5 7:af33521c:::benchmark_data_smithi081_73391_object54928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:54:46.461656+0000", 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 428.15662209099997, 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076295636, 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 54929 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461656+0000", 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461656+0000", 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461657+0000", 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.094e-06 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461720+0000", 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2421000000000004e-05 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461729+0000", 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9320000000000002e-06 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461735+0000", 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4479999999999999e-06 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461793+0000", 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7732000000000002e-05 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.461920+0000", 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000126957 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.462069+0000", 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014942500000000001 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.463573+0000", 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015035960000000001 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.502383+0000", 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.038809896000000003 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537843+0000", 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.035460301 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537910+0000", 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7265000000000001e-05 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:54:46.537952+0000", 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1569000000000001e-05 2024-09-21T19:01:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:99380 7.5 7:a5c9fcf3:::benchmark_data_smithi081_73391_object99379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:17.710751+0000", 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 396.90752765399998, 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079490342000000005, 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99380 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710751+0000", 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710751+0000", 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710752+0000", 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7129999999999999e-06 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710804+0000", 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1554999999999997e-05 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710813+0000", 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3589999999999997e-06 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710820+0000", 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7950000000000002e-06 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710870+0000", 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0315000000000002e-05 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.710979+0000", 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010852 2024-09-21T19:01:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.711073+0000", 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4493000000000005e-05 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.712997+0000", 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019239000000000001 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.713798+0000", 2024-09-21T19:01:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080073700000000004 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790157+0000", 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076358910000000002 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790215+0000", 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8378999999999998e-05 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:17.790241+0000", 2024-09-21T19:01:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5666000000000001e-05 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101889 7.f e105/82 7:f2f30f0c:::benchmark_data_smithi081_73391_object101888:head v 105'6311, mlcod=105'6308)", 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.908994+0000", 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 394.70928420400003, 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087156066000000004, 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908995+0000", 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9800000000000003e-07 2024-09-21T19:01:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909027+0000", 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2353e-05 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909033+0000", 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1859999999999997e-06 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909038+0000", 2024-09-21T19:01:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7249999999999997e-06 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909131+0000", 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2363999999999999e-05 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909149+0000", 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8450000000000001e-05 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996090+0000", 2024-09-21T19:01:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086940559000000001 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996150+0000", 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0730999999999998e-05 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101894 7.8 e105/79 7:1138a4d6:::benchmark_data_smithi081_73391_object101893:head v 105'6424, mlcod=105'6422)", 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910208+0000", 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 394.70807048400002, 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086428049000000007, 2024-09-21T19:01:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910209+0000", 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.24e-06 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910259+0000", 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9756000000000003e-05 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910269+0000", 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0035e-05 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910276+0000", 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8040000000000004e-06 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914457+0000", 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041814679999999998 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914478+0000", 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0528999999999999e-05 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.649 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996573+0000", 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082095020000000005 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996636+0000", 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3196999999999998e-05 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101892 7.f e105/82 7:f5e7532a:::benchmark_data_smithi081_73391_object101891:head v 105'6312, mlcod=105'6308)", 2024-09-21T19:01:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910906+0000", 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 394.70737262300003, 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085280135000000007, 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910907+0000", 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1990000000000001e-06 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910962+0000", 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5427000000000003e-05 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910972+0000", 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2520000000000005e-06 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910978+0000", 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.117e-06 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914620+0000", 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036426869999999999 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914639+0000", 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8848999999999999e-05 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996155+0000", 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.081515729999999995 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996186+0000", 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0874000000000003e-05 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101891 7.c e105/79 7:3b3bc22b:::benchmark_data_smithi081_73391_object101890:head v 105'6296, mlcod=105'6294)", 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.912575+0000", 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 394.70570303400001, 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084158549999999999, 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912575+0000", 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912575+0000", 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912577+0000", 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.316e-06 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912603+0000", 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5993999999999999e-05 2024-09-21T19:01:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912614+0000", 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1082e-05 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.912621+0000", 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.063e-06 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.913518+0000", 2024-09-21T19:01:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089733900000000004 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.913546+0000", 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7746000000000002e-05 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916419+0000", 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028730499999999998 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996734+0000", 2024-09-21T19:01:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080314960000000005 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101901 7.5 7:a8e9101c:::benchmark_data_smithi081_73391_object101900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916321+0000", 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 394.701957478, 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087318653999999996, 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101901 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916329+0000", 2024-09-21T19:01:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3990000000000004e-06 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916364+0000", 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4808999999999998e-05 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916374+0000", 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0298e-05 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916382+0000", 2024-09-21T19:01:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996259+0000", 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079877760000000006 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996412+0000", 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015201099999999999 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996586+0000", 2024-09-21T19:01:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017437 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003419+0000", 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068332979999999998 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003455+0000", 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5438000000000002e-05 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003478+0000", 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3048e-05 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003584+0000", 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106331 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003640+0000", 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5583e-05 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101902 7.5 7:a3b6d405:::benchmark_data_smithi081_73391_object101901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916400+0000", 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 394.70187846900001, 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087823038000000006, 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101902 2024-09-21T19:01:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916401+0000", 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5899999999999996e-07 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916430+0000", 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9076000000000001e-05 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996083+0000", 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079653529000000001 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996103+0000", 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0015999999999999e-05 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998560+0000", 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002457126 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998676+0000", 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115328 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998809+0000", 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133767 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003647+0000", 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048375040000000003 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003682+0000", 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4922999999999998e-05 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004144+0000", 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046231000000000001 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004190+0000", 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5958999999999998e-05 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004223+0000", 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2840999999999999e-05 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114519 7.6 7:6b5e3486:::benchmark_data_smithi081_73391_object114518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 383.909172652, 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12712907900000001, 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114519 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709107+0000", 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.66e-06 2024-09-21T19:01:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709157+0000", 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9911999999999999e-05 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709171+0000", 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3947e-05 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709179+0000", 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3529999999999997e-06 2024-09-21T19:01:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716464+0000", 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072858719999999997 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716590+0000", 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554999999999999 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716718+0000", 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012810499999999999 2024-09-21T19:01:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723181+0000", 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006462944 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723217+0000", 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5593000000000003e-05 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836081+0000", 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11286423700000001 2024-09-21T19:01:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836188+0000", 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107315 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836235+0000", 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6591000000000002e-05 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114521 7.6 7:6fe397a7:::benchmark_data_smithi081_73391_object114520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 383.90897810299998, 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12709793799999999, 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114521 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709301+0000", 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6199999999999996e-07 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709354+0000", 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2936999999999998e-05 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709361+0000", 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.097e-06 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709367+0000", 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6550000000000001e-06 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719193+0000", 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098261459999999991 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719453+0000", 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026041699999999998 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.720410+0000", 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095699200000000004 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723227+0000", 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002816888 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723832+0000", 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060487799999999999 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836280+0000", 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11244789099999999 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836362+0000", 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1967999999999999e-05 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836398+0000", 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6207000000000001e-05 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114538 7.6 7:657d97a1:::benchmark_data_smithi081_73391_object114537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723638+0000", 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 383.89464043100003, 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11594671600000001, 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114538 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723640+0000", 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.801e-06 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723678+0000", 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7873000000000001e-05 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723688+0000", 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0071000000000001e-05 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723695+0000", 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3429999999999998e-06 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723868+0000", 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017280900000000001 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724003+0000", 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013505 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724166+0000", 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162764 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729028+0000", 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048623590000000001 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729096+0000", 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7625000000000004e-05 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836438+0000", 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107342302 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839527+0000", 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030888589999999998 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839585+0000", 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7859999999999998e-05 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114539 7.6 7:6de988ff:::benchmark_data_smithi081_73391_object114538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:01:54.689 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723706+0000", 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 383.89457198000002, 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11604534699999999, 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114539 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723707+0000", 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4399999999999999e-07 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723743+0000", 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5395999999999998e-05 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723749+0000", 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1850000000000001e-06 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723754+0000", 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6219999999999997e-06 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726769+0000", 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030147289999999998 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726889+0000", 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012032999999999999 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727041+0000", 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015136699999999999 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729293+0000", 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002252085 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730090+0000", 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00079701999999999996 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839629+0000", 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10953908599999999 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839710+0000", 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0630000000000006e-05 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839752+0000", 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2153000000000001e-05 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121044 7.6 7:62f149b0:::benchmark_data_smithi081_73391_object121043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.421001+0000", 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 378.19727780099998, 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084735675999999996, 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121044 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421002+0000", 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.336e-06 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421062+0000", 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0475e-05 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421072+0000", 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9389999999999992e-06 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421078+0000", 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2870000000000004e-06 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421128+0000", 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9938000000000002e-05 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421252+0000", 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012434499999999999 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421447+0000", 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019541299999999999 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425265+0000", 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038176640000000001 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428690+0000", 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034252710000000001 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505655+0000", 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076964640000000001 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505691+0000", 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5887999999999999e-05 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505736+0000", 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5479999999999998e-05 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121051 7.6 7:619bd2cd:::benchmark_data_smithi081_73391_object121050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.425560+0000", 2024-09-21T19:01:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 378.19271858000002, 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088495110000000002, 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121051 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425562+0000", 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.154e-06 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425607+0000", 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5229999999999999e-05 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425620+0000", 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2346e-05 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425627+0000", 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425678+0000", 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1053000000000001e-05 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425785+0000", 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107099 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425916+0000", 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013072999999999999 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428156+0000", 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022408649999999999 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428832+0000", 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067540999999999999 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.513961+0000", 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085129596000000002 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514015+0000", 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3313000000000003e-05 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514055+0000", 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0156e-05 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:145390 7.5 7:a9dbf476:::benchmark_data_smithi081_73391_object6391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:59.350105+0000", 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 355.26817385999999, 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080355056999999994, 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145390 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350105+0000", 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350105+0000", 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350106+0000", 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5820000000000001e-06 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350111+0000", 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1220000000000003e-06 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350120+0000", 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6680000000000007e-06 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350125+0000", 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3329999999999996e-06 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350166+0000", 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0370000000000001e-05 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350248+0000", 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2588000000000003e-05 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1,2", 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.350334+0000", 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5597000000000001e-05 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.352607+0000", 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022733499999999999 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.353408+0000", 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080050800000000001 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.353423+0000", 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4961e-05 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.430264+0000", 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076841719000000003 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.430327+0000", 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3082000000000004e-05 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:59.430460+0000", 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013217700000000001 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211656 7.5 7:a3c1ba51:::benchmark_data_smithi081_73391_object72657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.348024+0000", 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 280.27025401899999, 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107860995, 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211656 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348024+0000", 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348024+0000", 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348027+0000", 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6709999999999999e-06 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348033+0000", 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4779999999999999e-06 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348046+0000", 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3638999999999999e-05 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348053+0000", 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2980000000000003e-06 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348113+0000", 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9852999999999998e-05 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348226+0000", 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011289 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348343+0000", 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116623 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.350450+0000", 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021071620000000001 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.357782+0000", 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073323420000000004 2024-09-21T19:01:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455803+0000", 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098020501999999995 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455840+0000", 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7206000000000003e-05 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455885+0000", 2024-09-21T19:01:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5330999999999997e-05 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211679 7.5 7:a0aaf166:::benchmark_data_smithi081_73391_object72680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.371480+0000", 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 280.24679836899998, 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084602176000000001, 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211679 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371480+0000", 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371480+0000", 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371482+0000", 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7400000000000001e-06 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371486+0000", 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5820000000000003e-06 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371497+0000", 2024-09-21T19:01:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0801e-05 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371503+0000", 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9000000000000003e-06 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371564+0000", 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0491000000000001e-05 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371687+0000", 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012302999999999999 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371815+0000", 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864899999999999 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.373861+0000", 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020455870000000002 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.375558+0000", 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016976879999999999 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455985+0000", 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080426912000000003 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.456037+0000", 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1109999999999997e-05 2024-09-21T19:01:54.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.456082+0000", 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5686000000000003e-05 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:241829 7.5 7:a585a8ba:::benchmark_data_smithi081_73391_object102830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.216357+0000", 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 245.40192144, 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.073916367999999996, 2024-09-21T19:01:54.726 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 241829 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216357+0000", 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216357+0000", 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216359+0000", 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1210000000000001e-06 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216366+0000", 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.669e-06 2024-09-21T19:01:54.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216381+0000", 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5705999999999998e-05 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216390+0000", 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5499999999999995e-06 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216448+0000", 2024-09-21T19:01:54.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7556000000000003e-05 2024-09-21T19:01:54.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216597+0000", 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014921199999999999 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:01:54.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.216707+0000", 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110716 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.219439+0000", 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027312679999999998 2024-09-21T19:01:54.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.287801+0000", 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.068362676999999997 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.290143+0000", 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023415229999999999 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.290216+0000", 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3202999999999997e-05 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.290273+0000", 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7166999999999997e-05 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.735 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242836 7.5 7:a22dd50b:::benchmark_data_smithi081_73391_object103837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:50.363052+0000", 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 244.255226884, 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.121274044, 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242836 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363052+0000", 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363052+0000", 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363054+0000", 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1160000000000002e-06 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363059+0000", 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6729999999999997e-06 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363072+0000", 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2897e-05 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363080+0000", 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4229999999999999e-06 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363131+0000", 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1759999999999999e-05 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363205+0000", 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3712000000000002e-05 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363271+0000", 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5877999999999995e-05 2024-09-21T19:01:54.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.365405+0000", 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021336670000000001 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.366435+0000", 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010303689999999999 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484220+0000", 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11778470100000001 2024-09-21T19:01:54.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484278+0000", 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8464000000000001e-05 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484326+0000", 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7383999999999999e-05 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.742 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:01:54.742 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:01:54.742 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:01:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.742 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:54.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:54.743 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T19:01:54.743 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:01:54.743 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T19:01:54.743 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T19:01:54.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6 2024-09-21T19:01:54.768 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:54 smithi104 ceph-mon[32438]: pgmap v414: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 859 B/s rd, 0 op/s; 8864/93864 objects misplaced (9.443%) 2024-09-21T19:01:54.768 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:54 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T19:01:54.768 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:54 smithi104 ceph-mon[32438]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T19:01:54.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:54.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:54.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:54.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:55.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:54 smithi081 ceph-mon[29384]: pgmap v414: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 859 B/s rd, 0 op/s; 8864/93864 objects misplaced (9.443%) 2024-09-21T19:01:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:54 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T19:01:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:54 smithi081 ceph-mon[29384]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T19:01:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:54 smithi081 ceph-mon[39192]: pgmap v414: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 859 B/s rd, 0 op/s; 8864/93864 objects misplaced (9.443%) 2024-09-21T19:01:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:54 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T19:01:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:54 smithi081 ceph-mon[39192]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T19:01:55.099 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:54 smithi104 systemd[1]: Stopping Ceph osd.6 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:01:55.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:55.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:55.290 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:55.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:55.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:55.563 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:55.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:55.691 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:01:55.704 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:55.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:55.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:55.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:01:55.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:55.950 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:55.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:56.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:55 smithi104 ceph-mon[32438]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T19:01:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3603586273' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:55 smithi104 ceph-mon[32438]: from='client.25513 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:55 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:55 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:56.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:56.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[39192]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3603586273' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[39192]: from='client.25513 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[29384]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3603586273' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[29384]: from='client.25513 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:01:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:56.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:55 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-21T19:01:56.073 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:56.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:56.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:56.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:56.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:56.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[47075]: 2024-09-21T19:01:56.186+0000 7f069620d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:01:56.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[47075]: 2024-09-21T19:01:56.186+0000 7f069620d640 -1 osd.6 238 *** Got signal Terminated *** 2024-09-21T19:01:56.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[47075]: 2024-09-21T19:01:56.186+0000 7f069620d640 -1 osd.6 238 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:01:56.543 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:56.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:56.572 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:01:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:56.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:56.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:56.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[29384]: pgmap v417: 178 pgs: 10 unknown, 4 active+remapped+backfill_toofull, 164 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 8864/93864 objects misplaced (9.443%) 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[29384]: from='client.25513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[29384]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[29384]: osd.6 marked itself down and dead 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[39192]: pgmap v417: 178 pgs: 10 unknown, 4 active+remapped+backfill_toofull, 164 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 8864/93864 objects misplaced (9.443%) 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[39192]: from='client.25513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-21T19:01:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[39192]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T19:01:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:56 smithi081 ceph-mon[39192]: osd.6 marked itself down and dead 2024-09-21T19:01:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:56.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:56.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:56.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:56.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-mon[32438]: pgmap v417: 178 pgs: 10 unknown, 4 active+remapped+backfill_toofull, 164 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 8864/93864 objects misplaced (9.443%) 2024-09-21T19:01:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:01:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-mon[32438]: from='client.25513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:01:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-21T19:01:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-mon[32438]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T19:01:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:56 smithi104 ceph-mon[32438]: osd.6 marked itself down and dead 2024-09-21T19:01:57.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:57.076 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:57.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:57.199 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:57.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:57.323 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:57.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:57.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:57.554 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:57 smithi104 podman[97056]: 2024-09-21 19:01:57.281812208 +0000 UTC m=+1.198024128 container died 2bee0454be3aa114c050cf19009649808a93e93942ecdf787bb0370f48ca3420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T19:01:57.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:01:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:01:57.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:57.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.713 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:57.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.838 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:57.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:57.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:01:57.940 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:57 smithi104 ceph-mon[32438]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T19:01:57.940 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:57 smithi104 ceph-mon[32438]: osdmap e239: 8 total, 7 up, 8 in 2024-09-21T19:01:57.940 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:57 smithi104 ceph-mon[32438]: osdmap e240: 8 total, 7 up, 8 in 2024-09-21T19:01:57.940 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:57 smithi104 podman[97056]: 2024-09-21 19:01:57.660731685 +0000 UTC m=+1.576943588 container cleanup 2bee0454be3aa114c050cf19009649808a93e93942ecdf787bb0370f48ca3420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T19:01:57.940 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:57 smithi104 bash[97056]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6 2024-09-21T19:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:57.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:01:57.988 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-mon[39192]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T19:01:57.988 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-mon[39192]: osdmap e239: 8 total, 7 up, 8 in 2024-09-21T19:01:57.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-mon[39192]: osdmap e240: 8 total, 7 up, 8 in 2024-09-21T19:01:57.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-mon[29384]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T19:01:57.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-mon[29384]: osdmap e239: 8 total, 7 up, 8 in 2024-09-21T19:01:57.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:57 smithi081 ceph-mon[29384]: osdmap e240: 8 total, 7 up, 8 in 2024-09-21T19:01:58.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:58.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:01:58.256 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:57 smithi104 podman[97139]: 2024-09-21 19:01:57.937524075 +0000 UTC m=+0.659391777 container remove 2bee0454be3aa114c050cf19009649808a93e93942ecdf787bb0370f48ca3420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T19:01:58.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:01:58.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:58.386 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:58.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:58.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:01:58.518 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:58.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:01:58.648 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:01:58.707 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:58 smithi104 podman[97257]: 2024-09-21 19:01:58.278192391 +0000 UTC m=+0.019165138 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:01:58.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:01:58.802 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:01:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: pgmap v420: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 1.08k op/s; 2531/81063 objects misplaced (3.122%); 39 KiB/s, 0 objects/s recovering 2024-09-21T19:01:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.11 continuing backfill to osd.5 from (206'8700,238'14272] MIN to 238'14272 2024-09-21T19:01:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.11 continuing backfill to osd.7 from (206'8700,238'14272] MIN to 238'14272 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.12 continuing backfill to osd.2 from (207'9100,238'14294] MIN to 238'14294 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.12 continuing backfill to osd.7 from (207'9100,238'14294] MIN to 238'14294 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1f starting backfill to osd.0 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1c starting backfill to osd.4 from (0'0,0'0] MAX to 237'14994 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.22 starting backfill to osd.1 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1a starting backfill to osd.0 from (0'0,0'0] MAX to 237'14680 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.20 starting backfill to osd.4 from (0'0,0'0] MAX to 237'14170 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.21 starting backfill to osd.5 from (0'0,0'0] MAX to 237'14273 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1e starting backfill to osd.3 from (0'0,0'0] MAX to 237'14856 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.23 starting backfill to osd.0 from (0'0,0'0] MAX to 237'14018 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1d starting backfill to osd.0 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1b starting backfill to osd.2 from (0'0,0'0] MAX to 237'14954 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.22 starting backfill to osd.3 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1c starting backfill to osd.5 from (0'0,0'0] MAX to 237'14994 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1d starting backfill to osd.1 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1a starting backfill to osd.7 from (0'0,0'0] MAX to 237'14680 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1f starting backfill to osd.5 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1d starting backfill to osd.3 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.22 starting backfill to osd.7 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1e starting backfill to osd.7 from (0'0,0'0] MAX to 237'14856 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: 8.1f starting backfill to osd.7 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:58.818 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: pgmap v420: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 1.08k op/s; 2531/81063 objects misplaced (3.122%); 39 KiB/s, 0 objects/s recovering 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.11 continuing backfill to osd.5 from (206'8700,238'14272] MIN to 238'14272 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.11 continuing backfill to osd.7 from (206'8700,238'14272] MIN to 238'14272 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.12 continuing backfill to osd.2 from (207'9100,238'14294] MIN to 238'14294 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.12 continuing backfill to osd.7 from (207'9100,238'14294] MIN to 238'14294 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1f starting backfill to osd.0 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1c starting backfill to osd.4 from (0'0,0'0] MAX to 237'14994 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.22 starting backfill to osd.1 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1a starting backfill to osd.0 from (0'0,0'0] MAX to 237'14680 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.20 starting backfill to osd.4 from (0'0,0'0] MAX to 237'14170 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.21 starting backfill to osd.5 from (0'0,0'0] MAX to 237'14273 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:01:58.819 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1e starting backfill to osd.3 from (0'0,0'0] MAX to 237'14856 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.23 starting backfill to osd.0 from (0'0,0'0] MAX to 237'14018 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1d starting backfill to osd.0 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1b starting backfill to osd.2 from (0'0,0'0] MAX to 237'14954 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.22 starting backfill to osd.3 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1c starting backfill to osd.5 from (0'0,0'0] MAX to 237'14994 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1d starting backfill to osd.1 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1a starting backfill to osd.7 from (0'0,0'0] MAX to 237'14680 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1f starting backfill to osd.5 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1d starting backfill to osd.3 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.22 starting backfill to osd.7 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1e starting backfill to osd.7 from (0'0,0'0] MAX to 237'14856 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: 8.1f starting backfill to osd.7 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:58.820 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:01:58 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:58.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:58.947 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:58.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPushReply(8.22 240/239 [PushReplyOp(8:4416f3fd:::benchmark_data_smithi081_105538_object117911:head)])", 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.143950+0000", 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.011923512000000001, 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011946807, 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:58.983 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.143950+0000", 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.143950+0000", 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.143951+0000", 2024-09-21T19:01:58.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9400000000000005e-07 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.143957+0000", 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6470000000000001e-06 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.143965+0000", 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8320000000000006e-06 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:58.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.143972+0000", 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5769999999999999e-06 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:238067 8.12 8:4b46ee56:::benchmark_data_smithi081_105538_object105835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.144235+0000", 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.011639033, 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011710631000000001, 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:58.986 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 238067 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144235+0000", 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144235+0000", 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144236+0000", 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8109999999999999e-06 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144242+0000", 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1529999999999996e-06 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:58.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144251+0000", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3749999999999992e-06 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144257+0000", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4869999999999999e-06 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144834+0000", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00057689799999999995 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144940+0000", 2024-09-21T19:01:58.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105703 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4,5,7", 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.145241+0000", 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030092500000000001 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150435+0000", 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051942869999999997 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:58.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150484+0000", 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8219e-05 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:238068 8.12 8:4f4fc786:::benchmark_data_smithi081_105538_object105836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.144266+0000", 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.011608065000000001, 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011749693, 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:01:58.991 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 238068 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144266+0000", 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144266+0000", 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144266+0000", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6700000000000003e-07 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144269+0000", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.418e-06 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144274+0000", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4759999999999997e-06 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:58.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144280+0000", 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.485e-06 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150519+0000", 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0062397069999999997 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150643+0000", 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123254 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,4,5,7", 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150900+0000", 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000257746 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPushReply(8.22 240/239 [PushReplyOp(8:4416f3fd:::benchmark_data_smithi081_105538_object117911:head)])", 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.144583+0000", 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.011291024, 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011488997000000001, 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:58.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144583+0000", 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144583+0000", 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144583+0000", 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3099999999999998e-07 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144586+0000", 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0199999999999999e-06 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144592+0000", 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9139999999999996e-06 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.144598+0000", 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.404e-06 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:58.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPushReply(8.22 240/239 [PushReplyOp(8:4416f3fd:::benchmark_data_smithi081_105538_object117911:head)])", 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.146654+0000", 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0092199589999999998, 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0094561590000000004, 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.146654+0000", 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.146654+0000", 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.146654+0000", 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.91e-07 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.146657+0000", 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1030000000000002e-06 2024-09-21T19:01:58.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.146663+0000", 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4500000000000003e-06 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.146668+0000", 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3369999999999999e-06 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.000 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:238064 8.11 e240/239 8:8e32e699:::benchmark_data_smithi081_105538_object105832:head v 240'14279, mlcod=240'14277)", 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.150062+0000", 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0058120330000000003, 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060875080000000002, 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150062+0000", 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150062+0000", 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150063+0000", 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9300000000000006e-07 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150069+0000", 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8939999999999998e-06 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150078+0000", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0469999999999999e-06 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150084+0000", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2600000000000002e-06 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150158+0000", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4396999999999999e-05 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150183+0000", 2024-09-21T19:01:59.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4519999999999999e-05 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:238071 8.1d e240/239 8:bd692ff6:::benchmark_data_smithi081_105538_object105839:head v 240'14921, mlcod=240'14919)", 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.150092+0000", 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0057817809999999997, 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061045719999999999, 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150092+0000", 2024-09-21T19:01:59.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150092+0000", 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150093+0000", 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.023e-06 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150099+0000", 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2369999999999997e-06 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150107+0000", 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8970000000000008e-06 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150113+0000", 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9460000000000003e-06 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150997+0000", 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00088404200000000003 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.151018+0000", 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1083000000000001e-05 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:238069 8.11 e240/239 8:8911553d:::benchmark_data_smithi081_105538_object105837:head v 240'14280, mlcod=240'14277)", 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.150106+0000", 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0057681570000000003, 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061370130000000002, 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:59.007 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150106+0000", 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150106+0000", 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150106+0000", 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3099999999999999e-07 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150109+0000", 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.208e-06 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150114+0000", 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6290000000000001e-06 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150118+0000", 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8739999999999997e-06 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.151836+0000", 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017183960000000001 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.151860+0000", 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3811e-05 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:238075 8.1c 8:3922494d:::benchmark_data_smithi081_105538_object105843:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.150140+0000", 2024-09-21T19:01:59.010 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0057341550000000003, 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061537500000000004, 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 238075 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150140+0000", 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150140+0000", 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150141+0000", 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3e-06 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150146+0000", 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6630000000000001e-06 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150153+0000", 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8469999999999996e-06 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150159+0000", 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.463e-06 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150277+0000", 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011846200000000001 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.151302+0000", 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001024612 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.151543+0000", 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024115399999999999 2024-09-21T19:01:59.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.154017+0000", 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024743399999999998 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.154064+0000", 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6227000000000003e-05 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:238076 8.1e 8:7b6b59e1:::benchmark_data_smithi081_105538_object105844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:01:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.150171+0000", 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0057024470000000002, 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061891519999999998, 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 238076 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150171+0000", 2024-09-21T19:01:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150171+0000", 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150172+0000", 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6899999999999997e-07 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150175+0000", 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8310000000000001e-06 2024-09-21T19:01:59.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150181+0000", 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0260000000000004e-06 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150186+0000", 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1549999999999998e-06 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150528+0000", 2024-09-21T19:01:59.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034265000000000001 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150641+0000", 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011267799999999999 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,3,4,7", 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150914+0000", 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027311000000000002 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.154699+0000", 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037847390000000001 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155316+0000", 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00061717900000000003 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:238077 8.6 8:67031128:::benchmark_data_smithi081_105538_object105845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.150199+0000", 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0056746160000000004, 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0062290710000000001, 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.020 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 238077 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150199+0000", 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150199+0000", 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150200+0000", 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150202+0000", 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8499999999999998e-06 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150208+0000", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.772e-06 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.150214+0000", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3410000000000002e-06 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152135+0000", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001921617 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152243+0000", 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010816599999999999 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 7", 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152566+0000", 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00032256 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.156058+0000", 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034924639999999998 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.35258.0:238067 8.12 e240/239)", 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.152553+0000", 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0033210800000000001, 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039397590000000001, 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152553+0000", 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152553+0000", 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152554+0000", 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1319999999999999e-06 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152559+0000", 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8060000000000002e-06 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152568+0000", 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3039999999999994e-06 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.152574+0000", 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6150000000000002e-06 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:238079 8.23 e240/239 8:c590cad1:::benchmark_data_smithi081_105538_object105847:head v 240'14025, mlcod=240'14023)", 2024-09-21T19:01:59.027 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.153769+0000", 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0021044840000000002, 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027611039999999999, 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153769+0000", 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153769+0000", 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153770+0000", 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8400000000000003e-07 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153776+0000", 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8359999999999997e-06 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153786+0000", 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9040000000000004e-06 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153792+0000", 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6059999999999999e-06 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155326+0000", 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015340460000000001 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:01:59.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155346+0000", 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9093000000000002e-05 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.35258.0:238068 8.12 e240/239)", 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.153928+0000", 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0019461630000000001, 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026478999999999999, 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153928+0000", 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153928+0000", 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153928+0000", 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6100000000000002e-07 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153930+0000", 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.272e-06 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153935+0000", 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.843e-06 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.153939+0000", 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3100000000000002e-06 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.033 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.35258.0:238068 8.12 e240/239)", 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.155149+0000", 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.000725065, 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001464098, 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155149+0000", 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155149+0000", 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155149+0000", 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7299999999999996e-07 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155152+0000", 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.146e-06 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155159+0000", 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1500000000000004e-06 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155164+0000", 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0440000000000003e-06 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.036 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDPGPush(8.1a 240/239 [PushOp(8:581a6943:::benchmark_data_smithi081_105538_object116278:head, version: 202'7241, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:581a6943:::benchmark_data_smithi081_105538_object116278:head@202'7241, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.155192+0000", 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00068135400000000001, 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014878020000000001, 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155192+0000", 2024-09-21T19:01:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155192+0000", 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155193+0000", 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4899999999999995e-07 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155225+0000", 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2181999999999998e-05 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155242+0000", 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6897000000000001e-05 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155258+0000", 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5852e-05 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_backfill(progress 8.1a e 240/240 lb 8:581a3dc4:::benchmark_data_smithi081_105538_object112707:head)", 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:58.155274+0000", 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00059946499999999998, 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014461630000000001, 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155274+0000", 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155274+0000", 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155275+0000", 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7899999999999999e-07 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155282+0000", 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3100000000000003e-06 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155293+0000", 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1357e-05 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:58.155299+0000", 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9889999999999997e-06 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 17 2024-09-21T19:01:59.042 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:01:59.043 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:58 smithi104 podman[97257]: 2024-09-21 19:01:58.705629343 +0000 UTC m=+0.446602083 container create ab4ee1504f8eeefdb0cd3bb0e992f58822138b8c44be45f28241dcd79959acee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True) 2024-09-21T19:01:59.043 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:58 smithi104 podman[97257]: 2024-09-21 19:01:58.942225338 +0000 UTC m=+0.683198094 container init ab4ee1504f8eeefdb0cd3bb0e992f58822138b8c44be45f28241dcd79959acee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T19:01:59.043 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:58 smithi104 podman[97257]: 2024-09-21 19:01:58.947361616 +0000 UTC m=+0.688334358 container start ab4ee1504f8eeefdb0cd3bb0e992f58822138b8c44be45f28241dcd79959acee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:01:59.043 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: pgmap v420: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 1.08k op/s; 2531/81063 objects misplaced (3.122%); 39 KiB/s, 0 objects/s recovering 2024-09-21T19:01:59.043 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.11 continuing backfill to osd.5 from (206'8700,238'14272] MIN to 238'14272 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.11 continuing backfill to osd.7 from (206'8700,238'14272] MIN to 238'14272 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.12 continuing backfill to osd.2 from (207'9100,238'14294] MIN to 238'14294 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.12 continuing backfill to osd.7 from (207'9100,238'14294] MIN to 238'14294 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1f starting backfill to osd.0 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1c starting backfill to osd.4 from (0'0,0'0] MAX to 237'14994 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.22 starting backfill to osd.1 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1a starting backfill to osd.0 from (0'0,0'0] MAX to 237'14680 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.20 starting backfill to osd.4 from (0'0,0'0] MAX to 237'14170 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.21 starting backfill to osd.5 from (0'0,0'0] MAX to 237'14273 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1e starting backfill to osd.3 from (0'0,0'0] MAX to 237'14856 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.23 starting backfill to osd.0 from (0'0,0'0] MAX to 237'14018 2024-09-21T19:01:59.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1d starting backfill to osd.0 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1b starting backfill to osd.2 from (0'0,0'0] MAX to 237'14954 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.22 starting backfill to osd.3 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1c starting backfill to osd.5 from (0'0,0'0] MAX to 237'14994 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1d starting backfill to osd.1 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1a starting backfill to osd.7 from (0'0,0'0] MAX to 237'14680 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1f starting backfill to osd.5 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1d starting backfill to osd.3 from (0'0,0'0] MAX to 237'14909 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.22 starting backfill to osd.7 from (0'0,0'0] MAX to 237'14362 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1e starting backfill to osd.7 from (0'0,0'0] MAX to 237'14856 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: 8.1f starting backfill to osd.7 from (0'0,0'0] MAX to 237'14863 2024-09-21T19:01:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:01:58 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:01:59.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:59.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:59.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:59.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:59.368 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:59.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:01:59.507 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:59.508 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:59 smithi104 podman[97257]: 2024-09-21 19:01:59.058683836 +0000 UTC m=+0.799656573 container attach ab4ee1504f8eeefdb0cd3bb0e992f58822138b8c44be45f28241dcd79959acee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T19:01:59.508 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:01:59 smithi104 podman[97257]: 2024-09-21 19:01:59.218135003 +0000 UTC m=+0.959107747 container died ab4ee1504f8eeefdb0cd3bb0e992f58822138b8c44be45f28241dcd79959acee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T19:01:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:01:59.609 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:01:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.632 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:59.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:01:59.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:01:59.769 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:01:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:01:59.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T19:01:59.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:01:59.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:01:59.894 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T19:01:59.895 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-09-21T19:01:59.895 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T19:01:59.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.6 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-09-21T19:01:59.903 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:01:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:01:59.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:00.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:00.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:00.149 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.6/config 2024-09-21T19:02:00.181 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:00.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:00.321 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:00.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:00.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:00.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:00.587 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:00.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:00.720 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:00.822 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.853 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:00.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:00.930 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:00 smithi081 ceph-mon[29384]: pgmap v422: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 9.2 KiB/s rd, 60 KiB/s wr, 879 op/s; 2531/81063 objects misplaced (3.122%); 32 KiB/s, 0 objects/s recovering 2024-09-21T19:02:00.930 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:00 smithi081 ceph-mon[29384]: osdmap e241: 8 total, 7 up, 8 in 2024-09-21T19:02:00.931 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:00 smithi081 ceph-mon[39192]: pgmap v422: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 9.2 KiB/s rd, 60 KiB/s wr, 879 op/s; 2531/81063 objects misplaced (3.122%); 32 KiB/s, 0 objects/s recovering 2024-09-21T19:02:00.931 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:00 smithi081 ceph-mon[39192]: osdmap e241: 8 total, 7 up, 8 in 2024-09-21T19:02:00.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:00.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:01.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:00 smithi104 ceph-mon[32438]: pgmap v422: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 9.2 KiB/s rd, 60 KiB/s wr, 879 op/s; 2531/81063 objects misplaced (3.122%); 32 KiB/s, 0 objects/s recovering 2024-09-21T19:02:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:00 smithi104 ceph-mon[32438]: osdmap e241: 8 total, 7 up, 8 in 2024-09-21T19:02:01.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:01.131 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:01.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:01.260 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:01.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:01.392 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:01.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:02:01.454 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:01.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:01.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:01 smithi081 ceph-mon[29384]: osdmap e242: 8 total, 7 up, 8 in 2024-09-21T19:02:01.635 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:01 smithi081 ceph-mon[39192]: osdmap e242: 8 total, 7 up, 8 in 2024-09-21T19:02:01.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:01.674 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:01.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.820 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:01.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:01.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:01.959 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:01.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:01 smithi104 ceph-mon[32438]: osdmap e242: 8 total, 7 up, 8 in 2024-09-21T19:02:02.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:02.119 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:02.134 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:02.134 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:02.176 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:02.176 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:02.185 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:02.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:02.211 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:02.211 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:02.237 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:02.248 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:02.277 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:02.277 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:02.310 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:02.310 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:02.312 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:02.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:02.343 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:02.343 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:02.380 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:02.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:02.512 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:02.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:02.638 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:02.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:02.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:02.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:02.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:02.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[29384]: pgmap v425: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 8.4 KiB/s rd, 4 op/s; 2531/81063 objects misplaced (3.122%); 32 KiB/s, 0 objects/s recovering 2024-09-21T19:02:02.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[29384]: osdmap e243: 8 total, 7 up, 8 in 2024-09-21T19:02:02.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2209065498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:02.779 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:02.779 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:02.779 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[39192]: pgmap v425: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 8.4 KiB/s rd, 4 op/s; 2531/81063 objects misplaced (3.122%); 32 KiB/s, 0 objects/s recovering 2024-09-21T19:02:02.779 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[39192]: osdmap e243: 8 total, 7 up, 8 in 2024-09-21T19:02:02.779 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2209065498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:02.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:02.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:02.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:02 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:02 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:02 smithi104 ceph-mon[32438]: pgmap v425: 178 pgs: 22 peering, 7 stale+active+clean, 10 remapped+peering, 2 unknown, 2 active+remapped+backfill_toofull, 135 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 8.4 KiB/s rd, 4 op/s; 2531/81063 objects misplaced (3.122%); 32 KiB/s, 0 objects/s recovering 2024-09-21T19:02:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:02 smithi104 ceph-mon[32438]: osdmap e243: 8 total, 7 up, 8 in 2024-09-21T19:02:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2209065498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:03.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:03.047 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:03.128 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:03.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:03.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:03.181 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:03.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:03.212 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:02:03.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:03.314 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:03.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:03.447 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:03.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:03.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:03.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:03.707 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:03.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:03 smithi104 ceph-mon[32438]: osdmap e244: 8 total, 7 up, 8 in 2024-09-21T19:02:03.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:03 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:03.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:03 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:03.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:03 smithi104 ceph-mon[32438]: osdmap e245: 8 total, 7 up, 8 in 2024-09-21T19:02:03.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[29384]: osdmap e244: 8 total, 7 up, 8 in 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[29384]: osdmap e245: 8 total, 7 up, 8 in 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[39192]: osdmap e244: 8 total, 7 up, 8 in 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:03.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:03 smithi081 ceph-mon[39192]: osdmap e245: 8 total, 7 up, 8 in 2024-09-21T19:02:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:03.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:03.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:03.973 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:03.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:04.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:04.096 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:04.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:04.223 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:04.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:04.345 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:04.447 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:04.470 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:04.515 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:04.515 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:04.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:04.588 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:02:04.602 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:04.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:04.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: pgmap v428: 178 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.24k op/s; 4590/66168 objects degraded (6.937%); 15507/66168 objects misplaced (23.436%); 29 MiB/s, 465 objects/s recovering 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: Health check failed: Degraded data redundancy: 4590/66168 objects degraded (6.937%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1787156227' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: from='client.35618 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:02:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[39192]: osdmap e246: 8 total, 7 up, 8 in 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: pgmap v428: 178 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.24k op/s; 4590/66168 objects degraded (6.937%); 15507/66168 objects misplaced (23.436%); 29 MiB/s, 465 objects/s recovering 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: Health check failed: Degraded data redundancy: 4590/66168 objects degraded (6.937%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1787156227' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: from='client.35618 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:02:04.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:04 smithi081 ceph-mon[29384]: osdmap e246: 8 total, 7 up, 8 in 2024-09-21T19:02:04.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.867 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:04.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:04.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:04.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: pgmap v428: 178 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.24k op/s; 4590/66168 objects degraded (6.937%); 15507/66168 objects misplaced (23.436%); 29 MiB/s, 465 objects/s recovering 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: Health check failed: Degraded data redundancy: 4590/66168 objects degraded (6.937%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1787156227' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: from='client.35618 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:02:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:04 smithi104 ceph-mon[32438]: osdmap e246: 8 total, 7 up, 8 in 2024-09-21T19:02:05.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:05.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:05.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:02:05.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:05.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:4.11 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:4.1d 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:4.1 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:4.18 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:4.10 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:4.12 2024-09-21T19:02:05.329 INFO:teuthology.orchestra.run.smithi104.stdout:1.3 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:8.3 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:1.6 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:5.f 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:6.1e 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:3.c 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:4.c 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:3.2 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:8.23 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:1.7 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:4.a 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:4.13 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:3.1d 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:3.d 2024-09-21T19:02:05.330 INFO:teuthology.orchestra.run.smithi104.stdout:3.0 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:2.2 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:4.e 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:6.a 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:8.5 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:3.a 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:3.14 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:4.5 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:4.1f 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:3.10 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:1.5 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:3.1e 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:5.10 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:3.3 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:3.11 2024-09-21T19:02:05.331 INFO:teuthology.orchestra.run.smithi104.stdout:5.2 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:8.11 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:1.0 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:5.18 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:5.1f 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:6.13 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:5.3 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:5.9 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:6.1b 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:5.11 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:6.1 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:6.c 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:6.3 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:1.a 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:6.5 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:8.12 2024-09-21T19:02:05.332 INFO:teuthology.orchestra.run.smithi104.stdout:8.9 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:8.15 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:8.6 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:6.1f 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:8.17 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:4.8 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:6.15 2024-09-21T19:02:05.333 INFO:teuthology.orchestra.run.smithi104.stdout:8.7 2024-09-21T19:02:05.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:05.460 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:05.480 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:05.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:05.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:05.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:05 smithi081 ceph-mon[39192]: 4.9 scrub starts 2024-09-21T19:02:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:05 smithi081 ceph-mon[39192]: 4.9 scrub ok 2024-09-21T19:02:05.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:05 smithi081 ceph-mon[39192]: osdmap e247: 8 total, 7 up, 8 in 2024-09-21T19:02:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:05 smithi081 ceph-mon[29384]: 4.9 scrub starts 2024-09-21T19:02:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:05 smithi081 ceph-mon[29384]: 4.9 scrub ok 2024-09-21T19:02:05.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:05 smithi081 ceph-mon[29384]: osdmap e247: 8 total, 7 up, 8 in 2024-09-21T19:02:05.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:05.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:05.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:05.894 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:05 smithi104 ceph-mon[32438]: 4.9 scrub starts 2024-09-21T19:02:05.894 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:05 smithi104 ceph-mon[32438]: 4.9 scrub ok 2024-09-21T19:02:05.894 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:05 smithi104 ceph-mon[32438]: osdmap e247: 8 total, 7 up, 8 in 2024-09-21T19:02:05.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:06.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:06.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:06.152 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:06.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:06.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:06.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:06.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:06.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.6 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 5.f --file /var/log/ceph/exp.5.f.6 2024-09-21T19:02:06.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:06.560 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:06.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:06.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:06.748 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.6/config 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: 8.e scrub starts 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: 8.e scrub ok 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: pgmap v431: 178 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.24k op/s; 4590/66168 objects degraded (6.937%); 15507/66168 objects misplaced (23.436%); 29 MiB/s, 465 objects/s recovering 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: 8.22 scrub starts 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: 8.1 scrub starts 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: 8.22 scrub ok 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: 8.1 scrub ok 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:06.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:06.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:06.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:06.956 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:06.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:07.057 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: 8.e scrub starts 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: 8.e scrub ok 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: pgmap v431: 178 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.24k op/s; 4590/66168 objects degraded (6.937%); 15507/66168 objects misplaced (23.436%); 29 MiB/s, 465 objects/s recovering 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: 8.22 scrub starts 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: 8.1 scrub starts 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: 8.22 scrub ok 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: 8.1 scrub ok 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: 8.e scrub starts 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: 8.e scrub ok 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: pgmap v431: 178 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.24k op/s; 4590/66168 objects degraded (6.937%); 15507/66168 objects misplaced (23.436%); 29 MiB/s, 465 objects/s recovering 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: 8.22 scrub starts 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: 8.1 scrub starts 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: 8.22 scrub ok 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: 8.1 scrub ok 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:07.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:07.089 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:07.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:07.219 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:07.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:07.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:02:07.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:07.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:07.505 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:02:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:02:07.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:02:07.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:07.617 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:07.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:07.747 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:07.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T19:02:07.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T19:02:07.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[29384]: 3.f scrub starts 2024-09-21T19:02:07.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[29384]: 3.f scrub ok 2024-09-21T19:02:07.811 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T19:02:07.811 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T19:02:07.811 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[39192]: 3.f scrub starts 2024-09-21T19:02:07.811 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:07 smithi081 ceph-mon[39192]: 3.f scrub ok 2024-09-21T19:02:07.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:07.880 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:02:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:07.900 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:07.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:07.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:07 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T19:02:07.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:07 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T19:02:07.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:07 smithi104 ceph-mon[32438]: 3.f scrub starts 2024-09-21T19:02:07.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:07 smithi104 ceph-mon[32438]: 3.f scrub ok 2024-09-21T19:02:08.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.046 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:08.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:08.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:08.174 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:08.201 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:08.201 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:08.239 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:08.240 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:08.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:02:08.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:08.277 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:08.295 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:08.316 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:08.316 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:08.347 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:08.349 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:08.349 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.350 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:08.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:08.400 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:08.400 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:08.401 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:08.437 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:08.438 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:08.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.492 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:08.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:08.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:08.642 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:08.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:08.773 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:08.875 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:08.896 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:08.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:08.952 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 8.20 scrub starts 2024-09-21T19:02:08.952 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 8.20 scrub ok 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 8.c scrub starts 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: pgmap v433: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 8.c scrub ok 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 8.a scrub starts 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: 8.a scrub ok 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/474602656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 8.20 scrub starts 2024-09-21T19:02:08.953 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 8.20 scrub ok 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 8.c scrub starts 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: pgmap v433: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 8.c scrub ok 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 8.a scrub starts 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: 8.a scrub ok 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:08.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/474602656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:08.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 8.20 scrub starts 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 8.20 scrub ok 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 8.c scrub starts 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: pgmap v433: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 8.c scrub ok 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 8.a scrub starts 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: 8.a scrub ok 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/474602656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:09.036 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:09.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:09.177 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:09.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:09.308 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:09.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:09.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:09.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:09.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:09.613 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:09.680 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:09.700 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:09 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 3549/50586 objects degraded (7.016%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:09.700 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:09 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 3549/50586 objects degraded (7.016%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:09.700 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:09 smithi081 ceph-mon[39192]: osdmap e248: 8 total, 7 up, 8 in 2024-09-21T19:02:09.706 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:09.807 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:09.834 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:09.936 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:09.963 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:09.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:09 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 3549/50586 objects degraded (7.016%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:09 smithi104 ceph-mon[32438]: osdmap e248: 8 total, 7 up, 8 in 2024-09-21T19:02:10.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:09 smithi081 ceph-mon[29384]: osdmap e248: 8 total, 7 up, 8 in 2024-09-21T19:02:10.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:10.105 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:10.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:10.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:10.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.518 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:10.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:10.619 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:10.647 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216350 7.6 7:693ca6d8:::benchmark_data_smithi081_73391_object77351:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 291.12910030099999, 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2616681970000001, 2024-09-21T19:02:10.652 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216350 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518164+0000", 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.593e-06 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518169+0000", 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6689999999999997e-06 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518176+0000", 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8180000000000004e-06 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518182+0000", 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2549999999999997e-06 2024-09-21T19:02:10.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518270+0000", 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8738000000000003e-05 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518279+0000", 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2660000000000001e-06 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779775+0000", 2024-09-21T19:02:10.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.26149613 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779830+0000", 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5727999999999999e-05 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216360 7.6 7:6a035ea2:::benchmark_data_smithi081_73391_object77361:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:02:10.657 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 291.12907424700001, 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2839865960000001, 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216360 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518189+0000", 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5400000000000002e-07 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518191+0000", 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518196+0000", 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.622e-06 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518200+0000", 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6190000000000002e-06 2024-09-21T19:02:10.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518299+0000", 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8587999999999999e-05 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518304+0000", 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6210000000000001e-06 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780077+0000", 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2617725909999999 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780187+0000", 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110469 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802102+0000", 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021914330999999999 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802175+0000", 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3116999999999998e-05 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216394 7.6 7:6d24eb15:::benchmark_data_smithi081_73391_object77395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 291.08825024100003, 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2297531310000001, 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216394 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559014+0000", 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0889999999999998e-06 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559020+0000", 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1669999999999998e-06 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559034+0000", 2024-09-21T19:02:10.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4385e-05 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559041+0000", 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.238e-06 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570552+0000", 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011510347000000001 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570561+0000", 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6209999999999999e-06 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780222+0000", 2024-09-21T19:02:10.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2096608149999999 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780290+0000", 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7988e-05 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780361+0000", 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1239e-05 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788612+0000", 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082509909999999992 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788622+0000", 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159999999999999e-05 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788671+0000", 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8939e-05 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788709+0000", 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7295999999999997e-05 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788765+0000", 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6855999999999999e-05 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216429 7.6 7:6000252a:::benchmark_data_smithi081_73391_object77430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.670 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 291.030786578, 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1724721330000001, 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216429 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616478+0000", 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616484+0000", 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616495+0000", 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1134999999999999e-05 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616502+0000", 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8569999999999997e-06 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616558+0000", 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6023999999999998e-05 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616567+0000", 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8980000000000002e-06 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782338+0000", 2024-09-21T19:02:10.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.165771565 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782505+0000", 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166558 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782607+0000", 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102357 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788626+0000", 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006018921 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788801+0000", 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000174852 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788853+0000", 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2006000000000003e-05 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788898+0000", 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4962000000000001e-05 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788948+0000", 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0006999999999997e-05 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.677 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216455 7.6 7:68c88556:::benchmark_data_smithi081_73391_object77456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 291.00228780200001, 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.144373474, 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216455 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644977+0000", 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5500000000000001e-06 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644983+0000", 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0360000000000003e-06 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644995+0000", 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2006e-05 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645003+0000", 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9459999999999998e-06 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645059+0000", 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5337999999999998e-05 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645068+0000", 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4320000000000005e-06 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.784935+0000", 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1398670499999999 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785006+0000", 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0840999999999995e-05 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785084+0000", 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8228999999999999e-05 2024-09-21T19:02:10.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788630+0000", 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035455730000000002 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789144+0000", 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00051452099999999999 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789199+0000", 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4484000000000002e-05 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789251+0000", 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2633000000000002e-05 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789348+0000", 2024-09-21T19:02:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6835000000000004e-05 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216459 7.6 7:66d9de63:::benchmark_data_smithi081_73391_object77460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.997699001, 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1586775789999999, 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.685 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216459 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649566+0000", 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0540000000000002e-06 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649586+0000", 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0472e-05 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649616+0000", 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9848e-05 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649637+0000", 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1114000000000001e-05 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649894+0000", 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025646600000000001 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649902+0000", 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4589999999999998e-06 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789632+0000", 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1397303350000001 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789788+0000", 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155852 2024-09-21T19:02:10.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789900+0000", 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111909 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802209+0000", 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012309334 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802251+0000", 2024-09-21T19:02:10.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1421000000000002e-05 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808109+0000", 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058583660000000003 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808145+0000", 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5374999999999999e-05 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808241+0000", 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6574000000000001e-05 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216478 7.6 7:69b4062c:::benchmark_data_smithi081_73391_object77479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.97331239499999, 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.137844316, 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216478 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673953+0000", 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3910000000000001e-06 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674028+0000", 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5197000000000005e-05 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674184+0000", 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156129 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674193+0000", 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0410000000000003e-06 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674323+0000", 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013050400000000001 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674368+0000", 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4211000000000001e-05 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792228+0000", 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.117859843 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792333+0000", 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105216 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792492+0000", 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015956400000000001 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802291+0000", 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097984360000000006 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802325+0000", 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4235e-05 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811655+0000", 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093300610000000006 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811704+0000", 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8520999999999998e-05 2024-09-21T19:02:10.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811795+0000", 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0966999999999998e-05 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216483 7.6 7:679f3a63:::benchmark_data_smithi081_73391_object77484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.96678445200001, 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1314383640000001, 2024-09-21T19:02:10.699 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216483 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680480+0000", 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3010000000000001e-06 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680487+0000", 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8090000000000004e-06 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680532+0000", 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4369999999999997e-05 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680540+0000", 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5969999999999992e-06 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680600+0000", 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0275000000000001e-05 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680610+0000", 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6090000000000007e-06 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798740+0000", 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1181301029999999 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798848+0000", 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010757 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798947+0000", 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9349e-05 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802432+0000", 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034852860000000002 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808361+0000", 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059282689999999999 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811802+0000", 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003440869 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811840+0000", 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8197e-05 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811916+0000", 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6760000000000004e-05 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.705 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216512 7.6 7:6f163cff:::benchmark_data_smithi081_73391_object77513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.93244515999999, 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0972389849999999, 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216512 2024-09-21T19:02:10.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714820+0000", 2024-09-21T19:02:10.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2450000000000001e-06 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714826+0000", 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1859999999999997e-06 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714841+0000", 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5288999999999999e-05 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714850+0000", 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6540000000000005e-06 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714963+0000", 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112893 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714971+0000", 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7120000000000006e-06 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802465+0000", 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0874939400000001 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802564+0000", 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.904e-05 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802667+0000", 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103101 2024-09-21T19:02:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808099+0000", 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054315889999999997 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808411+0000", 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031148299999999999 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811922+0000", 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035117199999999999 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811958+0000", 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5766999999999998e-05 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812056+0000", 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8366e-05 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.712 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216545 7.6 7:6b69332e:::benchmark_data_smithi081_73391_object77546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.89407364900001, 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0617420390000001, 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.713 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216545 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:02:10.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753191+0000", 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7680000000000001e-06 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753196+0000", 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1410000000000004e-06 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753207+0000", 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1143e-05 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753214+0000", 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.917e-06 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754192+0000", 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097769299999999996 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754200+0000", 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9400000000000002e-06 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.805892+0000", 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0516927620000001 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806005+0000", 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113143 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806125+0000", 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119331 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808476+0000", 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023510919999999999 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812065+0000", 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035891429999999999 2024-09-21T19:02:10.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814743+0000", 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026784489999999998 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814839+0000", 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5358000000000004e-05 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814931+0000", 2024-09-21T19:02:10.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2158999999999996e-05 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216552 7.6 7:63b154c8:::benchmark_data_smithi081_73391_object77553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.885221543, 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.060947284, 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.720 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216552 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762043+0000", 2024-09-21T19:02:10.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9889999999999999e-06 2024-09-21T19:02:10.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.722 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762056+0000", 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3441999999999999e-05 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762068+0000", 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-05 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762075+0000", 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1169999999999998e-06 2024-09-21T19:02:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762135+0000", 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9759000000000002e-05 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762143+0000", 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808532+0000", 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0463883469999999 2024-09-21T19:02:10.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808647+0000", 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011510600000000001 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808757+0000", 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109849 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814943+0000", 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006186558 2024-09-21T19:02:10.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814980+0000", 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6729e-05 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822841+0000", 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078607529999999998 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822895+0000", 2024-09-21T19:02:10.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4282e-05 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822988+0000", 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3422000000000001e-05 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216555 7.6 7:6310baba:::benchmark_data_smithi081_73391_object77556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:02:10.727 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 290.883895704, 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0695222289999999, 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216555 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:02:10.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763369+0000", 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.635e-06 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763374+0000", 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.101e-06 2024-09-21T19:02:10.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763503+0000", 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000129276 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763511+0000", 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5019999999999992e-06 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763815+0000", 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030339199999999998 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763823+0000", 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7600000000000002e-06 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812093+0000", 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0482702930000001 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812199+0000", 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010665100000000001 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812307+0000", 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107206 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819747+0000", 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074399499999999999 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819790+0000", 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3297999999999998e-05 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832684+0000", 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012894012 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832778+0000", 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4456999999999994e-05 2024-09-21T19:02:10.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832889+0000", 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110696 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235880 8.1e 8:7f5b8aa9:::benchmark_data_smithi081_105538_object103648:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.973244666999999, 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.088566871, 2024-09-21T19:02:10.734 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235880 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7300000000000001e-07 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674021+0000", 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1380000000000001e-06 2024-09-21T19:02:10.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674025+0000", 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3499999999999999e-06 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674029+0000", 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1690000000000002e-06 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842647+0000", 2024-09-21T19:02:10.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16861802400000001 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842654+0000", 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2509999999999997e-06 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758247+0000", 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91559294899999999 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758376+0000", 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012917699999999999 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762523+0000", 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041469219999999999 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762585+0000", 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1418e-05 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.739 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235892 8.c 8:34d14ac3:::benchmark_data_smithi081_105538_object103660:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.973229263, 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0724834350000001, 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.740 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235892 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674034+0000", 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.27e-07 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674036+0000", 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674040+0000", 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0249999999999996e-06 2024-09-21T19:02:10.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674044+0000", 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7160000000000001e-06 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.868789+0000", 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19474559399999999 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.868798+0000", 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5029999999999997e-06 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.720825+0000", 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.85202745899999999 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.720942+0000", 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117217 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.721225+0000", 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028250900000000002 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746281+0000", 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025056334 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746375+0000", 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4040000000000001e-05 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746416+0000", 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0538999999999999e-05 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746478+0000", 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2225999999999997e-05 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746517+0000", 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8761999999999999e-05 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237693 8.21 8:86cd0139:::benchmark_data_smithi081_105538_object105461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:02:10.746 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.947820474, 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.19476636, 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237693 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699444+0000", 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999999999999999e-06 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699448+0000", 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9110000000000003e-06 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699456+0000", 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3259999999999996e-06 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699471+0000", 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4681000000000001e-05 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700689+0000", 2024-09-21T19:02:10.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012186180000000001 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700697+0000", 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4739999999999999e-06 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698487+0000", 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.99779048800000003 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698497+0000", 2024-09-21T19:02:10.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1120000000000002e-06 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889014+0000", 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1905176980000001 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889174+0000", 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000159545 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:02:10.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889403+0000", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022944299999999999 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891432+0000", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002028448 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891637+0000", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020514200000000001 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891727+0000", 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0705000000000007e-05 2024-09-21T19:02:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.893997+0000", 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002269802 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894105+0000", 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107367 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894208+0000", 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001038 2024-09-21T19:02:10.753 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237741 8.21 8:86a63524:::benchmark_data_smithi081_105538_object105509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.924579845, 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1717911829999998, 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237741 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722684+0000", 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0270000000000001e-06 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722688+0000", 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7929999999999999e-06 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722697+0000", 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1290000000000002e-06 2024-09-21T19:02:10.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722703+0000", 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6200000000000001e-06 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722891+0000", 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018788600000000001 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722902+0000", 2024-09-21T19:02:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0541e-05 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702948+0000", 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98004672199999998 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702955+0000", 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0549999999999996e-06 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890451+0000", 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1874960109999999 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890572+0000", 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012139800000000001 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890731+0000", 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015913599999999999 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891711+0000", 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097963900000000003 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892154+0000", 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044374699999999998 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892258+0000", 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010389400000000001 2024-09-21T19:02:10.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894257+0000", 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019987550000000001 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894360+0000", 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010312200000000001 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894474+0000", 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113708 2024-09-21T19:02:10.761 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237675 8.1c 8:3e931efd:::benchmark_data_smithi081_105538_object105443:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.953911722000001, 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.207533752, 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.762 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.762 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237675 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693353+0000", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9609999999999999e-06 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693358+0000", 2024-09-21T19:02:10.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6030000000000004e-06 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694353+0000", 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00099412400000000009 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694362+0000", 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3500000000000003e-06 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698188+0000", 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038259919999999998 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698196+0000", 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7100000000000007e-06 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900800+0000", 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2026040019999999 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900885+0000", 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5010000000000001e-05 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237706 8.1c 8:39f9b21e:::benchmark_data_smithi081_105538_object105474:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.952890698999999, 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2136628949999999, 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.768 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237706 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694373+0000", 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7300000000000005e-07 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694377+0000", 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5800000000000002e-06 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694385+0000", 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3030000000000002e-06 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694390+0000", 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4140000000000002e-06 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698219+0000", 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038285289999999998 2024-09-21T19:02:10.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698225+0000", 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4269999999999996e-06 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901135+0000", 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2029100589999999 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901255+0000", 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120221 2024-09-21T19:02:10.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901527+0000", 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000272045 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907437+0000", 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059092579999999997 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907688+0000", 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025114199999999999 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907731+0000", 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3180000000000003e-05 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907774+0000", 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3256000000000001e-05 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907856+0000", 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1582000000000006e-05 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907917+0000", 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0930000000000001e-05 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908035+0000", 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118096 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.775 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237735 8.1c 8:3aa9ee71:::benchmark_data_smithi081_105538_object105503:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.952866115999999, 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2197030149999999, 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.776 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237735 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694397+0000", 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7000000000000005e-07 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694400+0000", 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7470000000000001e-06 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694405+0000", 2024-09-21T19:02:10.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6230000000000002e-06 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694410+0000", 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.972e-06 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698247+0000", 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038369789999999999 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698253+0000", 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9390000000000002e-06 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.904941+0000", 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.206687686 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905096+0000", 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155236 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:02:10.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905336+0000", 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024011 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907815+0000", 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024790139999999999 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908076+0000", 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026105799999999999 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908089+0000", 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2847000000000001e-05 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908126+0000", 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7177e-05 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913917+0000", 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057903410000000001 2024-09-21T19:02:10.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913979+0000", 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2253000000000002e-05 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.914099+0000", 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120463 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237756 8.22 8:47738879:::benchmark_data_smithi081_105538_object105524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.952846365999999, 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.228195653, 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237756 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694417+0000", 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7199999999999999e-07 2024-09-21T19:02:10.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694419+0000", 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.447e-06 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694425+0000", 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4689999999999996e-06 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694430+0000", 2024-09-21T19:02:10.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9099999999999996e-06 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701670+0000", 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0072406739999999999 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701679+0000", 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:10.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908648+0000", 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2069698309999999 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908761+0000", 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011227500000000001 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,3,4,5,7", 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908968+0000", 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020707700000000001 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912465+0000", 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003497207 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912477+0000", 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2086000000000001e-05 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913031+0000", 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055408400000000004 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913427+0000", 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039556599999999997 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913488+0000", 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1543000000000004e-05 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922372+0000", 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008883574 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922440+0000", 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7909000000000003e-05 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922612+0000", 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017206799999999999 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:10.791 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:02:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.793 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:10.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:10.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:10.922 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:10 smithi081 ceph-mon[29384]: pgmap v434: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T19:02:10.922 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2409761048' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:10.923 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:10 smithi081 ceph-mon[39192]: pgmap v434: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T19:02:10.923 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:10 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2409761048' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:10.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:11.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:10 smithi104 ceph-mon[32438]: pgmap v434: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T19:02:11.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:10 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2409761048' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:11.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:11.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:11.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:11.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:11.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:11.341 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:11.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:11.478 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:11.480 INFO:teuthology.orchestra.run.smithi104.stderr:Exporting 5.f info 5.f( empty local-lis/les=61/62 n=0 ec=61/61 lis/c=61/61 les/c/f=62/62/0 sis=61) 2024-09-21T19:02:11.483 INFO:teuthology.orchestra.run.smithi104.stderr:Export successful 2024-09-21T19:02:11.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:11.607 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:11.614 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:02:11.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:11.734 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:11 smithi104 ceph-mon[32438]: 8.23 scrub starts 2024-09-21T19:02:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:11 smithi104 ceph-mon[32438]: 8.23 scrub ok 2024-09-21T19:02:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:11 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:02:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2409761048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:11 smithi104 ceph-mon[32438]: osdmap e249: 8 total, 7 up, 8 in 2024-09-21T19:02:11.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.877 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:11.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:11.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:11.998 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:11.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[39192]: 8.23 scrub starts 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[39192]: 8.23 scrub ok 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2409761048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[39192]: osdmap e249: 8 total, 7 up, 8 in 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[29384]: 8.23 scrub starts 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[29384]: 8.23 scrub ok 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:02:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2409761048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:11 smithi081 ceph-mon[29384]: osdmap e249: 8 total, 7 up, 8 in 2024-09-21T19:02:12.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:12.140 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:12.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:12.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.414 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:12.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:12.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.550 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:12.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:12.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.6 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 5.f 2024-09-21T19:02:12.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:12.684 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:12.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:12.868 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.6/config 2024-09-21T19:02:12.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:12.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:12.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:12.998 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:12 smithi104 ceph-mon[32438]: pgmap v437: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T19:02:12.999 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:12 smithi104 ceph-mon[32438]: 6.2 scrub starts 2024-09-21T19:02:12.999 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:12 smithi104 ceph-mon[32438]: 6.2 scrub ok 2024-09-21T19:02:13.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:12 smithi081 ceph-mon[39192]: pgmap v437: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T19:02:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:12 smithi081 ceph-mon[39192]: 6.2 scrub starts 2024-09-21T19:02:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:12 smithi081 ceph-mon[39192]: 6.2 scrub ok 2024-09-21T19:02:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:12 smithi081 ceph-mon[29384]: pgmap v437: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 1.0 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 3549/50586 objects degraded (7.016%); 11092/50586 objects misplaced (21.927%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T19:02:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:12 smithi081 ceph-mon[29384]: 6.2 scrub starts 2024-09-21T19:02:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:12 smithi081 ceph-mon[29384]: 6.2 scrub ok 2024-09-21T19:02:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.092 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:13.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:13.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:13.230 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:13.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:13.367 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:13.469 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:13.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:13.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.775 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:13.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:13.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:13.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:13.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:13.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:02:14.016 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:14.048 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:14.086 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:02:14.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:14.180 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:14.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:14.309 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:14.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:14.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:14.539 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:14.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:14.623 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:14.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[39192]: pgmap v438: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 574 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 1997/28116 objects degraded (7.103%); 6259/28116 objects misplaced (22.261%) 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 1997/28116 objects degraded (7.103%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[39192]: osdmap e250: 8 total, 7 up, 8 in 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[29384]: pgmap v438: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 574 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 1997/28116 objects degraded (7.103%); 6259/28116 objects misplaced (22.261%) 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 1997/28116 objects degraded (7.103%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:14.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:14 smithi081 ceph-mon[29384]: osdmap e250: 8 total, 7 up, 8 in 2024-09-21T19:02:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.767 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:14.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:14.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:14.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:14 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T19:02:14.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:14 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T19:02:14.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:14 smithi104 ceph-mon[32438]: pgmap v438: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 574 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 1997/28116 objects degraded (7.103%); 6259/28116 objects misplaced (22.261%) 2024-09-21T19:02:14.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:14 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 1997/28116 objects degraded (7.103%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:14.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:14 smithi104 ceph-mon[32438]: osdmap e250: 8 total, 7 up, 8 in 2024-09-21T19:02:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:14.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:15.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:15.150 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:15.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:15.176 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:15.220 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:15.258 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:15.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:15.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:15.292 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:15.292 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:15.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:15.324 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:15.324 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:15.359 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:15.359 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:15.390 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:15.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:15.422 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:15.422 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:15.443 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:15.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:15.581 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:15.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.717 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:15.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:15.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:15.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:15.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:15.989 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:15.989 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:15.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:15 smithi081 ceph-mon[39192]: 8.1a scrub starts 2024-09-21T19:02:15.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:15 smithi081 ceph-mon[39192]: 8.1a scrub ok 2024-09-21T19:02:15.989 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/395943944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:15.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:15 smithi081 ceph-mon[29384]: 8.1a scrub starts 2024-09-21T19:02:15.989 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:15 smithi081 ceph-mon[29384]: 8.1a scrub ok 2024-09-21T19:02:15.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/395943944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:15.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:15.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:16.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:15 smithi104 ceph-mon[32438]: 8.1a scrub starts 2024-09-21T19:02:16.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:15 smithi104 ceph-mon[32438]: 8.1a scrub ok 2024-09-21T19:02:16.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/395943944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:16.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:16.133 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:16.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:16.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:16.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:16.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:16.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:16.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:16.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:02:16.542 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:16.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:16.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.836 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:16.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:16.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:16.975 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:16.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: 3.12 scrub starts 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: 3.12 scrub ok 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: pgmap v440: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 574 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 1997/28116 objects degraded (7.103%); 6259/28116 objects misplaced (22.261%) 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: 5.c scrub starts 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: 5.c scrub ok 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:02:17.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:16 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: 3.12 scrub starts 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: 3.12 scrub ok 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: pgmap v440: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 574 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 1997/28116 objects degraded (7.103%); 6259/28116 objects misplaced (22.261%) 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: 5.c scrub starts 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: 5.c scrub ok 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:02:17.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: 3.12 scrub starts 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: 3.12 scrub ok 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: pgmap v440: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 574 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 1997/28116 objects degraded (7.103%); 6259/28116 objects misplaced (22.261%) 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: 5.c scrub starts 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: 5.c scrub ok 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:02:17.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:16 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:02:17.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:17.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:17.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:17.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:17.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:17.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:17.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:17.524 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:17.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:02:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:02:17.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:17.659 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:17.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:17.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:17.874 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Removed 132229 objects 2024-09-21T19:02:17.874 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Clean up completed and total clean up time :115.063 2024-09-21T19:02:17.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:17.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:17 smithi104 ceph-mon[32438]: 8.16 scrub starts 2024-09-21T19:02:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:17 smithi104 ceph-mon[32438]: 8.16 scrub ok 2024-09-21T19:02:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:17 smithi104 ceph-mon[32438]: 1.2 scrub starts 2024-09-21T19:02:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:17 smithi104 ceph-mon[32438]: 1.2 scrub ok 2024-09-21T19:02:18.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:18.037 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[29384]: 8.16 scrub starts 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[29384]: 8.16 scrub ok 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[29384]: 1.2 scrub starts 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[29384]: 1.2 scrub ok 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[39192]: 8.16 scrub starts 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[39192]: 8.16 scrub ok 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[39192]: 1.2 scrub starts 2024-09-21T19:02:18.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:17 smithi081 ceph-mon[39192]: 1.2 scrub ok 2024-09-21T19:02:18.139 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:18.160 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:18.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:02:18.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:18.266 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:18.282 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:18.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:18.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:18.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:02:18.476 INFO:teuthology.orchestra.run.smithi104.stdout: marking collection for removal 2024-09-21T19:02:18.476 INFO:teuthology.orchestra.run.smithi104.stdout:setting '_remove' omap key 2024-09-21T19:02:18.476 INFO:teuthology.orchestra.run.smithi104.stdout:finish_remove_pgs 5.f_head removing 5.f 2024-09-21T19:02:18.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:02:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:02:18.476 INFO:teuthology.orchestra.run.smithi104.stdout:Remove successful 2024-09-21T19:02:18.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:18.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:18.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:18.651 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:18.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:18 smithi104 ceph-mon[32438]: pgmap v441: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 96 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 413/5163 objects degraded (7.999%); 1304/5163 objects misplaced (25.257%) 2024-09-21T19:02:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:18 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:18 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:02:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:18 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:02:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:18 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:18.862 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:18.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:18.898 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:18.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[29384]: pgmap v441: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 96 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 413/5163 objects degraded (7.999%); 1304/5163 objects misplaced (25.257%) 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[39192]: pgmap v441: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 96 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 413/5163 objects degraded (7.999%); 1304/5163 objects misplaced (25.257%) 2024-09-21T19:02:18.983 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:18.984 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:02:18.984 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:02:18.984 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:18 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:18.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:19.021 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:19.122 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:19.136 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:19.144 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:19.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:19.268 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:19.366 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:19.367 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:19.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:19.390 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:19.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:19.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:19.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:19.730 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:02:19.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:19.757 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:19.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:19.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:19.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:19.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.6 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.f.6 2024-09-21T19:02:19.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:20.002 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:20.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 6.0 scrub starts 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 6.0 scrub ok 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 8.f scrub starts 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 8.f scrub ok 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 5.1 scrub starts 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 5.1 scrub ok 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3656241984' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:02:20.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:19 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 413/5163 objects degraded (7.999%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 6.0 scrub starts 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 6.0 scrub ok 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 8.f scrub starts 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 8.f scrub ok 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T19:02:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 5.1 scrub starts 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 5.1 scrub ok 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3656241984' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 413/5163 objects degraded (7.999%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 6.0 scrub starts 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 6.0 scrub ok 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 8.f scrub starts 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 8.f scrub ok 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T19:02:20.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 5.1 scrub starts 2024-09-21T19:02:20.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 5.1 scrub ok 2024-09-21T19:02:20.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T19:02:20.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T19:02:20.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3656241984' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:02:20.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:19 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 413/5163 objects degraded (7.999%), 26 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:20.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:20.119 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.6/config 2024-09-21T19:02:20.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:20.176 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T19:02:20.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T19:02:20.226 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:20.249 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:20.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:20.373 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:20.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:20.496 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:20.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:20.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:20.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:20.743 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:20.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:20.846 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:20.847 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:20.847 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:20.847 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:20.847 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:20.847 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:20.847 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:20.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:20.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:20.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: 6.10 scrub starts 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: 6.10 scrub ok 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: pgmap v442: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 96 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 413/5163 objects degraded (7.999%); 1304/5163 objects misplaced (25.257%) 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:21.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3656241984' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:02:21.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: osdmap e251: 8 total, 7 up, 8 in 2024-09-21T19:02:21.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2454460159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: 6.10 scrub starts 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: 6.10 scrub ok 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: pgmap v442: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 96 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 413/5163 objects degraded (7.999%); 1304/5163 objects misplaced (25.257%) 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3656241984' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: osdmap e251: 8 total, 7 up, 8 in 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2454460159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: 6.10 scrub starts 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: 6.10 scrub ok 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: pgmap v442: 178 pgs: 3 active+clean+remapped, 30 active+undersized, 26 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 96 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 413/5163 objects degraded (7.999%); 1304/5163 objects misplaced (25.257%) 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3656241984' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:02:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: osdmap e251: 8 total, 7 up, 8 in 2024-09-21T19:02:21.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2454460159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:21.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:21.111 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:21.213 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:21.234 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:21.336 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:21.357 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:21.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:21.514 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:02:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:21.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:21.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.604 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:21.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:21.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:21.725 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:21 smithi081 ceph-mon[29384]: 8.11 scrub starts 2024-09-21T19:02:21.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:21 smithi081 ceph-mon[39192]: 8.11 scrub starts 2024-09-21T19:02:21.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:21.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:21.848 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:21.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:21.970 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:21 smithi104 ceph-mon[32438]: 8.11 scrub starts 2024-09-21T19:02:21.970 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:21.972 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:21.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:21.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:02:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:22.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.097 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:22.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:22.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:22.219 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:22.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.343 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:22.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:22.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:22.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:22.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:22.587 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:22.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:22.711 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:22.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:02:22.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:22.815 INFO:teuthology.orchestra.run.smithi104.stdout:get_pg_num_history pg_num_history pg_num_history(e238 pg_nums {1={19=1,168=11},2={56=3},3={57=32},4={59=32},5={61=32},6={63=32},7={79=16},8={172=16,222=26,236=36}} deleted_pools 170,7) 2024-09-21T19:02:22.815 INFO:teuthology.orchestra.run.smithi104.stdout:Importing pgid 5.f 2024-09-21T19:02:22.815 INFO:teuthology.orchestra.run.smithi104.stdout:write_pg epoch 216 info 5.f( empty local-lis/les=61/62 n=0 ec=61/61 lis/c=61/61 les/c/f=62/62/0 sis=61) 2024-09-21T19:02:22.815 INFO:teuthology.orchestra.run.smithi104.stdout:Import successful 2024-09-21T19:02:22.828 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_1' removed 2024-09-21T19:02:22.833 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:22.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:22.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:22.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:23.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:22 smithi104 ceph-mon[32438]: pgmap v444: 178 pgs: 3 active+clean+remapped, 33 active+undersized, 23 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 72 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 249/4008 objects degraded (6.213%); 1134/4008 objects misplaced (28.293%) 2024-09-21T19:02:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2210029396' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:02:23.057 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:23.065 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:23.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:22 smithi081 ceph-mon[29384]: pgmap v444: 178 pgs: 3 active+clean+remapped, 33 active+undersized, 23 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 72 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 249/4008 objects degraded (6.213%); 1134/4008 objects misplaced (28.293%) 2024-09-21T19:02:23.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2210029396' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:02:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:22 smithi081 ceph-mon[39192]: pgmap v444: 178 pgs: 3 active+clean+remapped, 33 active+undersized, 23 active+undersized+degraded, 9 active+remapped+backfill_toofull, 110 active+clean; 72 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 249/4008 objects degraded (6.213%); 1134/4008 objects misplaced (28.293%) 2024-09-21T19:02:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2210029396' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:02:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:23.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:23.201 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:23.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:23.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:23.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:23.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:23.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:23.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:02:23.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:23.570 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:23.670 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T19:02:23.671 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T19:02:23.671 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T19:02:23.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T19:02:23.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:23.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:23.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2210029396' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[29384]: osdmap e252: 8 total, 7 up, 8 in 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2210029396' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:02:23.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:23 smithi081 ceph-mon[39192]: osdmap e252: 8 total, 7 up, 8 in 2024-09-21T19:02:23.817 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:23.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:23.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:23.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:23 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:23 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:23 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2210029396' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:02:24.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:23 smithi104 ceph-mon[32438]: osdmap e252: 8 total, 7 up, 8 in 2024-09-21T19:02:24.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:24.063 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:24.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:24.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:24.211 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:24.239 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:02:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:02:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:02:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:02:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.239 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136126 7.b 7:d13dafa2:::benchmark_data_smithi081_73391_object136125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.161062+0000", 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 396.07475692000003, 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139333067, 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136126 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161062+0000", 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161063+0000", 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3500000000000005e-07 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161084+0000", 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0755e-05 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161091+0000", 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2169999999999997e-06 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.161097+0000", 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3230000000000003e-06 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164270+0000", 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031733239999999999 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164391+0000", 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120575 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.164535+0000", 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014399899999999999 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170661+0000", 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061256419999999997 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170700+0000", 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9578e-05 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300264+0000", 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12956358800000001 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300355+0000", 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1219999999999997e-05 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.300395+0000", 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0011e-05 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:136128 7.b 7:df396259:::benchmark_data_smithi081_73391_object136127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T19:02:24.245 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:55:48.162704+0000", 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 396.07311437300001, 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.139089409, 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 136128 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162704+0000", 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162707+0000", 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1229999999999998e-06 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162731+0000", 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4485e-05 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162747+0000", 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5452999999999999e-05 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.162755+0000", 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9920000000000007e-06 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167113+0000", 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043581749999999997 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167237+0000", 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012398699999999999 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.167386+0000", 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149672 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170665+0000", 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032787960000000001 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.170739+0000", 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4108000000000003e-05 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301654+0000", 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13091487199999999 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301749+0000", 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5187999999999997e-05 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:55:48.301794+0000", 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4558000000000003e-05 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:242088 7.b 7:d4630f67:::benchmark_data_smithi081_73391_object103089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512627+0000", 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 274.72319160400002, 2024-09-21T19:02:24.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.165892709, 2024-09-21T19:02:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:02:24.252 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 242088 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512627+0000", 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512627+0000", 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512629+0000", 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0770000000000002e-06 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512636+0000", 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.562e-06 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512649+0000", 2024-09-21T19:02:24.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3529e-05 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512657+0000", 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5639999999999999e-06 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512723+0000", 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6489999999999995e-05 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-09-21T19:02:24.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512729+0000", 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0240000000000003e-06 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514154+0000", 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001424771 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.672882+0000", 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.158727757 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.673023+0000", 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014103799999999999 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677654+0000", 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046308169999999997 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678082+0000", 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000428572 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678431+0000", 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034839400000000003 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678483+0000", 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1996999999999997e-05 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678520+0000", 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7116999999999997e-05 2024-09-21T19:02:24.258 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242087 7.d e149/114 7:b80d1913:::benchmark_data_smithi081_73391_object103088:head v 149'15301, mlcod=148'15299)", 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.512853+0000", 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 274.72296551599999, 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16432387000000001, 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512853+0000", 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512853+0000", 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512854+0000", 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.052e-06 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512862+0000", 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3749999999999997e-06 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512871+0000", 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5920000000000007e-06 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512878+0000", 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6120000000000004e-06 2024-09-21T19:02:24.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512943+0000", 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4738000000000003e-05 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.512966+0000", 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3591000000000002e-05 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677136+0000", 2024-09-21T19:02:24.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16416932200000001 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.677177+0000", 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1588000000000002e-05 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.34742.0:242084 7.0 e149/79 7:0c19e5c0:::benchmark_data_smithi081_73391_object103085:head v 149'15371, mlcod=148'15368)", 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:49.514283+0000", 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 274.72153571199999, 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16410017099999999, 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.263 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514283+0000", 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5900000000000003e-07 2024-09-21T19:02:24.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514287+0000", 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9770000000000002e-06 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.514294+0000", 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4200000000000004e-06 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674533+0000", 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.160238726 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674644+0000", 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111369 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.674663+0000", 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9338e-05 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678317+0000", 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003654175 2024-09-21T19:02:24.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:49.678383+0000", 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5807000000000002e-05 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235882 8.1 8:87f52933:::benchmark_data_smithi081_105538_object103650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.657857+0000", 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.577962088, 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136503771, 2024-09-21T19:02:24.267 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235882 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657857+0000", 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657857+0000", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657859+0000", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9130000000000001e-06 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657864+0000", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1719999999999998e-06 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657873+0000", 2024-09-21T19:02:24.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.986e-06 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657880+0000", 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1760000000000004e-06 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.657928+0000", 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8341999999999999e-05 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.658040+0000", 2024-09-21T19:02:24.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011157799999999999 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.658327+0000", 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028722600000000002 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.664749+0000", 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006422245 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.671141+0000", 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063914030000000004 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.794347+0000", 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.123206445 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.794361+0000", 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3285000000000001e-05 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235888 8.a 8:559d0eda:::benchmark_data_smithi081_105538_object103656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.661333+0000", 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.574485371000002, 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.214254058, 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235888 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661333+0000", 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661333+0000", 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661336+0000", 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.057e-06 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661341+0000", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5929999999999996e-06 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661354+0000", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3074999999999999e-05 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661366+0000", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1574e-05 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661422+0000", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5794999999999997e-05 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661531+0000", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109153 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661631+0000", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100083 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679383+0000", 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017752674 2024-09-21T19:02:24.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.680125+0000", 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00074180600000000002 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.875573+0000", 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19544782099999999 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.875588+0000", 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4426999999999999e-05 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235893 8.a 8:56f22e24:::benchmark_data_smithi081_105538_object103661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.670986+0000", 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.564833298, 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20505188799999999, 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235893 2024-09-21T19:02:24.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.670986+0000", 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.670986+0000", 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.670988+0000", 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3089999999999998e-06 2024-09-21T19:02:24.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.670995+0000", 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5780000000000001e-06 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.671010+0000", 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4166e-05 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.671018+0000", 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4630000000000001e-06 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.671081+0000", 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3417000000000006e-05 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.671206+0000", 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000124938 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.671323+0000", 2024-09-21T19:02:24.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011643599999999999 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.680134+0000", 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008811625 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876028+0000", 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.195893767 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876037+0000", 2024-09-21T19:02:24.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1889999999999996e-06 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235882 8.21 8:87f52933:::benchmark_data_smithi081_105538_object103650:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.559241405000002, 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0591238139999999, 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.283 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235882 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:02:24.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676578+0000", 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4799999999999997e-07 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676581+0000", 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0920000000000002e-06 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676588+0000", 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3090000000000001e-06 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676594+0000", 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1589999999999998e-06 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.795011+0000", 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.118417017 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.795019+0000", 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0039999999999999e-06 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.735295+0000", 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.94027655099999996 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.735701+0000", 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040593399999999999 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.287 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235888 8.a 8:559d0eda:::benchmark_data_smithi081_105538_object103656:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.559207915999998, 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.075440073, 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.288 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235888 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676614+0000", 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3970000000000001e-06 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676619+0000", 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2859999999999999e-06 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676624+0000", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.04e-06 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679419+0000", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027943460000000001 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679424+0000", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.215e-06 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876061+0000", 2024-09-21T19:02:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19663730400000001 2024-09-21T19:02:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876071+0000", 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6500000000000008e-06 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.751965+0000", 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87589449500000005 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752051+0000", 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5783000000000002e-05 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235893 8.a 8:56f22e24:::benchmark_data_smithi081_105538_object103661:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.559187100999999, 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1078101929999999, 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235893 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2e-07 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676634+0000", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9439999999999999e-06 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676640+0000", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6949999999999997e-06 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676644+0000", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6739999999999996e-06 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679477+0000", 2024-09-21T19:02:24.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028324510000000002 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting_for_map not empty", 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679482+0000", 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7180000000000004e-06 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876097+0000", 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19661515199999999 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876105+0000", 2024-09-21T19:02:24.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.524e-06 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752411+0000", 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87630529599999996 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752544+0000", 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013372500000000001 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784273+0000", 2024-09-21T19:02:24.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031728734000000001 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784442+0000", 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016886000000000001 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235905 8.1a 8:5a65b146:::benchmark_data_smithi081_105538_object103673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.906174+0000", 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.329645058000001, 2024-09-21T19:02:24.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.88163811199999997, 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235905 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906174+0000", 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906174+0000", 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906176+0000", 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8339999999999999e-06 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906182+0000", 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5100000000000004e-06 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906194+0000", 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2233e-05 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906201+0000", 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6710000000000002e-06 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906261+0000", 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9639000000000003e-05 2024-09-21T19:02:24.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.906270+0000", 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7659999999999993e-06 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.781949+0000", 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87567904500000004 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.782045+0000", 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5985999999999999e-05 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.782282+0000", 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023656999999999999 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.783563+0000", 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012810479999999999 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784361+0000", 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079829100000000002 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.787729+0000", 2024-09-21T19:02:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033673309999999999 2024-09-21T19:02:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.787776+0000", 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7549000000000002e-05 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.787812+0000", 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5639000000000002e-05 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237696 8.1a 8:5de8ae1c:::benchmark_data_smithi081_105538_object105464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 27.542953547, 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2294143280000001, 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.307 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237696 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692868+0000", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3070000000000001e-06 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692875+0000", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9779999999999999e-06 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692894+0000", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9734000000000001e-05 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692906+0000", 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1218e-05 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693046+0000", 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140288 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693056+0000", 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0366999999999999e-05 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.918656+0000", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2255999339999999 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.918766+0000", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109454 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.919037+0000", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027156700000000001 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921749+0000", 2024-09-21T19:02:24.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027122779999999998 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921789+0000", 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9567000000000003e-05 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921942+0000", 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000152612 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922003+0000", 2024-09-21T19:02:24.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1637999999999995e-05 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922094+0000", 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0926000000000003e-05 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922144+0000", 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0142000000000002e-05 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922280+0000", 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013531799999999999 2024-09-21T19:02:24.313 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237743 8.1a 8:5f844437:::benchmark_data_smithi081_105538_object105511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 27.542889483, 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2389039639999999, 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.314 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237743 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692930+0000", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4099999999999998e-07 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692933+0000", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.371e-06 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692940+0000", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4679999999999997e-06 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692945+0000", 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5600000000000001e-06 2024-09-21T19:02:24.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693080+0000", 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000134232 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693085+0000", 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.57e-06 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922497+0000", 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.229411636 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922649+0000", 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015161100000000001 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922904+0000", 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025589599999999998 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931394+0000", 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084900319999999998 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931456+0000", 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1847000000000006e-05 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931520+0000", 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3540999999999994e-05 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931559+0000", 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9418000000000001e-05 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931652+0000", 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3154999999999998e-05 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931710+0000", 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7414999999999998e-05 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931833+0000", 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123471 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237758 8.1a 8:5b1e9ef3:::benchmark_data_smithi081_105538_object105526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 27.542866353000001, 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2525915910000001, 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237758 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692953+0000", 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7999999999999996e-07 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692955+0000", 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3410000000000001e-06 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692961+0000", 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2209999999999996e-06 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692965+0000", 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7990000000000001e-06 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693105+0000", 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013992799999999999 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693111+0000", 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924108+0000", 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.230996792 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924214+0000", 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106278 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924401+0000", 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018696000000000001 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931404+0000", 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0070028060000000003 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931875+0000", 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00047151400000000003 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931917+0000", 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1340000000000001e-05 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.932034+0000", 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011719500000000001 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945070+0000", 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013035573 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945131+0000", 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1702e-05 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945544+0000", 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041286200000000001 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.327 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:239924 8.23 8:c6229102:::benchmark_data_smithi081_105538_object107692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:59.542645+0000", 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 24.693173731000002, 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.14318617, 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 239924 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.328 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542645+0000", 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542645+0000", 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542647+0000", 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.48e-06 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542651+0000", 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3460000000000004e-06 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542659+0000", 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8450000000000003e-06 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542665+0000", 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.835e-06 2024-09-21T19:02:24.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.542896+0000", 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023095799999999999 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.543046+0000", 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015003600000000001 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.543222+0000", 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000175928 2024-09-21T19:02:24.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545666+0000", 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024445550000000002 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.683266+0000", 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13759970099999999 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.685627+0000", 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002361274 2024-09-21T19:02:24.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.685683+0000", 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6187999999999998e-05 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.685831+0000", 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014802399999999999 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.333 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:239929 8.a 8:53a2def4:::benchmark_data_smithi081_105538_object107697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:59.545923+0000", 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 24.689895986, 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13676427099999999, 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 239929 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545923+0000", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545923+0000", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545924+0000", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6130000000000001e-06 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545929+0000", 2024-09-21T19:02:24.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6500000000000004e-06 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545940+0000", 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0455000000000001e-05 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.545947+0000", 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9000000000000006e-06 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546265+0000", 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031754899999999998 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546377+0000", 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011192399999999999 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,5", 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546675+0000", 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000298063 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.549289+0000", 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026141070000000001 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.682336+0000", 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13304733299999999 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.682596+0000", 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025994599999999999 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.682647+0000", 2024-09-21T19:02:24.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0497000000000001e-05 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.682687+0000", 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0234000000000001e-05 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:239933 8.7 8:e34690ca:::benchmark_data_smithi081_105538_object107701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:59.546537+0000", 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 24.689281900000001, 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.136760151, 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.339 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 239933 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546537+0000", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546537+0000", 2024-09-21T19:02:24.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546538+0000", 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8899999999999998e-07 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546541+0000", 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3869999999999999e-06 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546548+0000", 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1069999999999999e-06 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546554+0000", 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4070000000000001e-06 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546603+0000", 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8798000000000003e-05 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.546704+0000", 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101816 2024-09-21T19:02:24.342 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3", 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547008+0000", 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000303858 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.549254+0000", 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002245679 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.683083+0000", 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13382930300000001 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.683217+0000", 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013355299999999999 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.683297+0000", 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0353999999999997e-05 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.35258.0:239924 8.23 e240/239 ondisk, result = 0)", 2024-09-21T19:02:24.344 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:59.547940+0000", 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 24.687878494, 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13789662499999999, 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547940+0000", 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547940+0000", 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547941+0000", 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8000000000000004e-07 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547945+0000", 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9210000000000002e-06 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547954+0000", 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0180000000000007e-06 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.547960+0000", 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9329999999999998e-06 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.685611+0000", 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.13765113600000001 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.685620+0000", 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2029999999999998e-06 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.685837+0000", 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000216534 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:239937 8.11 8:89170039:::benchmark_data_smithi081_105538_object107705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:59.549531+0000", 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 24.686287721999999, 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.15898125699999999, 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:02:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 239937 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.549531+0000", 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:02:24.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.549531+0000", 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.549533+0000", 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.469e-06 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.549538+0000", 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0390000000000004e-06 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.683777+0000", 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.134239096 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.683788+0000", 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1165000000000001e-05 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.694158+0000", 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010370538 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.694275+0000", 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116328 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.694569+0000", 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029426700000000002 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.700115+0000", 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055460120000000003 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.700130+0000", 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4837999999999999e-05 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.700165+0000", 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5074000000000003e-05 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.700207+0000", 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2449e-05 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.707893+0000", 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0076860569999999996 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.708306+0000", 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000412772 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:59.708512+0000", 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020615299999999999 2024-09-21T19:02:24.354 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.355 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.355 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.355 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:24.355 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:24.355 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:02:24.380 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:24.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:24.503 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:24.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:24.630 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:24.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:24.755 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:24.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:24.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:24.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:25.001 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:25.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:24 smithi104 ceph-mon[32438]: pgmap v446: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:24 smithi104 ceph-mon[32438]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull) 2024-09-21T19:02:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:24 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 77/597 objects degraded (12.898%), 19 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:25.012 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:24 smithi081 ceph-mon[29384]: pgmap v446: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:25.012 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:24 smithi081 ceph-mon[29384]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull) 2024-09-21T19:02:25.012 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:24 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 77/597 objects degraded (12.898%), 19 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:25.013 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:24 smithi081 ceph-mon[39192]: pgmap v446: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:25.013 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:24 smithi081 ceph-mon[39192]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull) 2024-09-21T19:02:25.013 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:24 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 77/597 objects degraded (12.898%), 19 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:25.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:25.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:25.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:25.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:25.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/exp.5.f.6 2024-09-21T19:02:25.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:25.374 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:25.475 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:25.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:25.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:25.623 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:25.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:25.749 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:02:25.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:25.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:25.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:25.996 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:25.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:26.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2652194360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:26.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2652194360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:26.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:26.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:26.245 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:26.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:02:26.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2652194360' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:26.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:26.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:26.597 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:26.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:26.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:26.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:26.868 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:26.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:26.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:26.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:27.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[39192]: pgmap v447: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:27.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:27.017 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2652194360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:27.018 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[39192]: osdmap e253: 8 total, 7 up, 8 in 2024-09-21T19:02:27.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[29384]: pgmap v447: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:27.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:27.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2652194360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:27.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:26 smithi081 ceph-mon[29384]: osdmap e253: 8 total, 7 up, 8 in 2024-09-21T19:02:27.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:27.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:27.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:26 smithi104 ceph-mon[32438]: pgmap v447: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:26 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2652194360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:26 smithi104 ceph-mon[32438]: osdmap e253: 8 total, 7 up, 8 in 2024-09-21T19:02:27.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:02:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:02:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:02:27.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:27.367 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:27.467 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:27.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:27.484 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:27.484 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:27.492 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:27.500 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:27.501 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:27.517 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:27.517 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:27.534 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:27.534 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:27.550 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:27.551 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:27.551 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:27.568 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:27.568 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:27.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:27.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:27.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:27.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:27.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:27.865 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:27.898 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:27.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:27.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:27 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3433636671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:27 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3433636671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:28.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:28.098 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:27 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3433636671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:28.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:28.212 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:28.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:28.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:28.364 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:28.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:28.491 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:28.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:02:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:02:28.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:28.619 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:28.720 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:28.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:28.771 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_2' created 2024-09-21T19:02:28.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:28.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:02:28.872 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:28.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:28.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:28 smithi081 ceph-mon[29384]: pgmap v449: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:28 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4210156831' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T19:02:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:28 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:28 smithi081 ceph-mon[39192]: pgmap v449: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:28 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4210156831' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T19:02:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:28 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:29.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:29.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:29.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:29.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:28 smithi104 ceph-mon[32438]: pgmap v449: 142 pgs: 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:28 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4210156831' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T19:02:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:28 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:29.345 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:29.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:29.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:29.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:29.499 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:29.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:29.622 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:29.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:29.745 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:29.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.873 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:29.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:29.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:29.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:29.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4210156831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T19:02:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:29 smithi081 ceph-mon[29384]: osdmap e254: 8 total, 7 up, 8 in 2024-09-21T19:02:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4210156831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T19:02:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:29 smithi081 ceph-mon[39192]: osdmap e254: 8 total, 7 up, 8 in 2024-09-21T19:02:30.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:30.122 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:30.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T19:02:30.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:30.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:30.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4210156831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T19:02:30.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:29 smithi104 ceph-mon[32438]: osdmap e254: 8 total, 7 up, 8 in 2024-09-21T19:02:30.301 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-21T19:02:30.301 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:02:30.301 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-21T19:02:30.301 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-09-21T19:02:30.302 DEBUG:teuthology.orchestra.run.smithi104:osd.6> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T19:02:30.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl start ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6 2024-09-21T19:02:30.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:30.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:30.494 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:30.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:30.610 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 systemd[1]: Starting Ceph osd.6 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:02:30.617 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:30.695 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:30.695 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:30.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:30.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:30.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:30.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:30.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:30 smithi104 ceph-mon[32438]: pgmap v451: 158 pgs: 16 unknown, 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:30.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:30 smithi104 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:02:30.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:30 smithi104 ceph-mon[32438]: osdmap e255: 8 total, 7 up, 8 in 2024-09-21T19:02:30.913 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 podman[101032]: 2024-09-21 19:02:30.60839964 +0000 UTC m=+0.104670402 container create 1c8ee70b6ae5cc3b723adb4375ac6e5bc7840bc06f812809101dad3e25ae5f6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:02:30.913 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 podman[101032]: 2024-09-21 19:02:30.519581688 +0000 UTC m=+0.015852451 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:02:30.913 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 podman[101032]: 2024-09-21 19:02:30.819246079 +0000 UTC m=+0.315516845 container init 1c8ee70b6ae5cc3b723adb4375ac6e5bc7840bc06f812809101dad3e25ae5f6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T19:02:30.913 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 podman[101032]: 2024-09-21 19:02:30.822657863 +0000 UTC m=+0.318928635 container start 1c8ee70b6ae5cc3b723adb4375ac6e5bc7840bc06f812809101dad3e25ae5f6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:02:30.913 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 podman[101032]: 2024-09-21 19:02:30.910837906 +0000 UTC m=+0.407108694 container attach 1c8ee70b6ae5cc3b723adb4375ac6e5bc7840bc06f812809101dad3e25ae5f6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T19:02:30.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:30.986 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:30.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:31.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:30 smithi081 ceph-mon[39192]: pgmap v451: 158 pgs: 16 unknown, 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:31.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:30 smithi081 ceph-mon[39192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:02:31.047 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:30 smithi081 ceph-mon[39192]: osdmap e255: 8 total, 7 up, 8 in 2024-09-21T19:02:31.048 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:30 smithi081 ceph-mon[29384]: pgmap v451: 158 pgs: 16 unknown, 30 active+undersized, 19 active+undersized+degraded, 93 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 749 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:31.048 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:30 smithi081 ceph-mon[29384]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:02:31.048 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:30 smithi081 ceph-mon[29384]: osdmap e255: 8 total, 7 up, 8 in 2024-09-21T19:02:31.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:31.109 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:31.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:31.232 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:31.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:02:31.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 bash[101032]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:02:31.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:02:31.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:30 smithi104 bash[101032]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:02:31.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:31.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:02:31.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:31.478 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:31.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:31.603 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:31.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:31.726 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:31.736 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T19:02:31.736 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T19:02:31.736 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T19:02:31.736 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T19:02:31.736 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T19:02:31.736 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T19:02:31.828 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:02:31.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:31.896 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:31.935 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:31.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:31.973 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:31.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:32.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-mon[32438]: osdmap e256: 8 total, 7 up, 8 in 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate[101047]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 bash[101032]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-21T19:02:32.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:31 smithi104 podman[101032]: 2024-09-21 19:02:31.761251877 +0000 UTC m=+1.257522640 container died 1c8ee70b6ae5cc3b723adb4375ac6e5bc7840bc06f812809101dad3e25ae5f6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T19:02:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:31 smithi081 ceph-mon[29384]: osdmap e256: 8 total, 7 up, 8 in 2024-09-21T19:02:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:31 smithi081 ceph-mon[39192]: osdmap e256: 8 total, 7 up, 8 in 2024-09-21T19:02:32.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:32.096 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:32.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:32.224 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:32.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:32.348 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:32.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:32.470 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:32.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:32.594 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:32.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:32.718 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:32.756 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 podman[101269]: 2024-09-21 19:02:32.465316805 +0000 UTC m=+0.695035249 container remove 1c8ee70b6ae5cc3b723adb4375ac6e5bc7840bc06f812809101dad3e25ae5f6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T19:02:32.757 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 podman[101344]: 2024-09-21 19:02:32.66602618 +0000 UTC m=+0.104790191 container create 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T19:02:32.757 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 podman[101344]: 2024-09-21 19:02:32.577158036 +0000 UTC m=+0.015922056 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:02:32.780 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:32.795 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:02:32.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:32.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:32.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:32.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:32.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:32.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:02:33.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:32 smithi104 ceph-mon[32438]: pgmap v454: 158 pgs: 13 unknown, 30 active+undersized, 19 active+undersized+degraded, 96 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:33.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:32 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3677269101' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:33.057 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 podman[101344]: 2024-09-21 19:02:32.893630089 +0000 UTC m=+0.332394109 container init 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:02:33.058 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 podman[101344]: 2024-09-21 19:02:32.897171784 +0000 UTC m=+0.335935794 container start 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T19:02:33.058 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 bash[101344]: 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd 2024-09-21T19:02:33.058 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:32 smithi104 systemd[1]: Started Ceph osd.6 for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:02:33.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:32 smithi081 ceph-mon[29384]: pgmap v454: 158 pgs: 13 unknown, 30 active+undersized, 19 active+undersized+degraded, 96 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:33.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:32 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3677269101' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:32 smithi081 ceph-mon[39192]: pgmap v454: 158 pgs: 13 unknown, 30 active+undersized, 19 active+undersized+degraded, 96 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:33.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:32 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3677269101' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:02:33.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:33.089 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:33.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:33.213 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:33.222 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:33.299 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:02:33.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:33.338 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:33.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:33.445 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:33.445 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:33.445 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:33.445 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:33.459 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:33.476 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:33.476 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:33.493 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:33.494 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:33.511 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:33.511 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:33.511 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:33.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:33.529 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:33.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:33.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:33.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:no valid command found; 10 closest matches: 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:0 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:1 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:abort 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:assert 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:config diff 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:config diff get 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:config get 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:config help [] 2024-09-21T19:02:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:config set ... 2024-09-21T19:02:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:admin_socket: invalid command 2024-09-21T19:02:33.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:33.822 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T19:02:33.834 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:33.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:33.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:33 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3677269101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:02:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:33 smithi104 ceph-mon[32438]: osdmap e257: 8 total, 7 up, 8 in 2024-09-21T19:02:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2311800153' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:02:34.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1959554931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:34.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:34.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3677269101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[29384]: osdmap e257: 8 total, 7 up, 8 in 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2311800153' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1959554931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3677269101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[39192]: osdmap e257: 8 total, 7 up, 8 in 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2311800153' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:02:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1959554931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:34.080 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:34.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:34.202 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:34.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:34.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:34.448 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T19:02:34.496 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-21T19:02:34.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:34.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:34.696 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:34.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:34.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.822 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:34.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:34.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:34.935 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:02:34.935 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:02:34.935 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:02:34.935 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:02:34.935 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:02:34.935 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:34 smithi081 ceph-mon[29384]: pgmap v456: 158 pgs: 33 active+undersized, 19 active+undersized+degraded, 106 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:34.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2311800153' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:02:34.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:34 smithi081 ceph-mon[29384]: osdmap e258: 8 total, 7 up, 8 in 2024-09-21T19:02:34.936 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:34 smithi081 ceph-mon[39192]: pgmap v456: 158 pgs: 33 active+undersized, 19 active+undersized+degraded, 106 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:34.936 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2311800153' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:02:34.936 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:34 smithi081 ceph-mon[39192]: osdmap e258: 8 total, 7 up, 8 in 2024-09-21T19:02:34.944 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:34.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:35.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:35.067 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:35.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:35.189 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:34 smithi104 ceph-mon[32438]: pgmap v456: 158 pgs: 33 active+undersized, 19 active+undersized+degraded, 106 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2311800153' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:02:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:34 smithi104 ceph-mon[32438]: osdmap e258: 8 total, 7 up, 8 in 2024-09-21T19:02:35.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:02:34.803+0000 7f442bd9c640 -1 Falling back to public interface 2024-09-21T19:02:35.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:35.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:35.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:35.435 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:35.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:35.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:35.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:35.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:35.784 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:35.804 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:35.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:35.928 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:35.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:36.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[29384]: 3.5 scrub starts 2024-09-21T19:02:36.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[29384]: 3.5 scrub ok 2024-09-21T19:02:36.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:02:36.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:02:36.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T19:02:36.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T19:02:36.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[39192]: 3.5 scrub starts 2024-09-21T19:02:36.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[39192]: 3.5 scrub ok 2024-09-21T19:02:36.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:02:36.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:02:36.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T19:02:36.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:35 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T19:02:36.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:36.050 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:36.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:36.173 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:35 smithi104 ceph-mon[32438]: 3.5 scrub starts 2024-09-21T19:02:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:35 smithi104 ceph-mon[32438]: 3.5 scrub ok 2024-09-21T19:02:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:35 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:02:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:35 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:02:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:35 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T19:02:36.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:35 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T19:02:36.275 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:36.296 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:36.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:36.419 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:36.521 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.543 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:36.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:36.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.666 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:36.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:36.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:02:36.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:36.788 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:36.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:02:36.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:36.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:36.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:36.958 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:37.012 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:37.033 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: 6.11 scrub starts 2024-09-21T19:02:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: 6.11 scrub ok 2024-09-21T19:02:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: 3.1f scrub starts 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: 3.1f scrub ok 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: pgmap v458: 158 pgs: 33 active+undersized, 19 active+undersized+degraded, 106 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 881 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: 6.12 scrub starts 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: 6.12 scrub ok 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: 6.11 scrub starts 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: 6.11 scrub ok 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: 3.1f scrub starts 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: 3.1f scrub ok 2024-09-21T19:02:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: pgmap v458: 158 pgs: 33 active+undersized, 19 active+undersized+degraded, 106 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 881 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:02:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: 6.12 scrub starts 2024-09-21T19:02:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: 6.12 scrub ok 2024-09-21T19:02:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:37.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:37.156 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: 6.11 scrub starts 2024-09-21T19:02:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: 6.11 scrub ok 2024-09-21T19:02:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: 3.1f scrub starts 2024-09-21T19:02:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: 3.1f scrub ok 2024-09-21T19:02:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: pgmap v458: 158 pgs: 33 active+undersized, 19 active+undersized+degraded, 106 active+clean; 687 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 881 B/s rd, 0 op/s; 77/597 objects degraded (12.898%) 2024-09-21T19:02:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:02:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: 6.12 scrub starts 2024-09-21T19:02:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: 6.12 scrub ok 2024-09-21T19:02:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:37.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:37.279 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:37.349 DEBUG:teuthology.orchestra.run.smithi081:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T19:02:37.351 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T19:02:37.352 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T19:02:37.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:37.400 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:hints = 1 2024-09-21T19:02:37.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T19:02:37.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object prefix: benchmark_data_smithi081_131274 2024-09-21T19:02:37.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:02:37.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 0 0 0 0 0 0 - 0 2024-09-21T19:02:37.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:37.429 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:02:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:02:37.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:37.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:37.614 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:37.631 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:37.631 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:37.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:37.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:02:37.669 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:37.670 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:37.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:37.706 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:37.706 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:37.741 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:37.741 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:37.779 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:37.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:37.814 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:37.814 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:37.814 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T19:02:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:37.854 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:37.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:37.967 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:37.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: 5.13 scrub starts 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: 5.13 scrub ok 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: 6.17 scrub starts 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: 6.17 scrub ok 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3359168099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:37.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: 5.13 scrub starts 2024-09-21T19:02:37.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: 5.13 scrub ok 2024-09-21T19:02:37.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: 6.17 scrub starts 2024-09-21T19:02:37.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: 6.17 scrub ok 2024-09-21T19:02:37.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T19:02:37.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T19:02:37.992 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3359168099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:38.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:38.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: 5.13 scrub starts 2024-09-21T19:02:38.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: 5.13 scrub ok 2024-09-21T19:02:38.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: 6.17 scrub starts 2024-09-21T19:02:38.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: 6.17 scrub ok 2024-09-21T19:02:38.109 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T19:02:38.109 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T19:02:38.109 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3359168099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:38.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:38.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:38.249 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:38.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:38.385 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:38.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 1 16 3460 3444 215.238 215.25 0.00652689 0.00463127 2024-09-21T19:02:38.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:38.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:02:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:38.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:38.763 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.801 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:38.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:02:38.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:02:38.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:38.937 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:38.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:39.038 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: pgmap v459: 158 pgs: 32 active+undersized, 20 active+undersized+degraded, 106 active+clean; 4.9 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 630 KiB/s wr, 11 op/s; 94/798 objects degraded (11.779%) 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 94/798 objects degraded (11.779%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: osdmap e259: 8 total, 7 up, 8 in 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: from='osd.6 v1:172.21.15.104:6808/2795996229' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T19:02:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: pgmap v459: 158 pgs: 32 active+undersized, 20 active+undersized+degraded, 106 active+clean; 4.9 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 630 KiB/s wr, 11 op/s; 94/798 objects degraded (11.779%) 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 94/798 objects degraded (11.779%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: osdmap e259: 8 total, 7 up, 8 in 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: from='osd.6 v1:172.21.15.104:6808/2795996229' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:38 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T19:02:39.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:39.163 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:39.163 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:39.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:39.201 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:39.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:02:38.817+0000 7f442bd9c640 -1 osd.6 238 log_to_monitors true 2024-09-21T19:02:39.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:02:38.933+0000 7f441fb3e640 -1 osd.6 260 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T19:02:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: pgmap v459: 158 pgs: 32 active+undersized, 20 active+undersized+degraded, 106 active+clean; 4.9 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 630 KiB/s wr, 11 op/s; 94/798 objects degraded (11.779%) 2024-09-21T19:02:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 94/798 objects degraded (11.779%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: osdmap e259: 8 total, 7 up, 8 in 2024-09-21T19:02:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: from='osd.6 v1:172.21.15.104:6808/2795996229' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T19:02:39.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:38 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T19:02:39.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:39.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:39.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 2 16 6929 6913 216.016 216.812 0.00373307 0.00461934 2024-09-21T19:02:39.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:39.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:39.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:02:39.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:39.587 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:39.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:39.720 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:39.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:39.842 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:02:39.849 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:39.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:39.972 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:39.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:40.024 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:40.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.096 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:40.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:40.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: 5.5 scrub starts 2024-09-21T19:02:40.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: 5.5 scrub ok 2024-09-21T19:02:40.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T19:02:40.127 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T19:02:40.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T19:02:40.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: from='osd.6 v1:172.21.15.104:6808/2795996229' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:02:40.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: osdmap e260: 8 total, 7 up, 8 in 2024-09-21T19:02:40.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:39 smithi104 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:02:40.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: 5.5 scrub starts 2024-09-21T19:02:40.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: 5.5 scrub ok 2024-09-21T19:02:40.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T19:02:40.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T19:02:40.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T19:02:40.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: from='osd.6 v1:172.21.15.104:6808/2795996229' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: osdmap e260: 8 total, 7 up, 8 in 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[39192]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: 5.5 scrub starts 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: 5.5 scrub ok 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: from='osd.6 v1:172.21.15.104:6808/2795996229' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: osdmap e260: 8 total, 7 up, 8 in 2024-09-21T19:02:40.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:39 smithi081 ceph-mon[29384]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:02:40.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:40.218 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:40.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:40.340 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:40.374 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:02:40.374 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:02:40.374 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T19:02:40.374 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:02:40.401 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 3 16 8036 8020 167.069 69.1875 0.00132258 0.00498503 2024-09-21T19:02:40.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:40.463 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:40.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:02:40.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:40.590 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:40.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:40.716 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:40.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:40.841 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:40.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:02:40.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:40.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:40.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:41.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:41.128 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:02:41.128 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:02:41.128 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: pgmap v462: 158 pgs: 32 active+undersized, 20 active+undersized+degraded, 106 active+clean; 4.9 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 714 KiB/s wr, 11 op/s; 94/798 objects degraded (11.779%) 2024-09-21T19:02:41.128 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 4.16 scrub starts 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 4.16 scrub ok 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: osd.6 v1:172.21.15.104:6808/2795996229 boot 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 5.4 scrub starts 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: 5.4 scrub ok 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[29384]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T19:02:41.129 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: pgmap v462: 158 pgs: 32 active+undersized, 20 active+undersized+degraded, 106 active+clean; 4.9 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 714 KiB/s wr, 11 op/s; 94/798 objects degraded (11.779%) 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 4.16 scrub starts 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 4.16 scrub ok 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: osd.6 v1:172.21.15.104:6808/2795996229 boot 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 5.4 scrub starts 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: 5.4 scrub ok 2024-09-21T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:40 smithi081 ceph-mon[39192]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T19:02:41.170 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:41.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: pgmap v462: 158 pgs: 32 active+undersized, 20 active+undersized+degraded, 106 active+clean; 4.9 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 714 KiB/s wr, 11 op/s; 94/798 objects degraded (11.779%) 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 4.16 scrub starts 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 4.16 scrub ok 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: osd.6 v1:172.21.15.104:6808/2795996229 boot 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T19:02:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 5.4 scrub starts 2024-09-21T19:02:41.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: 5.4 scrub ok 2024-09-21T19:02:41.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:40 smithi104 ceph-mon[32438]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T19:02:41.314 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:41.338 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:41.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 4 16 8036 8020 125.302 0 - 0.00498503 2024-09-21T19:02:41.402 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:41.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:41.460 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:41.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:41.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:41.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:41.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:41.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:41.830 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:02:41.893 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:41.893 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:41.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:41.954 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:41.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:42.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: 3.1c scrub starts 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: 3.1c scrub ok 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: 6.15 scrub starts 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: 6.15 scrub ok 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3777765155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3777765155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[39192]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: 3.1c scrub starts 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: 3.1c scrub ok 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: 6.15 scrub starts 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: 6.15 scrub ok 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3777765155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3777765155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:41 smithi081 ceph-mon[29384]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T19:02:42.077 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:42.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.218 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:42.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: 3.1c scrub starts 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: 3.1c scrub ok 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: 6.15 scrub starts 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: 6.15 scrub ok 2024-09-21T19:02:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3777765155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:02:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:02:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3777765155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:02:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:41 smithi104 ceph-mon[32438]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T19:02:42.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:42.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:42.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 5 16 8767 8751 109.378 22.8438 0.00707244 0.00913515 2024-09-21T19:02:42.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:42.493 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:42.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:42.630 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:42.732 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:42.765 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:42.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:42.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:02:42.897 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:42.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:42.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:43.031 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[29384]: 4.10 scrub starts 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[29384]: 4.10 scrub ok 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[29384]: pgmap v465: 158 pgs: 21 active+undersized, 13 active+undersized+degraded, 124 active+clean; 99 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 375 op/s; 66/5295 objects degraded (1.246%) 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[39192]: 4.10 scrub starts 2024-09-21T19:02:43.069 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[39192]: 4.10 scrub ok 2024-09-21T19:02:43.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[39192]: pgmap v465: 158 pgs: 21 active+undersized, 13 active+undersized+degraded, 124 active+clean; 99 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 375 op/s; 66/5295 objects degraded (1.246%) 2024-09-21T19:02:43.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:43.070 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:43.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:43.154 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:43.155 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:43.162 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:43.194 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:43.194 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:43.229 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:43.229 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:42 smithi104 ceph-mon[32438]: 4.10 scrub starts 2024-09-21T19:02:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:42 smithi104 ceph-mon[32438]: 4.10 scrub ok 2024-09-21T19:02:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:42 smithi104 ceph-mon[32438]: pgmap v465: 158 pgs: 21 active+undersized, 13 active+undersized+degraded, 124 active+clean; 99 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 375 op/s; 66/5295 objects degraded (1.246%) 2024-09-21T19:02:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:43.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:43.265 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:43.265 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:43.296 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:43.298 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:43.332 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:43.332 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:43.365 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:02:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T19:02:43.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:43.399 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:43.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 6 16 11390 11374 118.469 163.938 0.00413802 0.00843142 2024-09-21T19:02:43.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:43.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:43.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:43.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:43.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:43.844 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:43.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:43.984 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:43.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T19:02:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T19:02:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[39192]: 4.1 scrub starts 2024-09-21T19:02:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[39192]: 4.1 scrub ok 2024-09-21T19:02:44.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2345619976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:44.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T19:02:44.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T19:02:44.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[29384]: 4.1 scrub starts 2024-09-21T19:02:44.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[29384]: 4.1 scrub ok 2024-09-21T19:02:44.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:43 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2345619976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:44.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:44.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:44.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:44.249 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:43 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T19:02:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:43 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T19:02:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:43 smithi104 ceph-mon[32438]: 4.1 scrub starts 2024-09-21T19:02:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:43 smithi104 ceph-mon[32438]: 4.1 scrub ok 2024-09-21T19:02:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:43 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2345619976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:44.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:44.386 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:44.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 7 16 14053 14037 125.32 166.438 0.00318783 0.00797245 2024-09-21T19:02:44.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.528 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:44.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:44.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:44.666 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:44.674 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:44.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:44.802 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:44.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:44.945 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:44.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:45.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[29384]: pgmap v467: 158 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 145 active+clean; 515 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 1474/25299 objects degraded (5.826%); 541 KiB/s, 8 objects/s recovering 2024-09-21T19:02:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[29384]: 3.3 scrub starts 2024-09-21T19:02:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[29384]: 3.3 scrub ok 2024-09-21T19:02:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 1474/25299 objects degraded (5.826%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:45.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[39192]: pgmap v467: 158 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 145 active+clean; 515 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 1474/25299 objects degraded (5.826%); 541 KiB/s, 8 objects/s recovering 2024-09-21T19:02:45.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[39192]: 3.3 scrub starts 2024-09-21T19:02:45.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[39192]: 3.3 scrub ok 2024-09-21T19:02:45.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:44 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 1474/25299 objects degraded (5.826%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:45.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:45.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.230 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:45.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:44 smithi104 ceph-mon[32438]: pgmap v467: 158 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized, 4 active+undersized+degraded, 145 active+clean; 515 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 1474/25299 objects degraded (5.826%); 541 KiB/s, 8 objects/s recovering 2024-09-21T19:02:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:44 smithi104 ceph-mon[32438]: 3.3 scrub starts 2024-09-21T19:02:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:44 smithi104 ceph-mon[32438]: 3.3 scrub ok 2024-09-21T19:02:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:44 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 1474/25299 objects degraded (5.826%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T19:02:45.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:45.354 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:45.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 8 16 16252 16236 126.833 137.438 0.0145527 0.00787863 2024-09-21T19:02:45.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:45.490 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:45.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:02:45.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:45.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:45.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.764 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:45.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:45.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:45.900 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:45.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:46.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:46.023 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:46.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:46.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:46.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.272 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:46.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:46.358 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:02:46.358 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:02:46.358 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T19:02:46.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T19:02:46.359 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:02:46.359 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:02:46.360 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:02:46.360 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:02:46.360 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:02:46.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:46.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 9 16 17572 17556 121.906 82.5 0.00154017 0.00777117 2024-09-21T19:02:46.422 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:46.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:46.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:02:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:46.647 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:46.672 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:46.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:46.795 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:46.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:46.928 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:46.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:47.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:47.066 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:47.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:47.168 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:47.203 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: pgmap v468: 158 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 515 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s; 1451/25299 objects degraded (5.735%); 427 KiB/s, 6 objects/s recovering 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: 1.5 scrub starts 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: 1.5 scrub ok 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: 5.9 scrub starts 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: 6.c scrub starts 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[29384]: 6.c scrub ok 2024-09-21T19:02:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: pgmap v468: 158 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 515 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s; 1451/25299 objects degraded (5.735%); 427 KiB/s, 6 objects/s recovering 2024-09-21T19:02:47.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: 1.5 scrub starts 2024-09-21T19:02:47.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: 1.5 scrub ok 2024-09-21T19:02:47.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T19:02:47.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: 5.9 scrub starts 2024-09-21T19:02:47.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: 6.c scrub starts 2024-09-21T19:02:47.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:46 smithi081 ceph-mon[39192]: 6.c scrub ok 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: pgmap v468: 158 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 515 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s; 1451/25299 objects degraded (5.735%); 427 KiB/s, 6 objects/s recovering 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: 1.5 scrub starts 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: 1.5 scrub ok 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: 5.9 scrub starts 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: 6.c scrub starts 2024-09-21T19:02:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:46 smithi104 ceph-mon[32438]: 6.c scrub ok 2024-09-21T19:02:47.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:47.340 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:47.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 10 16 18792 18776 117.34 76.25 0.00305766 0.00851715 2024-09-21T19:02:47.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:47.464 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:02:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:02:47.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:02:47.469 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:47.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:47.642 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:47.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:47.774 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:47.876 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:47.907 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:47.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 5.9 scrub ok 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 3.c scrub starts 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 3.c scrub ok 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 3.1e scrub starts 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 3.1e scrub ok 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T19:02:48.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 6.16 scrub starts 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[29384]: 6.16 scrub ok 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 5.9 scrub ok 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 3.c scrub starts 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 3.c scrub ok 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 3.1e scrub starts 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 3.1e scrub ok 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T19:02:48.003 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 6.16 scrub starts 2024-09-21T19:02:48.004 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:47 smithi081 ceph-mon[39192]: 6.16 scrub ok 2024-09-21T19:02:48.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.040 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:48.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:48.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.172 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:48.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:48.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 5.9 scrub ok 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 3.c scrub starts 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 3.c scrub ok 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 3.1e scrub starts 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 3.1e scrub ok 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T19:02:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T19:02:48.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 6.16 scrub starts 2024-09-21T19:02:48.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:47 smithi104 ceph-mon[32438]: 6.16 scrub ok 2024-09-21T19:02:48.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:48.275 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:48.303 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:48.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 11 15 20940 20925 118.882 134.312 0.0131573 0.00840494 2024-09-21T19:02:48.405 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:48.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:48.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:48.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:48.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:48.726 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:48.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:48.873 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:48.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:48.944 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 265, 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T19:02:46.901932+0000", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T19:02:39.856486+0000", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T18:55:50.490072+0000", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5799936, 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T19:02:48.945 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 19, 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 9, 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T19:02:48.946 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.947 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 11, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 11, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 11, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 11, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 11, 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.948 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "170", 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "168", 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:02:48.949 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.951 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.952 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.1800000667572021, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.1800000667572021, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T19:02:48.953 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.954 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.956 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.957 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.958 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6500000953674316, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6500000953674316, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.959 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.960 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.961 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.962 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.963 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.964 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.965 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.5, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.5, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T19:02:48.966 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.967 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.968 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.969 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.971 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.25, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.25, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.25, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.25, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.973 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.974 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.975 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.976 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.977 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.25, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.25, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.25, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.25, 2024-09-21T19:02:48.978 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.979 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.979 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.979 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.979 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.980 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:02:48.981 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:02:48.982 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.983 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.984 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.75, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.75, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.75, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.75, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.986 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 9, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_2", 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T19:02:27.858840+0000", 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:02:48.987 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 16, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 16, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 16, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 16, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 16, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:02:48.988 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "258", 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:02:48.989 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:02:48.990 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:02:48.991 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 1, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 1, 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 1 2024-09-21T19:02:48.992 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T19:02:48.993 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 264, 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.994 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T19:02:48.995 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:02:48.996 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T19:02:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:48.998 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:48.998 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:48.998 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:48.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:48.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:48.998 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:48.999 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 264, 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.000 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.001 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.003 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.003 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.003 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.003 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T19:02:49.003 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.004 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.005 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 239, 2024-09-21T19:02:49.006 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.008 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T19:02:49.009 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:49.010 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 264, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T19:02:49.012 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.013 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:49.014 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 261, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.017 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 37, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 261, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.019 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:02:49.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.021 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T19:02:49.021 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:49.022 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:02:49.022 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.022 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.022 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/2569144231", 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/2569144231", 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/2569144231", 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/2569144231", 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 45, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 238, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 261, 2024-09-21T19:02:49.024 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 261, 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 239, 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.026 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/2795996229", 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/2795996229", 2024-09-21T19:02:49.027 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/2795996229", 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/2795996229", 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:02:49.028 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 261, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.029 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.030 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.031 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.032 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:01:42.547688+0000", 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.033 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:02:03.242892+0000", 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:02:01.158175+0000", 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:02:22.449532+0000", 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:01:51.264234+0000", 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.036 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:01:50.961872+0000", 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:02:41.827113+0000", 2024-09-21T19:02:49.037 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 238 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:02:37.555006+0000", 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:02:49.038 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.7", 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:02:49.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.10", 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.13", 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.18", 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T19:02:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.1d", 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T19:02:49.042 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [], 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:02:49.043 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T19:02:49.044 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:02:49.045 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: pgmap v471: 158 pgs: 158 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 171 MiB/s wr, 2.74k op/s; 16 MiB/s, 248 objects/s recovering 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1451/25299 objects degraded (5.735%), 3 pgs degraded) 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:02:49.046 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3028399090' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: pgmap v471: 158 pgs: 158 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 171 MiB/s wr, 2.74k op/s; 16 MiB/s, 248 objects/s recovering 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1451/25299 objects degraded (5.735%), 3 pgs degraded) 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:02:49.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3028399090' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:02:49.079 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:49.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:49.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: pgmap v471: 158 pgs: 158 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 171 MiB/s wr, 2.74k op/s; 16 MiB/s, 248 objects/s recovering 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1451/25299 objects degraded (5.735%), 3 pgs degraded) 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:02:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3028399090' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:02:49.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:49.340 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:49.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 12 16 23140 23124 120.427 137.438 0.00627468 0.00829795 2024-09-21T19:02:49.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:49.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:49.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:49.617 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:49.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:49.756 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:49.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:49.895 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:49.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:49.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:50.029 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:50.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:50.167 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:50.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:49 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:50.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3854994241' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:02:50.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3028399090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:02:50.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:49 smithi104 ceph-mon[32438]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T19:02:50.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:50.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3854994241' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:02:50.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3028399090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:02:50.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[39192]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T19:02:50.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:02:50.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3854994241' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:02:50.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3028399090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:02:50.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:49 smithi081 ceph-mon[29384]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T19:02:50.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:50.305 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:50.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 13 16 25584 25568 122.913 152.75 0.00391479 0.00813034 2024-09-21T19:02:50.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:50.423 INFO:tasks.thrashosds.thrasher:j is {'epoch': 265, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T19:02:46.901932+0000', 'last_up_change': '2024-09-21T19:02:39.856486+0000', 'last_in_change': '2024-09-21T18:55:50.490072+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '168', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-21T19:02:27.858840+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '258', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 264, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 264, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 264, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 2569144231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 2569144231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 2569144231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 2569144231}]}, 'public_addr': '172.21.15.104:6804/2569144231', 'cluster_addr': '172.21.15.104:6805/2569144231', 'heartbeat_back_addr': '172.21.15.104:6807/2569144231', 'heartbeat_front_addr': '172.21.15.104:6806/2569144231', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 45, 'last_clean_end': 238, 'up_from': 261, 'up_thru': 261, 'down_at': 239, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 2795996229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 2795996229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 2795996229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 2795996229}]}, 'public_addr': '172.21.15.104:6808/2795996229', 'cluster_addr': '172.21.15.104:6809/2795996229', 'heartbeat_back_addr': '172.21.15.104:6811/2795996229', 'heartbeat_front_addr': '172.21.15.104:6810/2795996229', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:01:42.547688+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:02:03.242892+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:02:01.158175+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:02:22.449532+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:01:51.264234+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:01:50.961872+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:02:41.827113+0000', 'dead_epoch': 238}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:02:37.555006+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 4, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T19:02:50.423 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.10 2024-09-21T19:02:50.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd rm-pg-upmap-items 4.10 2024-09-21T19:02:50.439 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:50.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:50.586 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:50.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:50.716 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:50.716 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.739 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:50.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:50.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:50.788 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:50.823 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:50.823 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:50.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:50.855 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:50.856 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:50.873 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:50.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:50.889 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:50.889 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:50.921 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:02:50.921 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:02:50.956 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:50.956 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:50.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:51.016 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[29384]: pgmap v473: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s; 15 MiB/s, 241 objects/s recovering 2024-09-21T19:02:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[29384]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T19:02:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/882145017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:51.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[39192]: pgmap v473: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s; 15 MiB/s, 241 objects/s recovering 2024-09-21T19:02:51.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:51.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:51.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[39192]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T19:02:51.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/882145017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:51.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:51.150 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:51.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:50 smithi104 ceph-mon[32438]: pgmap v473: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s; 15 MiB/s, 241 objects/s recovering 2024-09-21T19:02:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:50 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:50 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:50 smithi104 ceph-mon[32438]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T19:02:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/882145017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:51.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:51.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:51.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 14 16 27734 27718 123.73 134.375 0.00623192 0.00807604 2024-09-21T19:02:51.411 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:51.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:51.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:51.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:51.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:51.813 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:51.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:51.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:52.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:52.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:52.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:52.248 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:52.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:52.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:52.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:52.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 15 16 29990 29974 124.881 141 0.00790636 0.00800196 2024-09-21T19:02:52.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:52.520 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:52.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:52.658 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:52.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:52 smithi104 ceph-mon[32438]: pgmap v475: 158 pgs: 158 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s; 16 MiB/s, 255 objects/s recovering 2024-09-21T19:02:52.759 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:52.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:52.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:52 smithi081 ceph-mon[29384]: pgmap v475: 158 pgs: 158 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s; 16 MiB/s, 255 objects/s recovering 2024-09-21T19:02:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:02:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:02:52.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:52 smithi081 ceph-mon[39192]: pgmap v475: 158 pgs: 158 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s; 16 MiB/s, 255 objects/s recovering 2024-09-21T19:02:52.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:52.917 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:52.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:53.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:53.054 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:53.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:53.192 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:53.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:53.326 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:53.402 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 16 16 32123 32107 125.407 133.312 0.00685342 0.00796896 2024-09-21T19:02:53.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:53.462 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:53.564 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:53.596 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:53.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:53.723 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:53.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:53.866 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:53.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:53.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:54.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:54.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:54.275 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:54.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:54.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 17 16 34070 34054 125.185 121.688 0.00481587 0.00798274 2024-09-21T19:02:54.415 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:54.517 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:54.551 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:54.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:54.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:54.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:54.824 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:54.926 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:54.965 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:54.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:54 smithi104 ceph-mon[32438]: pgmap v476: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s; 14 MiB/s, 216 objects/s recovering 2024-09-21T19:02:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:54 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:54 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:54 smithi104 ceph-mon[32438]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T19:02:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[29384]: pgmap v476: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s; 14 MiB/s, 216 objects/s recovering 2024-09-21T19:02:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[29384]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T19:02:55.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[39192]: pgmap v476: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s; 14 MiB/s, 216 objects/s recovering 2024-09-21T19:02:55.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:55.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:54 smithi081 ceph-mon[39192]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T19:02:55.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:55.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.236 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:55.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:55.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:55.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 18 16 36202 36186 125.632 133.25 0.00707466 0.00795498 2024-09-21T19:02:55.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.509 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:55.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:55.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:55.642 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:55.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:55.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:55.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:55.914 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:55.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:55.928 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:55 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:02:55.928 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:55 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:02:55.928 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:55 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:02:55.928 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:55 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:02:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:56.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:55 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:02:56.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:55 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:02:56.016 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.050 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:56.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:56.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:56.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.318 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:56.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:56.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 19 16 38241 38225 125.727 127.438 0.00876978 0.00794806 2024-09-21T19:02:56.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.462 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:56.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:56.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:56.575 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:56.575 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:56.616 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:56.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:56 smithi081 ceph-mon[29384]: pgmap v478: 158 pgs: 158 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 771 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-21T19:02:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:56 smithi081 ceph-mon[29384]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T19:02:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:56 smithi081 ceph-mon[39192]: pgmap v478: 158 pgs: 158 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 771 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-21T19:02:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:56 smithi081 ceph-mon[39192]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T19:02:56.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:56.876 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:56.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:56.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:56 smithi104 ceph-mon[32438]: pgmap v478: 158 pgs: 158 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 771 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-21T19:02:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:56 smithi104 ceph-mon[32438]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T19:02:57.010 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:57.112 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:02:57.156 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:02:57.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:02:57.297 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:02:57.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:57.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:02:57.402115+0000 min lat: 0.00117786 max lat: 2.27862 avg lat: 0.00796746 2024-09-21T19:02:57.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:02:57.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 20 16 40156 40140 125.424 119.688 0.0113396 0.00796746 2024-09-21T19:02:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.440 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:57.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:57.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:57.560 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:02:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:02:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:02:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:57.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:57.721 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:57.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:57.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:02:57.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:57.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:02:57.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:57.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:58.085 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:58.112 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:58.214 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:58.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:02:58.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:58.377 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:02:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:58.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:58.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 21 16 41820 41804 124.404 104 0.00437605 0.00803285 2024-09-21T19:02:58.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:58.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:02:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:02:58.526 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:58.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.669 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:58.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:58.686 INFO:teuthology.orchestra.run.smithi081.stderr:clear 4.10 pg_upmap_items mapping 2024-09-21T19:02:58.720 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:02:58.768 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:58 smithi081 ceph-mon[29384]: pgmap v480: 158 pgs: 158 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-21T19:02:58.768 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1196006919' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"} : dispatch 2024-09-21T19:02:58.768 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:58 smithi081 ceph-mon[29384]: from='client.25636 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"} : dispatch 2024-09-21T19:02:58.768 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:58 smithi081 ceph-mon[39192]: pgmap v480: 158 pgs: 158 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-21T19:02:58.768 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:58 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1196006919' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"} : dispatch 2024-09-21T19:02:58.768 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:58 smithi081 ceph-mon[39192]: from='client.25636 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"} : dispatch 2024-09-21T19:02:58.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:58.796 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:58.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:02:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:58.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:02:59.003 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:58 smithi104 ceph-mon[32438]: pgmap v480: 158 pgs: 158 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-21T19:02:59.003 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:58 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1196006919' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"} : dispatch 2024-09-21T19:02:59.003 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:58 smithi104 ceph-mon[32438]: from='client.25636 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"} : dispatch 2024-09-21T19:02:59.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:02:59.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:59.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:02:59.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:59.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:59.220 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:02:59.221 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:02:59.256 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:02:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:02:59.292 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:02:59.293 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:02:59.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:59.332 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:02:59.333 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:02:59.335 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:02:59.335 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:02:59.337 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:59.367 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:02:59.367 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:02:59.400 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:02:59.400 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:02:59.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 22 16 43544 43528 123.646 107.75 0.00947027 0.0080823 2024-09-21T19:02:59.434 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:02:59.435 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:02:59.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:02:59.468 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:02:59.469 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:02:59.474 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:02:59.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:02:59.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:02:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:02:59.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:02:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:02:59.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:02:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:02:59.883 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:02:59.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:02:59.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:02:59.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:59 smithi104 ceph-mon[32438]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"}]': finished 2024-09-21T19:02:59.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:59 smithi104 ceph-mon[32438]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T19:02:59.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1220769158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:02:59.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:02:59 smithi104 ceph-mon[32438]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T19:02:59.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:00.023 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:00.025 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[29384]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"}]': finished 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[29384]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1220769158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[29384]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[39192]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.10"}]': finished 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[39192]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1220769158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:02:59 smithi081 ceph-mon[39192]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T19:03:00.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:00.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:00.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:00.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:00.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 23 16 45111 45095 122.528 97.9375 0.014498 0.00815622 2024-09-21T19:03:00.432 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:00.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:00.568 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:00.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:00.706 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:00.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:03:00.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:00.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:00 smithi081 ceph-mon[29384]: pgmap v482: 158 pgs: 158 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-21T19:03:00.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:00 smithi081 ceph-mon[39192]: pgmap v482: 158 pgs: 158 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-21T19:03:00.848 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:00.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:00.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:00.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:01.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:00 smithi104 ceph-mon[32438]: pgmap v482: 158 pgs: 158 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-21T19:03:01.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:01.133 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:01.155 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:01.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:01.270 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:01.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:01.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 24 16 46845 46829 121.938 108.375 0.0155825 0.00819613 2024-09-21T19:03:01.410 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:01.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:01.546 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:01.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:01.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:01.719 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:03:01.777 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:03:01.777 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:03:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:03:01.778 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:03:01.778 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:03:01.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:01.818 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:01.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:01.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:01.959 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:01.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3135637035' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:01 smithi104 ceph-mon[32438]: from='client.35747 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:02.027 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3135637035' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:02.027 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:01 smithi081 ceph-mon[39192]: from='client.35747 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:02.027 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3135637035' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:02.027 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:01 smithi081 ceph-mon[29384]: from='client.35747 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:02.060 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.097 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:02.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:02.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.233 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:02.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:02.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:02.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:02.403 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 25 16 48639 48623 121.545 112.125 0.0079624 0.00822147 2024-09-21T19:03:02.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:02.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:02.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:03:02.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:02.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:03:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:02.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:02.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:02.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:02.901 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:02.943 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:02.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:02.990 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:03.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:02 smithi104 ceph-mon[32438]: pgmap v484: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-21T19:03:03.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:02 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:02 smithi104 ceph-mon[32438]: from='client.35747 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:02 smithi104 ceph-mon[32438]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T19:03:03.045 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:03.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[29384]: pgmap v484: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-21T19:03:03.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:03.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[29384]: from='client.35747 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:03.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[29384]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T19:03:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[39192]: pgmap v484: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-21T19:03:03.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:03.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[39192]: from='client.35747 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:03.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:02 smithi081 ceph-mon[39192]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T19:03:03.078 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:03.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:03.214 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:03.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:03.351 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:03.404 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 26 16 50148 50132 120.497 94.3125 0.00732078 0.00829379 2024-09-21T19:03:03.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:03.493 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:03.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:03.614 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102355 7.4 7:2a73ca6f:::benchmark_data_smithi081_73391_object102354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:03:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 463.292655674, 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16875653400000001, 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102355 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325049+0000", 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.754e-06 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325092+0000", 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3551999999999999e-05 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325112+0000", 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9306000000000001e-05 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325122+0000", 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7960000000000007e-06 2024-09-21T19:03:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325298+0000", 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000176014 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325430+0000", 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013203999999999999 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325589+0000", 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000159486 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.328322+0000", 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002732656 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329357+0000", 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010357400000000001 2024-09-21T19:03:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493698+0000", 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16434100300000001 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493758+0000", 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9515999999999997e-05 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493802+0000", 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3671000000000003e-05 2024-09-21T19:03:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102361 7.4 7:2b82beac:::benchmark_data_smithi081_73391_object102360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 463.28781534500001, 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.166344565, 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102361 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329887+0000", 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.545e-06 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329943+0000", 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5818e-05 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329952+0000", 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6729999999999998e-06 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329956+0000", 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8509999999999998e-06 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330052+0000", 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5827000000000003e-05 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330254+0000", 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020186800000000001 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330415+0000", 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016062699999999999 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338473+0000", 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080582750000000002 2024-09-21T19:03:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341282+0000", 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028086040000000001 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496118+0000", 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15483660299999999 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496188+0000", 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9626e-05 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496230+0000", 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2247999999999999e-05 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102367 7.4 7:2bc3d420:::benchmark_data_smithi081_73391_object102366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:03:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 463.28501418600001, 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16373927799999999, 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102367 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332689+0000", 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0810000000000001e-06 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332717+0000", 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8388999999999999e-05 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332731+0000", 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4158999999999999e-05 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332739+0000", 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1159999999999999e-06 2024-09-21T19:03:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332800+0000", 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0343999999999997e-05 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332928+0000", 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128341 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333068+0000", 2024-09-21T19:03:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013955200000000001 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338482+0000", 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054145959999999998 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341320+0000", 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028377950000000002 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496287+0000", 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15496721699999999 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496382+0000", 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4477999999999999e-05 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496426+0000", 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4209999999999999e-05 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102368 7.4 7:29e74c0a:::benchmark_data_smithi081_73391_object102367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 463.28415340700002, 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16300020900000001, 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102368 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333549+0000", 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1930000000000001e-06 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333748+0000", 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019946800000000001 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333759+0000", 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0645000000000001e-05 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333766+0000", 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0219999999999999e-06 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335818+0000", 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002052356 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335940+0000", 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012176999999999999 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336078+0000", 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137639 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341538+0000", 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054607579999999996 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341553+0000", 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4975e-05 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496464+0000", 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.154911199 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496506+0000", 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1530999999999999e-05 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496548+0000", 2024-09-21T19:03:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1653000000000003e-05 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102370 7.4 7:261d1db0:::benchmark_data_smithi081_73391_object102369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 463.28035364200002, 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15954671000000001, 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102370 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:03:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337352+0000", 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1070000000000004e-06 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337487+0000", 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134325 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337498+0000", 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1063999999999999e-05 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337505+0000", 2024-09-21T19:03:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3159999999999999e-06 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338628+0000", 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001123298 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338755+0000", 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127059 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:03:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338910+0000", 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154213 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341560+0000", 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026500429999999999 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.342134+0000", 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057463499999999999 2024-09-21T19:03:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496815+0000", 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15468104099999999 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496862+0000", 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7197999999999998e-05 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496894+0000", 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1411000000000001e-05 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102384 7.1 7:8c91c1ab:::benchmark_data_smithi081_73391_object102383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 463.27264326300002, 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.151452372, 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102384 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345059+0000", 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.714e-06 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345113+0000", 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3814999999999999e-05 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345121+0000", 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2479999999999996e-06 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345126+0000", 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9309999999999999e-06 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345176+0000", 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9310000000000001e-05 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345271+0000", 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4955000000000001e-05 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345358+0000", 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7319999999999997e-05 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347395+0000", 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020371719999999999 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.349879+0000", 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002483519 2024-09-21T19:03:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496421+0000", 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14654283400000001 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496473+0000", 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1610000000000002e-05 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496510+0000", 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6943999999999997e-05 2024-09-21T19:03:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242080 7.1 7:892cd020:::benchmark_data_smithi081_73391_object103081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 314.11199210799998, 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16669837000000001, 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242080 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505710+0000", 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5400000000000001e-06 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505716+0000", 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3170000000000001e-06 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505725+0000", 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7899999999999994e-06 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505732+0000", 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9789999999999996e-06 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505867+0000", 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134523 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505964+0000", 2024-09-21T19:03:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6680000000000003e-05 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.506050+0000", 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6395000000000002e-05 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.509085+0000", 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030351779999999999 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.509739+0000", 2024-09-21T19:03:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00065418699999999998 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672337+0000", 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16259812500000001 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672376+0000", 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8399999999999998e-05 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672407+0000", 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1256e-05 2024-09-21T19:03:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242082 7.1 7:83ef7caf:::benchmark_data_smithi081_73391_object103083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 314.10709294700001, 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16217166099999999, 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242082 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510609+0000", 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.345e-06 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510614+0000", 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3030000000000001e-06 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510623+0000", 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3559999999999999e-06 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510630+0000", 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9430000000000003e-06 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510673+0000", 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3180999999999999e-05 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510747+0000", 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4388999999999995e-05 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510818+0000", 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0288000000000002e-05 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515225+0000", 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044072699999999996 2024-09-21T19:03:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515269+0000", 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4165000000000002e-05 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672707+0000", 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15743800899999999 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672742+0000", 2024-09-21T19:03:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4878e-05 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672780+0000", 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7533999999999997e-05 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242083 7.1 7:8f0bc0f0:::benchmark_data_smithi081_73391_object103084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 314.10574372799999, 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18721689799999999, 2024-09-21T19:03:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242083 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511958+0000", 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7300000000000004e-07 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511962+0000", 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9199999999999997e-06 2024-09-21T19:03:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511973+0000", 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0577000000000001e-05 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511980+0000", 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5310000000000004e-06 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512698+0000", 2024-09-21T19:03:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071787200000000004 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512808+0000", 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110412 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512929+0000", 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012087999999999999 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515461+0000", 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002532191 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516028+0000", 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056677599999999996 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699096+0000", 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18306810000000001 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699133+0000", 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6837e-05 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699174+0000", 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0828999999999998e-05 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235896 8.22 8:45adf4b4:::benchmark_data_smithi081_105538_object103664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 69.942560964999998, 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039943074, 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235896 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675143+0000", 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.729e-06 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675150+0000", 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6820000000000003e-06 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675168+0000", 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7875000000000001e-05 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675178+0000", 2024-09-21T19:03:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5610000000000004e-06 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731748+0000", 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056570691999999999 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731754+0000", 2024-09-21T19:03:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.976e-06 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.700981+0000", 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96922663099999995 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701110+0000", 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864500000000001 2024-09-21T19:03:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701320+0000", 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021059200000000001 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702216+0000", 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089577400000000001 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702245+0000", 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8608000000000001e-05 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.714979+0000", 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012734354999999999 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715038+0000", 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8588999999999998e-05 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715083+0000", 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5364999999999997e-05 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235898 8.2 8:4337194e:::benchmark_data_smithi081_105538_object103666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 69.942507586999994, 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0181600850000001, 2024-09-21T19:03:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235898 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675194+0000", 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1699999999999997e-07 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675197+0000", 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675203+0000", 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2369999999999997e-06 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675208+0000", 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9989999999999999e-06 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731592+0000", 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056383961000000003 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731604+0000", 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2278e-05 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685684+0000", 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95407945800000005 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:03.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685861+0000", 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017756400000000001 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.686082+0000", 2024-09-21T19:03:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022110200000000001 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.687697+0000", 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016145230000000001 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.690382+0000", 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026848850000000001 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693261+0000", 2024-09-21T19:03:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879291 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693314+0000", 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2845e-05 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693353+0000", 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9369000000000002e-05 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237679 8.1b 8:deadaa1e:::benchmark_data_smithi081_105538_object105447:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.933055307000004, 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0964987129999999, 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237679 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684647+0000", 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.153e-06 2024-09-21T19:03:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684651+0000", 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3370000000000001e-06 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684660+0000", 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6470000000000004e-06 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684667+0000", 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684783+0000", 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011565600000000001 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684791+0000", 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7809999999999994e-06 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691384+0000", 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0065922899999999 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691392+0000", 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.636e-06 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781063+0000", 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0896703569999999 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781144+0000", 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1546999999999997e-05 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237686 8.1b 8:df3678b5:::benchmark_data_smithi081_105538_object105454:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.933016249999994, 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.173105171, 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237686 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1299999999999999e-07 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684688+0000", 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5830000000000001e-06 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684693+0000", 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5910000000000003e-06 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684699+0000", 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684822+0000", 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012324499999999999 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684830+0000", 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6750000000000002e-06 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691420+0000", 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006590262 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691426+0000", 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4559999999999997e-06 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781412+0000", 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.089985687 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for missing object", 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781478+0000", 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6272000000000005e-05 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857711+0000", 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076232366999999995 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857790+0000", 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9029999999999994e-05 2024-09-21T19:03:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237687 8.1b 8:dc29a777:::benchmark_data_smithi081_105538_object105455:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.932995911999996, 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1710659049999999, 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237687 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4899999999999995e-07 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684708+0000", 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1339999999999998e-06 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684713+0000", 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9749999999999998e-06 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684734+0000", 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1206e-05 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684861+0000", 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012699799999999999 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684870+0000", 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9770000000000006e-06 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691453+0000", 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006583418 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691459+0000", 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2580000000000001e-06 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781524+0000", 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0900641010000001 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781643+0000", 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119735 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781902+0000", 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025915999999999998 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784476+0000", 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025738229999999998 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784488+0000", 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1401e-05 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784513+0000", 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5462000000000001e-05 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855646+0000", 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071132476 2024-09-21T19:03:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855686+0000", 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0157999999999997e-05 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855771+0000", 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5074000000000006e-05 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237708 8.1b 8:dbdbeb85:::benchmark_data_smithi081_105538_object105476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.913308315999998, 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.151669386, 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237708 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704394+0000", 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.855e-06 2024-09-21T19:03:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704399+0000", 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9699999999999998e-06 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704413+0000", 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3560000000000001e-05 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704421+0000", 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6399999999999997e-06 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704473+0000", 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.27e-05 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704483+0000", 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4520000000000003e-06 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691484+0000", 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98700145800000005 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691491+0000", 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782825+0000", 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0913340810000001 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782935+0000", 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110647 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.783135+0000", 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020000399999999999 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784491+0000", 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013554319999999999 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784543+0000", 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2609999999999999e-05 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784566+0000", 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2945000000000001e-05 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855871+0000", 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071304949000000006 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855929+0000", 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7924999999999999e-05 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.856062+0000", 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013279700000000001 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237711 8.1b 8:dd96cf97:::benchmark_data_smithi081_105538_object105479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.911108729000006, 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1506975869999998, 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237711 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:03:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706594+0000", 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9099999999999999e-06 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706599+0000", 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0259999999999998e-06 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706618+0000", 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8499e-05 2024-09-21T19:03:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706633+0000", 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5931999999999999e-05 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706751+0000", 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117395 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706762+0000", 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1316e-05 2024-09-21T19:03:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691770+0000", 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.985007619 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691776+0000", 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5490000000000003e-06 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784627+0000", 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0928504649999999 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784699+0000", 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2539000000000005e-05 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784817+0000", 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118029 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785805+0000", 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098732000000000008 2024-09-21T19:03:03.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785853+0000", 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8554000000000003e-05 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785945+0000", 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1835000000000004e-05 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857195+0000", 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071249589000000002 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857226+0000", 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1705999999999998e-05 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857290+0000", 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3304000000000002e-05 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2026 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94695648315648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.345310923, 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46135494399999999, 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2026 2024-09-21T19:03:03.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9019999999999994e-06 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272421+0000", 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7499999999999998e-05 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0607e-05 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272504+0000", 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2151999999999996e-05 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272515+0000", 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1241e-05 2024-09-21T19:03:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702573+0000", 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.430058259 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702717+0000", 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143931 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.733745+0000", 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031027852000000002 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2027 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.345243224000001, 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.464594917, 2024-09-21T19:03:03.738 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2027 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4000000000000001e-07 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272462+0000", 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4199999999999999e-06 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272469+0000", 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272474+0000", 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4249999999999997e-06 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272539+0000", 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5066000000000003e-05 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272549+0000", 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8709999999999992e-06 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704278+0000", 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43172896500000002 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704401+0000", 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012265999999999999 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.737053+0000", 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.032651612000000003 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.743 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2034 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94695648315648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553712+0000", 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.063988667000004, 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14948307299999999, 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2034 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553712+0000", 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553712+0000", 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553715+0000", 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.582e-06 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553721+0000", 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7020000000000004e-06 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553741+0000", 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9167000000000001e-05 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553749+0000", 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6440000000000006e-06 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553815+0000", 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5877999999999995e-05 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553825+0000", 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6180000000000001e-06 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702976+0000", 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.149151433 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.703051+0000", 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5306e-05 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.703195+0000", 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143743 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2036 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.063883283999999, 2024-09-21T19:03:03.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15105808700000001, 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2036 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0399999999999996e-07 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553821+0000", 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6769999999999999e-06 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553827+0000", 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553831+0000", 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6550000000000003e-06 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553873+0000", 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1241e-05 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553879+0000", 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5429999999999999e-06 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704657+0000", 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150777833 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704730+0000", 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3398000000000004e-05 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704876+0000", 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014533600000000001 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:03.753 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:03:03.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.838 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:03.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:03.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:03.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:04.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:04.102 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:04.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:04.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:04.337 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:04.337 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:04.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:04.370 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:04.370 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:04.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:04.404 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 27 16 51704 51688 119.636 97.25 0.00642559 0.00835374 2024-09-21T19:03:04.405 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:04.405 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:04.439 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:04.439 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:04.470 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:04.470 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:04.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:04.507 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:04.508 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:04.508 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:04.543 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:04.543 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:04.574 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:04.574 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:04.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:04.649 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:04.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:04.775 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:03:04.775 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:03:04.775 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T19:03:04.775 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.169141 2024-09-21T19:03:04.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.16914061082524012 2024-09-21T19:03:04.783 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:04.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:03:04.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:04 smithi081 ceph-mon[29384]: pgmap v486: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:04.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2796847011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:04.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:04 smithi081 ceph-mon[29384]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T19:03:04.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:04 smithi081 ceph-mon[39192]: pgmap v486: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2796847011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:04.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:04 smithi081 ceph-mon[39192]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T19:03:04.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:04.924 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:04.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:04 smithi104 ceph-mon[32438]: pgmap v486: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2796847011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:04 smithi104 ceph-mon[32438]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T19:03:05.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:05.067 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:05.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:05.223 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:05.249 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:05.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:05.364 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:05.404 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 28 15 53228 53213 118.767 95.3125 0.0162357 0.00841486 2024-09-21T19:03:05.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:05.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:03:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:05.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:05.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:05.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.770 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:05.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:05.851 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:05.851 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:05.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:05.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:05.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:06.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:06.040 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:06.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:06.177 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:06.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:06.313 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:06.404 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 29 16 54787 54771 118.029 97.375 0.00858866 0.0084674 2024-09-21T19:03:06.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:06.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:06.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:03:06.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:06.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:06.692 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:06.732 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:06.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:06.855 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:06.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:06.933 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:06.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:07.002 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:07.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:06 smithi104 ceph-mon[32438]: pgmap v488: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:07.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:07.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:06 smithi081 ceph-mon[29384]: pgmap v488: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:07.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:07.008 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:06 smithi081 ceph-mon[39192]: pgmap v488: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:07.008 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:07.101 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:07.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:07.139 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:07.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.278 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:07.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:07.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:03:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:03:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:03:07.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:07.406 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 30 16 56263 56247 117.164 92.25 0.00975168 0.00852995 2024-09-21T19:03:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:07.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:07.544 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:07.544 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:07.565 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:07.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:07.700 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:07.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:07.942 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:07.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:08.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:08.116 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:08.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:08.247 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:08.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:03:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:03:08.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:03:08.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:08.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:08.406 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 31 15 57666 57651 116.215 87.75 0.0121139 0.00859978 2024-09-21T19:03:08.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:08.530 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:08.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:08.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:08.699 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:08.758 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.5 primary-affinity to 0.169141 (8110842) 2024-09-21T19:03:08.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:08.794 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:08.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:08.917 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:08.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:09.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:03:09.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:09.030 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:08 smithi104 ceph-mon[32438]: pgmap v489: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 128 MiB/s wr, 2.05k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:09.030 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/130994393' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012} : dispatch 2024-09-21T19:03:09.030 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:08 smithi104 ceph-mon[32438]: from='client.35753 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012} : dispatch 2024-09-21T19:03:09.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:08 smithi081 ceph-mon[29384]: pgmap v489: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 128 MiB/s wr, 2.05k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:09.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/130994393' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012} : dispatch 2024-09-21T19:03:09.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:08 smithi081 ceph-mon[29384]: from='client.35753 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012} : dispatch 2024-09-21T19:03:09.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:08 smithi081 ceph-mon[39192]: pgmap v489: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 128 MiB/s wr, 2.05k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:09.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/130994393' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012} : dispatch 2024-09-21T19:03:09.032 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:08 smithi081 ceph-mon[39192]: from='client.35753 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012} : dispatch 2024-09-21T19:03:09.053 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:09.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:09.176 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:09.235 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:09.236 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:09.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:09.300 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:09.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:09.406 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 32 16 58159 58143 113.544 30.75 0.00261299 0.00863052 2024-09-21T19:03:09.422 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:09.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:09.603 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:09.617 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:03:09.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.696 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:09.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:09.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:09.832 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:09.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:09.971 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:09.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:10.041 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: from='client.35753 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012}]': finished 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1155641808' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: from='client.35759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[29384]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: from='client.35753 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012}]': finished 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1155641808' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:10.042 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: from='client.35759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:10.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:10.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:03:10.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:09 smithi081 ceph-mon[39192]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T19:03:10.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.110 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:10.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:10.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: from='client.35753 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.16914061082524012}]': finished 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1155641808' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: from='client.35759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:03:10.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:09 smithi104 ceph-mon[32438]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T19:03:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:10.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:10.261 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:03:10.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:10.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:10.406 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 33 16 59401 59385 112.455 77.625 0.00689009 0.00888744 2024-09-21T19:03:10.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:10.536 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:10.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:10.682 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:10.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:10.809 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:10.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:10.943 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:10.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:11.045 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:11.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:10 smithi104 ceph-mon[32438]: pgmap v491: 158 pgs: 1 peering, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 649 B/s rd, 100 MiB/s wr, 1.60k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:11.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:10 smithi081 ceph-mon[39192]: pgmap v491: 158 pgs: 1 peering, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 649 B/s rd, 100 MiB/s wr, 1.60k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:11.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:10 smithi081 ceph-mon[29384]: pgmap v491: 158 pgs: 1 peering, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 649 B/s rd, 100 MiB/s wr, 1.60k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:03:11.068 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:11.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:11.180 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:11.180 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:11.192 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.200 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:11.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:11.223 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:11.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:11.259 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:11.259 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:11.290 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:11.290 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:11.294 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:11.294 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:11.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:11.326 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:11.327 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:11.338 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:11.360 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:11.361 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:11.391 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:11.391 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:11.406 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 34 16 60635 60619 111.416 77.125 0.00819869 0.00897008 2024-09-21T19:03:11.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:11.483 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:11.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:11.618 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:11.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:11.750 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:11.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:11.883 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:11.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: 3.1d scrub starts 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: 3.1d scrub ok 2024-09-21T19:03:11.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/825251779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:11.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:12.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:03:12.017 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: 3.1d scrub starts 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: 3.1d scrub ok 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/825251779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T19:03:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T19:03:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: 3.1d scrub starts 2024-09-21T19:03:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: 3.1d scrub ok 2024-09-21T19:03:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/825251779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:12.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:12.162 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:12.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:12.295 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:12.397 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:12.406 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 35 16 62196 62180 111.02 97.5625 0.00809911 0.00900207 2024-09-21T19:03:12.413 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:12.448 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:12.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:03:12.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.581 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:12.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:12.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:12.726 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:12.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: 5.15 scrub starts 2024-09-21T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: 5.15 scrub ok 2024-09-21T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: pgmap v493: 158 pgs: 1 peering, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 732 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T19:03:12.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T19:03:12.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T19:03:12.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: 5.15 scrub starts 2024-09-21T19:03:12.819 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: 5.15 scrub ok 2024-09-21T19:03:12.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: pgmap v493: 158 pgs: 1 peering, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 732 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T19:03:12.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T19:03:12.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:12 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T19:03:12.828 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:12.861 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:12.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T19:03:12.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T19:03:12.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: 5.15 scrub starts 2024-09-21T19:03:12.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: 5.15 scrub ok 2024-09-21T19:03:12.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: pgmap v493: 158 pgs: 1 peering, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 732 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T19:03:12.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T19:03:12.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:12 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T19:03:12.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:12.971 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:12.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:13.028 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:03:13.028 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:03:13.029 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:03:13.029 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:03:13.029 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:03:13.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:13.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:13.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:13.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:13.407 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 36 16 63591 63575 110.357 87.1875 0.0104092 0.00905635 2024-09-21T19:03:13.416 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:13.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:13.555 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:13.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:13.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:13.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:03:13.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:13.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:13.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:13 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T19:03:13.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:13 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T19:03:13.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:13.970 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:13.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:13 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T19:03:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:13 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T19:03:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:13 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T19:03:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:13 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T19:03:14.072 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:14.111 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:14.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:14.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:14.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:03:14.221 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:14.245 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:14.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:14.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:14.391 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:14.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:14.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 37 16 64988 64972 109.73 87.3125 0.00500459 0.00910835 2024-09-21T19:03:14.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:14.537 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:03:14.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T19:03:14.549 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:14.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:14.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:14.678 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:14.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:14.715 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:14.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:14.815 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:14.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:14.820 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:03:14.856 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T19:03:14.856 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: pgmap v494: 158 pgs: 1 peering, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 26 KiB/s, 0 objects/s recovering 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3994466469' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: pgmap v494: 158 pgs: 1 peering, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 26 KiB/s, 0 objects/s recovering 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T19:03:14.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3994466469' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:14.858 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T19:03:14.858 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:14 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T19:03:14.870 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101889 7.f e105/82 7:f2f30f0c:::benchmark_data_smithi081_73391_object101888:head v 105'6311, mlcod=105'6308)", 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.908994+0000", 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 474.95800564299998, 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087156066000000004, 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908994+0000", 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.908995+0000", 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9800000000000003e-07 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909027+0000", 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2353e-05 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909033+0000", 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1859999999999997e-06 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909038+0000", 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7249999999999997e-06 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909131+0000", 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2363999999999999e-05 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.909149+0000", 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8450000000000001e-05 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996090+0000", 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086940559000000001 2024-09-21T19:03:14.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996150+0000", 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0730999999999998e-05 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101894 7.8 e105/79 7:1138a4d6:::benchmark_data_smithi081_73391_object101893:head v 105'6424, mlcod=105'6422)", 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910208+0000", 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 474.95679192300003, 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.086428049000000007, 2024-09-21T19:03:14.875 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910208+0000", 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910209+0000", 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.24e-06 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910259+0000", 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9756000000000003e-05 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910269+0000", 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0035e-05 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910276+0000", 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8040000000000004e-06 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914457+0000", 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041814679999999998 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914478+0000", 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0528999999999999e-05 2024-09-21T19:03:14.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996573+0000", 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.082095020000000005 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996636+0000", 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3196999999999998e-05 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.879 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.34742.0:101892 7.f e105/82 7:f5e7532a:::benchmark_data_smithi081_73391_object101891:head v 105'6312, mlcod=105'6308)", 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.910906+0000", 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 474.95609406199998, 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085280135000000007, 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910906+0000", 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910907+0000", 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1990000000000001e-06 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910962+0000", 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5427000000000003e-05 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910972+0000", 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2520000000000005e-06 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.910978+0000", 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.117e-06 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914620+0000", 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036426869999999999 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.914639+0000", 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8848999999999999e-05 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996155+0000", 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.081515729999999995 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996186+0000", 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0874000000000003e-05 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101901 7.5 7:a8e9101c:::benchmark_data_smithi081_73391_object101900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916321+0000", 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 474.950678917, 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087318653999999996, 2024-09-21T19:03:14.884 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101901 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916321+0000", 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916329+0000", 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3990000000000004e-06 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916364+0000", 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4808999999999998e-05 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916374+0000", 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0298e-05 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916382+0000", 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996259+0000", 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079877760000000006 2024-09-21T19:03:14.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996412+0000", 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015201099999999999 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996586+0000", 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017437 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003419+0000", 2024-09-21T19:03:14.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068332979999999998 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003455+0000", 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5438000000000002e-05 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003478+0000", 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3048e-05 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003584+0000", 2024-09-21T19:03:14.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106331 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003640+0000", 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5583e-05 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:101902 7.5 7:a3b6d405:::benchmark_data_smithi081_73391_object101901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:19.916400+0000", 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 474.95059990800002, 2024-09-21T19:03:14.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.087823038000000006, 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 101902 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916400+0000", 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916401+0000", 2024-09-21T19:03:14.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5899999999999996e-07 2024-09-21T19:03:14.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.916430+0000", 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9076000000000001e-05 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996083+0000", 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.079653529000000001 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.996103+0000", 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0015999999999999e-05 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998560+0000", 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002457126 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998676+0000", 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115328 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:19.998809+0000", 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133767 2024-09-21T19:03:14.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003647+0000", 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048375040000000003 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.003682+0000", 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4922999999999998e-05 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004144+0000", 2024-09-21T19:03:14.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046231000000000001 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004190+0000", 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5958999999999998e-05 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.004223+0000", 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2840999999999999e-05 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114519 7.6 7:6b5e3486:::benchmark_data_smithi081_73391_object114518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 464.157894091, 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12712907900000001, 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114519 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.898 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709107+0000", 2024-09-21T19:03:14.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.66e-06 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709157+0000", 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9911999999999999e-05 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709171+0000", 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3947e-05 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709179+0000", 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3529999999999997e-06 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716464+0000", 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072858719999999997 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716590+0000", 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554999999999999 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716718+0000", 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012810499999999999 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723181+0000", 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006462944 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723217+0000", 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5593000000000003e-05 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836081+0000", 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11286423700000001 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836188+0000", 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107315 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836235+0000", 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6591000000000002e-05 2024-09-21T19:03:14.903 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114521 7.6 7:6fe397a7:::benchmark_data_smithi081_73391_object114520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 464.15769954199999, 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12709793799999999, 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.904 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114521 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709301+0000", 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6199999999999996e-07 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709354+0000", 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2936999999999998e-05 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709361+0000", 2024-09-21T19:03:14.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.097e-06 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709367+0000", 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6550000000000001e-06 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719193+0000", 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098261459999999991 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719453+0000", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026041699999999998 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.720410+0000", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095699200000000004 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723227+0000", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002816888 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723832+0000", 2024-09-21T19:03:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060487799999999999 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836280+0000", 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11244789099999999 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836362+0000", 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1967999999999999e-05 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836398+0000", 2024-09-21T19:03:14.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6207000000000001e-05 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114538 7.6 7:657d97a1:::benchmark_data_smithi081_73391_object114537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723638+0000", 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 464.14336186999998, 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11594671600000001, 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.910 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114538 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723638+0000", 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723640+0000", 2024-09-21T19:03:14.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.801e-06 2024-09-21T19:03:14.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723678+0000", 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7873000000000001e-05 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723688+0000", 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0071000000000001e-05 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723695+0000", 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3429999999999998e-06 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723868+0000", 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017280900000000001 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724003+0000", 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013505 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.724166+0000", 2024-09-21T19:03:14.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000162764 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729028+0000", 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048623590000000001 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729096+0000", 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7625000000000004e-05 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836438+0000", 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107342302 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839527+0000", 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030888589999999998 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839585+0000", 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7859999999999998e-05 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.916 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114539 7.6 7:6de988ff:::benchmark_data_smithi081_73391_object114538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.723706+0000", 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 464.14329341899997, 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11604534699999999, 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.917 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114539 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723706+0000", 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723707+0000", 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4399999999999999e-07 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723743+0000", 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5395999999999998e-05 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723749+0000", 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1850000000000001e-06 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723754+0000", 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6219999999999997e-06 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726769+0000", 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030147289999999998 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.726889+0000", 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012032999999999999 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.727041+0000", 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015136699999999999 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.729293+0000", 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002252085 2024-09-21T19:03:14.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.730090+0000", 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00079701999999999996 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839629+0000", 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10953908599999999 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839710+0000", 2024-09-21T19:03:14.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0630000000000006e-05 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.839752+0000", 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2153000000000001e-05 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121044 7.6 7:62f149b0:::benchmark_data_smithi081_73391_object121043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.421001+0000", 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 458.44599923999999, 2024-09-21T19:03:14.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084735675999999996, 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121044 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421001+0000", 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421002+0000", 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.336e-06 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421062+0000", 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0475e-05 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421072+0000", 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9389999999999992e-06 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421078+0000", 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2870000000000004e-06 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421128+0000", 2024-09-21T19:03:14.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9938000000000002e-05 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421252+0000", 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012434499999999999 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.421447+0000", 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019541299999999999 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425265+0000", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038176640000000001 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428690+0000", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034252710000000001 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505655+0000", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076964640000000001 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505691+0000", 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5887999999999999e-05 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.505736+0000", 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5479999999999998e-05 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:121051 7.6 7:619bd2cd:::benchmark_data_smithi081_73391_object121050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:36.425560+0000", 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 458.44144001900003, 2024-09-21T19:03:14.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088495110000000002, 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 121051 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425560+0000", 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425562+0000", 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.154e-06 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425607+0000", 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5229999999999999e-05 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425620+0000", 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2346e-05 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425627+0000", 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425678+0000", 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1053000000000001e-05 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425785+0000", 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107099 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.425916+0000", 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013072999999999999 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428156+0000", 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022408649999999999 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.428832+0000", 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067540999999999999 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.513961+0000", 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.085129596000000002 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514015+0000", 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3313000000000003e-05 2024-09-21T19:03:14.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:36.514055+0000", 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0156e-05 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211656 7.5 7:a3c1ba51:::benchmark_data_smithi081_73391_object72657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.348024+0000", 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 360.518975458, 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107860995, 2024-09-21T19:03:14.935 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211656 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348024+0000", 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348024+0000", 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348027+0000", 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6709999999999999e-06 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348033+0000", 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4779999999999999e-06 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348046+0000", 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3638999999999999e-05 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348053+0000", 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2980000000000003e-06 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348113+0000", 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9852999999999998e-05 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348226+0000", 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011289 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.348343+0000", 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116623 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.350450+0000", 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021071620000000001 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.357782+0000", 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073323420000000004 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455803+0000", 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.098020501999999995 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455840+0000", 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7206000000000003e-05 2024-09-21T19:03:14.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455885+0000", 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5330999999999997e-05 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:211679 7.5 7:a0aaf166:::benchmark_data_smithi081_73391_object72680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:14.371480+0000", 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 360.49551980799998, 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.084602176000000001, 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.941 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 211679 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371480+0000", 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371480+0000", 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371482+0000", 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7400000000000001e-06 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371486+0000", 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5820000000000003e-06 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371497+0000", 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0801e-05 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371503+0000", 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9000000000000003e-06 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371564+0000", 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0491000000000001e-05 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371687+0000", 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012302999999999999 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.371815+0000", 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864899999999999 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.373861+0000", 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020455870000000002 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.375558+0000", 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016976879999999999 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.455985+0000", 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.080426912000000003 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.456037+0000", 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1109999999999997e-05 2024-09-21T19:03:14.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:14.456082+0000", 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5686000000000003e-05 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242836 7.5 7:a22dd50b:::benchmark_data_smithi081_73391_object103837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:50.363052+0000", 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 324.50394832299997, 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.121274044, 2024-09-21T19:03:14.947 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242836 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363052+0000", 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363052+0000", 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363054+0000", 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1160000000000002e-06 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363059+0000", 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6729999999999997e-06 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363072+0000", 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2897e-05 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363080+0000", 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4229999999999999e-06 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363131+0000", 2024-09-21T19:03:14.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1759999999999999e-05 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363205+0000", 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3712000000000002e-05 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.363271+0000", 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5877999999999995e-05 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.365405+0000", 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021336670000000001 2024-09-21T19:03:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.366435+0000", 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010303689999999999 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484220+0000", 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11778470100000001 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:14.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484278+0000", 2024-09-21T19:03:14.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8464000000000001e-05 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:50.484326+0000", 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7383999999999999e-05 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237756 8.22 8.9e11cee2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 79.128935661, 2024-09-21T19:03:14.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94932720900000001, 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237756 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738066+0000", 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6950000000000001e-06 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738071+0000", 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1390000000000003e-06 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738083+0000", 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1609e-05 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738088+0000", 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3000000000000001e-06 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738133+0000", 2024-09-21T19:03:14.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5189999999999999e-05 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738139+0000", 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5860000000000004e-06 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.687381+0000", 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94924230799999998 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.687391+0000", 2024-09-21T19:03:14.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0382e-05 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2028 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94695648323712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 77.594609927999997, 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43766471099999998, 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.959 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2028 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5080000000000001e-06 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8600000000000005e-06 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272420+0000", 2024-09-21T19:03:14.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6631999999999999e-05 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1506e-05 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272509+0000", 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7401e-05 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:14.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272519+0000", 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0308e-05 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.701593+0000", 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.429073594 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.701753+0000", 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000160717 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.710055+0000", 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083011850000000009 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2038 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94695648323712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 77.313305280999998, 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14878480499999999, 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2038 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553698+0000", 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0649999999999999e-06 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553706+0000", 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3359999999999995e-06 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553727+0000", 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0899000000000001e-05 2024-09-21T19:03:14.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553737+0000", 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0056e-05 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553819+0000", 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2121999999999997e-05 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553828+0000", 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6789999999999994e-06 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702052+0000", 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.148224774 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702124+0000", 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1186000000000006e-05 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702479+0000", 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000355688 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.968 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:238532 8.15 8:ae744388:::benchmark_data_smithi081_105538_object106300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:58.509295+0000", 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 76.357704941999998, 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092772447999999993, 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 238532 2024-09-21T19:03:14.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509295+0000", 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509295+0000", 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509297+0000", 2024-09-21T19:03:14.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.841e-06 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509302+0000", 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0880000000000002e-06 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509313+0000", 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1041e-05 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509318+0000", 2024-09-21T19:03:14.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5949999999999998e-06 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509368+0000", 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9234999999999999e-05 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509443+0000", 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4889999999999996e-05 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1", 2024-09-21T19:03:14.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.509666+0000", 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000223367 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.510789+0000", 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001123088 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.601919+0000", 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.091130408999999996 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.601976+0000", 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6403999999999997e-05 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:58.602067+0000", 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1490000000000007e-05 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.974 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:239931 8.17 8:ec94a649:::benchmark_data_smithi081_105538_object107699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 75.320760800000002, 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13502588600000001, 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239931 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546241+0000", 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9300000000000002e-06 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546246+0000", 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3970000000000002e-06 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546258+0000", 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546265+0000", 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.99e-06 2024-09-21T19:03:14.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546441+0000", 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017672099999999999 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546556+0000", 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114626 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546806+0000", 2024-09-21T19:03:14.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025021199999999998 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.549068+0000", 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002261833 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681108+0000", 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13204001600000001 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681166+0000", 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8137999999999998e-05 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681265+0000", 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8869000000000005e-05 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:239934 8.17 8:e9fd7e5d:::benchmark_data_smithi081_105538_object107702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:59.547680+0000", 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 75.319320313999995, 2024-09-21T19:03:14.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.133722849, 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239934 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547680+0000", 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547680+0000", 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547681+0000", 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.564e-06 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547686+0000", 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6920000000000001e-06 2024-09-21T19:03:14.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547699+0000", 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2733e-05 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547706+0000", 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0670000000000003e-06 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547877+0000", 2024-09-21T19:03:14.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017096900000000001 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.548004+0000", 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127663 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.548231+0000", 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022652999999999999 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.549173+0000", 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094251700000000003 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681298+0000", 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.132124518 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681347+0000", 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8982e-05 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681402+0000", 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5614e-05 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.985 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.986 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.986 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:14.986 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:14.986 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: pgmap v494: 158 pgs: 1 peering, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 26 KiB/s, 0 objects/s recovering 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3994466469' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T19:03:15.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:14 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T19:03:15.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:15.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.175 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:15.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:15.309 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:15.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 38 16 66220 66204 108.868 77 0.0095383 0.00917928 2024-09-21T19:03:15.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:15.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:15.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:15.586 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:15.592 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:15.592 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:15.635 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:15.635 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:15.665 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:15.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:15.703 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:15.703 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:15.730 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:15.733 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:15.734 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:15.768 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:15.768 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:15.803 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:15.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:15.852 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:15.864 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:15.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:15.990 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:15.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: 4.15 scrub starts 2024-09-21T19:03:16.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: 4.15 scrub ok 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3994466469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1385480686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: 4.15 scrub starts 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: 4.15 scrub ok 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:03:16.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3994466469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T19:03:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T19:03:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T19:03:16.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1385480686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:16.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:16.117 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:16.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: 4.15 scrub starts 2024-09-21T19:03:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: 4.15 scrub ok 2024-09-21T19:03:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:03:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:03:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3994466469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:16.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T19:03:16.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T19:03:16.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T19:03:16.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1385480686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:16.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:16.402 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:16.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 39 16 67440 67424 108.032 76.25 0.0117837 0.00925153 2024-09-21T19:03:16.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:16.539 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:16.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:16.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:16.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:16.808 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:16.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:16.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:16.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:16.951 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:17.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:17.083 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:17.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T19:03:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T19:03:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T19:03:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: pgmap v496: 158 pgs: 158 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.6 KiB/s rd, 55 MiB/s wr, 879 op/s; 31 KiB/s, 0 objects/s recovering 2024-09-21T19:03:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: pgmap v496: 158 pgs: 158 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.6 KiB/s rd, 55 MiB/s wr, 879 op/s; 31 KiB/s, 0 objects/s recovering 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T19:03:17.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:03:17.209 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:16 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:03:17.216 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: pgmap v496: 158 pgs: 158 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.6 KiB/s rd, 55 MiB/s wr, 879 op/s; 31 KiB/s, 0 objects/s recovering 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:03:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:16 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:03:17.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:17.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:17.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:03:17.407049+0000 min lat: 0.00117786 max lat: 2.27862 avg lat: 0.00929865 2024-09-21T19:03:17.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:03:17.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 40 16 68817 68801 107.483 86.0625 0.0178993 0.00929865 2024-09-21T19:03:17.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:17.491 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:17.501 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:03:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:03:17.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:17.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:17.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:17.763 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:17.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:17.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:18.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:18.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: 4.19 scrub starts 2024-09-21T19:03:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: 4.19 scrub ok 2024-09-21T19:03:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T19:03:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T19:03:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T19:03:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T19:03:18.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:17 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T19:03:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:18.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:18.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:03:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:18.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: 4.19 scrub starts 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: 4.19 scrub ok 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: 4.19 scrub starts 2024-09-21T19:03:18.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: 4.19 scrub ok 2024-09-21T19:03:18.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T19:03:18.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T19:03:18.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T19:03:18.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T19:03:18.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:17 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T19:03:18.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:18.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:03:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.313 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:18.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:18.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 41 16 70097 70081 106.812 80 0.00745625 0.00935697 2024-09-21T19:03:18.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:18.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:18.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:03:18.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:18.602 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:18.619 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:18.686 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:18.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:18.752 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:18.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:18.884 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T19:03:18.911 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:18.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:18.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:03:19.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:19.051 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: pgmap v498: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 51 KiB/s rd, 99 MiB/s wr, 1.67k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3315416786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: from='client.25672 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: pgmap v498: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 51 KiB/s rd, 99 MiB/s wr, 1.67k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T19:03:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T19:03:19.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3315416786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:03:19.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: from='client.25672 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:03:19.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T19:03:19.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T19:03:19.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:18 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: pgmap v498: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 51 KiB/s rd, 99 MiB/s wr, 1.67k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3315416786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: from='client.25672 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T19:03:19.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:18 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:03:19.153 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:19.185 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:19.279 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:19.279 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:19.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:19.312 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:19.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 42 16 71156 71140 105.845 66.1875 0.00343286 0.00942278 2024-09-21T19:03:19.414 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:19.439 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:19.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:19.573 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:19.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:19.727 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:19.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:19.862 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:19.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:03:19.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:19.998 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:19.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:20.055 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:20.055 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: 3.18 scrub starts 2024-09-21T19:03:20.055 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: 3.18 scrub ok 2024-09-21T19:03:20.055 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T19:03:20.055 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: from='client.25672 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[39192]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: 3.18 scrub starts 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: 3.18 scrub ok 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: from='client.25672 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:03:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T19:03:20.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:03:20.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:03:20.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:03:20.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:19 smithi081 ceph-mon[29384]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T19:03:20.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:20.145 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:20.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: 3.18 scrub starts 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: 3.18 scrub ok 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: from='client.25672 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:03:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:19 smithi104 ceph-mon[32438]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T19:03:20.280 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:20.362 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:20.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:20.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 43 16 72368 72352 105.145 75.75 0.00506775 0.00950478 2024-09-21T19:03:20.427 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:20.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:20.543 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T19:03:20.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:20.585 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:20.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:20.692 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:20.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:20.837 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:20.932 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:03:20.934 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:20.934 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:20.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:20.981 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:20.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:20 smithi081 ceph-mon[39192]: pgmap v500: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 69 KiB/s rd, 69 MiB/s wr, 1.23k op/s 2024-09-21T19:03:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/56586837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:03:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:20 smithi081 ceph-mon[39192]: from='client.35783 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:03:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:20 smithi081 ceph-mon[29384]: pgmap v500: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 69 KiB/s rd, 69 MiB/s wr, 1.23k op/s 2024-09-21T19:03:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/56586837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:03:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:20 smithi081 ceph-mon[29384]: from='client.35783 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:03:21.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.119 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:21.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:21.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:20 smithi104 ceph-mon[32438]: pgmap v500: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 69 KiB/s rd, 69 MiB/s wr, 1.23k op/s 2024-09-21T19:03:21.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/56586837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:03:21.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:20 smithi104 ceph-mon[32438]: from='client.35783 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:03:21.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.246 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:21.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:21.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:21.382 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:21.408 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 44 16 73536 73520 104.414 73 0.0113506 0.00957234 2024-09-21T19:03:21.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.528 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:21.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:21.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:21.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:03:21.668 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:21.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:21.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:21.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:21.960 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:21.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:22.029 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:03:22.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: 6.b scrub starts 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: 6.b scrub ok 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:03:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2189392582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: 6.b scrub starts 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: 6.b scrub ok 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2189392582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:22.105 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:22.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:22.161 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:22.162 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:22.162 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:22.162 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:22.162 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:22.162 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:22.162 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:22.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: 6.b scrub starts 2024-09-21T19:03:22.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: 6.b scrub ok 2024-09-21T19:03:22.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:03:22.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:03:22.168 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:22.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:03:22.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T19:03:22.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:22.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2189392582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:22.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:22.244 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:22.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:22.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:22.381 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:22.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:22.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 45 16 74794 74778 103.841 78.625 0.0203595 0.00962491 2024-09-21T19:03:22.482 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:22.515 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:22.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:22.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:22.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:22.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:22.927 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:22.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:22.990 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:23.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:23.055 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:23.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:23.186 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:22 smithi104 ceph-mon[32438]: pgmap v503: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 KiB/s rd, 55 MiB/s wr, 970 op/s 2024-09-21T19:03:23.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:22 smithi081 ceph-mon[29384]: pgmap v503: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 KiB/s rd, 55 MiB/s wr, 970 op/s 2024-09-21T19:03:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:22 smithi081 ceph-mon[39192]: pgmap v503: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 KiB/s rd, 55 MiB/s wr, 970 op/s 2024-09-21T19:03:23.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:23.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 46 16 76116 76100 103.38 82.625 0.00728409 0.00966594 2024-09-21T19:03:23.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:23.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:23.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:23.585 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:23.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:23.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:23.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:23.865 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:23.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:23.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:24.005 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:24.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:03:24.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:24.139 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:24.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:24.273 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:24.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:24.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 47 16 77321 77305 102.782 75.3125 0.0155695 0.00972467 2024-09-21T19:03:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:24.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:24.453 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:03:24.453 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPush(9.b 263/262 [PushOp(9:d8c762a9:::benchmark_data_smithi081_131274_object3485:head, version: 259'205, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:d8c762a9:::benchmark_data_smithi081_131274_object3485:head@259'205, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:43.408063+0000", 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.040632166999998, 2024-09-21T19:03:24.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01784233, 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408063+0000", 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408063+0000", 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408065+0000", 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.215e-06 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408129+0000", 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4060999999999995e-05 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408142+0000", 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3523000000000001e-05 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408152+0000", 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4010000000000003e-06 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408231+0000", 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9809999999999997e-05 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408244+0000", 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2215000000000001e-05 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425906+0000", 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017662105000000001 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:11398 9.b e263/262 9:daeaad5e:::benchmark_data_smithi081_131274_object11397:head v 263'734, mlcod=259'204)", 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:43.408908+0000", 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.039787277000002, 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019368351999999998, 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408908+0000", 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408908+0000", 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408910+0000", 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.187e-06 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408946+0000", 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6644999999999999e-05 2024-09-21T19:03:24.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425482+0000", 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016535840999999999 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425494+0000", 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1590000000000001e-05 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425948+0000", 2024-09-21T19:03:24.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045490199999999998 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425993+0000", 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4421e-05 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.428248+0000", 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022553310000000002 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.428277+0000", 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8435000000000001e-05 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPush(9.1 263/262 [PushOp(9:8a955c6b:::benchmark_data_smithi081_131274_object2647:head, version: 259'161, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:8a955c6b:::benchmark_data_smithi081_131274_object2647:head@259'161, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:43.409046+0000", 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.039649808, 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017081439, 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.463 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.409046+0000", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.409046+0000", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.409046+0000", 2024-09-21T19:03:24.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8800000000000002e-07 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.409111+0000", 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4859999999999999e-05 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425478+0000", 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016366677 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425510+0000", 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2215000000000002e-05 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425618+0000", 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107604 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425636+0000", 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7827999999999999e-05 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.426127+0000", 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049166699999999999 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPush(9.0 263/262 [PushOp(9:093dc1bd:::benchmark_data_smithi081_131274_object7551:head, version: 260'459, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:093dc1bd:::benchmark_data_smithi081_131274_object7551:head@260'459, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:45.246951+0000", 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.201744386000001, 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013076838, 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.246951+0000", 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.246951+0000", 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.246952+0000", 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1790000000000001e-06 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259275+0000", 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012322866 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259299+0000", 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3524999999999999e-05 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259317+0000", 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7810000000000001e-05 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259420+0000", 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103625 2024-09-21T19:03:24.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259438+0000", 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8257000000000002e-05 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.260028+0000", 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00058957599999999997 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:16081 9.0 e263/262 9:0e88a00b:::benchmark_data_smithi081_131274_object16080:head v 263'995, mlcod=260'458)", 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:45.259344+0000", 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.189351152, 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020181035999999999, 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259344+0000", 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259344+0000", 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259345+0000", 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7899999999999997e-07 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278567+0000", 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019222132999999999 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278579+0000", 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1671999999999999e-05 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278585+0000", 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.302e-06 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278676+0000", 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0228000000000004e-05 2024-09-21T19:03:24.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278705+0000", 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9516e-05 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.279478+0000", 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00077334500000000004 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.279525+0000", 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7061000000000002e-05 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.475 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:16102 9.0 e263/262 9:05db5d84:::benchmark_data_smithi081_131274_object16101:head v 263'996, mlcod=260'458)", 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:45.278601+0000", 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.170094110999997, 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021694821999999999, 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278601+0000", 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278601+0000", 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278602+0000", 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7000000000000004e-07 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297657+0000", 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019054588000000001 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297677+0000", 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9950999999999999e-05 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297684+0000", 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.165e-06 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297763+0000", 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8876999999999997e-05 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297784+0000", 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1348e-05 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.300252+0000", 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024677879999999998 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.300296+0000", 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4335000000000002e-05 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20170 9.1 e265/264 9:855477f7:::benchmark_data_smithi081_131274_object20169:head v 265'1249, mlcod=265'1247)", 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.017697+0000", 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.430998744, 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.061766164999999998, 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.017697+0000", 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.017697+0000", 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.017698+0000", 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4810000000000001e-06 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078263+0000", 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060564918000000002 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078271+0000", 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6260000000000003e-06 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078276+0000", 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7210000000000002e-06 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078342+0000", 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6116000000000002e-05 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078361+0000", 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9527000000000001e-05 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.079387+0000", 2024-09-21T19:03:24.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010253180000000001 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.079463+0000", 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6458000000000006e-05 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20194 9.0 e265/264 9:0c9bf7c2:::benchmark_data_smithi081_131274_object20193:head v 265'1265, mlcod=265'1263)", 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.052087+0000", 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.396608624999999, 2024-09-21T19:03:24.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018127275000000002, 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.052087+0000", 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.052087+0000", 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.052089+0000", 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6700000000000001e-06 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.068924+0000", 2024-09-21T19:03:24.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016835209 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.068949+0000", 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5428000000000001e-05 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.068972+0000", 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2450999999999999e-05 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.069141+0000", 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016949000000000001 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.069174+0000", 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3253000000000002e-05 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.070164+0000", 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098973500000000005 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.070214+0000", 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0039e-05 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20380 9.b e265/264 9:db99f4de:::benchmark_data_smithi081_131274_object20379:head v 265'1288, mlcod=265'1286)", 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.143267+0000", 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.305428765000002, 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017892808, 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.490 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.143267+0000", 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.143267+0000", 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.143268+0000", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4279999999999999e-06 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159714+0000", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016446104999999999 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159727+0000", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.254e-05 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159733+0000", 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.66e-06 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159783+0000", 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0850999999999998e-05 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159800+0000", 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6263999999999999e-05 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.161113+0000", 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001313074 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.161160+0000", 2024-09-21T19:03:24.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6885999999999998e-05 2024-09-21T19:03:24.495 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:24.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.495 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20397 9.0 e265/264 9:02223bef:::benchmark_data_smithi081_131274_object20396:head v 265'1277, mlcod=265'1275)", 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.147416+0000", 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.301279528000002, 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013874393, 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.147416+0000", 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.147416+0000", 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.147418+0000", 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.454e-06 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159848+0000", 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012430293 2024-09-21T19:03:24.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159881+0000", 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3456e-05 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159897+0000", 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5512e-05 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159963+0000", 2024-09-21T19:03:24.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6494000000000004e-05 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159984+0000", 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1097000000000001e-05 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.161243+0000", 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001258667 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.161290+0000", 2024-09-21T19:03:24.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7420000000000003e-05 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20383 9.b e265/264 9:d078fbde:::benchmark_data_smithi081_131274_object20382:head v 265'1289, mlcod=265'1286)", 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.159973+0000", 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.288722171000003, 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020680995000000001, 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159973+0000", 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159973+0000", 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159974+0000", 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6499999999999999e-07 2024-09-21T19:03:24.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179511+0000", 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019537171999999998 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179517+0000", 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9699999999999996e-06 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179523+0000", 2024-09-21T19:03:24.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0379999999999996e-06 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179592+0000", 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8739999999999996e-05 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179609+0000", 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6538999999999999e-05 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.180619+0000", 2024-09-21T19:03:24.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010108389999999999 2024-09-21T19:03:24.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.180654+0000", 2024-09-21T19:03:24.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5031999999999999e-05 2024-09-21T19:03:24.505 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.505 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.505 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20432 9.1 e265/264 9:8c967db3:::benchmark_data_smithi081_131274_object20431:head v 265'1261, mlcod=265'1259)", 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.183935+0000", 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.264760543000001, 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015866846, 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.183935+0000", 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.183935+0000", 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.183941+0000", 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1399999999999997e-06 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.198319+0000", 2024-09-21T19:03:24.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014377757 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.198335+0000", 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5968e-05 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.198343+0000", 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2910000000000007e-06 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.198453+0000", 2024-09-21T19:03:24.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109511 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.198508+0000", 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5220000000000003e-05 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.199756+0000", 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001248019 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.199802+0000", 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5939999999999997e-05 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:45973 9.1 e271/264 9:8e01c6ff:::benchmark_data_smithi081_131274_object45972:head v 271'2817, mlcod=271'2813)", 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:00.871645+0000", 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 23.577050230000001, 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022307799, 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.871645+0000", 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.871645+0000", 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.871646+0000", 2024-09-21T19:03:24.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1699999999999998e-07 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.890826+0000", 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019180413 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.890846+0000", 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.961e-05 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.890855+0000", 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9709999999999993e-06 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.891035+0000", 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018034099999999999 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.891058+0000", 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2971000000000001e-05 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.893633+0000", 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025748469999999999 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.893953+0000", 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032002899999999999 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:51175 9.0 e272/264 9:008df47d:::benchmark_data_smithi081_131274_object51174:head v 272'3157, mlcod=272'3154)", 2024-09-21T19:03:24.514 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:04.037239+0000", 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.411457080000002, 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016122500000000001, 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.037239+0000", 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.037239+0000", 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.037240+0000", 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.883e-06 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051454+0000", 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014213537 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051476+0000", 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1977999999999999e-05 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051489+0000", 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2736e-05 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051571+0000", 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2794999999999996e-05 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051597+0000", 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5794000000000001e-05 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.053273+0000", 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016759559999999999 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.053361+0000", 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7820999999999998e-05 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:51179 9.0 e272/264 9:0fe09c03:::benchmark_data_smithi081_131274_object51178:head v 272'3158, mlcod=272'3155)", 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:04.051554+0000", 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.397141387000001, 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022420020999999998, 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.519 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051554+0000", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051554+0000", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051555+0000", 2024-09-21T19:03:24.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.074e-06 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070810+0000", 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019254934000000001 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070846+0000", 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5941e-05 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070873+0000", 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6699e-05 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070949+0000", 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5819e-05 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070971+0000", 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2586999999999999e-05 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.073850+0000", 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028786390000000001 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.073974+0000", 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012432799999999999 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:51187 9.0 e272/264 9:0207673d:::benchmark_data_smithi081_131274_object51186:head v 272'3160, mlcod=272'3155)", 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:04.071039+0000", 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.377656379000001, 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022849504, 2024-09-21T19:03:24.523 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.071039+0000", 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.071039+0000", 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.071040+0000", 2024-09-21T19:03:24.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.13e-07 2024-09-21T19:03:24.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090220+0000", 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019180064 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090239+0000", 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8753000000000002e-05 2024-09-21T19:03:24.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090247+0000", 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8070000000000006e-06 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090310+0000", 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2716000000000001e-05 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090332+0000", 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1603e-05 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.093595+0000", 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032630070000000001 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.093889+0000", 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029394099999999998 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:62048 9.0 e275/264 9:045801f9:::benchmark_data_smithi081_131274_object62047:head v 275'3863, mlcod=275'3861)", 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:12.314554+0000", 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 12.134141889, 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011259409999999999, 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314554+0000", 2024-09-21T19:03:24.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314554+0000", 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314555+0000", 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.668e-06 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314599+0000", 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3952000000000002e-05 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314616+0000", 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6274e-05 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314624+0000", 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1300000000000001e-06 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314776+0000", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015216900000000001 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.314821+0000", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4889999999999999e-05 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.325677+0000", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010856558000000001 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:12.325813+0000", 2024-09-21T19:03:24.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013576899999999999 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:65785 9.b e276/264 9:d56336b3:::benchmark_data_smithi081_131274_object65784:head v 276'4179, mlcod=276'4177)", 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:15.083433+0000", 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.3652626219999995, 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022701378000000001, 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.083433+0000", 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.083433+0000", 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.083434+0000", 2024-09-21T19:03:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102055+0000", 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018620384 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102066+0000", 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0659000000000001e-05 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102071+0000", 2024-09-21T19:03:24.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.135e-06 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102144+0000", 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3219000000000005e-05 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102177+0000", 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2728999999999998e-05 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.105996+0000", 2024-09-21T19:03:24.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038189399999999998 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.106134+0000", 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013881200000000001 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76679 9.b e280/264 9:d2675948:::benchmark_data_smithi081_131274_object76678:head v 280'4854, mlcod=280'4851)", 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.879754+0000", 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.568941425, 2024-09-21T19:03:24.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023429959, 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.879754+0000", 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.879754+0000", 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.879755+0000", 2024-09-21T19:03:24.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5900000000000003e-07 2024-09-21T19:03:24.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898779+0000", 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019024946000000001 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898799+0000", 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9938000000000001e-05 2024-09-21T19:03:24.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898809+0000", 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3489999999999998e-06 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898983+0000", 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000174558 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.899011+0000", 2024-09-21T19:03:24.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7339e-05 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.903155+0000", 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041439129999999999 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.903184+0000", 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9556999999999998e-05 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76716 9.b e280/264 9:def1090e:::benchmark_data_smithi081_131274_object76715:head v 280'4856, mlcod=280'4852)", 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.898926+0000", 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.54976910599999995, 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021674242, 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898926+0000", 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898926+0000", 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898927+0000", 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918136+0000", 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019208920000000001 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918147+0000", 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0685e-05 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918153+0000", 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6900000000000003e-06 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918208+0000", 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5034000000000002e-05 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918228+0000", 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9646000000000001e-05 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.920521+0000", 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022929339999999999 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:03:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.920601+0000", 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9748999999999998e-05 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:03:24.548 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:03:24.578 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:24.620 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T19:03:24.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:24.712 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:24.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.850 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:24.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:24.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:24.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:03:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:24.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:24.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:25.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:25.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:25.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:25.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:24 smithi104 ceph-mon[32438]: pgmap v504: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 6.3 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-09-21T19:03:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2861703862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:03:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2861703862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:03:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:24 smithi104 ceph-mon[32438]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T19:03:25.259 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[29384]: pgmap v504: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 6.3 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-09-21T19:03:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2861703862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:03:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2861703862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:03:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[29384]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T19:03:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[39192]: pgmap v504: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 6.3 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-09-21T19:03:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2861703862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:03:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2861703862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:03:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:24 smithi081 ceph-mon[39192]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T19:03:25.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:25.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:03:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.399 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:25.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:25.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 48 16 78580 78564 102.28 78.6875 0.0073801 0.00977244 2024-09-21T19:03:25.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:25.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:25.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.693 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:25.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:25.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:25.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:25.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:25.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:25.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:26.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:26.095 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:26.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.220 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:26.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:26.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:26.341 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:26.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 49 15 79701 79686 101.624 70.125 0.005091 0.00983573 2024-09-21T19:03:26.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:26.481 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:26.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.620 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:26.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:26.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:26.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:26.864 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:26.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:27.000 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:27.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:27.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T19:03:27.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:27.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:27.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:26 smithi081 ceph-mon[29384]: pgmap v506: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T19:03:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:26 smithi081 ceph-mon[39192]: pgmap v506: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T19:03:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:26 smithi104 ceph-mon[32438]: pgmap v506: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T19:03:27.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:27.327 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:27.406 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:27.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 50 16 81012 80996 101.229 81.875 0.0180617 0.00987169 2024-09-21T19:03:27.417 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:27.429 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:27.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:27.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:27.558 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:03:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:03:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:03:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:27.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:27.700 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.728 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:27.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:27.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:27.874 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:27.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:27.959 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:03:27.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:28.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:28.111 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:28.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:28.226 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:27 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2481151109' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:28.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:27 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2481151109' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:28.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:28.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:27 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2481151109' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:28.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:03:28.281 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:28.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:28.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 51 16 82179 82163 100.674 72.9375 0.00992754 0.00992818 2024-09-21T19:03:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.415 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:28.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:28.516 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:28.547 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:28.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:28.689 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:28.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:28.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:28.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:28.970 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:28.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:29.022 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:29.045 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:29.045 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:29.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:29.089 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:29.111 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:29.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:03:29.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:03:29.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[39192]: pgmap v507: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-21T19:03:29.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:29.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2481151109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:29.124 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[39192]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T19:03:29.124 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:29.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:03:29.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:03:29.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[29384]: pgmap v507: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-21T19:03:29.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:29.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2481151109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:29.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:28 smithi081 ceph-mon[29384]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T19:03:29.153 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:03:29.153 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:03:29.153 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.153 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:82513 9.9 9:94e76e90:::benchmark_data_smithi081_131274_object82512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:03:28.666189+0000", 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0047943150000000004, 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048258299999999997, 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 82513 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.154 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666189+0000", 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666189+0000", 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666191+0000", 2024-09-21T19:03:29.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.832e-06 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666212+0000", 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1061e-05 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666223+0000", 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0716999999999999e-05 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:29.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666230+0000", 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2949999999999996e-06 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.667012+0000", 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00078239399999999995 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.667132+0000", 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119218 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.667306+0000", 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017457599999999999 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670522+0000", 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032157990000000001 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:82514 9.5 9:ab39da52:::benchmark_data_smithi081_131274_object82513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:03:28.666913+0000", 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.004070527, 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041818549999999999, 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 82514 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666913+0000", 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666913+0000", 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666914+0000", 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1459999999999999e-06 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666940+0000", 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5680000000000001e-05 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666954+0000", 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3857e-05 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.666961+0000", 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1779999999999997e-06 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670154+0000", 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031936579999999998 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670291+0000", 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013712800000000001 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670438+0000", 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000146704 2024-09-21T19:03:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.15816.0:82513 9.9 e282/254)", 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:03:28.670604+0000", 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00037898499999999998, 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00054687099999999999, 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670604+0000", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670604+0000", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670606+0000", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2610000000000001e-06 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:29.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670611+0000", 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9749999999999998e-06 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670620+0000", 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1050000000000001e-06 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.670626+0000", 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3849999999999999e-06 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:82516 9.b 9:d79a54a3:::benchmark_data_smithi081_131274_object82515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:03:28.671054+0000", 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4294967295.999929, 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013982, 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 82516 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.166 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671054+0000", 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671054+0000", 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:29.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671056+0000", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.564e-06 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671076+0000", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0489999999999999e-05 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671087+0000", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0767e-05 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671094+0000", 2024-09-21T19:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9099999999999999e-06 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:03:28.671151+0000", 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6657999999999999e-05 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 4 2024-09-21T19:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:03:29.170 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:29.170 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:29.189 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:29.213 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:29.226 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:29.226 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:29.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:29.254 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:29.255 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:03:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:03:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-mon[32438]: pgmap v507: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-21T19:03:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2481151109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:28 smithi104 ceph-mon[32438]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T19:03:29.291 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:29.291 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:29.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:29.389 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:29.409 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 52 16 83429 83413 100.24 78.125 0.0108361 0.00997107 2024-09-21T19:03:29.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:29.522 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:29.624 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:29.658 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:29.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.796 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:29.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:29.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:29.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:29.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:30.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:30.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:30.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:30.200 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:29 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:03:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:29 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:03:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:29 smithi104 ceph-mon[32438]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T19:03:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3234649413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:30.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[39192]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3234649413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[29384]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T19:03:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3234649413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:30.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:30.410 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 53 16 84845 84829 100.018 88.5 0.0082452 0.00999302 2024-09-21T19:03:30.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:30.473 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:30.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:30.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:30.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:30.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:30.887 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:30.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:30.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:31.020 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:31.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:31.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:30 smithi104 ceph-mon[32438]: pgmap v510: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-21T19:03:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.294 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:31.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:31.302 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:30 smithi081 ceph-mon[39192]: pgmap v510: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-21T19:03:31.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:30 smithi081 ceph-mon[29384]: pgmap v510: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-21T19:03:31.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:31.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 54 16 86176 86160 99.7047 83.1875 0.0131631 0.0100249 2024-09-21T19:03:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:31.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:31.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:31.575 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:31.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.719 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:31.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:31.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:31.868 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:31.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:31.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:03:31.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:32.004 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:32.107 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:32.135 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:32.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:32.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:32.357 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:32.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:32.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 55 16 87387 87371 99.2674 75.6875 0.0055405 0.0100681 2024-09-21T19:03:32.418 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:32.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:03:32.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.568 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:32.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:32.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:32.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:32.800 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:32.833 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:32.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:32.963 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:32.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:32.991 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:32.991 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:32.997 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T19:03:33.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:33.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:03:33.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:33.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:32 smithi081 ceph-mon[29384]: pgmap v511: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 732 B/s rd, 56 MiB/s wr, 893 op/s 2024-09-21T19:03:33.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:32 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/663291101' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:03:33.138 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:32 smithi081 ceph-mon[39192]: pgmap v511: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 732 B/s rd, 56 MiB/s wr, 893 op/s 2024-09-21T19:03:33.138 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:32 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/663291101' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:03:33.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:33.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:32 smithi104 ceph-mon[32438]: pgmap v511: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 732 B/s rd, 56 MiB/s wr, 893 op/s 2024-09-21T19:03:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:32 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/663291101' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:03:33.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:33.368 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:33.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 56 16 88574 88558 98.8194 74.1875 0.0158329 0.0101148 2024-09-21T19:03:33.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:33.499 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:33.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:33.632 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:33.734 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.769 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:33.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:33.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:33.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:34.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:34.039 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:34.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:34.179 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/663291101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:03:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:34 smithi104 ceph-mon[32438]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T19:03:34.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:34.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/663291101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:03:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:34 smithi081 ceph-mon[29384]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T19:03:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/663291101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:03:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:34 smithi081 ceph-mon[39192]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T19:03:34.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 57 16 89894 89878 98.533 82.5 0.0087722 0.0101445 2024-09-21T19:03:34.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:34.455 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:34.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:34.697 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:34.728 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:34.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:34.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:34.967 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:34.989 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:34.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:35.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:35.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:35.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:35.253 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:35 smithi081 ceph-mon[39192]: pgmap v513: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T19:03:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:35 smithi081 ceph-mon[39192]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T19:03:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:35 smithi081 ceph-mon[29384]: pgmap v513: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T19:03:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:35 smithi081 ceph-mon[29384]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T19:03:35.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.395 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:35.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:35.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 58 16 91128 91112 98.1639 77.125 0.0102735 0.0101822 2024-09-21T19:03:35.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:35.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:35 smithi104 ceph-mon[32438]: pgmap v513: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T19:03:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:35 smithi104 ceph-mon[32438]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T19:03:35.536 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:35.638 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:35.669 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:35.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.803 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:35.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:35.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:35.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:35.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:36.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.080 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:36.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:36.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:36.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:36.350 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:36.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 59 16 92512 92496 97.9661 86.5 0.00800584 0.010203 2024-09-21T19:03:36.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:36.475 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:36.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:36.616 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:36.717 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:36.751 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:36.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:36.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:36.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:36.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:36.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:37.131 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.167 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:37.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:37.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:37.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:37 smithi081 ceph-mon[29384]: pgmap v515: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 771 B/s rd, 60 MiB/s wr, 961 op/s 2024-09-21T19:03:37.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:37.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:37 smithi081 ceph-mon[39192]: pgmap v515: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 771 B/s rd, 60 MiB/s wr, 961 op/s 2024-09-21T19:03:37.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:37.291 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:03:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:03:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:03:37.302 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:37.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T19:03:37.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:37.408 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:37.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:03:37.410369+0000 min lat: 0.00117786 max lat: 2.27862 avg lat: 0.0102326 2024-09-21T19:03:37.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:03:37.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 60 16 93806 93790 97.6811 80.875 0.0136719 0.0102326 2024-09-21T19:03:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:37.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:37.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:03:37.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:37 smithi104 ceph-mon[32438]: pgmap v515: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 771 B/s rd, 60 MiB/s wr, 961 op/s 2024-09-21T19:03:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:37.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.606 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:37.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:37.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.761 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:37.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:37.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:37.921 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:37.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:38.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:38.051 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:38.090 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:38.153 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:38.180 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:38.180 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:38.182 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:38.221 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:38.222 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:38.257 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:38.257 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:38.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:38.287 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:38.288 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:38.323 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:38.323 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:38.326 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:38.362 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:38.362 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:38.397 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:38.397 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:38.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 61 16 94856 94840 97.1555 65.625 0.0231162 0.010288 2024-09-21T19:03:38.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:38.435 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:38.436 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:38.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:38.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:03:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:03:38.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:38.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:38.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:38.742 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:38.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:38.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:38.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:38.987 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:38.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:39.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.120 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:39.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:39.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:39.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:39.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:39 smithi081 ceph-mon[39192]: pgmap v516: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T19:03:39.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1723559308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:39 smithi081 ceph-mon[29384]: pgmap v516: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T19:03:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1723559308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:39.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:39.392 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:39.411 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 62 16 95729 95713 96.4685 54.5625 0.0178109 0.0103617 2024-09-21T19:03:39.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:39.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:39 smithi104 ceph-mon[32438]: pgmap v516: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T19:03:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1723559308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:39.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:39.751 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:39.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:39.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:39.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:40.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:40.033 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:03:40.040 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:40.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:40.184 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:40.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:40 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/681318724' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:40 smithi081 ceph-mon[29384]: from='client.35819 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:40 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/681318724' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:40 smithi081 ceph-mon[39192]: from='client.35819 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.318 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:40.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:40.407 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:40 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:40.407 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/681318724' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:40.407 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:40 smithi104 ceph-mon[32438]: from='client.35819 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:03:40.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 63 16 96841 96825 96.0403 69.5 0.00963487 0.0104075 2024-09-21T19:03:40.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:40.453 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:40.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:40.589 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:40.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:40.724 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:40.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:40.864 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:40.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:40.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:41.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:41.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:41.137 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[39192]: pgmap v517: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T19:03:41.137 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:41.137 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[39192]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:03:41.137 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[39192]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T19:03:41.138 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:03:41.138 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:03:41.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[29384]: pgmap v517: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T19:03:41.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:41.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[29384]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:03:41.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[29384]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T19:03:41.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:03:41.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:41 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:03:41.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:41.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:41 smithi104 ceph-mon[32438]: pgmap v517: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T19:03:41.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:41 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:03:41.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:41 smithi104 ceph-mon[32438]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:03:41.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:41 smithi104 ceph-mon[32438]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T19:03:41.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:41 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:03:41.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:41 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:03:41.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.287 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:41.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:41.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:41.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 64 16 98060 98044 95.7301 76.1875 0.0154991 0.0104407 2024-09-21T19:03:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:41.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:41.453 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:41.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:41.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:41.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:41.726 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:41.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:41.874 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:41.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:41.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:42.006 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:42.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:42.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:42.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:42.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:03:42.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:03:42.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T19:03:42.263 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T19:03:42.263 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:03:42.263 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:03:42.263 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T19:03:42.263 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:42 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T19:03:42.278 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216350 7.6 7:693ca6d8:::benchmark_data_smithi081_73391_object77351:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:03:42.315 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.792026846, 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2616681970000001, 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216350 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518164+0000", 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.593e-06 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518169+0000", 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6689999999999997e-06 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518176+0000", 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8180000000000004e-06 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518182+0000", 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2549999999999997e-06 2024-09-21T19:03:42.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518270+0000", 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8738000000000003e-05 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518279+0000", 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2660000000000001e-06 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779775+0000", 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.26149613 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779830+0000", 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5727999999999999e-05 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.320 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216360 7.6 7:6a035ea2:::benchmark_data_smithi081_73391_object77361:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.79200079200001, 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2839865960000001, 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216360 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518189+0000", 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5400000000000002e-07 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518191+0000", 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518196+0000", 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.622e-06 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518200+0000", 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6190000000000002e-06 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518299+0000", 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8587999999999999e-05 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518304+0000", 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6210000000000001e-06 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780077+0000", 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2617725909999999 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780187+0000", 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110469 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802102+0000", 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021914330999999999 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802175+0000", 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3116999999999998e-05 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216394 7.6 7:6d24eb15:::benchmark_data_smithi081_73391_object77395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.75117678599997, 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2297531310000001, 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216394 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559014+0000", 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0889999999999998e-06 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559020+0000", 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1669999999999998e-06 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559034+0000", 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4385e-05 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559041+0000", 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.238e-06 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570552+0000", 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011510347000000001 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570561+0000", 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6209999999999999e-06 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780222+0000", 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2096608149999999 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780290+0000", 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7988e-05 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780361+0000", 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1239e-05 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788612+0000", 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082509909999999992 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788622+0000", 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159999999999999e-05 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788671+0000", 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8939e-05 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788709+0000", 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7295999999999997e-05 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788765+0000", 2024-09-21T19:03:42.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6855999999999999e-05 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216429 7.6 7:6000252a:::benchmark_data_smithi081_73391_object77430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.69371312300001, 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1724721330000001, 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.333 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216429 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:03:42.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616478+0000", 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616484+0000", 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616495+0000", 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1134999999999999e-05 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616502+0000", 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8569999999999997e-06 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616558+0000", 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6023999999999998e-05 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616567+0000", 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8980000000000002e-06 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782338+0000", 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.165771565 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782505+0000", 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166558 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782607+0000", 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102357 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788626+0000", 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006018921 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788801+0000", 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000174852 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788853+0000", 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2006000000000003e-05 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788898+0000", 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4962000000000001e-05 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788948+0000", 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0006999999999997e-05 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216455 7.6 7:68c88556:::benchmark_data_smithi081_73391_object77456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.66521434700002, 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.144373474, 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216455 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644977+0000", 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5500000000000001e-06 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644983+0000", 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0360000000000003e-06 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.342 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644995+0000", 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2006e-05 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645003+0000", 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9459999999999998e-06 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645059+0000", 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5337999999999998e-05 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645068+0000", 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4320000000000005e-06 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.784935+0000", 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1398670499999999 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785006+0000", 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0840999999999995e-05 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785084+0000", 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8228999999999999e-05 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788630+0000", 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035455730000000002 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789144+0000", 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00051452099999999999 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789199+0000", 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4484000000000002e-05 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789251+0000", 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2633000000000002e-05 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789348+0000", 2024-09-21T19:03:42.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6835000000000004e-05 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216459 7.6 7:66d9de63:::benchmark_data_smithi081_73391_object77460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.66062554600001, 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1586775789999999, 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.347 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216459 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649566+0000", 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0540000000000002e-06 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649586+0000", 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0472e-05 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649616+0000", 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9848e-05 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649637+0000", 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1114000000000001e-05 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649894+0000", 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025646600000000001 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649902+0000", 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4589999999999998e-06 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789632+0000", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1397303350000001 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789788+0000", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155852 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789900+0000", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111909 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802209+0000", 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012309334 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802251+0000", 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1421000000000002e-05 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808109+0000", 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058583660000000003 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808145+0000", 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5374999999999999e-05 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808241+0000", 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6574000000000001e-05 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216478 7.6 7:69b4062c:::benchmark_data_smithi081_73391_object77479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.63623894, 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.137844316, 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216478 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673953+0000", 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3910000000000001e-06 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674028+0000", 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5197000000000005e-05 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674184+0000", 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156129 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674193+0000", 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0410000000000003e-06 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674323+0000", 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013050400000000001 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674368+0000", 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4211000000000001e-05 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792228+0000", 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.117859843 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792333+0000", 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105216 2024-09-21T19:03:42.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792492+0000", 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015956400000000001 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802291+0000", 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097984360000000006 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802325+0000", 2024-09-21T19:03:42.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4235e-05 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811655+0000", 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093300610000000006 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811704+0000", 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8520999999999998e-05 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811795+0000", 2024-09-21T19:03:42.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0966999999999998e-05 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216483 7.6 7:679f3a63:::benchmark_data_smithi081_73391_object77484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.62971099700002, 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1314383640000001, 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.361 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216483 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680480+0000", 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3010000000000001e-06 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680487+0000", 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8090000000000004e-06 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680532+0000", 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4369999999999997e-05 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680540+0000", 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5969999999999992e-06 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680600+0000", 2024-09-21T19:03:42.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0275000000000001e-05 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680610+0000", 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6090000000000007e-06 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798740+0000", 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1181301029999999 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798848+0000", 2024-09-21T19:03:42.365 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010757 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798947+0000", 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9349e-05 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802432+0000", 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034852860000000002 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808361+0000", 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059282689999999999 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811802+0000", 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003440869 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811840+0000", 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8197e-05 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811916+0000", 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6760000000000004e-05 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216512 7.6 7:6f163cff:::benchmark_data_smithi081_73391_object77513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.59537170499999, 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0972389849999999, 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.368 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216512 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714820+0000", 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2450000000000001e-06 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714826+0000", 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1859999999999997e-06 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714841+0000", 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5288999999999999e-05 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714850+0000", 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6540000000000005e-06 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714963+0000", 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112893 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714971+0000", 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7120000000000006e-06 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802465+0000", 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0874939400000001 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802564+0000", 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.904e-05 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802667+0000", 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103101 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808099+0000", 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054315889999999997 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808411+0000", 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031148299999999999 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811922+0000", 2024-09-21T19:03:42.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035117199999999999 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811958+0000", 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5766999999999998e-05 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812056+0000", 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8366e-05 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.374 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216545 7.6 7:6b69332e:::benchmark_data_smithi081_73391_object77546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.55700019400001, 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0617420390000001, 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216545 2024-09-21T19:03:42.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753191+0000", 2024-09-21T19:03:42.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7680000000000001e-06 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753196+0000", 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1410000000000004e-06 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753207+0000", 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1143e-05 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753214+0000", 2024-09-21T19:03:42.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.917e-06 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754192+0000", 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097769299999999996 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754200+0000", 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9400000000000002e-06 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.805892+0000", 2024-09-21T19:03:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0516927620000001 2024-09-21T19:03:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806005+0000", 2024-09-21T19:03:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113143 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806125+0000", 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119331 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.381 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808476+0000", 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023510919999999999 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812065+0000", 2024-09-21T19:03:42.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035891429999999999 2024-09-21T19:03:42.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.383 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.383 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814743+0000", 2024-09-21T19:03:42.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026784489999999998 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814839+0000", 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5358000000000004e-05 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814931+0000", 2024-09-21T19:03:42.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2158999999999996e-05 2024-09-21T19:03:42.385 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.385 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.385 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.386 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216552 7.6 7:63b154c8:::benchmark_data_smithi081_73391_object77553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.386 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:03:42.386 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.548148088, 2024-09-21T19:03:42.386 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.060947284, 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.387 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216552 2024-09-21T19:03:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:03:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762043+0000", 2024-09-21T19:03:42.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9889999999999999e-06 2024-09-21T19:03:42.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.390 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.391 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:42.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:42 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:03:42.392 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:42 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:03:42.393 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:42 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T19:03:42.393 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:42 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T19:03:42.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762056+0000", 2024-09-21T19:03:42.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3441999999999999e-05 2024-09-21T19:03:42.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.394 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762068+0000", 2024-09-21T19:03:42.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-05 2024-09-21T19:03:42.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762075+0000", 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1169999999999998e-06 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762135+0000", 2024-09-21T19:03:42.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9759000000000002e-05 2024-09-21T19:03:42.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762143+0000", 2024-09-21T19:03:42.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:03:42.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808532+0000", 2024-09-21T19:03:42.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0463883469999999 2024-09-21T19:03:42.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808647+0000", 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011510600000000001 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808757+0000", 2024-09-21T19:03:42.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109849 2024-09-21T19:03:42.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814943+0000", 2024-09-21T19:03:42.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006186558 2024-09-21T19:03:42.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814980+0000", 2024-09-21T19:03:42.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6729e-05 2024-09-21T19:03:42.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822841+0000", 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078607529999999998 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822895+0000", 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4282e-05 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822988+0000", 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3422000000000001e-05 2024-09-21T19:03:42.403 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216555 7.6 7:6310baba:::benchmark_data_smithi081_73391_object77556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 382.546822249, 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0695222289999999, 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.404 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216555 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763369+0000", 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.635e-06 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763374+0000", 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.101e-06 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763503+0000", 2024-09-21T19:03:42.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000129276 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763511+0000", 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5019999999999992e-06 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763815+0000", 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030339199999999998 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763823+0000", 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7600000000000002e-06 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812093+0000", 2024-09-21T19:03:42.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0482702930000001 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812199+0000", 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010665100000000001 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812307+0000", 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107206 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819747+0000", 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074399499999999999 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819790+0000", 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3297999999999998e-05 2024-09-21T19:03:42.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832684+0000", 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012894012 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832778+0000", 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4456999999999994e-05 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832889+0000", 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110696 2024-09-21T19:03:42.411 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235880 8.1e 8:7f5b8aa9:::benchmark_data_smithi081_105538_object103648:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 108.63617121199999, 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.088566871, 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:03:42.412 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235880 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7300000000000001e-07 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674021+0000", 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1380000000000001e-06 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674025+0000", 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3499999999999999e-06 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674029+0000", 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1690000000000002e-06 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842647+0000", 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16861802400000001 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842654+0000", 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2509999999999997e-06 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758247+0000", 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91559294899999999 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758376+0000", 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012917699999999999 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762523+0000", 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041469219999999999 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762585+0000", 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1418e-05 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235892 8.c 8:34d14ac3:::benchmark_data_smithi081_105538_object103660:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:03:42.417 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 108.636155808, 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0724834350000001, 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235892 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674034+0000", 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.27e-07 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674036+0000", 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674040+0000", 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0249999999999996e-06 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674044+0000", 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7160000000000001e-06 2024-09-21T19:03:42.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.868789+0000", 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19474559399999999 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.868798+0000", 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5029999999999997e-06 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.720825+0000", 2024-09-21T19:03:42.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.85202745899999999 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.720942+0000", 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117217 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.721225+0000", 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028250900000000002 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746281+0000", 2024-09-21T19:03:42.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025056334 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746375+0000", 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4040000000000001e-05 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746416+0000", 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0538999999999999e-05 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746478+0000", 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2225999999999997e-05 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746517+0000", 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8761999999999999e-05 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237693 8.21 8:86cd0139:::benchmark_data_smithi081_105538_object105461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 106.610747019, 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.19476636, 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.424 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237693 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699444+0000", 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999999999999999e-06 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699448+0000", 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9110000000000003e-06 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699456+0000", 2024-09-21T19:03:42.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3259999999999996e-06 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699471+0000", 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4681000000000001e-05 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700689+0000", 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012186180000000001 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700697+0000", 2024-09-21T19:03:42.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4739999999999999e-06 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698487+0000", 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.99779048800000003 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698497+0000", 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1120000000000002e-06 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889014+0000", 2024-09-21T19:03:42.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1905176980000001 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889174+0000", 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000159545 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889403+0000", 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022944299999999999 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891432+0000", 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002028448 2024-09-21T19:03:42.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891637+0000", 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020514200000000001 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891727+0000", 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0705000000000007e-05 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.893997+0000", 2024-09-21T19:03:42.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002269802 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894105+0000", 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107367 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894208+0000", 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001038 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237741 8.21 8:86a63524:::benchmark_data_smithi081_105538_object105509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 106.58750639, 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1717911829999998, 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237741 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722684+0000", 2024-09-21T19:03:42.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0270000000000001e-06 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722688+0000", 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7929999999999999e-06 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722697+0000", 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1290000000000002e-06 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722703+0000", 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6200000000000001e-06 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722891+0000", 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018788600000000001 2024-09-21T19:03:42.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722902+0000", 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0541e-05 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702948+0000", 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98004672199999998 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702955+0000", 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0549999999999996e-06 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890451+0000", 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1874960109999999 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890572+0000", 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012139800000000001 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890731+0000", 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015913599999999999 2024-09-21T19:03:42.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891711+0000", 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097963900000000003 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892154+0000", 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044374699999999998 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892258+0000", 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010389400000000001 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894257+0000", 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019987550000000001 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894360+0000", 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010312200000000001 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894474+0000", 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113708 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237675 8.1c 8:3e931efd:::benchmark_data_smithi081_105538_object105443:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 105.61683826700001, 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.207533752, 2024-09-21T19:03:42.441 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237675 2024-09-21T19:03:42.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693353+0000", 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9609999999999999e-06 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693358+0000", 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6030000000000004e-06 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694353+0000", 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00099412400000000009 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694362+0000", 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3500000000000003e-06 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698188+0000", 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038259919999999998 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698196+0000", 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7100000000000007e-06 2024-09-21T19:03:42.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900800+0000", 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2026040019999999 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900885+0000", 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5010000000000001e-05 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237706 8.1c 8:39f9b21e:::benchmark_data_smithi081_105538_object105474:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 105.615817244, 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2136628949999999, 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237706 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.447 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694373+0000", 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7300000000000005e-07 2024-09-21T19:03:42.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694377+0000", 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5800000000000002e-06 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694385+0000", 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3030000000000002e-06 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694390+0000", 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4140000000000002e-06 2024-09-21T19:03:42.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698219+0000", 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038285289999999998 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698225+0000", 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4269999999999996e-06 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901135+0000", 2024-09-21T19:03:42.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2029100589999999 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901255+0000", 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120221 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901527+0000", 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000272045 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907437+0000", 2024-09-21T19:03:42.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059092579999999997 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907688+0000", 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025114199999999999 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907731+0000", 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3180000000000003e-05 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907774+0000", 2024-09-21T19:03:42.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3256000000000001e-05 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907856+0000", 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1582000000000006e-05 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907917+0000", 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0930000000000001e-05 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908035+0000", 2024-09-21T19:03:42.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118096 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237735 8.1c 8:3aa9ee71:::benchmark_data_smithi081_105538_object105503:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 105.615792661, 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2197030149999999, 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.454 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237735 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694397+0000", 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7000000000000005e-07 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694400+0000", 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7470000000000001e-06 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694405+0000", 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6230000000000002e-06 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694410+0000", 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.972e-06 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698247+0000", 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038369789999999999 2024-09-21T19:03:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698253+0000", 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9390000000000002e-06 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.904941+0000", 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.206687686 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905096+0000", 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155236 2024-09-21T19:03:42.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905336+0000", 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024011 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907815+0000", 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024790139999999999 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908076+0000", 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026105799999999999 2024-09-21T19:03:42.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908089+0000", 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2847000000000001e-05 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908126+0000", 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7177e-05 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913917+0000", 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057903410000000001 2024-09-21T19:03:42.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913979+0000", 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2253000000000002e-05 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.914099+0000", 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120463 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237756 8.22 8:47738879:::benchmark_data_smithi081_105538_object105524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 105.61577291099999, 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.228195653, 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237756 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.463 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694417+0000", 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7199999999999999e-07 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694419+0000", 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.447e-06 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694425+0000", 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4689999999999996e-06 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694430+0000", 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9099999999999996e-06 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701670+0000", 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0072406739999999999 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701679+0000", 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908648+0000", 2024-09-21T19:03:42.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2069698309999999 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908761+0000", 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011227500000000001 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,3,4,5,7", 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908968+0000", 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020707700000000001 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912465+0000", 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003497207 2024-09-21T19:03:42.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912477+0000", 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2086000000000001e-05 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913031+0000", 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055408400000000004 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913427+0000", 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039556599999999997 2024-09-21T19:03:42.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913488+0000", 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1543000000000004e-05 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922372+0000", 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008883574 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922440+0000", 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7909000000000003e-05 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:03:42.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922612+0000", 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017206799999999999 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:03:42.470 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:03:42.470 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 65 16 99224 99208 95.3765 72.75 0.0079224 0.0104798 2024-09-21T19:03:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.471 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:42.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:42.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.594 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:42.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:42.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:42.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:42.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:42.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:42.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:03:42.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:43.055 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T19:03:43.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:43.127 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:43.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.270 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:43.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:43 smithi081 ceph-mon[39192]: pgmap v519: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s 2024-09-21T19:03:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:43 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/397221731' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:03:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:43 smithi081 ceph-mon[39192]: from='client.25723 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:43 smithi081 ceph-mon[29384]: pgmap v519: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s 2024-09-21T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:43 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/397221731' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:43 smithi081 ceph-mon[29384]: from='client.25723 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:03:43.372 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:43.406 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:43.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 66 16 100270 100254 94.9218 65.375 0.0195536 0.0105305 2024-09-21T19:03:43.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:03:43.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:43 smithi104 ceph-mon[32438]: pgmap v519: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s 2024-09-21T19:03:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:43 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/397221731' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:03:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:43 smithi104 ceph-mon[32438]: from='client.25723 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:03:43.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:43.592 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:43.694 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:43.724 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:43.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:43.861 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:43.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:43.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:43.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:44.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:44.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.275 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:44.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[29384]: 5.c scrub starts 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[29384]: 5.c scrub ok 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[29384]: from='client.25723 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[29384]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[29384]: 9.d scrub starts 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[39192]: 5.c scrub starts 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[39192]: 5.c scrub ok 2024-09-21T19:03:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[39192]: from='client.25723 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:03:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[39192]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T19:03:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:44 smithi081 ceph-mon[39192]: 9.d scrub starts 2024-09-21T19:03:44.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:44.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 67 16 101368 101352 94.5292 68.625 0.0142035 0.010574 2024-09-21T19:03:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:44 smithi104 ceph-mon[32438]: 5.c scrub starts 2024-09-21T19:03:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:44 smithi104 ceph-mon[32438]: 5.c scrub ok 2024-09-21T19:03:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:44 smithi104 ceph-mon[32438]: from='client.25723 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:03:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:44 smithi104 ceph-mon[32438]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T19:03:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:44 smithi104 ceph-mon[32438]: 9.d scrub starts 2024-09-21T19:03:44.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:44.551 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:44.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:44.673 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:44.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:44.811 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:44.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:44.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:45.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:45.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:45.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:45.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:45.314 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[29384]: pgmap v521: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[39192]: pgmap v521: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:03:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:45 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:03:45.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:45.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 68 15 102515 102500 94.1942 71.75 0.00675703 0.0106115 2024-09-21T19:03:45.453 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:45.484 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:45 smithi104 ceph-mon[32438]: pgmap v521: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T19:03:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:45 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T19:03:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:45 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T19:03:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:45 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:03:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:45 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:03:45.586 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:45.627 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:45.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:45.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:45.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:45.884 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:45.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:45.985 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:46.018 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:46.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:46.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:46.307 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:46.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:46.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 69 16 103690 103674 93.8924 73.375 0.00704785 0.0106448 2024-09-21T19:03:46.437 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:46 smithi104 ceph-mon[32438]: 9.f scrub starts 2024-09-21T19:03:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:46 smithi104 ceph-mon[32438]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T19:03:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:46 smithi104 ceph-mon[32438]: 9.a scrub starts 2024-09-21T19:03:46.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:46.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:46 smithi081 ceph-mon[29384]: 9.f scrub starts 2024-09-21T19:03:46.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:46 smithi081 ceph-mon[29384]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T19:03:46.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:46 smithi081 ceph-mon[29384]: 9.a scrub starts 2024-09-21T19:03:46.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:46 smithi081 ceph-mon[39192]: 9.f scrub starts 2024-09-21T19:03:46.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:46 smithi081 ceph-mon[39192]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T19:03:46.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:46 smithi081 ceph-mon[39192]: 9.a scrub starts 2024-09-21T19:03:46.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:46.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:46.710 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:46.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.846 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:46.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:46.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:46.968 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:46.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:47.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:47.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:47.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:47.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:47.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.388 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:47.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[29384]: 9.d scrub ok 2024-09-21T19:03:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[29384]: pgmap v523: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 922 op/s 2024-09-21T19:03:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[29384]: 5.1 scrub starts 2024-09-21T19:03:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[29384]: 5.1 scrub ok 2024-09-21T19:03:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[29384]: 9.7 scrub starts 2024-09-21T19:03:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[29384]: 9.f scrub ok 2024-09-21T19:03:47.406 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:03:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:03:47.406 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[39192]: 9.d scrub ok 2024-09-21T19:03:47.406 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[39192]: pgmap v523: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 922 op/s 2024-09-21T19:03:47.406 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[39192]: 5.1 scrub starts 2024-09-21T19:03:47.407 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[39192]: 5.1 scrub ok 2024-09-21T19:03:47.407 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[39192]: 9.7 scrub starts 2024-09-21T19:03:47.407 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:47 smithi081 ceph-mon[39192]: 9.f scrub ok 2024-09-21T19:03:47.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:03:47.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 70 15 104646 104631 93.4055 59.8125 0.0138742 0.0107009 2024-09-21T19:03:47.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:47.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:47 smithi104 ceph-mon[32438]: 9.d scrub ok 2024-09-21T19:03:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:47 smithi104 ceph-mon[32438]: pgmap v523: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 922 op/s 2024-09-21T19:03:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:47 smithi104 ceph-mon[32438]: 5.1 scrub starts 2024-09-21T19:03:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:47 smithi104 ceph-mon[32438]: 5.1 scrub ok 2024-09-21T19:03:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:47 smithi104 ceph-mon[32438]: 9.7 scrub starts 2024-09-21T19:03:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:47 smithi104 ceph-mon[32438]: 9.f scrub ok 2024-09-21T19:03:47.523 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:47.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.661 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:47.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:47.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:47.804 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:47.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:47.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:47.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:48.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T19:03:48.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:48.074 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:48.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:48.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:48.159 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:48.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:48.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:48 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:03:48.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:48 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:03:48.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:48 smithi081 ceph-mon[39192]: 9.2 scrub starts 2024-09-21T19:03:48.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:48 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:03:48.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:48 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:03:48.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:48 smithi081 ceph-mon[29384]: 9.2 scrub starts 2024-09-21T19:03:48.321 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:48.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:48.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 71 16 105655 105639 92.9772 63 0.00652304 0.0107496 2024-09-21T19:03:48.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:48.482 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:48.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:48.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:48 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:03:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:48 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:03:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:48 smithi104 ceph-mon[32438]: 9.2 scrub starts 2024-09-21T19:03:48.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:03:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:03:48.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:48.609 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:48.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:48.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:48.748 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:48.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:48.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:48.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:48.984 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:49.010 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:49.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:49.154 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:49.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:49.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:49.390 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:49.412 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 72 15 106785 106770 92.6676 70.6875 0.00622632 0.0107869 2024-09-21T19:03:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.426 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:49.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: pgmap v524: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: 9.7 scrub ok 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: 9.a scrub ok 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T19:03:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:49.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: pgmap v524: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: 9.7 scrub ok 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: 9.a scrub ok 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T19:03:49.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: pgmap v524: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: 9.7 scrub ok 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: 9.a scrub ok 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T19:03:49.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:49.565 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:49.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:49.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:49.766 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:49.789 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:49.789 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:49.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:49.827 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:49.827 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:49.846 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.858 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:49.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:49.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:49.879 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:49.912 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:49.912 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:49.943 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:49.944 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:49.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:49.982 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:49.983 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:49.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:50.015 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:50.015 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:50.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:50.132 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:50.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:50.262 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:50.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:50.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 73 16 107841 107825 92.3013 65.9375 0.0108642 0.0108296 2024-09-21T19:03:50.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:50.534 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:50.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:50.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:50.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:50.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[29384]: pgmap v526: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 779 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[29384]: 9.9 scrub starts 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1815210416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[39192]: pgmap v526: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 779 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[39192]: 9.9 scrub starts 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1815210416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T19:03:50.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:50 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T19:03:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.826 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:50.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:50.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:50.961 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:50.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:50 smithi104 ceph-mon[32438]: pgmap v526: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 779 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-21T19:03:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:50 smithi104 ceph-mon[32438]: 9.9 scrub starts 2024-09-21T19:03:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1815210416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:50 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T19:03:51.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:50 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T19:03:51.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:51.102 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:51.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:51.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:51.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:51.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 74 16 109000 108984 92.0328 72.4375 0.00769625 0.0108613 2024-09-21T19:03:51.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:51.497 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:51.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:51.632 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: 9.2 scrub ok 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3402310932' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: 3.11 scrub starts 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: 3.11 scrub ok 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: 6.1 scrub starts 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: 6.1 scrub ok 2024-09-21T19:03:51.728 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: 9.2 scrub ok 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3402310932' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: 3.11 scrub starts 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: 3.11 scrub ok 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: 6.1 scrub starts 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: 6.1 scrub ok 2024-09-21T19:03:51.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:51.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.760 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:51.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:51.762 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:03:51.762 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:03:51.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:51.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:51.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:51.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: 9.2 scrub ok 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3402310932' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: 3.11 scrub starts 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: 3.11 scrub ok 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: 6.1 scrub starts 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: 6.1 scrub ok 2024-09-21T19:03:52.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:03:52.025 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:52.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:52.160 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:52.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:52.301 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:52.402 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:52.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 75 16 110052 110036 91.6824 65.75 0.00639343 0.0109004 2024-09-21T19:03:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.424 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:52.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:52.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:52.564 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:52.666 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:52.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: pgmap v527: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 681 op/s 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3402310932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: 9.9 scrub ok 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: 5.1a scrub starts 2024-09-21T19:03:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:52 smithi104 ceph-mon[32438]: 5.1a scrub ok 2024-09-21T19:03:52.801 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:52.838 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:52.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:52.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: pgmap v527: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 681 op/s 2024-09-21T19:03:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3402310932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: 9.9 scrub ok 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: 5.1a scrub starts 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[39192]: 5.1a scrub ok 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: pgmap v527: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 681 op/s 2024-09-21T19:03:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:03:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:03:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3402310932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:03:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T19:03:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: 9.9 scrub ok 2024-09-21T19:03:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: 5.1a scrub starts 2024-09-21T19:03:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:52 smithi081 ceph-mon[29384]: 5.1a scrub ok 2024-09-21T19:03:53.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:53.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:53.158 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:03:53.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:53.233 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:53.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.373 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:53.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:53.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 76 16 111290 111274 91.494 77.375 0.0131862 0.0109252 2024-09-21T19:03:53.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.523 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:53.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:53.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:53.672 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:53.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:53.810 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:53.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:53.945 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:53.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:54.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:53 smithi104 ceph-mon[32438]: 4.b scrub starts 2024-09-21T19:03:54.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:53 smithi104 ceph-mon[32438]: 4.b scrub ok 2024-09-21T19:03:54.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:53 smithi081 ceph-mon[29384]: 4.b scrub starts 2024-09-21T19:03:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:53 smithi081 ceph-mon[29384]: 4.b scrub ok 2024-09-21T19:03:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:53 smithi081 ceph-mon[39192]: 4.b scrub starts 2024-09-21T19:03:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:53 smithi081 ceph-mon[39192]: 4.b scrub ok 2024-09-21T19:03:54.088 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:54.190 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:54.224 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:54.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:54.349 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:54.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:54.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 77 16 112422 112406 91.2246 70.75 0.0137552 0.0109574 2024-09-21T19:03:54.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:54.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:54.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:54.617 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:54.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:54.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:54.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:54.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:54.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:54.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:54 smithi104 ceph-mon[32438]: pgmap v529: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:54 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:54 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:54 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:03:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:54 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:03:55.027 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[39192]: pgmap v529: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T19:03:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T19:03:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T19:03:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:03:55.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:03:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[29384]: pgmap v529: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T19:03:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T19:03:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T19:03:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:03:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:54 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:03:55.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:55.160 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:55.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.298 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:55.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:55.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:55.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 78 16 113722 113706 91.0966 81.25 0.0137522 0.0109729 2024-09-21T19:03:55.430 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:55.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:55.569 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:55.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:55.674 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:55.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:55.800 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:55.846 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:55.948 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:55.983 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:55.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:56.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:56.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:56.235 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:03:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:56.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:56.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:56.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:56.413 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 79 16 114982 114966 90.9396 78.75 0.0105952 0.0109916 2024-09-21T19:03:56.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:56.547 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:56.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:03:56.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:56.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:56.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:56.809 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: 1.7 scrub starts 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: pgmap v530: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 675 op/s 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: 1.7 scrub ok 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: 3.17 scrub starts 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: 3.17 scrub ok 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: 1.7 scrub starts 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: pgmap v530: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 675 op/s 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: 1.7 scrub ok 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: 3.17 scrub starts 2024-09-21T19:03:56.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: 3.17 scrub ok 2024-09-21T19:03:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T19:03:56.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:56 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T19:03:56.882 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:56.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:56.969 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:03:56.977 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:56.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:57.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: 1.7 scrub starts 2024-09-21T19:03:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: pgmap v530: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 675 op/s 2024-09-21T19:03:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: 1.7 scrub ok 2024-09-21T19:03:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: 3.17 scrub starts 2024-09-21T19:03:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: 3.17 scrub ok 2024-09-21T19:03:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T19:03:57.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:56 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T19:03:57.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:57.099 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:57.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:57.232 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:57.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:57.367 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:57.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:03:57.412685+0000 min lat: 0.00117786 max lat: 2.27862 avg lat: 0.0110297 2024-09-21T19:03:57.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:03:57.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 80 16 116040 116024 90.6294 66.125 0.00506793 0.0110297 2024-09-21T19:03:57.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:57.503 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:57.533 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:03:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:03:57.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:57.628 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:57.700 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T19:03:57.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.768 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:57.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:03:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:03:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:03:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:03:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3046472221' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:03:57.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[39192]: from='client.25732 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:03:57.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:03:57.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:03:57.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:03:57.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:03:57.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3046472221' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:03:57.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:57 smithi081 ceph-mon[29384]: from='client.25732 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:03:57.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:57.894 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:57.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:03:57.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:57 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:03:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:57 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:03:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:57 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:03:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:57 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:03:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3046472221' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:03:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:57 smithi104 ceph-mon[32438]: from='client.25732 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:03:58.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:58.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:03:58.167 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:03:58.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:58.309 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:58.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:03:58.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 81 16 117247 117231 90.4418 75.4375 0.0126684 0.0110523 2024-09-21T19:03:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:58.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:03:58.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:03:58.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:58.566 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:03:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:03:58.582 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:58.595 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:03:58.596 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:03:58.624 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:03:58.625 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:03:58.662 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:03:58.662 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:03:58.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:58.693 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:03:58.693 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:03:58.714 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:58.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:58.726 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:03:58.727 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:03:58.755 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:03:58.756 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:03:58.793 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:03:58.793 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:03:58.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:58.842 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:58.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:58.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: pgmap v531: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: from='client.25732 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T19:03:59.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:58 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1053865890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: pgmap v531: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: from='client.25732 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T19:03:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1053865890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: pgmap v531: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: from='client.25732 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T19:03:59.058 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:58 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1053865890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:03:59.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:59.118 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:59.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:03:59.257 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:03:59.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:03:59.393 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:03:59.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 82 16 118520 118504 90.309 79.5625 0.00544663 0.0110689 2024-09-21T19:03:59.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:03:59.538 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:03:59.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:03:59.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T19:03:59.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:03:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:03:59.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:03:59.804 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:03:59.876 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:59.876 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: 6.1f scrub starts 2024-09-21T19:03:59.876 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: 6.1f scrub ok 2024-09-21T19:03:59.876 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: 5.1c scrub starts 2024-09-21T19:03:59.876 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: 5.1c scrub ok 2024-09-21T19:03:59.876 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: 5.4 scrub starts 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[39192]: 5.4 scrub ok 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: 6.1f scrub starts 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: 6.1f scrub ok 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: 5.1c scrub starts 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: 5.1c scrub ok 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: 5.4 scrub starts 2024-09-21T19:03:59.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:03:59 smithi081 ceph-mon[29384]: 5.4 scrub ok 2024-09-21T19:03:59.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:03:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:03:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:03:59.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: 6.1f scrub starts 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: 6.1f scrub ok 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: 5.1c scrub starts 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: 5.1c scrub ok 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: 5.4 scrub starts 2024-09-21T19:04:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:03:59 smithi104 ceph-mon[32438]: 5.4 scrub ok 2024-09-21T19:04:00.039 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:00.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:00.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:00.198 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:00.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:00.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:00.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 83 16 119653 119637 90.0741 70.8125 0.0145322 0.0110978 2024-09-21T19:04:00.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:00.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:00.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:00.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:00.710 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:04:00.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:00.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:00.884 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:00.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:00.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:00 smithi104 ceph-mon[32438]: pgmap v533: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T19:04:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:00 smithi104 ceph-mon[32438]: 3.1c scrub starts 2024-09-21T19:04:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:00 smithi104 ceph-mon[32438]: 3.1c scrub ok 2024-09-21T19:04:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:00 smithi104 ceph-mon[32438]: 2.0 scrub starts 2024-09-21T19:04:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:00 smithi104 ceph-mon[32438]: 2.0 scrub ok 2024-09-21T19:04:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1383162837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:04:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:01.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[39192]: pgmap v533: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[39192]: 3.1c scrub starts 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[39192]: 3.1c scrub ok 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[39192]: 2.0 scrub starts 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[39192]: 2.0 scrub ok 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1383162837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:04:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[29384]: pgmap v533: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T19:04:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[29384]: 3.1c scrub starts 2024-09-21T19:04:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[29384]: 3.1c scrub ok 2024-09-21T19:04:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[29384]: 2.0 scrub starts 2024-09-21T19:04:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[29384]: 2.0 scrub ok 2024-09-21T19:04:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1383162837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:04:01.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:01.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:01.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.280 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:01.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:01.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:01.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:01.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 84 16 120745 120729 89.8142 68.25 0.0157389 0.0111296 2024-09-21T19:04:01.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:01.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:01.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:01.683 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:01.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:01.828 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:01.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:01.966 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:01.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:02.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:01 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1383162837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:04:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:01 smithi104 ceph-mon[32438]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T19:04:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:01 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1383162837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:04:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:01 smithi081 ceph-mon[39192]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T19:04:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:01 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1383162837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:04:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:01 smithi081 ceph-mon[29384]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T19:04:02.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:02.103 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:02.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:02.249 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:02.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:02.389 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:02.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 85 16 122065 122049 89.7281 82.5 0.00968913 0.0111402 2024-09-21T19:04:02.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:02.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:02.636 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:02.666 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:02.744 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:02 smithi081 ceph-mon[39192]: pgmap v535: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 760 op/s 2024-09-21T19:04:02.745 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:02 smithi081 ceph-mon[29384]: pgmap v535: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 760 op/s 2024-09-21T19:04:02.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:02.795 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:04:02.795 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:04:02.795 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:04:02.795 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:04:02.795 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:04:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.824 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:02.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:02.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:02.963 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:02.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:02 smithi104 ceph-mon[32438]: pgmap v535: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 760 op/s 2024-09-21T19:04:03.059 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:03.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:03.095 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:03.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:03.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:03.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:03.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:03.362 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:03.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:03.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 86 16 123335 123319 89.6076 79.375 0.0101245 0.0111553 2024-09-21T19:04:03.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:03.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:04:03.501 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:03.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:03.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:04:03.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:03.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:03.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:03.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:03.886 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:03.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:03.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:04.047 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:04.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:04.170 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:04.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:04:04.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:04.298 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:04.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:04.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 87 16 124283 124267 89.2586 59.25 0.0114676 0.011199 2024-09-21T19:04:04.430 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:04.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:04.576 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:04.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:04.717 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:04.818 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:04.850 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:04.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:04.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:05.006 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T19:04:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:04 smithi104 ceph-mon[32438]: pgmap v536: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T19:04:05.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/395107167' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:04:05.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:04 smithi081 ceph-mon[39192]: pgmap v536: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T19:04:05.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/395107167' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:04:05.010 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:04 smithi081 ceph-mon[29384]: pgmap v536: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T19:04:05.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/395107167' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:04:05.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:05.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:05.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:05.243 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:05.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:05.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 88 16 125542 125526 89.1384 78.6875 0.00887959 0.0112141 2024-09-21T19:04:05.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:05.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:05.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:05.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:05.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:05.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:05.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:05.924 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:05.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/395107167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:04:05.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:05 smithi081 ceph-mon[29384]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T19:04:05.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/395107167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:04:05.998 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:05 smithi081 ceph-mon[39192]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T19:04:06.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/395107167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:04:06.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:05 smithi104 ceph-mon[32438]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T19:04:06.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:06.038 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:06.051 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:06.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T19:04:06.153 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:06.182 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:06.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:06.332 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:06.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 89 16 126796 126780 89.0174 78.375 0.0105783 0.0112294 2024-09-21T19:04:06.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:06.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.586 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:06.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:06.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:06.712 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:06.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:06.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:06 smithi081 ceph-mon[29384]: pgmap v538: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 47 MiB/s wr, 743 op/s 2024-09-21T19:04:06.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:06.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/964633058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:06.842 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:06 smithi081 ceph-mon[39192]: pgmap v538: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 47 MiB/s wr, 743 op/s 2024-09-21T19:04:06.842 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:06.842 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/964633058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:06.848 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:06.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:06.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:07.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:07.119 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:07.221 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:07.242 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:06 smithi104 ceph-mon[32438]: pgmap v538: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 47 MiB/s wr, 743 op/s 2024-09-21T19:04:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/964633058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:07.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:07.381 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:07.414 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 90 14 127852 127838 88.763 66.125 0.0575855 0.0112584 2024-09-21T19:04:07.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:07.503 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:07.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:04:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:04:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:04:07.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.627 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:07.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:07.727 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:07.749 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:07.755 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:07.755 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:07.755 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:07.851 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:07.872 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:07.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:07.998 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:07.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:08.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:08.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:08.225 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:08.248 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:08.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.373 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:08.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total time run: 90.0174 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total writes made: 127852 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Write size: 65536 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object size: 65536 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Bandwidth (MB/sec): 88.769 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Bandwidth: 33.4782 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max bandwidth (MB/sec): 216.812 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min bandwidth (MB/sec): 0 2024-09-21T19:04:08.415 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average IOPS: 1420 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev IOPS: 535.662 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max IOPS: 3469 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min IOPS: 0 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average Latency(s): 0.0112613 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Latency(s): 0.0303609 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max latency(s): 2.27862 2024-09-21T19:04:08.416 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min latency(s): 0.00117786 2024-09-21T19:04:08.420 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T19:04:08.474 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:08.506 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:08.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:04:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:04:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:08.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:08.649 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:08.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:08.788 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:08.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:08.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:08.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:08.973 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:09.037 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:09.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:08 smithi081 ceph-mon[29384]: pgmap v539: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T19:04:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:08 smithi081 ceph-mon[39192]: pgmap v539: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T19:04:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:09.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:09.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:09.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:09.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:08 smithi104 ceph-mon[32438]: pgmap v539: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T19:04:09.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:09.399 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:09.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:09.486 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:09.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:09.616 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:09.717 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:09.750 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:09.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:09.881 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:09.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:09.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:10.022 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:10.023 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:10.034 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:10.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:10.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:10.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:10.172 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:10.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:10.302 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:10.337 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:04:10.337 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:04:10.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:10.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:10.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:10.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:10.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:10.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:10.801 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:04:10.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:10.843 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:10.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:10.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:10 smithi104 ceph-mon[32438]: pgmap v540: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-21T19:04:10.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:10 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2511323882' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:10.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:10 smithi104 ceph-mon[32438]: from='client.25756 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:10.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:04:10.945 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:10.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:10 smithi081 ceph-mon[39192]: pgmap v540: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-21T19:04:10.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:10 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2511323882' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:10.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:10 smithi081 ceph-mon[39192]: from='client.25756 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:10.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:10 smithi081 ceph-mon[29384]: pgmap v540: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-21T19:04:10.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2511323882' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:10.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:10 smithi081 ceph-mon[29384]: from='client.25756 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:11.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:11.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:11.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.293 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:11.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:11.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:11.429 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:11.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:11.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:11.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:11.792 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:11.824 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:11.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:11.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:12.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:12.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:11 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:11 smithi081 ceph-mon[39192]: from='client.25756 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:11 smithi081 ceph-mon[39192]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T19:04:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:11 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:11 smithi081 ceph-mon[29384]: from='client.25756 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:11 smithi081 ceph-mon[29384]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T19:04:12.096 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:12.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:12.234 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:12.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:11 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:11 smithi104 ceph-mon[32438]: from='client.25756 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:11 smithi104 ceph-mon[32438]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T19:04:12.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:04:12.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:12.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:12.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:12.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:12.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:12.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:12.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:12.767 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:12.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:12.905 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:12.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:13.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:13.037 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:12 smithi081 ceph-mon[29384]: pgmap v542: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T19:04:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:12 smithi081 ceph-mon[39192]: pgmap v542: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T19:04:13.138 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:13.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:12 smithi104 ceph-mon[32438]: pgmap v542: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T19:04:13.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:13.296 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:13.396 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:13.397 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.422 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:13.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:13.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.560 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:13.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:13.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:04:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore.transactions t=2024-09-21T19:04:13.395874397Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T19:04:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:04:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=cleanup t=2024-09-21T19:04:13.39617828Z level=info msg="Completed cleanup jobs" duration=10.739431ms 2024-09-21T19:04:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:04:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=plugins.update.checker t=2024-09-21T19:04:13.533688785Z level=info msg="Update check succeeded" duration=61.703238ms 2024-09-21T19:04:13.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:13.819 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T19:04:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:13.911 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:13.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:14.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:14.044 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:14.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2659040931' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:04:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:13 smithi081 ceph-mon[29384]: from='client.25762 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:04:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2659040931' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:04:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:13 smithi081 ceph-mon[39192]: from='client.25762 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:04:14.146 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:14.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2659040931' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:04:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:13 smithi104 ceph-mon[32438]: from='client.25762 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:04:14.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.306 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:14.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:14.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:14.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:14.578 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:14.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:14.717 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:14.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.856 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:14.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:14.958 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:14.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:04:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:04:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[29384]: pgmap v543: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 1.22k op/s 2024-09-21T19:04:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[29384]: from='client.25762 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:04:15.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[29384]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T19:04:15.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:04:15.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:04:15.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[39192]: pgmap v543: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 1.22k op/s 2024-09-21T19:04:15.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[39192]: from='client.25762 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:04:15.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:14 smithi081 ceph-mon[39192]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T19:04:15.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:15.122 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:15.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:14 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:04:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:14 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:04:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:14 smithi104 ceph-mon[32438]: pgmap v543: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 1.22k op/s 2024-09-21T19:04:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:14 smithi104 ceph-mon[32438]: from='client.25762 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:04:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:14 smithi104 ceph-mon[32438]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T19:04:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.260 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:15.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:15.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:15.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:15.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:15.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:15.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:15.677 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:15.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T19:04:15.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:15.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:15.943 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:15.977 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:15.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:16.078 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:16.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.235 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:16.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:16.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.377 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:16.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:16.444 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:16.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:16.483 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:16.483 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:16.507 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:16.515 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:16.515 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:16.550 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:16.550 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:16.583 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:16.584 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:16.608 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:16.613 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:16.614 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:16.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:16.647 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:16.647 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:16.681 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:16.681 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:16.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:16.779 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:16.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:16.914 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:16.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:16.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:04:17.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:17.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:16 smithi081 ceph-mon[29384]: pgmap v545: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 14 MiB/s wr, 806 op/s 2024-09-21T19:04:17.102 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/493379515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:17.102 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:16 smithi081 ceph-mon[39192]: pgmap v545: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 14 MiB/s wr, 806 op/s 2024-09-21T19:04:17.102 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/493379515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:17.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:17.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:17.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:16 smithi104 ceph-mon[32438]: pgmap v545: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 14 MiB/s wr, 806 op/s 2024-09-21T19:04:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/493379515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:17.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:17.339 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:17.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:17.477 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:17.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:04:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:04:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:04:17.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:17.619 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:17.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:17.751 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:17.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:17.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:17.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:17.912 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:139678 9.6 9:6a737472:::benchmark_data_smithi081_131274_object11823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:04:17.945040+0000", 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.011055102000000001, 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011086062000000001, 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:04:17.961 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 139678 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945040+0000", 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945040+0000", 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945043+0000", 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2189999999999998e-06 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945048+0000", 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5470000000000004e-06 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945061+0000", 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2860000000000001e-05 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945068+0000", 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5889999999999999e-06 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945265+0000", 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019736499999999999 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945534+0000", 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026935000000000001 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945639+0000", 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104893 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:139679 9.6 9:6afb70eb:::benchmark_data_smithi081_131274_object11824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-21T19:04:17.965 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:04:17.945085+0000", 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.011010937, 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01111581, 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 139679 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945085+0000", 2024-09-21T19:04:17.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945085+0000", 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945085+0000", 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4099999999999998e-07 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:04:17.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945089+0000", 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.377e-06 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945095+0000", 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5849999999999996e-06 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.945101+0000", 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0780000000000001e-06 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.952340+0000", 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0072387750000000002 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.952781+0000", 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044121200000000002 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953304+0000", 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00052322700000000002 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.969 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.15816.0:139678 9.6 e295/274)", 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:04:17.949438+0000", 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0066573220000000002, 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006823544, 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:04:17.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949438+0000", 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949438+0000", 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949439+0000", 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0729999999999999e-06 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:04:17.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949444+0000", 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6659999999999999e-06 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949452+0000", 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2959999999999999e-06 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949463+0000", 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0716999999999999e-05 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.15816.0:139681 9.0 e295/264 9:0b9138ac:::benchmark_data_smithi081_131274_object11826:head v 295'8700, mlcod=295'8696)", 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:04:17.949642+0000", 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0064535199999999999, 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066598910000000002, 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949642+0000", 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949642+0000", 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949643+0000", 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1699999999999997e-07 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949648+0000", 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9570000000000001e-06 2024-09-21T19:04:17.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949658+0000", 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0601000000000001e-05 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.949664+0000", 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5609999999999998e-06 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.951916+0000", 2024-09-21T19:04:17.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022522729999999999 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.951967+0000", 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1102e-05 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:139686 9.6 9.5bd7b8e6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:04:17.953237+0000", 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0028579569999999999, 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031139190000000001, 2024-09-21T19:04:17.976 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 139686 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953237+0000", 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953237+0000", 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953240+0000", 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2259999999999999e-06 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953245+0000", 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.463e-06 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953257+0000", 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2242000000000001e-05 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:04:17.953264+0000", 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7909999999999999e-06 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 5 2024-09-21T19:04:17.979 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:04:17.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:18.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:18.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:18.168 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:18.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:18.308 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:18.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:18.443 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:18.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:04:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:04:18.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:18.580 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:18.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:18.712 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:18.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:18.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:18.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:04:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:18.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:18.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:18.968 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:19.005 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:19.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:18 smithi081 ceph-mon[29384]: pgmap v546: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.8 MiB/s wr, 1.20k op/s 2024-09-21T19:04:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:18 smithi081 ceph-mon[39192]: pgmap v546: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.8 MiB/s wr, 1.20k op/s 2024-09-21T19:04:19.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:19.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:19.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:18 smithi104 ceph-mon[32438]: pgmap v546: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.8 MiB/s wr, 1.20k op/s 2024-09-21T19:04:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:19.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:19.391 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:19.428 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:19.531 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:19.566 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:19.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:19.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:04:19.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:19.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.846 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:19.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:19.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:19.981 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:19.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:20.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:20.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:20.107 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:20.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.248 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:20.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:20.349 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:20.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:20.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:20.515 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:20.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:20.657 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:20.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:20.790 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:20.891 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:20.931 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:20.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:21.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:21.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: pgmap v547: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 MiB/s wr, 1.26k op/s 2024-09-21T19:04:21.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:04:21.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: pgmap v547: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 MiB/s wr, 1.26k op/s 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.075 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.076 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.076 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.076 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.076 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.076 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:20 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.079 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:21.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:21.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:21.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:21.247 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: pgmap v547: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 MiB/s wr, 1.26k op/s 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.248 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:20 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:21.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:21.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:21.465 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:21.524 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:21.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:21.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:21.810 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:21.840 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T19:04:21.848 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:04:21.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:21.950 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:21.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:21.958 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi104 to 3248M 2024-09-21T19:04:21.958 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4123971328' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='client.35879 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: Adjusting osd_memory_target on smithi081 to 3248M 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:04:21.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3322736254' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi104 to 3248M 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4123971328' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='client.35879 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: Adjusting osd_memory_target on smithi081 to 3248M 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:21.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3322736254' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:04:22.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:22.082 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:22.123 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:04:22.123 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:04:22.123 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:04:22.123 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:04:22.123 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:04:22.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.219 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:22.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi104 to 3248M 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4123971328' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='client.35879 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: Adjusting osd_memory_target on smithi081 to 3248M 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:22.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3322736254' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:04:22.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:22.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:22.490 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:22.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:22.636 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:22.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.771 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:22.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:22.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:22.907 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:22.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:23.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:23.038 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: pgmap v548: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.2 MiB/s wr, 1.11k op/s 2024-09-21T19:04:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:04:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:04:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3322736254' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: 3.14 scrub starts 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[39192]: 3.14 scrub ok 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: pgmap v548: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.2 MiB/s wr, 1.11k op/s 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:04:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:23.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:04:23.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3322736254' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:04:23.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T19:04:23.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: 3.14 scrub starts 2024-09-21T19:04:23.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:22 smithi081 ceph-mon[29384]: 3.14 scrub ok 2024-09-21T19:04:23.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:23.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:23.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:04:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: pgmap v548: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.2 MiB/s wr, 1.11k op/s 2024-09-21T19:04:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:04:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:04:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:04:23.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3322736254' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:04:23.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T19:04:23.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: 3.14 scrub starts 2024-09-21T19:04:23.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:22 smithi104 ceph-mon[32438]: 3.14 scrub ok 2024-09-21T19:04:23.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.334 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:23.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:23.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:23.486 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:23.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:23.725 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:23.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:23.897 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:23.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:23.910 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T19:04:23.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:04:23.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:24.044 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:24.146 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:24.175 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:24.276 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:24.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:24.445 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:24.484 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:24.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.580 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:24.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:24.720 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:24.822 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:24.850 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:24.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:24.986 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:24.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: 6.13 scrub starts 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: 6.13 scrub ok 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: pgmap v550: 158 pgs: 158 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T19:04:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: 6.13 scrub starts 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: 6.13 scrub ok 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: pgmap v550: 158 pgs: 158 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T19:04:25.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:24 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T19:04:25.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.122 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:25.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:25.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: 6.13 scrub starts 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: 6.13 scrub ok 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: pgmap v550: 158 pgs: 158 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T19:04:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:24 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T19:04:25.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:25.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:25.404 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:25.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:25.537 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:25.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:25.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:25.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:25.797 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:25.899 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:25.933 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:25.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:26.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:26.066 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:26.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.205 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:26.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:26.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3751425390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:26.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:26.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3751425390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:26.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3751425390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.339 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:26.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:26.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:26.460 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:26.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:04:26.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:26.591 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:26.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:26.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:26.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:26.856 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:26.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:26.993 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:26.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:27.094 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:27.125 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:26 smithi081 ceph-mon[39192]: pgmap v552: 158 pgs: 158 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s 2024-09-21T19:04:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:26 smithi081 ceph-mon[39192]: 3.6 scrub starts 2024-09-21T19:04:27.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:26 smithi081 ceph-mon[39192]: 3.6 scrub ok 2024-09-21T19:04:27.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:26 smithi081 ceph-mon[29384]: pgmap v552: 158 pgs: 158 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s 2024-09-21T19:04:27.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:26 smithi081 ceph-mon[29384]: 3.6 scrub starts 2024-09-21T19:04:27.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:26 smithi081 ceph-mon[29384]: 3.6 scrub ok 2024-09-21T19:04:27.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:27.254 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:26 smithi104 ceph-mon[32438]: pgmap v552: 158 pgs: 158 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s 2024-09-21T19:04:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:26 smithi104 ceph-mon[32438]: 3.6 scrub starts 2024-09-21T19:04:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:26 smithi104 ceph-mon[32438]: 3.6 scrub ok 2024-09-21T19:04:27.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:27.390 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:27.492 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:27.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.527 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:27.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:27.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:04:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:04:27.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:27.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:27.765 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:27.798 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:27.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:27.931 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:27.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:28.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:28.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:28.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:28.203 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:28.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:27 smithi104 ceph-mon[32438]: 5.0 scrub starts 2024-09-21T19:04:28.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:27 smithi104 ceph-mon[32438]: 5.0 scrub ok 2024-09-21T19:04:28.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:27 smithi104 ceph-mon[32438]: 4.4 scrub starts 2024-09-21T19:04:28.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:27 smithi104 ceph-mon[32438]: 4.4 scrub ok 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[29384]: 5.0 scrub starts 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[29384]: 5.0 scrub ok 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[29384]: 4.4 scrub starts 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[29384]: 4.4 scrub ok 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[39192]: 5.0 scrub starts 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[39192]: 5.0 scrub ok 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[39192]: 4.4 scrub starts 2024-09-21T19:04:28.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:27 smithi081 ceph-mon[39192]: 4.4 scrub ok 2024-09-21T19:04:28.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.358 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:28.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:28.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:28.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:28.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:04:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:04:28.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:28.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:28.800 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:28.879 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:28.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:28.946 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:28.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:28.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:29.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:29.099 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:29.178 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:28 smithi081 ceph-mon[29384]: pgmap v553: 158 pgs: 158 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T19:04:29.178 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:28 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T19:04:29.178 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:28 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T19:04:29.178 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:28 smithi081 ceph-mon[39192]: pgmap v553: 158 pgs: 158 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T19:04:29.178 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:28 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T19:04:29.178 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:28 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T19:04:29.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.229 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:29.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:28 smithi104 ceph-mon[32438]: pgmap v553: 158 pgs: 158 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T19:04:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:28 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T19:04:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:28 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T19:04:29.282 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:04:29.298 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:04:29.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.375 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:29.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:29.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:29.517 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:29.619 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:29.654 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:29.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.791 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:29.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:29.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:29.919 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:29.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:30.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:30.045 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:04:30.045 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:04:30.056 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:30.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:30.202 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:30.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:30.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:29 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:04:30.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:29 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:04:30.204 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:30.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:29 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:04:30.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:29 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:04:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:29 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:04:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:29 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:04:30.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:30.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:04:30.347 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:30.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:30.477 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:30.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.609 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:30.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:30.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:30.851 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:30.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:30.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:30.931 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:04:30.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: 3.d scrub starts 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: 3.d scrub ok 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: from='client.25783 v1:172.21.15.81:0/1722851263' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: pgmap v554: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1431702511' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:31.021 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: 3.d scrub starts 2024-09-21T19:04:31.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: 3.d scrub ok 2024-09-21T19:04:31.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: from='client.25783 v1:172.21.15.81:0/1722851263' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:31.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: pgmap v554: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:04:31.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T19:04:31.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T19:04:31.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1431702511' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.023 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:31.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:31.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:04:31.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:31.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: 3.d scrub starts 2024-09-21T19:04:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: 3.d scrub ok 2024-09-21T19:04:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: from='client.25783 v1:172.21.15.81:0/1722851263' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: pgmap v554: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T19:04:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T19:04:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T19:04:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1431702511' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.302 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:31.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:31.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:31.438 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:31.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.575 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:31.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:31.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:31.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.716 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:31.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:31.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:31.844 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:31.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:31.992 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:31.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: 4.14 scrub starts 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: 4.14 scrub ok 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: 9.6 scrub starts 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1431702511' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[29384]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: 4.14 scrub starts 2024-09-21T19:04:32.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: 4.14 scrub ok 2024-09-21T19:04:32.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: 9.6 scrub starts 2024-09-21T19:04:32.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:04:32.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:04:32.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1431702511' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:32.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:31 smithi081 ceph-mon[39192]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T19:04:32.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:32.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:32.239 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: 4.14 scrub starts 2024-09-21T19:04:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: 4.14 scrub ok 2024-09-21T19:04:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: 9.6 scrub starts 2024-09-21T19:04:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:04:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:04:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1431702511' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:31 smithi104 ceph-mon[32438]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T19:04:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:32.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:32.369 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:32.403 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:32.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:32.535 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:32.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:32.672 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:32.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:32.809 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:32.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:32.941 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:32.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:33.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:33.050 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2024-09-21T19:04:27.627835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6968580659,"num_objects":106523,"num_object_clones":0,"num_object_copies":319569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106523,"num_whiteouts":0,"num_read":1810,"num_read_kb":1562,"num_write":150104,"num_write_kb":8182574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":97720406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80425,"ondisk_log_size":80425,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23783672,"kb_used_data":20458160,"kb_used_omap":15,"kb_used_meta":3325232,"kb_avail":726177544,"statfs":{"total":767960285184,"available":743605805056,"internally_reserved":0,"allocated":20949155840,"data_stored":20946064729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":3405038354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,2,8,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-707002368,"num_objects":-10788,"num_object_clones":0,"num_object_copies":-32364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10788,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10788,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002852"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":406,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881423+0000","last_change":"2024-09-21T19:02:35.574023+0000","last_active":"2024-09-21T19:04:23.881423+0000","last_peered":"2024-09-21T19:04:23.881423+0000","last_clean":"2024-09-21T19:04:23.881423+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:04:23.881423+0000","last_undegraded":"2024-09-21T19:04:23.881423+0000","last_fullsized":"2024-09-21T19:04:23.881423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.711076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":574,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880771+0000","last_change":"2024-09-21T19:03:15.600063+0000","last_active":"2024-09-21T19:04:23.880771+0000","last_peered":"2024-09-21T19:04:23.880771+0000","last_clean":"2024-09-21T19:04:23.880771+0000","last_became_active":"2024-09-21T19:02:40.874205+0000","last_became_peered":"2024-09-21T19:02:40.874205+0000","last_unstale":"2024-09-21T19:04:23.880771+0000","last_undegraded":"2024-09-21T19:04:23.880771+0000","last_fullsized":"2024-09-21T19:04:23.880771+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:15.600006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.600006+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.243933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":509,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876336+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:04:23.876336+0000","last_peered":"2024-09-21T19:04:23.876336+0000","last_clean":"2024-09-21T19:04:23.876336+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:04:23.876336+0000","last_undegraded":"2024-09-21T19:04:23.876336+0000","last_fullsized":"2024-09-21T19:04:23.876336+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:34.519529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":497,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880981+0000","last_change":"2024-09-21T19:01:48.626265+0000","last_active":"2024-09-21T19:04:23.880981+0000","last_peered":"2024-09-21T19:04:23.880981+0000","last_clean":"2024-09-21T19:04:23.880981+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:04:23.880981+0000","last_undegraded":"2024-09-21T19:04:23.880981+0000","last_fullsized":"2024-09-21T19:04:23.880981+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.626214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.626214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:27.844445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":515,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848837+0000","last_change":"2024-09-21T19:02:40.877489+0000","last_active":"2024-09-21T19:04:21.848837+0000","last_peered":"2024-09-21T19:04:21.848837+0000","last_clean":"2024-09-21T19:04:21.848837+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:04:21.848837+0000","last_undegraded":"2024-09-21T19:04:21.848837+0000","last_fullsized":"2024-09-21T19:04:21.848837+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.831931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.831931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:24.454813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":528,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.859546+0000","last_change":"2024-09-21T19:02:47.515077+0000","last_active":"2024-09-21T19:04:21.859546+0000","last_peered":"2024-09-21T19:04:21.859546+0000","last_clean":"2024-09-21T19:04:21.859546+0000","last_became_active":"2024-09-21T19:02:40.879934+0000","last_became_peered":"2024-09-21T19:02:40.879934+0000","last_unstale":"2024-09-21T19:04:21.859546+0000","last_undegraded":"2024-09-21T19:04:21.859546+0000","last_fullsized":"2024-09-21T19:04:21.859546+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.514985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.514985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":593,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849113+0000","last_change":"2024-09-21T19:03:16.432252+0000","last_active":"2024-09-21T19:04:21.849113+0000","last_peered":"2024-09-21T19:04:21.849113+0000","last_clean":"2024-09-21T19:04:21.849113+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:04:21.849113+0000","last_undegraded":"2024-09-21T19:04:21.849113+0000","last_fullsized":"2024-09-21T19:04:21.849113+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:16.432117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.432117+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:25.798625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":528,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880740+0000","last_change":"2024-09-21T19:03:13.601474+0000","last_active":"2024-09-21T19:04:23.880740+0000","last_peered":"2024-09-21T19:04:23.880740+0000","last_clean":"2024-09-21T19:04:23.880740+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:04:23.880740+0000","last_undegraded":"2024-09-21T19:04:23.880740+0000","last_fullsized":"2024-09-21T19:04:23.880740+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:13.601407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.601407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":504,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881730+0000","last_change":"2024-09-21T19:03:10.612487+0000","last_active":"2024-09-21T19:04:23.881730+0000","last_peered":"2024-09-21T19:04:23.881730+0000","last_clean":"2024-09-21T19:04:23.881730+0000","last_became_active":"2024-09-21T19:03:09.619056+0000","last_became_peered":"2024-09-21T19:03:09.619056+0000","last_unstale":"2024-09-21T19:04:23.881730+0000","last_undegraded":"2024-09-21T19:04:23.881730+0000","last_fullsized":"2024-09-21T19:04:23.881730+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:10.612408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:10.612408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":572,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881137+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:04:23.881137+0000","last_peered":"2024-09-21T19:04:23.881137+0000","last_clean":"2024-09-21T19:04:23.881137+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:04:23.881137+0000","last_undegraded":"2024-09-21T19:04:23.881137+0000","last_fullsized":"2024-09-21T19:04:23.881137+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.858575+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":539,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.871524+0000","last_change":"2024-09-21T19:03:56.486775+0000","last_active":"2024-09-21T19:04:23.871524+0000","last_peered":"2024-09-21T19:04:23.871524+0000","last_clean":"2024-09-21T19:04:23.871524+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:04:23.871524+0000","last_undegraded":"2024-09-21T19:04:23.871524+0000","last_fullsized":"2024-09-21T19:04:23.871524+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:56.486496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.486496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.359373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":389,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882045+0000","last_change":"2024-09-21T18:59:51.189725+0000","last_active":"2024-09-21T19:04:23.882045+0000","last_peered":"2024-09-21T19:04:23.882045+0000","last_clean":"2024-09-21T19:04:23.882045+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:04:23.882045+0000","last_undegraded":"2024-09-21T19:04:23.882045+0000","last_fullsized":"2024-09-21T19:04:23.882045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.189659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.189659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.671622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":65,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882877+0000","last_change":"2024-09-21T19:03:59.885217+0000","last_active":"2024-09-21T19:04:23.882877+0000","last_peered":"2024-09-21T19:04:23.882877+0000","last_clean":"2024-09-21T19:04:23.882877+0000","last_became_active":"2024-09-21T19:03:09.628920+0000","last_became_peered":"2024-09-21T19:03:09.628920+0000","last_unstale":"2024-09-21T19:04:23.882877+0000","last_undegraded":"2024-09-21T19:04:23.882877+0000","last_fullsized":"2024-09-21T19:04:23.882877+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.276502+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":518,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881370+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:04:23.881370+0000","last_peered":"2024-09-21T19:04:23.881370+0000","last_clean":"2024-09-21T19:04:23.881370+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:04:23.881370+0000","last_undegraded":"2024-09-21T19:04:23.881370+0000","last_fullsized":"2024-09-21T19:04:23.881370+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.873852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":508,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869797+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:04:23.869797+0000","last_peered":"2024-09-21T19:04:23.869797+0000","last_clean":"2024-09-21T19:04:23.869797+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:04:23.869797+0000","last_undegraded":"2024-09-21T19:04:23.869797+0000","last_fullsized":"2024-09-21T19:04:23.869797+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:57.782768+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":65,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876165+0000","last_change":"2024-09-21T19:03:16.368532+0000","last_active":"2024-09-21T19:04:23.876165+0000","last_peered":"2024-09-21T19:04:23.876165+0000","last_clean":"2024-09-21T19:04:23.876165+0000","last_became_active":"2024-09-21T19:03:09.627894+0000","last_became_peered":"2024-09-21T19:03:09.627894+0000","last_unstale":"2024-09-21T19:04:23.876165+0000","last_undegraded":"2024-09-21T19:04:23.876165+0000","last_fullsized":"2024-09-21T19:04:23.876165+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:16.368451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.368451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.953461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":525,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881798+0000","last_change":"2024-09-21T19:03:12.639309+0000","last_active":"2024-09-21T19:04:23.881798+0000","last_peered":"2024-09-21T19:04:23.881798+0000","last_clean":"2024-09-21T19:04:23.881798+0000","last_became_active":"2024-09-21T19:02:40.879452+0000","last_became_peered":"2024-09-21T19:02:40.879452+0000","last_unstale":"2024-09-21T19:04:23.881798+0000","last_undegraded":"2024-09-21T19:04:23.881798+0000","last_fullsized":"2024-09-21T19:04:23.881798+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:12.638987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:12.638987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.019636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":397,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880615+0000","last_change":"2024-09-21T19:01:16.593955+0000","last_active":"2024-09-21T19:04:23.880615+0000","last_peered":"2024-09-21T19:04:23.880615+0000","last_clean":"2024-09-21T19:04:23.880615+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:04:23.880615+0000","last_undegraded":"2024-09-21T19:04:23.880615+0000","last_fullsized":"2024-09-21T19:04:23.880615+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:16.593895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.593895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.577306+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":556,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.878740+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:04:23.878740+0000","last_peered":"2024-09-21T19:04:23.878740+0000","last_clean":"2024-09-21T19:04:23.878740+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:04:23.878740+0000","last_undegraded":"2024-09-21T19:04:23.878740+0000","last_fullsized":"2024-09-21T19:04:23.878740+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.400512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":531,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876878+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:04:23.876878+0000","last_peered":"2024-09-21T19:04:23.876878+0000","last_clean":"2024-09-21T19:04:23.876878+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:04:23.876878+0000","last_undegraded":"2024-09-21T19:04:23.876878+0000","last_fullsized":"2024-09-21T19:04:23.876878+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.710160+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":508,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.864491+0000","last_change":"2024-09-21T19:03:58.416244+0000","last_active":"2024-09-21T19:04:21.864491+0000","last_peered":"2024-09-21T19:04:21.864491+0000","last_clean":"2024-09-21T19:04:21.864491+0000","last_became_active":"2024-09-21T19:02:40.879991+0000","last_became_peered":"2024-09-21T19:02:40.879991+0000","last_unstale":"2024-09-21T19:04:21.864491+0000","last_undegraded":"2024-09-21T19:04:21.864491+0000","last_fullsized":"2024-09-21T19:04:21.864491+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:01.286671+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":525,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881225+0000","last_change":"2024-09-21T19:01:33.521063+0000","last_active":"2024-09-21T19:04:23.881225+0000","last_peered":"2024-09-21T19:04:23.881225+0000","last_clean":"2024-09-21T19:04:23.881225+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:04:23.881225+0000","last_undegraded":"2024-09-21T19:04:23.881225+0000","last_fullsized":"2024-09-21T19:04:23.881225+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:33.520989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:33.520989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":582,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849171+0000","last_change":"2024-09-21T19:03:13.490415+0000","last_active":"2024-09-21T19:04:21.849171+0000","last_peered":"2024-09-21T19:04:21.849171+0000","last_clean":"2024-09-21T19:04:21.849171+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:04:21.849171+0000","last_undegraded":"2024-09-21T19:04:21.849171+0000","last_fullsized":"2024-09-21T19:04:21.849171+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:03:13.490332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.490332+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.917938+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":509,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882703+0000","last_change":"2024-09-21T19:03:58.898230+0000","last_active":"2024-09-21T19:04:23.882703+0000","last_peered":"2024-09-21T19:04:23.882703+0000","last_clean":"2024-09-21T19:04:23.882703+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:04:23.882703+0000","last_undegraded":"2024-09-21T19:04:23.882703+0000","last_fullsized":"2024-09-21T19:04:23.882703+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.898154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.898154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.123418+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":521,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869711+0000","last_change":"2024-09-21T19:03:41.162782+0000","last_active":"2024-09-21T19:04:23.869711+0000","last_peered":"2024-09-21T19:04:23.869711+0000","last_clean":"2024-09-21T19:04:23.869711+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:04:23.869711+0000","last_undegraded":"2024-09-21T19:04:23.869711+0000","last_fullsized":"2024-09-21T19:04:23.869711+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:44.255552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":529,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876413+0000","last_change":"2024-09-21T19:03:40.622949+0000","last_active":"2024-09-21T19:04:23.876413+0000","last_peered":"2024-09-21T19:04:23.876413+0000","last_clean":"2024-09-21T19:04:23.876413+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:04:23.876413+0000","last_undegraded":"2024-09-21T19:04:23.876413+0000","last_fullsized":"2024-09-21T19:04:23.876413+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:03:40.622880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.622880+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:49.026818+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":433,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880579+0000","last_change":"2024-09-21T19:01:37.492543+0000","last_active":"2024-09-21T19:04:23.880579+0000","last_peered":"2024-09-21T19:04:23.880579+0000","last_clean":"2024-09-21T19:04:23.880579+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:04:23.880579+0000","last_undegraded":"2024-09-21T19:04:23.880579+0000","last_fullsized":"2024-09-21T19:04:23.880579+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:01:37.492484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:37.492484+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.666040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"263'10","reported_seq":532,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.895436+0000","last_change":"2024-09-21T19:02:40.879205+0000","last_active":"2024-09-21T19:04:23.895436+0000","last_peered":"2024-09-21T19:04:23.895436+0000","last_clean":"2024-09-21T19:04:23.895436+0000","last_became_active":"2024-09-21T19:02:40.878691+0000","last_became_peered":"2024-09-21T19:02:40.878691+0000","last_unstale":"2024-09-21T19:04:23.895436+0000","last_undegraded":"2024-09-21T19:04:23.895436+0000","last_fullsized":"2024-09-21T19:04:23.895436+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.734152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"296'8970","reported_seq":9140,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:22.669093+0000","last_change":"2024-09-21T19:03:50.138454+0000","last_active":"2024-09-21T19:04:22.669093+0000","last_peered":"2024-09-21T19:04:22.669093+0000","last_clean":"2024-09-21T19:04:22.669093+0000","last_became_active":"2024-09-21T19:02:29.785679+0000","last_became_peered":"2024-09-21T19:02:29.785679+0000","last_unstale":"2024-09-21T19:04:22.669093+0000","last_undegraded":"2024-09-21T19:04:22.669093+0000","last_fullsized":"2024-09-21T19:04:22.669093+0000","mapping_epoch":254,"log_start":"280'4600","ondisk_log_start":"280'4600","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'6599","last_scrub_stamp":"2024-09-21T19:03:50.138203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.138203+0000","objects_scrubbed":6517,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:55.055653+0000","scrub_duration":2697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444071936,"num_objects":6776,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":503872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":66,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881988+0000","last_change":"2024-09-21T19:03:10.415757+0000","last_active":"2024-09-21T19:04:23.881988+0000","last_peered":"2024-09-21T19:04:23.881988+0000","last_clean":"2024-09-21T19:04:23.881988+0000","last_became_active":"2024-09-21T19:03:09.622733+0000","last_became_peered":"2024-09-21T19:03:09.622733+0000","last_unstale":"2024-09-21T19:04:23.881988+0000","last_undegraded":"2024-09-21T19:04:23.881988+0000","last_fullsized":"2024-09-21T19:04:23.881988+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:10.415628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:10.415628+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:30.260948+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":565,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.879858+0000","last_change":"2024-09-21T19:03:57.634402+0000","last_active":"2024-09-21T19:04:23.879858+0000","last_peered":"2024-09-21T19:04:23.879858+0000","last_clean":"2024-09-21T19:04:23.879858+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:04:23.879858+0000","last_undegraded":"2024-09-21T19:04:23.879858+0000","last_fullsized":"2024-09-21T19:04:23.879858+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:57.634230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.634230+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:58.234423+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":517,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.860826+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:04:21.860826+0000","last_peered":"2024-09-21T19:04:21.860826+0000","last_clean":"2024-09-21T19:04:21.860826+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:04:21.860826+0000","last_undegraded":"2024-09-21T19:04:21.860826+0000","last_fullsized":"2024-09-21T19:04:21.860826+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.469739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":691,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881871+0000","last_change":"2024-09-21T19:03:53.851419+0000","last_active":"2024-09-21T19:04:23.881871+0000","last_peered":"2024-09-21T19:04:23.881871+0000","last_clean":"2024-09-21T19:04:23.881871+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:04:23.881871+0000","last_undegraded":"2024-09-21T19:04:23.881871+0000","last_fullsized":"2024-09-21T19:04:23.881871+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:55.131138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":189,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876797+0000","last_change":"2024-09-21T19:03:14.420119+0000","last_active":"2024-09-21T19:04:23.876797+0000","last_peered":"2024-09-21T19:04:23.876797+0000","last_clean":"2024-09-21T19:04:23.876797+0000","last_became_active":"2024-09-21T19:03:09.623771+0000","last_became_peered":"2024-09-21T19:03:09.623771+0000","last_unstale":"2024-09-21T19:04:23.876797+0000","last_undegraded":"2024-09-21T19:04:23.876797+0000","last_fullsized":"2024-09-21T19:04:23.876797+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:14.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:25.608093+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"297'9386","reported_seq":9558,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.658070+0000","last_change":"2024-09-21T19:03:45.264553+0000","last_active":"2024-09-21T19:04:26.658070+0000","last_peered":"2024-09-21T19:04:26.658070+0000","last_clean":"2024-09-21T19:04:26.658070+0000","last_became_active":"2024-09-21T19:02:29.786797+0000","last_became_peered":"2024-09-21T19:02:29.786797+0000","last_unstale":"2024-09-21T19:04:26.658070+0000","last_undegraded":"2024-09-21T19:04:26.658070+0000","last_fullsized":"2024-09-21T19:04:26.658070+0000","mapping_epoch":254,"log_start":"280'4800","ondisk_log_start":"280'4800","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6439","last_scrub_stamp":"2024-09-21T19:03:45.264423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.264423+0000","objects_scrubbed":6363,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.021933+0000","scrub_duration":2070,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435748888,"num_objects":6650,"num_object_clones":0,"num_object_copies":19950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6650,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9386,"num_write_kb":513089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":293,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881695+0000","last_change":"2024-09-21T19:02:45.696512+0000","last_active":"2024-09-21T19:04:23.881695+0000","last_peered":"2024-09-21T19:04:23.881695+0000","last_clean":"2024-09-21T19:04:23.881695+0000","last_became_active":"2024-09-21T19:02:40.874426+0000","last_became_peered":"2024-09-21T19:02:40.874426+0000","last_unstale":"2024-09-21T19:04:23.881695+0000","last_undegraded":"2024-09-21T19:04:23.881695+0000","last_fullsized":"2024-09-21T19:04:23.881695+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":605,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.860478+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:04:21.860478+0000","last_peered":"2024-09-21T19:04:21.860478+0000","last_clean":"2024-09-21T19:04:21.860478+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:04:21.860478+0000","last_undegraded":"2024-09-21T19:04:21.860478+0000","last_fullsized":"2024-09-21T19:04:21.860478+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.535087+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":531,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.864750+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:04:21.864750+0000","last_peered":"2024-09-21T19:04:21.864750+0000","last_clean":"2024-09-21T19:04:21.864750+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:04:21.864750+0000","last_undegraded":"2024-09-21T19:04:21.864750+0000","last_fullsized":"2024-09-21T19:04:21.864750+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:28.448107+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":518,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881308+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:04:23.881308+0000","last_peered":"2024-09-21T19:04:23.881308+0000","last_clean":"2024-09-21T19:04:23.881308+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:04:23.881308+0000","last_undegraded":"2024-09-21T19:04:23.881308+0000","last_fullsized":"2024-09-21T19:04:23.881308+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:52.440101+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":498,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881102+0000","last_change":"2024-09-21T19:02:11.853722+0000","last_active":"2024-09-21T19:04:23.881102+0000","last_peered":"2024-09-21T19:04:23.881102+0000","last_clean":"2024-09-21T19:04:23.881102+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:04:23.881102+0000","last_undegraded":"2024-09-21T19:04:23.881102+0000","last_fullsized":"2024-09-21T19:04:23.881102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:11.853678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:11.853678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.736684+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"296'9149","reported_seq":9244,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:22.670662+0000","last_change":"2024-09-21T19:02:30.709345+0000","last_active":"2024-09-21T19:04:22.670662+0000","last_peered":"2024-09-21T19:04:22.670662+0000","last_clean":"2024-09-21T19:04:22.670662+0000","last_became_active":"2024-09-21T19:02:30.709045+0000","last_became_peered":"2024-09-21T19:02:30.709045+0000","last_unstale":"2024-09-21T19:04:22.670662+0000","last_undegraded":"2024-09-21T19:04:22.670662+0000","last_fullsized":"2024-09-21T19:04:22.670662+0000","mapping_epoch":254,"log_start":"280'4800","ondisk_log_start":"280'4800","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452657152,"num_objects":6907,"num_object_clones":0,"num_object_copies":20721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":277,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.860090+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:04:21.860090+0000","last_peered":"2024-09-21T19:04:21.860090+0000","last_clean":"2024-09-21T19:04:21.860090+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:04:21.860090+0000","last_undegraded":"2024-09-21T19:04:21.860090+0000","last_fullsized":"2024-09-21T19:04:21.860090+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:01.404232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":567,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880985+0000","last_change":"2024-09-21T19:02:42.663946+0000","last_active":"2024-09-21T19:04:23.880985+0000","last_peered":"2024-09-21T19:04:23.880985+0000","last_clean":"2024-09-21T19:04:23.880985+0000","last_became_active":"2024-09-21T19:02:40.885348+0000","last_became_peered":"2024-09-21T19:02:40.885348+0000","last_unstale":"2024-09-21T19:04:23.880985+0000","last_undegraded":"2024-09-21T19:04:23.880985+0000","last_fullsized":"2024-09-21T19:04:23.880985+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T19:02:42.663859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:42.663859+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":409,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:25.838960+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:04:25.838960+0000","last_peered":"2024-09-21T19:04:25.838960+0000","last_clean":"2024-09-21T19:04:25.838960+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:04:25.838960+0000","last_undegraded":"2024-09-21T19:04:25.838960+0000","last_fullsized":"2024-09-21T19:04:25.838960+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.439642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":637,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848737+0000","last_change":"2024-09-21T19:00:22.845024+0000","last_active":"2024-09-21T19:04:21.848737+0000","last_peered":"2024-09-21T19:04:21.848737+0000","last_clean":"2024-09-21T19:04:21.848737+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:04:21.848737+0000","last_undegraded":"2024-09-21T19:04:21.848737+0000","last_fullsized":"2024-09-21T19:04:21.848737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:22.844953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.844953+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":508,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869670+0000","last_change":"2024-09-21T19:02:40.887898+0000","last_active":"2024-09-21T19:04:23.869670+0000","last_peered":"2024-09-21T19:04:23.869670+0000","last_clean":"2024-09-21T19:04:23.869670+0000","last_became_active":"2024-09-21T19:02:40.886800+0000","last_became_peered":"2024-09-21T19:02:40.886800+0000","last_unstale":"2024-09-21T19:04:23.869670+0000","last_undegraded":"2024-09-21T19:04:23.869670+0000","last_fullsized":"2024-09-21T19:04:23.869670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.644848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.644848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:28.334259+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"297'9365","reported_seq":9535,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.679877+0000","last_change":"2024-09-21T19:03:46.619927+0000","last_active":"2024-09-21T19:04:26.679877+0000","last_peered":"2024-09-21T19:04:26.679877+0000","last_clean":"2024-09-21T19:04:26.679877+0000","last_became_active":"2024-09-21T19:02:29.787909+0000","last_became_peered":"2024-09-21T19:02:29.787909+0000","last_unstale":"2024-09-21T19:04:26.679877+0000","last_undegraded":"2024-09-21T19:04:26.679877+0000","last_fullsized":"2024-09-21T19:04:26.679877+0000","mapping_epoch":254,"log_start":"278'4400","ondisk_log_start":"278'4400","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6487","last_scrub_stamp":"2024-09-21T19:03:46.619847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:46.619847+0000","objects_scrubbed":6415,"log_size":4965,"log_dups_size":0,"ondisk_log_size":4965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.392510+0000","scrub_duration":2131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428408832,"num_objects":6537,"num_object_clones":0,"num_object_copies":19611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":508864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":322,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.895662+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:04:23.895662+0000","last_peered":"2024-09-21T19:04:23.895662+0000","last_clean":"2024-09-21T19:04:23.895662+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:04:23.895662+0000","last_undegraded":"2024-09-21T19:04:23.895662+0000","last_fullsized":"2024-09-21T19:04:23.895662+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:23.574104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":544,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.877140+0000","last_change":"2024-09-21T19:03:50.509414+0000","last_active":"2024-09-21T19:04:23.877140+0000","last_peered":"2024-09-21T19:04:23.877140+0000","last_clean":"2024-09-21T19:04:23.877140+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:04:23.877140+0000","last_undegraded":"2024-09-21T19:04:23.877140+0000","last_fullsized":"2024-09-21T19:04:23.877140+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:03:50.509275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.509275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:57.379074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":55,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849262+0000","last_change":"2024-09-21T19:03:09.624294+0000","last_active":"2024-09-21T19:04:21.849262+0000","last_peered":"2024-09-21T19:04:21.849262+0000","last_clean":"2024-09-21T19:04:21.849262+0000","last_became_active":"2024-09-21T19:03:09.623941+0000","last_became_peered":"2024-09-21T19:03:09.623941+0000","last_unstale":"2024-09-21T19:04:21.849262+0000","last_undegraded":"2024-09-21T19:04:21.849262+0000","last_fullsized":"2024-09-21T19:04:21.849262+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"263'12","reported_seq":510,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880517+0000","last_change":"2024-09-21T19:02:40.897242+0000","last_active":"2024-09-21T19:04:23.880517+0000","last_peered":"2024-09-21T19:04:23.880517+0000","last_clean":"2024-09-21T19:04:23.880517+0000","last_became_active":"2024-09-21T19:02:40.896856+0000","last_became_peered":"2024-09-21T19:02:40.896856+0000","last_unstale":"2024-09-21T19:04:23.880517+0000","last_undegraded":"2024-09-21T19:04:23.880517+0000","last_fullsized":"2024-09-21T19:04:23.880517+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:01:36.539314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:36.539314+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.638141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":398,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882183+0000","last_change":"2024-09-21T19:02:18.444667+0000","last_active":"2024-09-21T19:04:23.882183+0000","last_peered":"2024-09-21T19:04:23.882183+0000","last_clean":"2024-09-21T19:04:23.882183+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:04:23.882183+0000","last_undegraded":"2024-09-21T19:04:23.882183+0000","last_fullsized":"2024-09-21T19:04:23.882183+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.662660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"297'9613","reported_seq":9710,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.674141+0000","last_change":"2024-09-21T19:02:30.708018+0000","last_active":"2024-09-21T19:04:26.674141+0000","last_peered":"2024-09-21T19:04:26.674141+0000","last_clean":"2024-09-21T19:04:26.674141+0000","last_became_active":"2024-09-21T19:02:30.707708+0000","last_became_peered":"2024-09-21T19:02:30.707708+0000","last_unstale":"2024-09-21T19:04:26.674141+0000","last_undegraded":"2024-09-21T19:04:26.674141+0000","last_fullsized":"2024-09-21T19:04:26.674141+0000","mapping_epoch":254,"log_start":"280'5000","ondisk_log_start":"280'5000","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.181156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442826752,"num_objects":6757,"num_object_clones":0,"num_object_copies":20271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":523840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":300,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.895580+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:04:23.895580+0000","last_peered":"2024-09-21T19:04:23.895580+0000","last_clean":"2024-09-21T19:04:23.895580+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:04:23.895580+0000","last_undegraded":"2024-09-21T19:04:23.895580+0000","last_fullsized":"2024-09-21T19:04:23.895580+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.034739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":439,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880698+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:04:23.880698+0000","last_peered":"2024-09-21T19:04:23.880698+0000","last_clean":"2024-09-21T19:04:23.880698+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:04:23.880698+0000","last_undegraded":"2024-09-21T19:04:23.880698+0000","last_fullsized":"2024-09-21T19:04:23.880698+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.511233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":518,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.886689+0000","last_change":"2024-09-21T19:00:56.467643+0000","last_active":"2024-09-21T19:04:23.886689+0000","last_peered":"2024-09-21T19:04:23.886689+0000","last_clean":"2024-09-21T19:04:23.886689+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:04:23.886689+0000","last_undegraded":"2024-09-21T19:04:23.886689+0000","last_fullsized":"2024-09-21T19:04:23.886689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:03.696298+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":416,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.894616+0000","last_change":"2024-09-21T19:02:40.882592+0000","last_active":"2024-09-21T19:04:23.894616+0000","last_peered":"2024-09-21T19:04:23.894616+0000","last_clean":"2024-09-21T19:04:23.894616+0000","last_became_active":"2024-09-21T19:02:40.882174+0000","last_became_peered":"2024-09-21T19:02:40.882174+0000","last_unstale":"2024-09-21T19:04:23.894616+0000","last_undegraded":"2024-09-21T19:04:23.894616+0000","last_fullsized":"2024-09-21T19:04:23.894616+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:18.736361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":517,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876259+0000","last_change":"2024-09-21T19:03:51.466892+0000","last_active":"2024-09-21T19:04:23.876259+0000","last_peered":"2024-09-21T19:04:23.876259+0000","last_clean":"2024-09-21T19:04:23.876259+0000","last_became_active":"2024-09-21T19:02:40.882759+0000","last_became_peered":"2024-09-21T19:02:40.882759+0000","last_unstale":"2024-09-21T19:04:23.876259+0000","last_undegraded":"2024-09-21T19:04:23.876259+0000","last_fullsized":"2024-09-21T19:04:23.876259+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.466833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.466833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.333404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"297'9465","reported_seq":9638,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:27.154574+0000","last_change":"2024-09-21T19:03:51.831929+0000","last_active":"2024-09-21T19:04:27.154574+0000","last_peered":"2024-09-21T19:04:27.154574+0000","last_clean":"2024-09-21T19:04:27.154574+0000","last_became_active":"2024-09-21T19:02:29.787771+0000","last_became_peered":"2024-09-21T19:02:29.787771+0000","last_unstale":"2024-09-21T19:04:27.154574+0000","last_undegraded":"2024-09-21T19:04:27.154574+0000","last_fullsized":"2024-09-21T19:04:27.154574+0000","mapping_epoch":254,"log_start":"275'3800","ondisk_log_start":"275'3800","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6909","last_scrub_stamp":"2024-09-21T19:03:51.831669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.831669+0000","objects_scrubbed":6830,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.352296+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429719552,"num_objects":6557,"num_object_clones":0,"num_object_copies":19671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9465,"num_write_kb":512704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":243,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.879991+0000","last_change":"2024-09-21T19:03:17.696156+0000","last_active":"2024-09-21T19:04:23.879991+0000","last_peered":"2024-09-21T19:04:23.879991+0000","last_clean":"2024-09-21T19:04:23.879991+0000","last_became_active":"2024-09-21T19:03:09.627416+0000","last_became_peered":"2024-09-21T19:03:09.627416+0000","last_unstale":"2024-09-21T19:04:23.879991+0000","last_undegraded":"2024-09-21T19:04:23.879991+0000","last_fullsized":"2024-09-21T19:04:23.879991+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:17.696074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.696074+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:34.444138+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"276'40","reported_seq":716,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.993917+0000","last_change":"2024-09-21T19:04:26.993917+0000","last_active":"2024-09-21T19:04:26.993917+0000","last_peered":"2024-09-21T19:04:26.993917+0000","last_clean":"2024-09-21T19:04:26.993917+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:04:26.993917+0000","last_undegraded":"2024-09-21T19:04:26.993917+0000","last_fullsized":"2024-09-21T19:04:26.993917+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.205140+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":551,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881573+0000","last_change":"2024-09-21T19:02:43.617505+0000","last_active":"2024-09-21T19:04:23.881573+0000","last_peered":"2024-09-21T19:04:23.881573+0000","last_clean":"2024-09-21T19:04:23.881573+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:04:23.881573+0000","last_undegraded":"2024-09-21T19:04:23.881573+0000","last_fullsized":"2024-09-21T19:04:23.881573+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:43.617435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:43.617435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.861728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":87,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.877382+0000","last_change":"2024-09-21T19:03:15.425428+0000","last_active":"2024-09-21T19:04:23.877382+0000","last_peered":"2024-09-21T19:04:23.877382+0000","last_clean":"2024-09-21T19:04:23.877382+0000","last_became_active":"2024-09-21T19:03:09.623214+0000","last_became_peered":"2024-09-21T19:03:09.623214+0000","last_unstale":"2024-09-21T19:04:23.877382+0000","last_undegraded":"2024-09-21T19:04:23.877382+0000","last_fullsized":"2024-09-21T19:04:23.877382+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:03:15.425359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.425359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:26.203970+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":398,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882742+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:04:23.882742+0000","last_peered":"2024-09-21T19:04:23.882742+0000","last_clean":"2024-09-21T19:04:23.882742+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:04:23.882742+0000","last_undegraded":"2024-09-21T19:04:23.882742+0000","last_fullsized":"2024-09-21T19:04:23.882742+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.712781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":497,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849526+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:04:21.849526+0000","last_peered":"2024-09-21T19:04:21.849526+0000","last_clean":"2024-09-21T19:04:21.849526+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:04:21.849526+0000","last_undegraded":"2024-09-21T19:04:21.849526+0000","last_fullsized":"2024-09-21T19:04:21.849526+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.494920+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"297'9468","reported_seq":9645,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:27.317141+0000","last_change":"2024-09-21T19:03:48.136769+0000","last_active":"2024-09-21T19:04:27.317141+0000","last_peered":"2024-09-21T19:04:27.317141+0000","last_clean":"2024-09-21T19:04:27.317141+0000","last_became_active":"2024-09-21T19:03:09.629373+0000","last_became_peered":"2024-09-21T19:03:09.629373+0000","last_unstale":"2024-09-21T19:04:27.317141+0000","last_undegraded":"2024-09-21T19:04:27.317141+0000","last_fullsized":"2024-09-21T19:04:27.317141+0000","mapping_epoch":274,"log_start":"280'4700","ondisk_log_start":"280'4700","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6614","last_scrub_stamp":"2024-09-21T19:03:48.136538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.136538+0000","objects_scrubbed":6508,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.921314+0000","scrub_duration":2592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435159040,"num_objects":6640,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":515456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":603,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869691+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:04:23.869691+0000","last_peered":"2024-09-21T19:04:23.869691+0000","last_clean":"2024-09-21T19:04:23.869691+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:04:23.869691+0000","last_undegraded":"2024-09-21T19:04:23.869691+0000","last_fullsized":"2024-09-21T19:04:23.869691+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:33.289219+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":507,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.878886+0000","last_change":"2024-09-21T19:03:40.251914+0000","last_active":"2024-09-21T19:04:23.878886+0000","last_peered":"2024-09-21T19:04:23.878886+0000","last_clean":"2024-09-21T19:04:23.878886+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:04:23.878886+0000","last_undegraded":"2024-09-21T19:04:23.878886+0000","last_fullsized":"2024-09-21T19:04:23.878886+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:03:40.251822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.251822+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:43.197688+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":535,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880163+0000","last_change":"2024-09-21T19:02:40.874431+0000","last_active":"2024-09-21T19:04:23.880163+0000","last_peered":"2024-09-21T19:04:23.880163+0000","last_clean":"2024-09-21T19:04:23.880163+0000","last_became_active":"2024-09-21T19:02:40.874003+0000","last_became_peered":"2024-09-21T19:02:40.874003+0000","last_unstale":"2024-09-21T19:04:23.880163+0000","last_undegraded":"2024-09-21T19:04:23.880163+0000","last_fullsized":"2024-09-21T19:04:23.880163+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.743742+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":557,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.878018+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:04:23.878018+0000","last_peered":"2024-09-21T19:04:23.878018+0000","last_clean":"2024-09-21T19:04:23.878018+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:04:23.878018+0000","last_undegraded":"2024-09-21T19:04:23.878018+0000","last_fullsized":"2024-09-21T19:04:23.878018+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.413921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":526,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881353+0000","last_change":"2024-09-21T19:04:23.530834+0000","last_active":"2024-09-21T19:04:23.881353+0000","last_peered":"2024-09-21T19:04:23.881353+0000","last_clean":"2024-09-21T19:04:23.881353+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:04:23.881353+0000","last_undegraded":"2024-09-21T19:04:23.881353+0000","last_fullsized":"2024-09-21T19:04:23.881353+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:38.789330+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":520,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870494+0000","last_change":"2024-09-21T19:03:17.183599+0000","last_active":"2024-09-21T19:04:23.870494+0000","last_peered":"2024-09-21T19:04:23.870494+0000","last_clean":"2024-09-21T19:04:23.870494+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:04:23.870494+0000","last_undegraded":"2024-09-21T19:04:23.870494+0000","last_fullsized":"2024-09-21T19:04:23.870494+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:17.183521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.183521+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.527335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"297'9518","reported_seq":10649,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:27.173622+0000","last_change":"2024-09-21T19:02:46.916132+0000","last_active":"2024-09-21T19:04:27.173622+0000","last_peered":"2024-09-21T19:04:27.173622+0000","last_clean":"2024-09-21T19:04:27.173622+0000","last_became_active":"2024-09-21T19:02:46.913818+0000","last_became_peered":"2024-09-21T19:02:46.913818+0000","last_unstale":"2024-09-21T19:04:27.173622+0000","last_undegraded":"2024-09-21T19:04:27.173622+0000","last_fullsized":"2024-09-21T19:04:27.173622+0000","mapping_epoch":264,"log_start":"275'3800","ondisk_log_start":"275'3800","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.707299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432930816,"num_objects":6606,"num_object_clones":0,"num_object_copies":19818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.885351+0000","last_change":"2024-09-21T19:03:56.613953+0000","last_active":"2024-09-21T19:04:23.885351+0000","last_peered":"2024-09-21T19:04:23.885351+0000","last_clean":"2024-09-21T19:04:23.885351+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:04:23.885351+0000","last_undegraded":"2024-09-21T19:04:23.885351+0000","last_fullsized":"2024-09-21T19:04:23.885351+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:56.613898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.613898+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.158839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":437,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880664+0000","last_change":"2024-09-21T19:01:13.569543+0000","last_active":"2024-09-21T19:04:23.880664+0000","last_peered":"2024-09-21T19:04:23.880664+0000","last_clean":"2024-09-21T19:04:23.880664+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:04:23.880664+0000","last_undegraded":"2024-09-21T19:04:23.880664+0000","last_fullsized":"2024-09-21T19:04:23.880664+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:01:13.569464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.569464+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.601121+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":398,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880351+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:04:23.880351+0000","last_peered":"2024-09-21T19:04:23.880351+0000","last_clean":"2024-09-21T19:04:23.880351+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:04:23.880351+0000","last_undegraded":"2024-09-21T19:04:23.880351+0000","last_fullsized":"2024-09-21T19:04:23.880351+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.668577+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1223,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:25.322869+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:04:25.322869+0000","last_peered":"2024-09-21T19:04:25.322869+0000","last_clean":"2024-09-21T19:04:25.322869+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:04:25.322869+0000","last_undegraded":"2024-09-21T19:04:25.322869+0000","last_fullsized":"2024-09-21T19:04:25.322869+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.975843+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":668,"num_read_kb":663,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":540,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.872115+0000","last_change":"2024-09-21T19:00:56.950003+0000","last_active":"2024-09-21T19:04:23.872115+0000","last_peered":"2024-09-21T19:04:23.872115+0000","last_clean":"2024-09-21T19:04:23.872115+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:04:23.872115+0000","last_undegraded":"2024-09-21T19:04:23.872115+0000","last_fullsized":"2024-09-21T19:04:23.872115+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.949931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.949931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.450029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":514,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.885018+0000","last_change":"2024-09-21T19:03:54.568908+0000","last_active":"2024-09-21T19:04:23.885018+0000","last_peered":"2024-09-21T19:04:23.885018+0000","last_clean":"2024-09-21T19:04:23.885018+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:04:23.885018+0000","last_undegraded":"2024-09-21T19:04:23.885018+0000","last_fullsized":"2024-09-21T19:04:23.885018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:54.568845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:54.568845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.700759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"296'9117","reported_seq":9212,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:22.627091+0000","last_change":"2024-09-21T19:02:29.785323+0000","last_active":"2024-09-21T19:04:22.627091+0000","last_peered":"2024-09-21T19:04:22.627091+0000","last_clean":"2024-09-21T19:04:22.627091+0000","last_became_active":"2024-09-21T19:02:29.785050+0000","last_became_peered":"2024-09-21T19:02:29.785050+0000","last_unstale":"2024-09-21T19:04:22.627091+0000","last_undegraded":"2024-09-21T19:04:22.627091+0000","last_fullsized":"2024-09-21T19:04:22.627091+0000","mapping_epoch":254,"log_start":"280'4700","ondisk_log_start":"280'4700","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454623232,"num_objects":6937,"num_object_clones":0,"num_object_copies":20811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9117,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":634,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.895254+0000","last_change":"2024-09-21T19:03:18.538009+0000","last_active":"2024-09-21T19:04:23.895254+0000","last_peered":"2024-09-21T19:04:23.895254+0000","last_clean":"2024-09-21T19:04:23.895254+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:04:23.895254+0000","last_undegraded":"2024-09-21T19:04:23.895254+0000","last_fullsized":"2024-09-21T19:04:23.895254+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:03:18.537912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.537912+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:27.037502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":520,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849187+0000","last_change":"2024-09-21T19:02:40.877780+0000","last_active":"2024-09-21T19:04:21.849187+0000","last_peered":"2024-09-21T19:04:21.849187+0000","last_clean":"2024-09-21T19:04:21.849187+0000","last_became_active":"2024-09-21T19:02:40.877452+0000","last_became_peered":"2024-09-21T19:02:40.877452+0000","last_unstale":"2024-09-21T19:04:21.849187+0000","last_undegraded":"2024-09-21T19:04:21.849187+0000","last_fullsized":"2024-09-21T19:04:21.849187+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.441173+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":601,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869502+0000","last_change":"2024-09-21T19:02:40.887141+0000","last_active":"2024-09-21T19:04:23.869502+0000","last_peered":"2024-09-21T19:04:23.869502+0000","last_clean":"2024-09-21T19:04:23.869502+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:04:23.869502+0000","last_undegraded":"2024-09-21T19:04:23.869502+0000","last_fullsized":"2024-09-21T19:04:23.869502+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:00:57.551119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:00:57.551119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":517,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869813+0000","last_change":"2024-09-21T19:03:59.160104+0000","last_active":"2024-09-21T19:04:23.869813+0000","last_peered":"2024-09-21T19:04:23.869813+0000","last_clean":"2024-09-21T19:04:23.869813+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:04:23.869813+0000","last_undegraded":"2024-09-21T19:04:23.869813+0000","last_fullsized":"2024-09-21T19:04:23.869813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.850193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":55,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.864425+0000","last_change":"2024-09-21T19:03:09.625168+0000","last_active":"2024-09-21T19:04:21.864425+0000","last_peered":"2024-09-21T19:04:21.864425+0000","last_clean":"2024-09-21T19:04:21.864425+0000","last_became_active":"2024-09-21T19:03:09.624424+0000","last_became_peered":"2024-09-21T19:03:09.624424+0000","last_unstale":"2024-09-21T19:04:21.864425+0000","last_undegraded":"2024-09-21T19:04:21.864425+0000","last_fullsized":"2024-09-21T19:04:21.864425+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.459943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"297'9280","reported_seq":9377,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.670349+0000","last_change":"2024-09-21T19:02:29.785879+0000","last_active":"2024-09-21T19:04:26.670349+0000","last_peered":"2024-09-21T19:04:26.670349+0000","last_clean":"2024-09-21T19:04:26.670349+0000","last_became_active":"2024-09-21T19:02:29.785492+0000","last_became_peered":"2024-09-21T19:02:29.785492+0000","last_unstale":"2024-09-21T19:04:26.670349+0000","last_undegraded":"2024-09-21T19:04:26.670349+0000","last_fullsized":"2024-09-21T19:04:26.670349+0000","mapping_epoch":254,"log_start":"280'4700","ondisk_log_start":"280'4700","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.229667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427687936,"num_objects":6526,"num_object_clones":0,"num_object_copies":19578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":572,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881944+0000","last_change":"2024-09-21T19:03:18.675564+0000","last_active":"2024-09-21T19:04:23.881944+0000","last_peered":"2024-09-21T19:04:23.881944+0000","last_clean":"2024-09-21T19:04:23.881944+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:04:23.881944+0000","last_undegraded":"2024-09-21T19:04:23.881944+0000","last_fullsized":"2024-09-21T19:04:23.881944+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:18.675329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.675329+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:42.981301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":526,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881296+0000","last_change":"2024-09-21T19:03:20.643868+0000","last_active":"2024-09-21T19:04:23.881296+0000","last_peered":"2024-09-21T19:04:23.881296+0000","last_clean":"2024-09-21T19:04:23.881296+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:04:23.881296+0000","last_undegraded":"2024-09-21T19:04:23.881296+0000","last_fullsized":"2024-09-21T19:04:23.881296+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.643800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.643800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:25.510489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":521,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881824+0000","last_change":"2024-09-21T19:03:11.634133+0000","last_active":"2024-09-21T19:04:23.881824+0000","last_peered":"2024-09-21T19:04:23.881824+0000","last_clean":"2024-09-21T19:04:23.881824+0000","last_became_active":"2024-09-21T19:03:09.619609+0000","last_became_peered":"2024-09-21T19:03:09.619609+0000","last_unstale":"2024-09-21T19:04:23.881824+0000","last_undegraded":"2024-09-21T19:04:23.881824+0000","last_fullsized":"2024-09-21T19:04:23.881824+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":499,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.860170+0000","last_change":"2024-09-21T19:02:46.521753+0000","last_active":"2024-09-21T19:04:21.860170+0000","last_peered":"2024-09-21T19:04:21.860170+0000","last_clean":"2024-09-21T19:04:21.860170+0000","last_became_active":"2024-09-21T19:02:40.877238+0000","last_became_peered":"2024-09-21T19:02:40.877238+0000","last_unstale":"2024-09-21T19:04:21.860170+0000","last_undegraded":"2024-09-21T19:04:21.860170+0000","last_fullsized":"2024-09-21T19:04:21.860170+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"297'9439","reported_seq":10516,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:27.302585+0000","last_change":"2024-09-21T19:02:46.916256+0000","last_active":"2024-09-21T19:04:27.302585+0000","last_peered":"2024-09-21T19:04:27.302585+0000","last_clean":"2024-09-21T19:04:27.302585+0000","last_became_active":"2024-09-21T19:02:46.913875+0000","last_became_peered":"2024-09-21T19:02:46.913875+0000","last_unstale":"2024-09-21T19:04:27.302585+0000","last_undegraded":"2024-09-21T19:04:27.302585+0000","last_fullsized":"2024-09-21T19:04:27.302585+0000","mapping_epoch":264,"log_start":"280'4600","ondisk_log_start":"280'4600","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.725035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428015616,"num_objects":6531,"num_object_clones":0,"num_object_copies":19593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9439,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":31850496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":66,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876354+0000","last_change":"2024-09-21T19:03:15.475191+0000","last_active":"2024-09-21T19:04:23.876354+0000","last_peered":"2024-09-21T19:04:23.876354+0000","last_clean":"2024-09-21T19:04:23.876354+0000","last_became_active":"2024-09-21T19:03:09.626413+0000","last_became_peered":"2024-09-21T19:03:09.626413+0000","last_unstale":"2024-09-21T19:04:23.876354+0000","last_undegraded":"2024-09-21T19:04:23.876354+0000","last_fullsized":"2024-09-21T19:04:23.876354+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:15.475112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.475112+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:43.597628+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":563,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881682+0000","last_change":"2024-09-21T19:02:07.716667+0000","last_active":"2024-09-21T19:04:23.881682+0000","last_peered":"2024-09-21T19:04:23.881682+0000","last_clean":"2024-09-21T19:04:23.881682+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:04:23.881682+0000","last_undegraded":"2024-09-21T19:04:23.881682+0000","last_fullsized":"2024-09-21T19:04:23.881682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T19:02:07.716607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:07.716607+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:34.816871+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":438,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.894812+0000","last_change":"2024-09-21T19:02:40.879761+0000","last_active":"2024-09-21T19:04:23.894812+0000","last_peered":"2024-09-21T19:04:23.894812+0000","last_clean":"2024-09-21T19:04:23.894812+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:04:23.894812+0000","last_undegraded":"2024-09-21T19:04:23.894812+0000","last_fullsized":"2024-09-21T19:04:23.894812+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:14.376671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:14.376671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.783406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":524,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.885107+0000","last_change":"2024-09-21T19:03:42.617727+0000","last_active":"2024-09-21T19:04:23.885107+0000","last_peered":"2024-09-21T19:04:23.885107+0000","last_clean":"2024-09-21T19:04:23.885107+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:04:23.885107+0000","last_undegraded":"2024-09-21T19:04:23.885107+0000","last_fullsized":"2024-09-21T19:04:23.885107+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.806920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":497,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881541+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:04:23.881541+0000","last_peered":"2024-09-21T19:04:23.881541+0000","last_clean":"2024-09-21T19:04:23.881541+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:04:23.881541+0000","last_undegraded":"2024-09-21T19:04:23.881541+0000","last_fullsized":"2024-09-21T19:04:23.881541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.397993+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"296'9020","reported_seq":10108,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:22.672576+0000","last_change":"2024-09-21T19:02:46.953650+0000","last_active":"2024-09-21T19:04:22.672576+0000","last_peered":"2024-09-21T19:04:22.672576+0000","last_clean":"2024-09-21T19:04:22.672576+0000","last_became_active":"2024-09-21T19:02:46.951397+0000","last_became_peered":"2024-09-21T19:02:46.951397+0000","last_unstale":"2024-09-21T19:04:22.672576+0000","last_undegraded":"2024-09-21T19:04:22.672576+0000","last_fullsized":"2024-09-21T19:04:22.672576+0000","mapping_epoch":264,"log_start":"280'4600","ondisk_log_start":"280'4600","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450101248,"num_objects":6868,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":508416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":32243712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":603,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869798+0000","last_change":"2024-09-21T19:03:48.127190+0000","last_active":"2024-09-21T19:04:23.869798+0000","last_peered":"2024-09-21T19:04:23.869798+0000","last_clean":"2024-09-21T19:04:23.869798+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:04:23.869798+0000","last_undegraded":"2024-09-21T19:04:23.869798+0000","last_fullsized":"2024-09-21T19:04:23.869798+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:48.127135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.127135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.997742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":547,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882786+0000","last_change":"2024-09-21T19:02:40.883840+0000","last_active":"2024-09-21T19:04:23.882786+0000","last_peered":"2024-09-21T19:04:23.882786+0000","last_clean":"2024-09-21T19:04:23.882786+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:04:23.882786+0000","last_undegraded":"2024-09-21T19:04:23.882786+0000","last_fullsized":"2024-09-21T19:04:23.882786+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.862643+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":544,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849171+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:04:21.849171+0000","last_peered":"2024-09-21T19:04:21.849171+0000","last_clean":"2024-09-21T19:04:21.849171+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:04:21.849171+0000","last_undegraded":"2024-09-21T19:04:21.849171+0000","last_fullsized":"2024-09-21T19:04:21.849171+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:26.403110+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":662,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.878650+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:04:23.878650+0000","last_peered":"2024-09-21T19:04:23.878650+0000","last_clean":"2024-09-21T19:04:23.878650+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:04:23.878650+0000","last_undegraded":"2024-09-21T19:04:23.878650+0000","last_fullsized":"2024-09-21T19:04:23.878650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.454335+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":497,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881800+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:04:23.881800+0000","last_peered":"2024-09-21T19:04:23.881800+0000","last_clean":"2024-09-21T19:04:23.881800+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:04:23.881800+0000","last_undegraded":"2024-09-21T19:04:23.881800+0000","last_fullsized":"2024-09-21T19:04:23.881800+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.831345+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"297'9544","reported_seq":9647,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.844190+0000","last_change":"2024-09-21T19:03:09.629024+0000","last_active":"2024-09-21T19:04:26.844190+0000","last_peered":"2024-09-21T19:04:26.844190+0000","last_clean":"2024-09-21T19:04:26.844190+0000","last_became_active":"2024-09-21T19:03:09.623358+0000","last_became_peered":"2024-09-21T19:03:09.623358+0000","last_unstale":"2024-09-21T19:04:26.844190+0000","last_undegraded":"2024-09-21T19:04:26.844190+0000","last_fullsized":"2024-09-21T19:04:26.844190+0000","mapping_epoch":274,"log_start":"272'3100","ondisk_log_start":"272'3100","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:29.199328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":408,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880879+0000","last_change":"2024-09-21T19:03:52.722880+0000","last_active":"2024-09-21T19:04:23.880879+0000","last_peered":"2024-09-21T19:04:23.880879+0000","last_clean":"2024-09-21T19:04:23.880879+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:04:23.880879+0000","last_undegraded":"2024-09-21T19:04:23.880879+0000","last_fullsized":"2024-09-21T19:04:23.880879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:52.722797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.722797+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.206303+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":503,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.861235+0000","last_change":"2024-09-21T19:03:09.630384+0000","last_active":"2024-09-21T19:04:21.861235+0000","last_peered":"2024-09-21T19:04:21.861235+0000","last_clean":"2024-09-21T19:04:21.861235+0000","last_became_active":"2024-09-21T19:03:09.629753+0000","last_became_peered":"2024-09-21T19:03:09.629753+0000","last_unstale":"2024-09-21T19:04:21.861235+0000","last_undegraded":"2024-09-21T19:04:21.861235+0000","last_fullsized":"2024-09-21T19:04:21.861235+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":510,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882182+0000","last_change":"2024-09-21T19:03:57.305837+0000","last_active":"2024-09-21T19:04:23.882182+0000","last_peered":"2024-09-21T19:04:23.882182+0000","last_clean":"2024-09-21T19:04:23.882182+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:04:23.882182+0000","last_undegraded":"2024-09-21T19:04:23.882182+0000","last_fullsized":"2024-09-21T19:04:23.882182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:26.798148+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":389,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880257+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:04:23.880257+0000","last_peered":"2024-09-21T19:04:23.880257+0000","last_clean":"2024-09-21T19:04:23.880257+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:04:23.880257+0000","last_undegraded":"2024-09-21T19:04:23.880257+0000","last_fullsized":"2024-09-21T19:04:23.880257+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.597338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"297'9446","reported_seq":9623,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:26.672841+0000","last_change":"2024-09-21T19:03:47.989682+0000","last_active":"2024-09-21T19:04:26.672841+0000","last_peered":"2024-09-21T19:04:26.672841+0000","last_clean":"2024-09-21T19:04:26.672841+0000","last_became_active":"2024-09-21T19:03:09.625334+0000","last_became_peered":"2024-09-21T19:03:09.625334+0000","last_unstale":"2024-09-21T19:04:26.672841+0000","last_undegraded":"2024-09-21T19:04:26.672841+0000","last_fullsized":"2024-09-21T19:04:26.672841+0000","mapping_epoch":274,"log_start":"281'4900","ondisk_log_start":"281'4900","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6556","last_scrub_stamp":"2024-09-21T19:03:47.989483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.989483+0000","objects_scrubbed":6505,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:51.168208+0000","scrub_duration":1845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432275456,"num_objects":6596,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9446,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":457,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.894750+0000","last_change":"2024-09-21T19:02:40.878888+0000","last_active":"2024-09-21T19:04:23.894750+0000","last_peered":"2024-09-21T19:04:23.894750+0000","last_clean":"2024-09-21T19:04:23.894750+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:04:23.894750+0000","last_undegraded":"2024-09-21T19:04:23.894750+0000","last_fullsized":"2024-09-21T19:04:23.894750+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.891844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":528,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870807+0000","last_change":"2024-09-21T19:02:40.889239+0000","last_active":"2024-09-21T19:04:23.870807+0000","last_peered":"2024-09-21T19:04:23.870807+0000","last_clean":"2024-09-21T19:04:23.870807+0000","last_became_active":"2024-09-21T19:02:40.888713+0000","last_became_peered":"2024-09-21T19:02:40.888713+0000","last_unstale":"2024-09-21T19:04:23.870807+0000","last_undegraded":"2024-09-21T19:04:23.870807+0000","last_fullsized":"2024-09-21T19:04:23.870807+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.619235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.619235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:27.379249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":532,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.878422+0000","last_change":"2024-09-21T19:02:47.357147+0000","last_active":"2024-09-21T19:04:23.878422+0000","last_peered":"2024-09-21T19:04:23.878422+0000","last_clean":"2024-09-21T19:04:23.878422+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:04:23.878422+0000","last_undegraded":"2024-09-21T19:04:23.878422+0000","last_fullsized":"2024-09-21T19:04:23.878422+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.448219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":497,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869407+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:04:23.869407+0000","last_peered":"2024-09-21T19:04:23.869407+0000","last_clean":"2024-09-21T19:04:23.869407+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:04:23.869407+0000","last_undegraded":"2024-09-21T19:04:23.869407+0000","last_fullsized":"2024-09-21T19:04:23.869407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.183675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"297'9266","reported_seq":9363,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:27.339328+0000","last_change":"2024-09-21T19:02:30.708593+0000","last_active":"2024-09-21T19:04:27.339328+0000","last_peered":"2024-09-21T19:04:27.339328+0000","last_clean":"2024-09-21T19:04:27.339328+0000","last_became_active":"2024-09-21T19:02:30.708207+0000","last_became_peered":"2024-09-21T19:02:30.708207+0000","last_unstale":"2024-09-21T19:04:27.339328+0000","last_undegraded":"2024-09-21T19:04:27.339328+0000","last_fullsized":"2024-09-21T19:04:27.339328+0000","mapping_epoch":254,"log_start":"280'4500","ondisk_log_start":"280'4500","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.754754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424411136,"num_objects":6476,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9266,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":565,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881147+0000","last_change":"2024-09-21T19:02:04.675992+0000","last_active":"2024-09-21T19:04:23.881147+0000","last_peered":"2024-09-21T19:04:23.881147+0000","last_clean":"2024-09-21T19:04:23.881147+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:04:23.881147+0000","last_undegraded":"2024-09-21T19:04:23.881147+0000","last_fullsized":"2024-09-21T19:04:23.881147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:02:04.675911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:04.675911+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:33.799727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":533,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870471+0000","last_change":"2024-09-21T19:03:47.143498+0000","last_active":"2024-09-21T19:04:23.870471+0000","last_peered":"2024-09-21T19:04:23.870471+0000","last_clean":"2024-09-21T19:04:23.870471+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:04:23.870471+0000","last_undegraded":"2024-09-21T19:04:23.870471+0000","last_fullsized":"2024-09-21T19:04:23.870471+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:47.143439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.143439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:52.903583+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":532,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876603+0000","last_change":"2024-09-21T19:01:48.464855+0000","last_active":"2024-09-21T19:04:23.876603+0000","last_peered":"2024-09-21T19:04:23.876603+0000","last_clean":"2024-09-21T19:04:23.876603+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:04:23.876603+0000","last_undegraded":"2024-09-21T19:04:23.876603+0000","last_fullsized":"2024-09-21T19:04:23.876603+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.370827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":496,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848849+0000","last_change":"2024-09-21T19:03:20.446083+0000","last_active":"2024-09-21T19:04:21.848849+0000","last_peered":"2024-09-21T19:04:21.848849+0000","last_clean":"2024-09-21T19:04:21.848849+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:04:21.848849+0000","last_undegraded":"2024-09-21T19:04:21.848849+0000","last_fullsized":"2024-09-21T19:04:21.848849+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.446014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.446014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.270682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"297'9336","reported_seq":9432,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:27.146456+0000","last_change":"2024-09-21T19:02:29.787669+0000","last_active":"2024-09-21T19:04:27.146456+0000","last_peered":"2024-09-21T19:04:27.146456+0000","last_clean":"2024-09-21T19:04:27.146456+0000","last_became_active":"2024-09-21T19:02:29.787259+0000","last_became_peered":"2024-09-21T19:02:29.787259+0000","last_unstale":"2024-09-21T19:04:27.146456+0000","last_undegraded":"2024-09-21T19:04:27.146456+0000","last_fullsized":"2024-09-21T19:04:27.146456+0000","mapping_epoch":254,"log_start":"273'3600","ondisk_log_start":"273'3600","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:39.635993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419299328,"num_objects":6398,"num_object_clones":0,"num_object_copies":19194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9336,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":128,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869296+0000","last_change":"2024-09-21T19:03:56.136641+0000","last_active":"2024-09-21T19:04:23.869296+0000","last_peered":"2024-09-21T19:04:23.869296+0000","last_clean":"2024-09-21T19:04:23.869296+0000","last_became_active":"2024-09-21T19:03:09.619439+0000","last_became_peered":"2024-09-21T19:03:09.619439+0000","last_unstale":"2024-09-21T19:04:23.869296+0000","last_undegraded":"2024-09-21T19:04:23.869296+0000","last_fullsized":"2024-09-21T19:04:23.869296+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:03.843546+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":547,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870120+0000","last_change":"2024-09-21T19:02:06.757027+0000","last_active":"2024-09-21T19:04:23.870120+0000","last_peered":"2024-09-21T19:04:23.870120+0000","last_clean":"2024-09-21T19:04:23.870120+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:04:23.870120+0000","last_undegraded":"2024-09-21T19:04:23.870120+0000","last_fullsized":"2024-09-21T19:04:23.870120+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:02:06.756954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:06.756954+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:30.321572+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"263'10","reported_seq":677,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870093+0000","last_change":"2024-09-21T19:02:46.144415+0000","last_active":"2024-09-21T19:04:23.870093+0000","last_peered":"2024-09-21T19:04:23.870093+0000","last_clean":"2024-09-21T19:04:23.870093+0000","last_became_active":"2024-09-21T19:02:40.887248+0000","last_became_peered":"2024-09-21T19:02:40.887248+0000","last_unstale":"2024-09-21T19:04:23.870093+0000","last_undegraded":"2024-09-21T19:04:23.870093+0000","last_fullsized":"2024-09-21T19:04:23.870093+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":523,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.894691+0000","last_change":"2024-09-21T19:03:10.560004+0000","last_active":"2024-09-21T19:04:23.894691+0000","last_peered":"2024-09-21T19:04:23.894691+0000","last_clean":"2024-09-21T19:04:23.894691+0000","last_became_active":"2024-09-21T19:03:09.618375+0000","last_became_peered":"2024-09-21T19:03:09.618375+0000","last_unstale":"2024-09-21T19:04:23.894691+0000","last_undegraded":"2024-09-21T19:04:23.894691+0000","last_fullsized":"2024-09-21T19:04:23.894691+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:10.559934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:10.559934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":458,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.895928+0000","last_change":"2024-09-21T19:02:40.882545+0000","last_active":"2024-09-21T19:04:23.895928+0000","last_peered":"2024-09-21T19:04:23.895928+0000","last_clean":"2024-09-21T19:04:23.895928+0000","last_became_active":"2024-09-21T19:02:40.882216+0000","last_became_peered":"2024-09-21T19:02:40.882216+0000","last_unstale":"2024-09-21T19:04:23.895928+0000","last_undegraded":"2024-09-21T19:04:23.895928+0000","last_fullsized":"2024-09-21T19:04:23.895928+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.745928+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":530,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.864653+0000","last_change":"2024-09-21T19:01:06.925416+0000","last_active":"2024-09-21T19:04:21.864653+0000","last_peered":"2024-09-21T19:04:21.864653+0000","last_clean":"2024-09-21T19:04:21.864653+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:04:21.864653+0000","last_undegraded":"2024-09-21T19:04:21.864653+0000","last_fullsized":"2024-09-21T19:04:21.864653+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-21T19:01:06.925267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.925267+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:22.465289+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":63,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849446+0000","last_change":"2024-09-21T19:03:11.520901+0000","last_active":"2024-09-21T19:04:21.849446+0000","last_peered":"2024-09-21T19:04:21.849446+0000","last_clean":"2024-09-21T19:04:21.849446+0000","last_became_active":"2024-09-21T19:03:09.623017+0000","last_became_peered":"2024-09-21T19:03:09.623017+0000","last_unstale":"2024-09-21T19:04:21.849446+0000","last_undegraded":"2024-09-21T19:04:21.849446+0000","last_fullsized":"2024-09-21T19:04:21.849446+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.264605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":507,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869602+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:04:23.869602+0000","last_peered":"2024-09-21T19:04:23.869602+0000","last_clean":"2024-09-21T19:04:23.869602+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:04:23.869602+0000","last_undegraded":"2024-09-21T19:04:23.869602+0000","last_fullsized":"2024-09-21T19:04:23.869602+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":416,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880747+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:04:23.880747+0000","last_peered":"2024-09-21T19:04:23.880747+0000","last_clean":"2024-09-21T19:04:23.880747+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:04:23.880747+0000","last_undegraded":"2024-09-21T19:04:23.880747+0000","last_fullsized":"2024-09-21T19:04:23.880747+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:15.701573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":527,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870340+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:04:23.870340+0000","last_peered":"2024-09-21T19:04:23.870340+0000","last_clean":"2024-09-21T19:04:23.870340+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:04:23.870340+0000","last_undegraded":"2024-09-21T19:04:23.870340+0000","last_fullsized":"2024-09-21T19:04:23.870340+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.889409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":500,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.864524+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:04:21.864524+0000","last_peered":"2024-09-21T19:04:21.864524+0000","last_clean":"2024-09-21T19:04:21.864524+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:04:21.864524+0000","last_undegraded":"2024-09-21T19:04:21.864524+0000","last_fullsized":"2024-09-21T19:04:21.864524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.488303+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":521,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.878390+0000","last_change":"2024-09-21T19:03:44.187925+0000","last_active":"2024-09-21T19:04:23.878390+0000","last_peered":"2024-09-21T19:04:23.878390+0000","last_clean":"2024-09-21T19:04:23.878390+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:04:23.878390+0000","last_undegraded":"2024-09-21T19:04:23.878390+0000","last_fullsized":"2024-09-21T19:04:23.878390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.242342+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":110,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870614+0000","last_change":"2024-09-21T19:03:14.187382+0000","last_active":"2024-09-21T19:04:23.870614+0000","last_peered":"2024-09-21T19:04:23.870614+0000","last_clean":"2024-09-21T19:04:23.870614+0000","last_became_active":"2024-09-21T19:03:09.624551+0000","last_became_peered":"2024-09-21T19:03:09.624551+0000","last_unstale":"2024-09-21T19:04:23.870614+0000","last_undegraded":"2024-09-21T19:04:23.870614+0000","last_fullsized":"2024-09-21T19:04:23.870614+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:03:14.187189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.187189+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":397,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880112+0000","last_change":"2024-09-21T19:02:15.401234+0000","last_active":"2024-09-21T19:04:23.880112+0000","last_peered":"2024-09-21T19:04:23.880112+0000","last_clean":"2024-09-21T19:04:23.880112+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:04:23.880112+0000","last_undegraded":"2024-09-21T19:04:23.880112+0000","last_fullsized":"2024-09-21T19:04:23.880112+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:15.401172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:15.401172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:27.684972+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":652,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.860283+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:04:21.860283+0000","last_peered":"2024-09-21T19:04:21.860283+0000","last_clean":"2024-09-21T19:04:21.860283+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:04:21.860283+0000","last_undegraded":"2024-09-21T19:04:21.860283+0000","last_fullsized":"2024-09-21T19:04:21.860283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.472526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":505,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881650+0000","last_change":"2024-09-21T19:02:36.651427+0000","last_active":"2024-09-21T19:04:23.881650+0000","last_peered":"2024-09-21T19:04:23.881650+0000","last_clean":"2024-09-21T19:04:23.881650+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:04:23.881650+0000","last_undegraded":"2024-09-21T19:04:23.881650+0000","last_fullsized":"2024-09-21T19:04:23.881650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":526,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.861092+0000","last_change":"2024-09-21T19:00:28.893623+0000","last_active":"2024-09-21T19:04:21.861092+0000","last_peered":"2024-09-21T19:04:21.861092+0000","last_clean":"2024-09-21T19:04:21.861092+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:04:21.861092+0000","last_undegraded":"2024-09-21T19:04:21.861092+0000","last_fullsized":"2024-09-21T19:04:21.861092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:00:28.893403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:28.893403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":508,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870063+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:04:23.870063+0000","last_peered":"2024-09-21T19:04:23.870063+0000","last_clean":"2024-09-21T19:04:23.870063+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:04:23.870063+0000","last_undegraded":"2024-09-21T19:04:23.870063+0000","last_fullsized":"2024-09-21T19:04:23.870063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.236097+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"275'11","reported_seq":106,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876714+0000","last_change":"2024-09-21T19:03:11.527080+0000","last_active":"2024-09-21T19:04:23.876714+0000","last_peered":"2024-09-21T19:04:23.876714+0000","last_clean":"2024-09-21T19:04:23.876714+0000","last_became_active":"2024-09-21T19:03:09.624143+0000","last_became_peered":"2024-09-21T19:03:09.624143+0000","last_unstale":"2024-09-21T19:04:23.876714+0000","last_undegraded":"2024-09-21T19:04:23.876714+0000","last_fullsized":"2024-09-21T19:04:23.876714+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:11.527007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.527007+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.827620+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":405,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880981+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:04:23.880981+0000","last_peered":"2024-09-21T19:04:23.880981+0000","last_clean":"2024-09-21T19:04:23.880981+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:04:23.880981+0000","last_undegraded":"2024-09-21T19:04:23.880981+0000","last_fullsized":"2024-09-21T19:04:23.880981+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.717779+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":574,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881409+0000","last_change":"2024-09-21T19:02:40.880896+0000","last_active":"2024-09-21T19:04:23.881409+0000","last_peered":"2024-09-21T19:04:23.881409+0000","last_clean":"2024-09-21T19:04:23.881409+0000","last_became_active":"2024-09-21T19:02:40.880515+0000","last_became_peered":"2024-09-21T19:02:40.880515+0000","last_unstale":"2024-09-21T19:04:23.881409+0000","last_undegraded":"2024-09-21T19:04:23.881409+0000","last_fullsized":"2024-09-21T19:04:23.881409+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:01:04.724868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:04.724868+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":528,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882937+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:04:23.882937+0000","last_peered":"2024-09-21T19:04:23.882937+0000","last_clean":"2024-09-21T19:04:23.882937+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:04:23.882937+0000","last_undegraded":"2024-09-21T19:04:23.882937+0000","last_fullsized":"2024-09-21T19:04:23.882937+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:48.105596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":509,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.877010+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:04:23.877010+0000","last_peered":"2024-09-21T19:04:23.877010+0000","last_clean":"2024-09-21T19:04:23.877010+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:04:23.877010+0000","last_undegraded":"2024-09-21T19:04:23.877010+0000","last_fullsized":"2024-09-21T19:04:23.877010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:26.378842+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":510,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848675+0000","last_change":"2024-09-21T19:02:35.435594+0000","last_active":"2024-09-21T19:04:21.848675+0000","last_peered":"2024-09-21T19:04:21.848675+0000","last_clean":"2024-09-21T19:04:21.848675+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:04:21.848675+0000","last_undegraded":"2024-09-21T19:04:21.848675+0000","last_fullsized":"2024-09-21T19:04:21.848675+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.435525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.435525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":557,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.877261+0000","last_change":"2024-09-21T19:02:40.879297+0000","last_active":"2024-09-21T19:04:23.877261+0000","last_peered":"2024-09-21T19:04:23.877261+0000","last_clean":"2024-09-21T19:04:23.877261+0000","last_became_active":"2024-09-21T19:02:40.878893+0000","last_became_peered":"2024-09-21T19:02:40.878893+0000","last_unstale":"2024-09-21T19:04:23.877261+0000","last_undegraded":"2024-09-21T19:04:23.877261+0000","last_fullsized":"2024-09-21T19:04:23.877261+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.762110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":524,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.870706+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:04:23.870706+0000","last_peered":"2024-09-21T19:04:23.870706+0000","last_clean":"2024-09-21T19:04:23.870706+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:04:23.870706+0000","last_undegraded":"2024-09-21T19:04:23.870706+0000","last_fullsized":"2024-09-21T19:04:23.870706+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.156153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":522,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848933+0000","last_change":"2024-09-21T19:02:36.480829+0000","last_active":"2024-09-21T19:04:21.848933+0000","last_peered":"2024-09-21T19:04:21.848933+0000","last_clean":"2024-09-21T19:04:21.848933+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:04:21.848933+0000","last_undegraded":"2024-09-21T19:04:21.848933+0000","last_fullsized":"2024-09-21T19:04:21.848933+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.480784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.480784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":405,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.882268+0000","last_change":"2024-09-21T19:02:19.440104+0000","last_active":"2024-09-21T19:04:23.882268+0000","last_peered":"2024-09-21T19:04:23.882268+0000","last_clean":"2024-09-21T19:04:23.882268+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:04:23.882268+0000","last_undegraded":"2024-09-21T19:04:23.882268+0000","last_fullsized":"2024-09-21T19:04:23.882268+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:19.440059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:19.440059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.686807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":546,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848891+0000","last_change":"2024-09-21T19:02:40.882171+0000","last_active":"2024-09-21T19:04:21.848891+0000","last_peered":"2024-09-21T19:04:21.848891+0000","last_clean":"2024-09-21T19:04:21.848891+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:04:21.848891+0000","last_undegraded":"2024-09-21T19:04:21.848891+0000","last_fullsized":"2024-09-21T19:04:21.848891+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.365192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":509,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.871956+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:04:23.871956+0000","last_peered":"2024-09-21T19:04:23.871956+0000","last_clean":"2024-09-21T19:04:23.871956+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:04:23.871956+0000","last_undegraded":"2024-09-21T19:04:23.871956+0000","last_fullsized":"2024-09-21T19:04:23.871956+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:32.379751+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":527,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869197+0000","last_change":"2024-09-21T19:02:48.126289+0000","last_active":"2024-09-21T19:04:23.869197+0000","last_peered":"2024-09-21T19:04:23.869197+0000","last_clean":"2024-09-21T19:04:23.869197+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:04:23.869197+0000","last_undegraded":"2024-09-21T19:04:23.869197+0000","last_fullsized":"2024-09-21T19:04:23.869197+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:48.126230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:48.126230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.849083+0000","last_change":"2024-09-21T19:02:40.877561+0000","last_active":"2024-09-21T19:04:21.849083+0000","last_peered":"2024-09-21T19:04:21.849083+0000","last_clean":"2024-09-21T19:04:21.849083+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:04:21.849083+0000","last_undegraded":"2024-09-21T19:04:21.849083+0000","last_fullsized":"2024-09-21T19:04:21.849083+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:58.945631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:58.945631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":545,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.848786+0000","last_change":"2024-09-21T19:02:59.672588+0000","last_active":"2024-09-21T19:04:21.848786+0000","last_peered":"2024-09-21T19:04:21.848786+0000","last_clean":"2024-09-21T19:04:21.848786+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:04:21.848786+0000","last_undegraded":"2024-09-21T19:04:21.848786+0000","last_fullsized":"2024-09-21T19:04:21.848786+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-21T19:02:41.535456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.535456+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":397,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.881250+0000","last_change":"2024-09-21T19:03:55.627621+0000","last_active":"2024-09-21T19:04:23.881250+0000","last_peered":"2024-09-21T19:04:23.881250+0000","last_clean":"2024-09-21T19:04:23.881250+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:04:23.881250+0000","last_undegraded":"2024-09-21T19:04:23.881250+0000","last_fullsized":"2024-09-21T19:04:23.881250+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.004468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":436,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.894462+0000","last_change":"2024-09-21T19:02:40.881419+0000","last_active":"2024-09-21T19:04:23.894462+0000","last_peered":"2024-09-21T19:04:23.894462+0000","last_clean":"2024-09-21T19:04:23.894462+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:04:23.894462+0000","last_undegraded":"2024-09-21T19:04:23.894462+0000","last_fullsized":"2024-09-21T19:04:23.894462+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.789192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":504,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.869794+0000","last_change":"2024-09-21T19:02:36.128893+0000","last_active":"2024-09-21T19:04:23.869794+0000","last_peered":"2024-09-21T19:04:23.869794+0000","last_clean":"2024-09-21T19:04:23.869794+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:04:23.869794+0000","last_undegraded":"2024-09-21T19:04:23.869794+0000","last_fullsized":"2024-09-21T19:04:23.869794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":497,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.876587+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:04:23.876587+0000","last_peered":"2024-09-21T19:04:23.876587+0000","last_clean":"2024-09-21T19:04:23.876587+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:04:23.876587+0000","last_undegraded":"2024-09-21T19:04:23.876587+0000","last_fullsized":"2024-09-21T19:04:23.876587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.485168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":538,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-21T19:04:21.860742+0000","last_change":"2024-09-21T19:03:18.435824+0000","last_active":"2024-09-21T19:04:21.860742+0000","last_peered":"2024-09-21T19:04:21.860742+0000","last_clean":"2024-09-21T19:04:21.860742+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:04:21.860742+0000","last_undegraded":"2024-09-21T19:04:21.860742+0000","last_fullsized":"2024-09-21T19:04:21.860742+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:18.435733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.435733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.735344+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":498,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.895541+0000","last_change":"2024-09-21T19:03:17.580525+0000","last_active":"2024-09-21T19:04:23.895541+0000","last_peered":"2024-09-21T19:04:23.895541+0000","last_clean":"2024-09-21T19:04:23.895541+0000","last_became_active":"2024-09-21T19:02:40.882966+0000","last_became_peered":"2024-09-21T19:02:40.882966+0000","last_unstale":"2024-09-21T19:04:23.895541+0000","last_undegraded":"2024-09-21T19:04:23.895541+0000","last_fullsized":"2024-09-21T19:04:23.895541+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.669525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":406,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-21T19:04:23.880907+0000","last_change":"2024-09-21T19:01:11.531925+0000","last_active":"2024-09-21T19:04:23.880907+0000","last_peered":"2024-09-21T19:04:23.880907+0000","last_clean":"2024-09-21T19:04:23.880907+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:04:23.880907+0000","last_undegraded":"2024-09-21T19:04:23.880907+0000","last_fullsized":"2024-09-21T19:04:23.880907+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.606767+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6967984152,"num_objects":106324,"num_object_clones":0,"num_object_copies":318972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149382,"num_write_kb":8182529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":97714176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20874801152,"data_stored":20874788936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78782,"ondisk_log_size":78782,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":668,"num_read_kb":663,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266828,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4232552,"kb_used_data":3814040,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":89512600,"statfs":{"total":95995035648,"available":91660902400,"internally_reserved":0,"allocated":3905576960,"data_stored":3905189924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464278,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1406864,"kb_used_data":1308224,"kb_used_omap":4,"kb_used_meta":98619,"kb_avail":92338288,"statfs":{"total":95995035648,"available":94554406912,"internally_reserved":0,"allocated":1339621376,"data_stored":1339251499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":100986532},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790100,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4310304,"kb_used_data":3805712,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":89434848,"statfs":{"total":95995035648,"available":91581284352,"internally_reserved":0,"allocated":3897049088,"data_stored":3896670269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019034,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621928,"kb_used_data":1283924,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92123224,"statfs":{"total":95995035648,"available":94334181376,"internally_reserved":0,"allocated":1314738176,"data_stored":1314355492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182558,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4062836,"kb_used_data":3522080,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":89682316,"statfs":{"total":95995035648,"available":91834691584,"internally_reserved":0,"allocated":3606609920,"data_stored":3606209047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411493,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1610724,"kb_used_data":1258836,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92134428,"statfs":{"total":95995035648,"available":94345654272,"internally_reserved":0,"allocated":1289048064,"data_stored":1288665680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575017,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4335992,"kb_used_data":3781352,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":89409160,"statfs":{"total":95995035648,"available":91554979840,"internally_reserved":0,"allocated":3872104448,"data_stored":3871701381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2202472,"kb_used_data":1683992,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91542680,"statfs":{"total":95995035648,"available":93739704320,"internally_reserved":0,"allocated":1724407808,"data_stored":1724021437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1714159616,"data_stored":1714159616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3853389824,"data_stored":3853385752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1282277376,"data_stored":1282277376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3595304960,"data_stored":3595304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1306329088,"data_stored":1306329088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3891073024,"data_stored":3891068952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1336213504,"data_stored":1336213504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3896053760,"data_stored":3896049688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:04:33.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:04:33.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:33.110 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:33.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:33.183 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.198 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:33.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:33.249 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[29384]: 4.17 scrub starts 2024-09-21T19:04:33.249 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[29384]: 4.17 scrub ok 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[29384]: pgmap v556: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 770 op/s 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[29384]: 6.18 scrub starts 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[29384]: 6.18 scrub ok 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[39192]: 4.17 scrub starts 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[39192]: 4.17 scrub ok 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[39192]: pgmap v556: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 770 op/s 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[39192]: 6.18 scrub starts 2024-09-21T19:04:33.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:32 smithi081 ceph-mon[39192]: 6.18 scrub ok 2024-09-21T19:04:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:32 smithi104 ceph-mon[32438]: 4.17 scrub starts 2024-09-21T19:04:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:32 smithi104 ceph-mon[32438]: 4.17 scrub ok 2024-09-21T19:04:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:32 smithi104 ceph-mon[32438]: pgmap v556: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 770 op/s 2024-09-21T19:04:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:32 smithi104 ceph-mon[32438]: 6.18 scrub starts 2024-09-21T19:04:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:32 smithi104 ceph-mon[32438]: 6.18 scrub ok 2024-09-21T19:04:33.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:33.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.486 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:33.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:33.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:33.629 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:33.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:33.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:33.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:04:33.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:33.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:33.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:34.038 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:04:34.038 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:04:34.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 3.16 scrub starts 2024-09-21T19:04:34.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:04:34.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:04:34.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 3.16 scrub ok 2024-09-21T19:04:34.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 9.6 scrub ok 2024-09-21T19:04:34.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T19:04:34.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T19:04:34.114 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 3.16 scrub starts 2024-09-21T19:04:34.114 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:04:34.114 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:04:34.114 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 3.16 scrub ok 2024-09-21T19:04:34.115 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 9.6 scrub ok 2024-09-21T19:04:34.115 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T19:04:34.115 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:33 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T19:04:34.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.172 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:34.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 3.16 scrub starts 2024-09-21T19:04:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:04:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:04:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 3.16 scrub ok 2024-09-21T19:04:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 9.6 scrub ok 2024-09-21T19:04:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T19:04:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:33 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T19:04:34.273 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:34.291 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:34.359 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:34.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:34.361 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:34.362 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:34.362 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:34.362 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:34.398 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:34.398 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:34.432 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:34.432 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:34.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:34.469 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:34.469 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:34.501 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:34.515 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:34.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:34.534 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:34.534 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:34.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:34.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:34.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:34.818 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:34.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:34.956 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:34.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:35.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:04:35.057 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:35.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:35.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.224 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:35.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:34 smithi104 ceph-mon[32438]: pgmap v557: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-21T19:04:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:34 smithi104 ceph-mon[32438]: 4.1a scrub starts 2024-09-21T19:04:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:34 smithi104 ceph-mon[32438]: 4.1a scrub ok 2024-09-21T19:04:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:34 smithi104 ceph-mon[32438]: 3.12 scrub starts 2024-09-21T19:04:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:34 smithi104 ceph-mon[32438]: 3.12 scrub ok 2024-09-21T19:04:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2840530136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[39192]: pgmap v557: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[39192]: 4.1a scrub starts 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[39192]: 4.1a scrub ok 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[39192]: 3.12 scrub starts 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[39192]: 3.12 scrub ok 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2840530136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[29384]: pgmap v557: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-21T19:04:35.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[29384]: 4.1a scrub starts 2024-09-21T19:04:35.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[29384]: 4.1a scrub ok 2024-09-21T19:04:35.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[29384]: 3.12 scrub starts 2024-09-21T19:04:35.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[29384]: 3.12 scrub ok 2024-09-21T19:04:35.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2840530136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:35.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:35.367 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:35.469 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:35.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:35.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:35.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:35.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:35.781 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:35.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:35.917 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:35.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:36.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:36.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:36.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.208 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:36.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:36.255 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:35 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T19:04:36.256 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:35 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T19:04:36.256 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:35 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T19:04:36.256 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:35 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T19:04:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:35 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T19:04:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:35 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T19:04:36.271 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:36.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:36.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:36.349 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:36.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:36.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:36.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:36.670 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:04:36.674 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:04:36.713 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:36.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:36.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:36.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:36.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:37.120 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:37.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: 6.3 scrub starts 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: 6.3 scrub ok 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: pgmap v558: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: 3.1a scrub starts 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: 3.1a scrub ok 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: 6.3 scrub starts 2024-09-21T19:04:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: 6.3 scrub ok 2024-09-21T19:04:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: pgmap v558: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: 3.1a scrub starts 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: 3.1a scrub ok 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:37.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:37.318 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:04:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:04:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: 6.3 scrub starts 2024-09-21T19:04:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: 6.3 scrub ok 2024-09-21T19:04:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:04:37.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:04:37.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: pgmap v558: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-21T19:04:37.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: 3.1a scrub starts 2024-09-21T19:04:37.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: 3.1a scrub ok 2024-09-21T19:04:37.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T19:04:37.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:37.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:37.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:37.420 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:37.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:37.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:37.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:37.691 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:37.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:37.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:37.949 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:04:37.949 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:04:37.949 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:04:37.949 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:04:37.949 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:04:37.952 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:37.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:37.988 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:04:38.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:38.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:04:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: 6.1b scrub starts 2024-09-21T19:04:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: 6.1b scrub ok 2024-09-21T19:04:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: from='client.16032 v1:172.21.15.81:0/3738219514' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T19:04:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2211168852' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: 6.1b scrub starts 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: 6.1b scrub ok 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: from='client.16032 v1:172.21.15.81:0/3738219514' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T19:04:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2211168852' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:04:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:38.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:38.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:38.241 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:38.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:04:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: 6.1b scrub starts 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: 6.1b scrub ok 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: from='client.16032 v1:172.21.15.81:0/3738219514' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T19:04:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2211168852' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:04:38.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:38.365 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:38.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:38.499 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:38.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:38.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:38.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:38.765 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:38.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:38.903 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:38.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:38.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:39.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:39.039 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:39.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:39.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:39.204 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2024-09-21T19:04:35.969443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6441278003,"num_objects":98477,"num_object_clones":0,"num_object_copies":295431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98477,"num_whiteouts":0,"num_read":1815,"num_read_kb":1567,"num_write":158150,"num_write_kb":8182574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":97720406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80671,"ondisk_log_size":80671,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22337896,"kb_used_data":18959324,"kb_used_omap":15,"kb_used_meta":3377968,"kb_avail":727623320,"statfs":{"total":767960285184,"available":745086279680,"internally_reserved":0,"allocated":19414347776,"data_stored":19411208399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":3459040018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-527302656,"num_objects":-8046,"num_object_clones":0,"num_object_copies":-24138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8046,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.341608"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":408,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933824+0000","last_change":"2024-09-21T19:02:35.574023+0000","last_active":"2024-09-21T19:04:30.933824+0000","last_peered":"2024-09-21T19:04:30.933824+0000","last_clean":"2024-09-21T19:04:30.933824+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:04:30.933824+0000","last_undegraded":"2024-09-21T19:04:30.933824+0000","last_fullsized":"2024-09-21T19:04:30.933824+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.711076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":576,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942483+0000","last_change":"2024-09-21T19:03:15.600063+0000","last_active":"2024-09-21T19:04:30.942483+0000","last_peered":"2024-09-21T19:04:30.942483+0000","last_clean":"2024-09-21T19:04:30.942483+0000","last_became_active":"2024-09-21T19:02:40.874205+0000","last_became_peered":"2024-09-21T19:02:40.874205+0000","last_unstale":"2024-09-21T19:04:30.942483+0000","last_undegraded":"2024-09-21T19:04:30.942483+0000","last_fullsized":"2024-09-21T19:04:30.942483+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:15.600006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.600006+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.243933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":511,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942043+0000","last_change":"2024-09-21T18:59:46.342260+0000","last_active":"2024-09-21T19:04:30.942043+0000","last_peered":"2024-09-21T19:04:30.942043+0000","last_clean":"2024-09-21T19:04:30.942043+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:04:30.942043+0000","last_undegraded":"2024-09-21T19:04:30.942043+0000","last_fullsized":"2024-09-21T19:04:30.942043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:46.342203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.342203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:34.519529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":499,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942762+0000","last_change":"2024-09-21T19:01:48.626265+0000","last_active":"2024-09-21T19:04:30.942762+0000","last_peered":"2024-09-21T19:04:30.942762+0000","last_clean":"2024-09-21T19:04:30.942762+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:04:30.942762+0000","last_undegraded":"2024-09-21T19:04:30.942762+0000","last_fullsized":"2024-09-21T19:04:30.942762+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.626214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.626214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930388+0000","last_change":"2024-09-21T19:02:40.877489+0000","last_active":"2024-09-21T19:04:30.930388+0000","last_peered":"2024-09-21T19:04:30.930388+0000","last_clean":"2024-09-21T19:04:30.930388+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:04:30.930388+0000","last_undegraded":"2024-09-21T19:04:30.930388+0000","last_fullsized":"2024-09-21T19:04:30.930388+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.831931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.831931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930923+0000","last_change":"2024-09-21T19:02:47.515077+0000","last_active":"2024-09-21T19:04:30.930923+0000","last_peered":"2024-09-21T19:04:30.930923+0000","last_clean":"2024-09-21T19:04:30.930923+0000","last_became_active":"2024-09-21T19:02:40.879934+0000","last_became_peered":"2024-09-21T19:02:40.879934+0000","last_unstale":"2024-09-21T19:04:30.930923+0000","last_undegraded":"2024-09-21T19:04:30.930923+0000","last_fullsized":"2024-09-21T19:04:30.930923+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.514985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.514985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":597,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931444+0000","last_change":"2024-09-21T19:03:16.432252+0000","last_active":"2024-09-21T19:04:30.931444+0000","last_peered":"2024-09-21T19:04:30.931444+0000","last_clean":"2024-09-21T19:04:30.931444+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:04:30.931444+0000","last_undegraded":"2024-09-21T19:04:30.931444+0000","last_fullsized":"2024-09-21T19:04:30.931444+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:16.432117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.432117+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":530,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942431+0000","last_change":"2024-09-21T19:03:13.601474+0000","last_active":"2024-09-21T19:04:30.942431+0000","last_peered":"2024-09-21T19:04:30.942431+0000","last_clean":"2024-09-21T19:04:30.942431+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:04:30.942431+0000","last_undegraded":"2024-09-21T19:04:30.942431+0000","last_fullsized":"2024-09-21T19:04:30.942431+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:13.601407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.601407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":506,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943605+0000","last_change":"2024-09-21T19:03:10.612487+0000","last_active":"2024-09-21T19:04:30.943605+0000","last_peered":"2024-09-21T19:04:30.943605+0000","last_clean":"2024-09-21T19:04:30.943605+0000","last_became_active":"2024-09-21T19:03:09.619056+0000","last_became_peered":"2024-09-21T19:03:09.619056+0000","last_unstale":"2024-09-21T19:04:30.943605+0000","last_undegraded":"2024-09-21T19:04:30.943605+0000","last_fullsized":"2024-09-21T19:04:30.943605+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:10.612408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:10.612408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":574,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943059+0000","last_change":"2024-09-21T18:59:50.775766+0000","last_active":"2024-09-21T19:04:30.943059+0000","last_peered":"2024-09-21T19:04:30.943059+0000","last_clean":"2024-09-21T19:04:30.943059+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:04:30.943059+0000","last_undegraded":"2024-09-21T19:04:30.943059+0000","last_fullsized":"2024-09-21T19:04:30.943059+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T18:59:50.775696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.775696+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.858575+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":541,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.928273+0000","last_change":"2024-09-21T19:03:56.486775+0000","last_active":"2024-09-21T19:04:30.928273+0000","last_peered":"2024-09-21T19:04:30.928273+0000","last_clean":"2024-09-21T19:04:30.928273+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:04:30.928273+0000","last_undegraded":"2024-09-21T19:04:30.928273+0000","last_fullsized":"2024-09-21T19:04:30.928273+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:56.486496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.486496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.359373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":399,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.864313+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:04:31.864313+0000","last_peered":"2024-09-21T19:04:31.864313+0000","last_clean":"2024-09-21T19:04:31.864313+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:04:31.864313+0000","last_undegraded":"2024-09-21T19:04:31.864313+0000","last_fullsized":"2024-09-21T19:04:31.864313+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:44.758167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":67,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.944782+0000","last_change":"2024-09-21T19:03:59.885217+0000","last_active":"2024-09-21T19:04:30.944782+0000","last_peered":"2024-09-21T19:04:30.944782+0000","last_clean":"2024-09-21T19:04:30.944782+0000","last_became_active":"2024-09-21T19:03:09.628920+0000","last_became_peered":"2024-09-21T19:03:09.628920+0000","last_unstale":"2024-09-21T19:04:30.944782+0000","last_undegraded":"2024-09-21T19:04:30.944782+0000","last_fullsized":"2024-09-21T19:04:30.944782+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.276502+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":520,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942782+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:04:30.942782+0000","last_peered":"2024-09-21T19:04:30.942782+0000","last_clean":"2024-09-21T19:04:30.942782+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:04:30.942782+0000","last_undegraded":"2024-09-21T19:04:30.942782+0000","last_fullsized":"2024-09-21T19:04:30.942782+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.873852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":510,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929717+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:04:30.929717+0000","last_peered":"2024-09-21T19:04:30.929717+0000","last_clean":"2024-09-21T19:04:30.929717+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:04:30.929717+0000","last_undegraded":"2024-09-21T19:04:30.929717+0000","last_fullsized":"2024-09-21T19:04:30.929717+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:57.782768+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":67,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.941948+0000","last_change":"2024-09-21T19:03:16.368532+0000","last_active":"2024-09-21T19:04:30.941948+0000","last_peered":"2024-09-21T19:04:30.941948+0000","last_clean":"2024-09-21T19:04:30.941948+0000","last_became_active":"2024-09-21T19:03:09.627894+0000","last_became_peered":"2024-09-21T19:03:09.627894+0000","last_unstale":"2024-09-21T19:04:30.941948+0000","last_undegraded":"2024-09-21T19:04:30.941948+0000","last_fullsized":"2024-09-21T19:04:30.941948+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:16.368451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.368451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.953461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943720+0000","last_change":"2024-09-21T19:03:12.639309+0000","last_active":"2024-09-21T19:04:30.943720+0000","last_peered":"2024-09-21T19:04:30.943720+0000","last_clean":"2024-09-21T19:04:30.943720+0000","last_became_active":"2024-09-21T19:02:40.879452+0000","last_became_peered":"2024-09-21T19:02:40.879452+0000","last_unstale":"2024-09-21T19:04:30.943720+0000","last_undegraded":"2024-09-21T19:04:30.943720+0000","last_fullsized":"2024-09-21T19:04:30.943720+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:12.638987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:12.638987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.019636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":399,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933993+0000","last_change":"2024-09-21T19:01:16.593955+0000","last_active":"2024-09-21T19:04:30.933993+0000","last_peered":"2024-09-21T19:04:30.933993+0000","last_clean":"2024-09-21T19:04:30.933993+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:04:30.933993+0000","last_undegraded":"2024-09-21T19:04:30.933993+0000","last_fullsized":"2024-09-21T19:04:30.933993+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:16.593895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.593895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.577306+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":558,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927397+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:04:30.927397+0000","last_peered":"2024-09-21T19:04:30.927397+0000","last_clean":"2024-09-21T19:04:30.927397+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:04:30.927397+0000","last_undegraded":"2024-09-21T19:04:30.927397+0000","last_fullsized":"2024-09-21T19:04:30.927397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.400512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942981+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:04:30.942981+0000","last_peered":"2024-09-21T19:04:30.942981+0000","last_clean":"2024-09-21T19:04:30.942981+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:04:30.942981+0000","last_undegraded":"2024-09-21T19:04:30.942981+0000","last_fullsized":"2024-09-21T19:04:30.942981+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.710160+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":512,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.945879+0000","last_change":"2024-09-21T19:03:58.416244+0000","last_active":"2024-09-21T19:04:30.945879+0000","last_peered":"2024-09-21T19:04:30.945879+0000","last_clean":"2024-09-21T19:04:30.945879+0000","last_became_active":"2024-09-21T19:02:40.879991+0000","last_became_peered":"2024-09-21T19:02:40.879991+0000","last_unstale":"2024-09-21T19:04:30.945879+0000","last_undegraded":"2024-09-21T19:04:30.945879+0000","last_fullsized":"2024-09-21T19:04:30.945879+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:01.286671+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943203+0000","last_change":"2024-09-21T19:01:33.521063+0000","last_active":"2024-09-21T19:04:30.943203+0000","last_peered":"2024-09-21T19:04:30.943203+0000","last_clean":"2024-09-21T19:04:30.943203+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:04:30.943203+0000","last_undegraded":"2024-09-21T19:04:30.943203+0000","last_fullsized":"2024-09-21T19:04:30.943203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:33.520989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:33.520989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":586,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932516+0000","last_change":"2024-09-21T19:03:13.490415+0000","last_active":"2024-09-21T19:04:30.932516+0000","last_peered":"2024-09-21T19:04:30.932516+0000","last_clean":"2024-09-21T19:04:30.932516+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:04:30.932516+0000","last_undegraded":"2024-09-21T19:04:30.932516+0000","last_fullsized":"2024-09-21T19:04:30.932516+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:03:13.490332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.490332+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:31.917938+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":511,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942976+0000","last_change":"2024-09-21T19:03:58.898230+0000","last_active":"2024-09-21T19:04:30.942976+0000","last_peered":"2024-09-21T19:04:30.942976+0000","last_clean":"2024-09-21T19:04:30.942976+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:04:30.942976+0000","last_undegraded":"2024-09-21T19:04:30.942976+0000","last_fullsized":"2024-09-21T19:04:30.942976+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.898154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.898154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.123418+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":523,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929969+0000","last_change":"2024-09-21T19:03:41.162782+0000","last_active":"2024-09-21T19:04:30.929969+0000","last_peered":"2024-09-21T19:04:30.929969+0000","last_clean":"2024-09-21T19:04:30.929969+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:04:30.929969+0000","last_undegraded":"2024-09-21T19:04:30.929969+0000","last_fullsized":"2024-09-21T19:04:30.929969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:44.255552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":531,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942125+0000","last_change":"2024-09-21T19:03:40.622949+0000","last_active":"2024-09-21T19:04:30.942125+0000","last_peered":"2024-09-21T19:04:30.942125+0000","last_clean":"2024-09-21T19:04:30.942125+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:04:30.942125+0000","last_undegraded":"2024-09-21T19:04:30.942125+0000","last_fullsized":"2024-09-21T19:04:30.942125+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:03:40.622880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.622880+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:49.026818+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":435,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932151+0000","last_change":"2024-09-21T19:01:37.492543+0000","last_active":"2024-09-21T19:04:30.932151+0000","last_peered":"2024-09-21T19:04:30.932151+0000","last_clean":"2024-09-21T19:04:30.932151+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:04:30.932151+0000","last_undegraded":"2024-09-21T19:04:30.932151+0000","last_fullsized":"2024-09-21T19:04:30.932151+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:01:37.492484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:37.492484+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.666040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"263'10","reported_seq":536,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.939987+0000","last_change":"2024-09-21T19:02:40.879205+0000","last_active":"2024-09-21T19:04:30.939987+0000","last_peered":"2024-09-21T19:04:30.939987+0000","last_clean":"2024-09-21T19:04:30.939987+0000","last_became_active":"2024-09-21T19:02:40.878691+0000","last_became_peered":"2024-09-21T19:02:40.878691+0000","last_unstale":"2024-09-21T19:04:30.939987+0000","last_undegraded":"2024-09-21T19:04:30.939987+0000","last_fullsized":"2024-09-21T19:04:30.939987+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.734152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"298'9729","reported_seq":9903,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.678992+0000","last_change":"2024-09-21T19:03:50.138454+0000","last_active":"2024-09-21T19:04:32.678992+0000","last_peered":"2024-09-21T19:04:32.678992+0000","last_clean":"2024-09-21T19:04:32.678992+0000","last_became_active":"2024-09-21T19:02:29.785679+0000","last_became_peered":"2024-09-21T19:02:29.785679+0000","last_unstale":"2024-09-21T19:04:32.678992+0000","last_undegraded":"2024-09-21T19:04:32.678992+0000","last_fullsized":"2024-09-21T19:04:32.678992+0000","mapping_epoch":254,"log_start":"283'5300","ondisk_log_start":"283'5300","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'6599","last_scrub_stamp":"2024-09-21T19:03:50.138203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.138203+0000","objects_scrubbed":6517,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:55.055653+0000","scrub_duration":2697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394330112,"num_objects":6017,"num_object_clones":0,"num_object_copies":18051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":503872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":68,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927109+0000","last_change":"2024-09-21T19:03:10.415757+0000","last_active":"2024-09-21T19:04:30.927109+0000","last_peered":"2024-09-21T19:04:30.927109+0000","last_clean":"2024-09-21T19:04:30.927109+0000","last_became_active":"2024-09-21T19:03:09.622733+0000","last_became_peered":"2024-09-21T19:03:09.622733+0000","last_unstale":"2024-09-21T19:04:30.927109+0000","last_undegraded":"2024-09-21T19:04:30.927109+0000","last_fullsized":"2024-09-21T19:04:30.927109+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:10.415628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:10.415628+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":567,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.941562+0000","last_change":"2024-09-21T19:03:57.634402+0000","last_active":"2024-09-21T19:04:30.941562+0000","last_peered":"2024-09-21T19:04:30.941562+0000","last_clean":"2024-09-21T19:04:30.941562+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:04:30.941562+0000","last_undegraded":"2024-09-21T19:04:30.941562+0000","last_fullsized":"2024-09-21T19:04:30.941562+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:57.634230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.634230+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:58.234423+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":521,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931542+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:04:30.931542+0000","last_peered":"2024-09-21T19:04:30.931542+0000","last_clean":"2024-09-21T19:04:30.931542+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:04:30.931542+0000","last_undegraded":"2024-09-21T19:04:30.931542+0000","last_fullsized":"2024-09-21T19:04:30.931542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.469739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":695,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.944080+0000","last_change":"2024-09-21T19:03:53.851419+0000","last_active":"2024-09-21T19:04:30.944080+0000","last_peered":"2024-09-21T19:04:30.944080+0000","last_clean":"2024-09-21T19:04:30.944080+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:04:30.944080+0000","last_undegraded":"2024-09-21T19:04:30.944080+0000","last_fullsized":"2024-09-21T19:04:30.944080+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:55.131138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":191,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942372+0000","last_change":"2024-09-21T19:03:14.420119+0000","last_active":"2024-09-21T19:04:30.942372+0000","last_peered":"2024-09-21T19:04:30.942372+0000","last_clean":"2024-09-21T19:04:30.942372+0000","last_became_active":"2024-09-21T19:03:09.623771+0000","last_became_peered":"2024-09-21T19:03:09.623771+0000","last_unstale":"2024-09-21T19:04:30.942372+0000","last_undegraded":"2024-09-21T19:04:30.942372+0000","last_fullsized":"2024-09-21T19:04:30.942372+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:14.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"298'9793","reported_seq":9965,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.672844+0000","last_change":"2024-09-21T19:03:45.264553+0000","last_active":"2024-09-21T19:04:31.672844+0000","last_peered":"2024-09-21T19:04:31.672844+0000","last_clean":"2024-09-21T19:04:31.672844+0000","last_became_active":"2024-09-21T19:02:29.786797+0000","last_became_peered":"2024-09-21T19:02:29.786797+0000","last_unstale":"2024-09-21T19:04:31.672844+0000","last_undegraded":"2024-09-21T19:04:31.672844+0000","last_fullsized":"2024-09-21T19:04:31.672844+0000","mapping_epoch":254,"log_start":"283'5200","ondisk_log_start":"283'5200","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6439","last_scrub_stamp":"2024-09-21T19:03:45.264423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.264423+0000","objects_scrubbed":6363,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.021933+0000","scrub_duration":2070,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409075736,"num_objects":6243,"num_object_clones":0,"num_object_copies":18729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9793,"num_write_kb":513089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":295,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943519+0000","last_change":"2024-09-21T19:02:45.696512+0000","last_active":"2024-09-21T19:04:30.943519+0000","last_peered":"2024-09-21T19:04:30.943519+0000","last_clean":"2024-09-21T19:04:30.943519+0000","last_became_active":"2024-09-21T19:02:40.874426+0000","last_became_peered":"2024-09-21T19:02:40.874426+0000","last_unstale":"2024-09-21T19:04:30.943519+0000","last_undegraded":"2024-09-21T19:04:30.943519+0000","last_fullsized":"2024-09-21T19:04:30.943519+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":609,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931202+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:04:30.931202+0000","last_peered":"2024-09-21T19:04:30.931202+0000","last_clean":"2024-09-21T19:04:30.931202+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:04:30.931202+0000","last_undegraded":"2024-09-21T19:04:30.931202+0000","last_fullsized":"2024-09-21T19:04:30.931202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.535087+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":535,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.946298+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:04:30.946298+0000","last_peered":"2024-09-21T19:04:30.946298+0000","last_clean":"2024-09-21T19:04:30.946298+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:04:30.946298+0000","last_undegraded":"2024-09-21T19:04:30.946298+0000","last_fullsized":"2024-09-21T19:04:30.946298+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":520,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942526+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:04:30.942526+0000","last_peered":"2024-09-21T19:04:30.942526+0000","last_clean":"2024-09-21T19:04:30.942526+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:04:30.942526+0000","last_undegraded":"2024-09-21T19:04:30.942526+0000","last_fullsized":"2024-09-21T19:04:30.942526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:52.440101+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":500,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942978+0000","last_change":"2024-09-21T19:02:11.853722+0000","last_active":"2024-09-21T19:04:30.942978+0000","last_peered":"2024-09-21T19:04:30.942978+0000","last_clean":"2024-09-21T19:04:30.942978+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:04:30.942978+0000","last_undegraded":"2024-09-21T19:04:30.942978+0000","last_fullsized":"2024-09-21T19:04:30.942978+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:11.853678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:11.853678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.736684+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"298'9900","reported_seq":9999,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.700486+0000","last_change":"2024-09-21T19:02:30.709345+0000","last_active":"2024-09-21T19:04:32.700486+0000","last_peered":"2024-09-21T19:04:32.700486+0000","last_clean":"2024-09-21T19:04:32.700486+0000","last_became_active":"2024-09-21T19:02:30.709045+0000","last_became_peered":"2024-09-21T19:02:30.709045+0000","last_unstale":"2024-09-21T19:04:32.700486+0000","last_undegraded":"2024-09-21T19:04:32.700486+0000","last_fullsized":"2024-09-21T19:04:32.700486+0000","mapping_epoch":254,"log_start":"284'5500","ondisk_log_start":"284'5500","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4400,"log_dups_size":0,"ondisk_log_size":4400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403439616,"num_objects":6156,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":281,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931091+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:04:30.931091+0000","last_peered":"2024-09-21T19:04:30.931091+0000","last_clean":"2024-09-21T19:04:30.931091+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:04:30.931091+0000","last_undegraded":"2024-09-21T19:04:30.931091+0000","last_fullsized":"2024-09-21T19:04:30.931091+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:01.404232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":569,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942663+0000","last_change":"2024-09-21T19:02:42.663946+0000","last_active":"2024-09-21T19:04:30.942663+0000","last_peered":"2024-09-21T19:04:30.942663+0000","last_clean":"2024-09-21T19:04:30.942663+0000","last_became_active":"2024-09-21T19:02:40.885348+0000","last_became_peered":"2024-09-21T19:02:40.885348+0000","last_unstale":"2024-09-21T19:04:30.942663+0000","last_undegraded":"2024-09-21T19:04:30.942663+0000","last_fullsized":"2024-09-21T19:04:30.942663+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T19:02:42.663859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:42.663859+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":412,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933622+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:04:30.933622+0000","last_peered":"2024-09-21T19:04:30.933622+0000","last_clean":"2024-09-21T19:04:30.933622+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:04:30.933622+0000","last_undegraded":"2024-09-21T19:04:30.933622+0000","last_fullsized":"2024-09-21T19:04:30.933622+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.439642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":651,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931043+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:04:30.931043+0000","last_peered":"2024-09-21T19:04:30.931043+0000","last_clean":"2024-09-21T19:04:30.931043+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:04:30.931043+0000","last_undegraded":"2024-09-21T19:04:30.931043+0000","last_fullsized":"2024-09-21T19:04:30.931043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:54.771825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":510,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929721+0000","last_change":"2024-09-21T19:02:40.887898+0000","last_active":"2024-09-21T19:04:30.929721+0000","last_peered":"2024-09-21T19:04:30.929721+0000","last_clean":"2024-09-21T19:04:30.929721+0000","last_became_active":"2024-09-21T19:02:40.886800+0000","last_became_peered":"2024-09-21T19:02:40.886800+0000","last_unstale":"2024-09-21T19:04:30.929721+0000","last_undegraded":"2024-09-21T19:04:30.929721+0000","last_fullsized":"2024-09-21T19:04:30.929721+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:06.644848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:06.644848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"298'9780","reported_seq":9952,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.682093+0000","last_change":"2024-09-21T19:03:46.619927+0000","last_active":"2024-09-21T19:04:31.682093+0000","last_peered":"2024-09-21T19:04:31.682093+0000","last_clean":"2024-09-21T19:04:31.682093+0000","last_became_active":"2024-09-21T19:02:29.787909+0000","last_became_peered":"2024-09-21T19:02:29.787909+0000","last_unstale":"2024-09-21T19:04:31.682093+0000","last_undegraded":"2024-09-21T19:04:31.682093+0000","last_fullsized":"2024-09-21T19:04:31.682093+0000","mapping_epoch":254,"log_start":"280'4800","ondisk_log_start":"280'4800","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6487","last_scrub_stamp":"2024-09-21T19:03:46.619847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:46.619847+0000","objects_scrubbed":6415,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.392510+0000","scrub_duration":2131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401211392,"num_objects":6122,"num_object_clones":0,"num_object_copies":18366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":508864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":324,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.940126+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:04:30.940126+0000","last_peered":"2024-09-21T19:04:30.940126+0000","last_clean":"2024-09-21T19:04:30.940126+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:04:30.940126+0000","last_undegraded":"2024-09-21T19:04:30.940126+0000","last_fullsized":"2024-09-21T19:04:30.940126+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:23.574104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":546,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.941062+0000","last_change":"2024-09-21T19:03:50.509414+0000","last_active":"2024-09-21T19:04:30.941062+0000","last_peered":"2024-09-21T19:04:30.941062+0000","last_clean":"2024-09-21T19:04:30.941062+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:04:30.941062+0000","last_undegraded":"2024-09-21T19:04:30.941062+0000","last_fullsized":"2024-09-21T19:04:30.941062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:03:50.509275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.509275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:57.379074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":59,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932602+0000","last_change":"2024-09-21T19:03:09.624294+0000","last_active":"2024-09-21T19:04:30.932602+0000","last_peered":"2024-09-21T19:04:30.932602+0000","last_clean":"2024-09-21T19:04:30.932602+0000","last_became_active":"2024-09-21T19:03:09.623941+0000","last_became_peered":"2024-09-21T19:03:09.623941+0000","last_unstale":"2024-09-21T19:04:30.932602+0000","last_undegraded":"2024-09-21T19:04:30.932602+0000","last_fullsized":"2024-09-21T19:04:30.932602+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"263'12","reported_seq":513,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933913+0000","last_change":"2024-09-21T19:02:40.897242+0000","last_active":"2024-09-21T19:04:30.933913+0000","last_peered":"2024-09-21T19:04:30.933913+0000","last_clean":"2024-09-21T19:04:30.933913+0000","last_became_active":"2024-09-21T19:02:40.896856+0000","last_became_peered":"2024-09-21T19:02:40.896856+0000","last_unstale":"2024-09-21T19:04:30.933913+0000","last_undegraded":"2024-09-21T19:04:30.933913+0000","last_fullsized":"2024-09-21T19:04:30.933913+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:01:36.539314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:36.539314+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.638141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":400,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932117+0000","last_change":"2024-09-21T19:02:18.444667+0000","last_active":"2024-09-21T19:04:30.932117+0000","last_peered":"2024-09-21T19:04:30.932117+0000","last_clean":"2024-09-21T19:04:30.932117+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:04:30.932117+0000","last_undegraded":"2024-09-21T19:04:30.932117+0000","last_fullsized":"2024-09-21T19:04:30.932117+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.662660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"298'10033","reported_seq":10132,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.644261+0000","last_change":"2024-09-21T19:02:30.708018+0000","last_active":"2024-09-21T19:04:31.644261+0000","last_peered":"2024-09-21T19:04:31.644261+0000","last_clean":"2024-09-21T19:04:31.644261+0000","last_became_active":"2024-09-21T19:02:30.707708+0000","last_became_peered":"2024-09-21T19:02:30.707708+0000","last_unstale":"2024-09-21T19:04:31.644261+0000","last_undegraded":"2024-09-21T19:04:31.644261+0000","last_fullsized":"2024-09-21T19:04:31.644261+0000","mapping_epoch":254,"log_start":"283'5400","ondisk_log_start":"283'5400","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.181156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415301632,"num_objects":6337,"num_object_clones":0,"num_object_copies":19011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":523840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":302,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.939790+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:04:30.939790+0000","last_peered":"2024-09-21T19:04:30.939790+0000","last_clean":"2024-09-21T19:04:30.939790+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:04:30.939790+0000","last_undegraded":"2024-09-21T19:04:30.939790+0000","last_fullsized":"2024-09-21T19:04:30.939790+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.034739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":441,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934089+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:04:30.934089+0000","last_peered":"2024-09-21T19:04:30.934089+0000","last_clean":"2024-09-21T19:04:30.934089+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:04:30.934089+0000","last_undegraded":"2024-09-21T19:04:30.934089+0000","last_fullsized":"2024-09-21T19:04:30.934089+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.511233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":520,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942910+0000","last_change":"2024-09-21T19:00:56.467643+0000","last_active":"2024-09-21T19:04:30.942910+0000","last_peered":"2024-09-21T19:04:30.942910+0000","last_clean":"2024-09-21T19:04:30.942910+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:04:30.942910+0000","last_undegraded":"2024-09-21T19:04:30.942910+0000","last_fullsized":"2024-09-21T19:04:30.942910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:03.696298+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":418,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.938991+0000","last_change":"2024-09-21T19:02:40.882592+0000","last_active":"2024-09-21T19:04:30.938991+0000","last_peered":"2024-09-21T19:04:30.938991+0000","last_clean":"2024-09-21T19:04:30.938991+0000","last_became_active":"2024-09-21T19:02:40.882174+0000","last_became_peered":"2024-09-21T19:02:40.882174+0000","last_unstale":"2024-09-21T19:04:30.938991+0000","last_undegraded":"2024-09-21T19:04:30.938991+0000","last_fullsized":"2024-09-21T19:04:30.938991+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:18.736361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942191+0000","last_change":"2024-09-21T19:03:51.466892+0000","last_active":"2024-09-21T19:04:30.942191+0000","last_peered":"2024-09-21T19:04:30.942191+0000","last_clean":"2024-09-21T19:04:30.942191+0000","last_became_active":"2024-09-21T19:02:40.882759+0000","last_became_peered":"2024-09-21T19:02:40.882759+0000","last_unstale":"2024-09-21T19:04:30.942191+0000","last_undegraded":"2024-09-21T19:04:30.942191+0000","last_fullsized":"2024-09-21T19:04:30.942191+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.466833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.466833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.333404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"298'9892","reported_seq":10067,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.183902+0000","last_change":"2024-09-21T19:03:51.831929+0000","last_active":"2024-09-21T19:04:32.183902+0000","last_peered":"2024-09-21T19:04:32.183902+0000","last_clean":"2024-09-21T19:04:32.183902+0000","last_became_active":"2024-09-21T19:02:29.787771+0000","last_became_peered":"2024-09-21T19:02:29.787771+0000","last_unstale":"2024-09-21T19:04:32.183902+0000","last_undegraded":"2024-09-21T19:04:32.183902+0000","last_fullsized":"2024-09-21T19:04:32.183902+0000","mapping_epoch":254,"log_start":"276'4200","ondisk_log_start":"276'4200","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6909","last_scrub_stamp":"2024-09-21T19:03:51.831669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.831669+0000","objects_scrubbed":6830,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.352296+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":512704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":245,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934280+0000","last_change":"2024-09-21T19:03:17.696156+0000","last_active":"2024-09-21T19:04:30.934280+0000","last_peered":"2024-09-21T19:04:30.934280+0000","last_clean":"2024-09-21T19:04:30.934280+0000","last_became_active":"2024-09-21T19:03:09.627416+0000","last_became_peered":"2024-09-21T19:03:09.627416+0000","last_unstale":"2024-09-21T19:04:30.934280+0000","last_undegraded":"2024-09-21T19:04:30.934280+0000","last_fullsized":"2024-09-21T19:04:30.934280+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:17.696074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.696074+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:34.444138+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"276'40","reported_seq":719,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942782+0000","last_change":"2024-09-21T19:04:26.993917+0000","last_active":"2024-09-21T19:04:30.942782+0000","last_peered":"2024-09-21T19:04:30.942782+0000","last_clean":"2024-09-21T19:04:30.942782+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:04:30.942782+0000","last_undegraded":"2024-09-21T19:04:30.942782+0000","last_fullsized":"2024-09-21T19:04:30.942782+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.205140+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":553,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943269+0000","last_change":"2024-09-21T19:02:43.617505+0000","last_active":"2024-09-21T19:04:30.943269+0000","last_peered":"2024-09-21T19:04:30.943269+0000","last_clean":"2024-09-21T19:04:30.943269+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:04:30.943269+0000","last_undegraded":"2024-09-21T19:04:30.943269+0000","last_fullsized":"2024-09-21T19:04:30.943269+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:43.617435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:43.617435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.861728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":89,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.941692+0000","last_change":"2024-09-21T19:03:15.425428+0000","last_active":"2024-09-21T19:04:30.941692+0000","last_peered":"2024-09-21T19:04:30.941692+0000","last_clean":"2024-09-21T19:04:30.941692+0000","last_became_active":"2024-09-21T19:03:09.623214+0000","last_became_peered":"2024-09-21T19:03:09.623214+0000","last_unstale":"2024-09-21T19:04:30.941692+0000","last_undegraded":"2024-09-21T19:04:30.941692+0000","last_fullsized":"2024-09-21T19:04:30.941692+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:03:15.425359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.425359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":400,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934407+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:04:30.934407+0000","last_peered":"2024-09-21T19:04:30.934407+0000","last_clean":"2024-09-21T19:04:30.934407+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:04:30.934407+0000","last_undegraded":"2024-09-21T19:04:30.934407+0000","last_fullsized":"2024-09-21T19:04:30.934407+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.712781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":501,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930680+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:04:30.930680+0000","last_peered":"2024-09-21T19:04:30.930680+0000","last_clean":"2024-09-21T19:04:30.930680+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:04:30.930680+0000","last_undegraded":"2024-09-21T19:04:30.930680+0000","last_fullsized":"2024-09-21T19:04:30.930680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.494920+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"298'9885","reported_seq":10064,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.249840+0000","last_change":"2024-09-21T19:03:48.136769+0000","last_active":"2024-09-21T19:04:32.249840+0000","last_peered":"2024-09-21T19:04:32.249840+0000","last_clean":"2024-09-21T19:04:32.249840+0000","last_became_active":"2024-09-21T19:03:09.629373+0000","last_became_peered":"2024-09-21T19:03:09.629373+0000","last_unstale":"2024-09-21T19:04:32.249840+0000","last_undegraded":"2024-09-21T19:04:32.249840+0000","last_fullsized":"2024-09-21T19:04:32.249840+0000","mapping_epoch":274,"log_start":"282'5100","ondisk_log_start":"282'5100","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6614","last_scrub_stamp":"2024-09-21T19:03:48.136538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.136538+0000","objects_scrubbed":6508,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.921314+0000","scrub_duration":2592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407830528,"num_objects":6223,"num_object_clones":0,"num_object_copies":18669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9885,"num_write_kb":515456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":605,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929600+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:04:30.929600+0000","last_peered":"2024-09-21T19:04:30.929600+0000","last_clean":"2024-09-21T19:04:30.929600+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:04:30.929600+0000","last_undegraded":"2024-09-21T19:04:30.929600+0000","last_fullsized":"2024-09-21T19:04:30.929600+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:33.289219+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":509,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927835+0000","last_change":"2024-09-21T19:03:40.251914+0000","last_active":"2024-09-21T19:04:30.927835+0000","last_peered":"2024-09-21T19:04:30.927835+0000","last_clean":"2024-09-21T19:04:30.927835+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:04:30.927835+0000","last_undegraded":"2024-09-21T19:04:30.927835+0000","last_fullsized":"2024-09-21T19:04:30.927835+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:03:40.251822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.251822+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:43.197688+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":537,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.944450+0000","last_change":"2024-09-21T19:02:40.874431+0000","last_active":"2024-09-21T19:04:30.944450+0000","last_peered":"2024-09-21T19:04:30.944450+0000","last_clean":"2024-09-21T19:04:30.944450+0000","last_became_active":"2024-09-21T19:02:40.874003+0000","last_became_peered":"2024-09-21T19:02:40.874003+0000","last_unstale":"2024-09-21T19:04:30.944450+0000","last_undegraded":"2024-09-21T19:04:30.944450+0000","last_fullsized":"2024-09-21T19:04:30.944450+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.743742+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":559,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.926724+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:04:30.926724+0000","last_peered":"2024-09-21T19:04:30.926724+0000","last_clean":"2024-09-21T19:04:30.926724+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:04:30.926724+0000","last_undegraded":"2024-09-21T19:04:30.926724+0000","last_fullsized":"2024-09-21T19:04:30.926724+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.413921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":528,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927049+0000","last_change":"2024-09-21T19:04:23.530834+0000","last_active":"2024-09-21T19:04:30.927049+0000","last_peered":"2024-09-21T19:04:30.927049+0000","last_clean":"2024-09-21T19:04:30.927049+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:04:30.927049+0000","last_undegraded":"2024-09-21T19:04:30.927049+0000","last_fullsized":"2024-09-21T19:04:30.927049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:38.789330+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":522,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931506+0000","last_change":"2024-09-21T19:03:17.183599+0000","last_active":"2024-09-21T19:04:30.931506+0000","last_peered":"2024-09-21T19:04:30.931506+0000","last_clean":"2024-09-21T19:04:30.931506+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:04:30.931506+0000","last_undegraded":"2024-09-21T19:04:30.931506+0000","last_fullsized":"2024-09-21T19:04:30.931506+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:17.183521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.183521+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.527335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"298'9942","reported_seq":11076,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.166325+0000","last_change":"2024-09-21T19:02:46.916132+0000","last_active":"2024-09-21T19:04:32.166325+0000","last_peered":"2024-09-21T19:04:32.166325+0000","last_clean":"2024-09-21T19:04:32.166325+0000","last_became_active":"2024-09-21T19:02:46.913818+0000","last_became_peered":"2024-09-21T19:02:46.913818+0000","last_unstale":"2024-09-21T19:04:32.166325+0000","last_undegraded":"2024-09-21T19:04:32.166325+0000","last_fullsized":"2024-09-21T19:04:32.166325+0000","mapping_epoch":264,"log_start":"276'4200","ondisk_log_start":"276'4200","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.707299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9942,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":303,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.940309+0000","last_change":"2024-09-21T19:03:56.613953+0000","last_active":"2024-09-21T19:04:30.940309+0000","last_peered":"2024-09-21T19:04:30.940309+0000","last_clean":"2024-09-21T19:04:30.940309+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:04:30.940309+0000","last_undegraded":"2024-09-21T19:04:30.940309+0000","last_fullsized":"2024-09-21T19:04:30.940309+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:56.613898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.613898+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.158839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":440,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932319+0000","last_change":"2024-09-21T19:01:13.569543+0000","last_active":"2024-09-21T19:04:30.932319+0000","last_peered":"2024-09-21T19:04:30.932319+0000","last_clean":"2024-09-21T19:04:30.932319+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:04:30.932319+0000","last_undegraded":"2024-09-21T19:04:30.932319+0000","last_fullsized":"2024-09-21T19:04:30.932319+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:01:13.569464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.569464+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.601121+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":400,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934589+0000","last_change":"2024-09-21T18:59:49.188876+0000","last_active":"2024-09-21T19:04:30.934589+0000","last_peered":"2024-09-21T19:04:30.934589+0000","last_clean":"2024-09-21T19:04:30.934589+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:04:30.934589+0000","last_undegraded":"2024-09-21T19:04:30.934589+0000","last_fullsized":"2024-09-21T19:04:30.934589+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:49.188813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:49.188813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.668577+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1230,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929311+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:04:30.929311+0000","last_peered":"2024-09-21T19:04:30.929311+0000","last_clean":"2024-09-21T19:04:30.929311+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:04:30.929311+0000","last_undegraded":"2024-09-21T19:04:30.929311+0000","last_fullsized":"2024-09-21T19:04:30.929311+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.975843+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":673,"num_read_kb":668,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":542,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931262+0000","last_change":"2024-09-21T19:00:56.950003+0000","last_active":"2024-09-21T19:04:30.931262+0000","last_peered":"2024-09-21T19:04:30.931262+0000","last_clean":"2024-09-21T19:04:30.931262+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:04:30.931262+0000","last_undegraded":"2024-09-21T19:04:30.931262+0000","last_fullsized":"2024-09-21T19:04:30.931262+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.949931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.949931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.450029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":516,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.940836+0000","last_change":"2024-09-21T19:03:54.568908+0000","last_active":"2024-09-21T19:04:30.940836+0000","last_peered":"2024-09-21T19:04:30.940836+0000","last_clean":"2024-09-21T19:04:30.940836+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:04:30.940836+0000","last_undegraded":"2024-09-21T19:04:30.940836+0000","last_fullsized":"2024-09-21T19:04:30.940836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:54.568845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:54.568845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.700759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"298'9929","reported_seq":10028,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.703218+0000","last_change":"2024-09-21T19:02:29.785323+0000","last_active":"2024-09-21T19:04:32.703218+0000","last_peered":"2024-09-21T19:04:32.703218+0000","last_clean":"2024-09-21T19:04:32.703218+0000","last_became_active":"2024-09-21T19:02:29.785050+0000","last_became_peered":"2024-09-21T19:02:29.785050+0000","last_unstale":"2024-09-21T19:04:32.703218+0000","last_undegraded":"2024-09-21T19:04:32.703218+0000","last_fullsized":"2024-09-21T19:04:32.703218+0000","mapping_epoch":254,"log_start":"283'5500","ondisk_log_start":"283'5500","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401408000,"num_objects":6125,"num_object_clones":0,"num_object_copies":18375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":646,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.939368+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:04:30.939368+0000","last_peered":"2024-09-21T19:04:30.939368+0000","last_clean":"2024-09-21T19:04:30.939368+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:04:30.939368+0000","last_undegraded":"2024-09-21T19:04:30.939368+0000","last_fullsized":"2024-09-21T19:04:30.939368+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:56.838535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":524,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934214+0000","last_change":"2024-09-21T19:02:40.877780+0000","last_active":"2024-09-21T19:04:30.934214+0000","last_peered":"2024-09-21T19:04:30.934214+0000","last_clean":"2024-09-21T19:04:30.934214+0000","last_became_active":"2024-09-21T19:02:40.877452+0000","last_became_peered":"2024-09-21T19:02:40.877452+0000","last_unstale":"2024-09-21T19:04:30.934214+0000","last_undegraded":"2024-09-21T19:04:30.934214+0000","last_fullsized":"2024-09-21T19:04:30.934214+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.441173+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":611,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929356+0000","last_change":"2024-09-21T19:04:28.281992+0000","last_active":"2024-09-21T19:04:30.929356+0000","last_peered":"2024-09-21T19:04:30.929356+0000","last_clean":"2024-09-21T19:04:30.929356+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:04:30.929356+0000","last_undegraded":"2024-09-21T19:04:30.929356+0000","last_fullsized":"2024-09-21T19:04:30.929356+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:28.281931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.281931+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.661197+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930040+0000","last_change":"2024-09-21T19:03:59.160104+0000","last_active":"2024-09-21T19:04:30.930040+0000","last_peered":"2024-09-21T19:04:30.930040+0000","last_clean":"2024-09-21T19:04:30.930040+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:04:30.930040+0000","last_undegraded":"2024-09-21T19:04:30.930040+0000","last_fullsized":"2024-09-21T19:04:30.930040+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.850193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":59,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.945789+0000","last_change":"2024-09-21T19:03:09.625168+0000","last_active":"2024-09-21T19:04:30.945789+0000","last_peered":"2024-09-21T19:04:30.945789+0000","last_clean":"2024-09-21T19:04:30.945789+0000","last_became_active":"2024-09-21T19:03:09.624424+0000","last_became_peered":"2024-09-21T19:03:09.624424+0000","last_unstale":"2024-09-21T19:04:30.945789+0000","last_undegraded":"2024-09-21T19:04:30.945789+0000","last_fullsized":"2024-09-21T19:04:30.945789+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.459943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"298'9685","reported_seq":9784,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.675054+0000","last_change":"2024-09-21T19:02:29.785879+0000","last_active":"2024-09-21T19:04:31.675054+0000","last_peered":"2024-09-21T19:04:31.675054+0000","last_clean":"2024-09-21T19:04:31.675054+0000","last_became_active":"2024-09-21T19:02:29.785492+0000","last_became_peered":"2024-09-21T19:02:29.785492+0000","last_unstale":"2024-09-21T19:04:31.675054+0000","last_undegraded":"2024-09-21T19:04:31.675054+0000","last_fullsized":"2024-09-21T19:04:31.675054+0000","mapping_epoch":254,"log_start":"281'5100","ondisk_log_start":"281'5100","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.229667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401145856,"num_objects":6121,"num_object_clones":0,"num_object_copies":18363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9685,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":574,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943816+0000","last_change":"2024-09-21T19:03:18.675564+0000","last_active":"2024-09-21T19:04:30.943816+0000","last_peered":"2024-09-21T19:04:30.943816+0000","last_clean":"2024-09-21T19:04:30.943816+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:04:30.943816+0000","last_undegraded":"2024-09-21T19:04:30.943816+0000","last_fullsized":"2024-09-21T19:04:30.943816+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:18.675329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.675329+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:42.981301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":528,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943192+0000","last_change":"2024-09-21T19:03:20.643868+0000","last_active":"2024-09-21T19:04:30.943192+0000","last_peered":"2024-09-21T19:04:30.943192+0000","last_clean":"2024-09-21T19:04:30.943192+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:04:30.943192+0000","last_undegraded":"2024-09-21T19:04:30.943192+0000","last_fullsized":"2024-09-21T19:04:30.943192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.643800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.643800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":523,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.944029+0000","last_change":"2024-09-21T19:03:11.634133+0000","last_active":"2024-09-21T19:04:30.944029+0000","last_peered":"2024-09-21T19:04:30.944029+0000","last_clean":"2024-09-21T19:04:30.944029+0000","last_became_active":"2024-09-21T19:03:09.619609+0000","last_became_peered":"2024-09-21T19:03:09.619609+0000","last_unstale":"2024-09-21T19:04:30.944029+0000","last_undegraded":"2024-09-21T19:04:30.944029+0000","last_fullsized":"2024-09-21T19:04:30.944029+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":504,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931721+0000","last_change":"2024-09-21T19:02:46.521753+0000","last_active":"2024-09-21T19:04:30.931721+0000","last_peered":"2024-09-21T19:04:30.931721+0000","last_clean":"2024-09-21T19:04:30.931721+0000","last_became_active":"2024-09-21T19:02:40.877238+0000","last_became_peered":"2024-09-21T19:02:40.877238+0000","last_unstale":"2024-09-21T19:04:30.931721+0000","last_undegraded":"2024-09-21T19:04:30.931721+0000","last_fullsized":"2024-09-21T19:04:30.931721+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"298'9829","reported_seq":10908,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.385075+0000","last_change":"2024-09-21T19:02:46.916256+0000","last_active":"2024-09-21T19:04:32.385075+0000","last_peered":"2024-09-21T19:04:32.385075+0000","last_clean":"2024-09-21T19:04:32.385075+0000","last_became_active":"2024-09-21T19:02:46.913875+0000","last_became_peered":"2024-09-21T19:02:46.913875+0000","last_unstale":"2024-09-21T19:04:32.385075+0000","last_undegraded":"2024-09-21T19:04:32.385075+0000","last_fullsized":"2024-09-21T19:04:32.385075+0000","mapping_epoch":264,"log_start":"281'5000","ondisk_log_start":"281'5000","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4829,"log_dups_size":0,"ondisk_log_size":4829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.725035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402456576,"num_objects":6141,"num_object_clones":0,"num_object_copies":18423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":31850496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":68,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930463+0000","last_change":"2024-09-21T19:03:15.475191+0000","last_active":"2024-09-21T19:04:30.930463+0000","last_peered":"2024-09-21T19:04:30.930463+0000","last_clean":"2024-09-21T19:04:30.930463+0000","last_became_active":"2024-09-21T19:03:09.626413+0000","last_became_peered":"2024-09-21T19:03:09.626413+0000","last_unstale":"2024-09-21T19:04:30.930463+0000","last_undegraded":"2024-09-21T19:04:30.930463+0000","last_fullsized":"2024-09-21T19:04:30.930463+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:15.475112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.475112+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:43.597628+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":565,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943395+0000","last_change":"2024-09-21T19:02:07.716667+0000","last_active":"2024-09-21T19:04:30.943395+0000","last_peered":"2024-09-21T19:04:30.943395+0000","last_clean":"2024-09-21T19:04:30.943395+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:04:30.943395+0000","last_undegraded":"2024-09-21T19:04:30.943395+0000","last_fullsized":"2024-09-21T19:04:30.943395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T19:02:07.716607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:07.716607+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:34.816871+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":440,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.939346+0000","last_change":"2024-09-21T19:02:40.879761+0000","last_active":"2024-09-21T19:04:30.939346+0000","last_peered":"2024-09-21T19:04:30.939346+0000","last_clean":"2024-09-21T19:04:30.939346+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:04:30.939346+0000","last_undegraded":"2024-09-21T19:04:30.939346+0000","last_fullsized":"2024-09-21T19:04:30.939346+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:14.376671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:14.376671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.783406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942312+0000","last_change":"2024-09-21T19:03:42.617727+0000","last_active":"2024-09-21T19:04:30.942312+0000","last_peered":"2024-09-21T19:04:30.942312+0000","last_clean":"2024-09-21T19:04:30.942312+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:04:30.942312+0000","last_undegraded":"2024-09-21T19:04:30.942312+0000","last_fullsized":"2024-09-21T19:04:30.942312+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.806920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":499,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927948+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:04:30.927948+0000","last_peered":"2024-09-21T19:04:30.927948+0000","last_clean":"2024-09-21T19:04:30.927948+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:04:30.927948+0000","last_undegraded":"2024-09-21T19:04:30.927948+0000","last_fullsized":"2024-09-21T19:04:30.927948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.397993+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"298'9850","reported_seq":10942,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.665961+0000","last_change":"2024-09-21T19:02:46.953650+0000","last_active":"2024-09-21T19:04:32.665961+0000","last_peered":"2024-09-21T19:04:32.665961+0000","last_clean":"2024-09-21T19:04:32.665961+0000","last_became_active":"2024-09-21T19:02:46.951397+0000","last_became_peered":"2024-09-21T19:02:46.951397+0000","last_unstale":"2024-09-21T19:04:32.665961+0000","last_undegraded":"2024-09-21T19:04:32.665961+0000","last_fullsized":"2024-09-21T19:04:32.665961+0000","mapping_epoch":264,"log_start":"284'5500","ondisk_log_start":"284'5500","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395706368,"num_objects":6038,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":508416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":32243712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":605,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930308+0000","last_change":"2024-09-21T19:03:48.127190+0000","last_active":"2024-09-21T19:04:30.930308+0000","last_peered":"2024-09-21T19:04:30.930308+0000","last_clean":"2024-09-21T19:04:30.930308+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:04:30.930308+0000","last_undegraded":"2024-09-21T19:04:30.930308+0000","last_fullsized":"2024-09-21T19:04:30.930308+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:48.127135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.127135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.997742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":549,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943059+0000","last_change":"2024-09-21T19:02:40.883840+0000","last_active":"2024-09-21T19:04:30.943059+0000","last_peered":"2024-09-21T19:04:30.943059+0000","last_clean":"2024-09-21T19:04:30.943059+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:04:30.943059+0000","last_undegraded":"2024-09-21T19:04:30.943059+0000","last_fullsized":"2024-09-21T19:04:30.943059+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.862643+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":549,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930799+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:04:30.930799+0000","last_peered":"2024-09-21T19:04:30.930799+0000","last_clean":"2024-09-21T19:04:30.930799+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:04:30.930799+0000","last_undegraded":"2024-09-21T19:04:30.930799+0000","last_fullsized":"2024-09-21T19:04:30.930799+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":665,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927615+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:04:30.927615+0000","last_peered":"2024-09-21T19:04:30.927615+0000","last_clean":"2024-09-21T19:04:30.927615+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:04:30.927615+0000","last_undegraded":"2024-09-21T19:04:30.927615+0000","last_fullsized":"2024-09-21T19:04:30.927615+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.454335+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":500,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943745+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:04:30.943745+0000","last_peered":"2024-09-21T19:04:30.943745+0000","last_clean":"2024-09-21T19:04:30.943745+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:04:30.943745+0000","last_undegraded":"2024-09-21T19:04:30.943745+0000","last_fullsized":"2024-09-21T19:04:30.943745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.831345+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"298'9944","reported_seq":10094,"reported_epoch":298,"state":"active+clean+scrubbing","last_fresh":"2024-09-21T19:04:31.870596+0000","last_change":"2024-09-21T19:04:30.573508+0000","last_active":"2024-09-21T19:04:31.870596+0000","last_peered":"2024-09-21T19:04:31.870596+0000","last_clean":"2024-09-21T19:04:31.870596+0000","last_became_active":"2024-09-21T19:03:09.623358+0000","last_became_peered":"2024-09-21T19:03:09.623358+0000","last_unstale":"2024-09-21T19:04:31.870596+0000","last_undegraded":"2024-09-21T19:04:31.870596+0000","last_fullsized":"2024-09-21T19:04:31.870596+0000","mapping_epoch":274,"log_start":"273'3500","ondisk_log_start":"273'3500","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":4049,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403832832,"num_objects":6162,"num_object_clones":0,"num_object_copies":18486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":410,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934312+0000","last_change":"2024-09-21T19:03:52.722880+0000","last_active":"2024-09-21T19:04:30.934312+0000","last_peered":"2024-09-21T19:04:30.934312+0000","last_clean":"2024-09-21T19:04:30.934312+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:04:30.934312+0000","last_undegraded":"2024-09-21T19:04:30.934312+0000","last_fullsized":"2024-09-21T19:04:30.934312+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:52.722797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.722797+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.206303+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":507,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933162+0000","last_change":"2024-09-21T19:03:09.630384+0000","last_active":"2024-09-21T19:04:30.933162+0000","last_peered":"2024-09-21T19:04:30.933162+0000","last_clean":"2024-09-21T19:04:30.933162+0000","last_became_active":"2024-09-21T19:03:09.629753+0000","last_became_peered":"2024-09-21T19:03:09.629753+0000","last_unstale":"2024-09-21T19:04:30.933162+0000","last_undegraded":"2024-09-21T19:04:30.933162+0000","last_fullsized":"2024-09-21T19:04:30.933162+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":512,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927302+0000","last_change":"2024-09-21T19:03:57.305837+0000","last_active":"2024-09-21T19:04:30.927302+0000","last_peered":"2024-09-21T19:04:30.927302+0000","last_clean":"2024-09-21T19:04:30.927302+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:04:30.927302+0000","last_undegraded":"2024-09-21T19:04:30.927302+0000","last_fullsized":"2024-09-21T19:04:30.927302+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:26.798148+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":392,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.934543+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:04:30.934543+0000","last_peered":"2024-09-21T19:04:30.934543+0000","last_clean":"2024-09-21T19:04:30.934543+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:04:30.934543+0000","last_undegraded":"2024-09-21T19:04:30.934543+0000","last_fullsized":"2024-09-21T19:04:30.934543+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.597338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"298'9892","reported_seq":10071,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.670759+0000","last_change":"2024-09-21T19:03:47.989682+0000","last_active":"2024-09-21T19:04:31.670759+0000","last_peered":"2024-09-21T19:04:31.670759+0000","last_clean":"2024-09-21T19:04:31.670759+0000","last_became_active":"2024-09-21T19:03:09.625334+0000","last_became_peered":"2024-09-21T19:03:09.625334+0000","last_unstale":"2024-09-21T19:04:31.670759+0000","last_undegraded":"2024-09-21T19:04:31.670759+0000","last_fullsized":"2024-09-21T19:04:31.670759+0000","mapping_epoch":274,"log_start":"283'5300","ondisk_log_start":"283'5300","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6556","last_scrub_stamp":"2024-09-21T19:03:47.989483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.989483+0000","objects_scrubbed":6505,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:51.168208+0000","scrub_duration":1845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403046400,"num_objects":6150,"num_object_clones":0,"num_object_copies":18450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":459,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.939221+0000","last_change":"2024-09-21T19:02:40.878888+0000","last_active":"2024-09-21T19:04:30.939221+0000","last_peered":"2024-09-21T19:04:30.939221+0000","last_clean":"2024-09-21T19:04:30.939221+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:04:30.939221+0000","last_undegraded":"2024-09-21T19:04:30.939221+0000","last_fullsized":"2024-09-21T19:04:30.939221+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.891844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":538,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931858+0000","last_change":"2024-09-21T19:04:29.259617+0000","last_active":"2024-09-21T19:04:30.931858+0000","last_peered":"2024-09-21T19:04:30.931858+0000","last_clean":"2024-09-21T19:04:30.931858+0000","last_became_active":"2024-09-21T19:02:40.888713+0000","last_became_peered":"2024-09-21T19:02:40.888713+0000","last_unstale":"2024-09-21T19:04:30.931858+0000","last_undegraded":"2024-09-21T19:04:30.931858+0000","last_fullsized":"2024-09-21T19:04:30.931858+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:57.089123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":534,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.926947+0000","last_change":"2024-09-21T19:02:47.357147+0000","last_active":"2024-09-21T19:04:30.926947+0000","last_peered":"2024-09-21T19:04:30.926947+0000","last_clean":"2024-09-21T19:04:30.926947+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:04:30.926947+0000","last_undegraded":"2024-09-21T19:04:30.926947+0000","last_fullsized":"2024-09-21T19:04:30.926947+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.448219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":499,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929331+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:04:30.929331+0000","last_peered":"2024-09-21T19:04:30.929331+0000","last_clean":"2024-09-21T19:04:30.929331+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:04:30.929331+0000","last_undegraded":"2024-09-21T19:04:30.929331+0000","last_fullsized":"2024-09-21T19:04:30.929331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.183675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"298'9647","reported_seq":9746,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.383818+0000","last_change":"2024-09-21T19:02:30.708593+0000","last_active":"2024-09-21T19:04:32.383818+0000","last_peered":"2024-09-21T19:04:32.383818+0000","last_clean":"2024-09-21T19:04:32.383818+0000","last_became_active":"2024-09-21T19:02:30.708207+0000","last_became_peered":"2024-09-21T19:02:30.708207+0000","last_unstale":"2024-09-21T19:04:32.383818+0000","last_undegraded":"2024-09-21T19:04:32.383818+0000","last_fullsized":"2024-09-21T19:04:32.383818+0000","mapping_epoch":254,"log_start":"281'4800","ondisk_log_start":"281'4800","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.754754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399441920,"num_objects":6095,"num_object_clones":0,"num_object_copies":18285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":567,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942796+0000","last_change":"2024-09-21T19:02:04.675992+0000","last_active":"2024-09-21T19:04:30.942796+0000","last_peered":"2024-09-21T19:04:30.942796+0000","last_clean":"2024-09-21T19:04:30.942796+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:04:30.942796+0000","last_undegraded":"2024-09-21T19:04:30.942796+0000","last_fullsized":"2024-09-21T19:04:30.942796+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:02:04.675911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:04.675911+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:33.799727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":535,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930508+0000","last_change":"2024-09-21T19:03:47.143498+0000","last_active":"2024-09-21T19:04:30.930508+0000","last_peered":"2024-09-21T19:04:30.930508+0000","last_clean":"2024-09-21T19:04:30.930508+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:04:30.930508+0000","last_undegraded":"2024-09-21T19:04:30.930508+0000","last_fullsized":"2024-09-21T19:04:30.930508+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:47.143439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.143439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:52.903583+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":534,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.935248+0000","last_change":"2024-09-21T19:01:48.464855+0000","last_active":"2024-09-21T19:04:30.935248+0000","last_peered":"2024-09-21T19:04:30.935248+0000","last_clean":"2024-09-21T19:04:30.935248+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:04:30.935248+0000","last_undegraded":"2024-09-21T19:04:30.935248+0000","last_fullsized":"2024-09-21T19:04:30.935248+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.370827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":501,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930536+0000","last_change":"2024-09-21T19:03:20.446083+0000","last_active":"2024-09-21T19:04:30.930536+0000","last_peered":"2024-09-21T19:04:30.930536+0000","last_clean":"2024-09-21T19:04:30.930536+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:04:30.930536+0000","last_undegraded":"2024-09-21T19:04:30.930536+0000","last_fullsized":"2024-09-21T19:04:30.930536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.446014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.446014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.270682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"298'9698","reported_seq":9797,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:32.096088+0000","last_change":"2024-09-21T19:02:29.787669+0000","last_active":"2024-09-21T19:04:32.096088+0000","last_peered":"2024-09-21T19:04:32.096088+0000","last_clean":"2024-09-21T19:04:32.096088+0000","last_became_active":"2024-09-21T19:02:29.787259+0000","last_became_peered":"2024-09-21T19:02:29.787259+0000","last_unstale":"2024-09-21T19:04:32.096088+0000","last_undegraded":"2024-09-21T19:04:32.096088+0000","last_fullsized":"2024-09-21T19:04:32.096088+0000","mapping_epoch":254,"log_start":"275'4000","ondisk_log_start":"275'4000","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:39.635993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395575296,"num_objects":6036,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9698,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":130,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929229+0000","last_change":"2024-09-21T19:03:56.136641+0000","last_active":"2024-09-21T19:04:30.929229+0000","last_peered":"2024-09-21T19:04:30.929229+0000","last_clean":"2024-09-21T19:04:30.929229+0000","last_became_active":"2024-09-21T19:03:09.619439+0000","last_became_peered":"2024-09-21T19:03:09.619439+0000","last_unstale":"2024-09-21T19:04:30.929229+0000","last_undegraded":"2024-09-21T19:04:30.929229+0000","last_fullsized":"2024-09-21T19:04:30.929229+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:03.843546+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":549,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932216+0000","last_change":"2024-09-21T19:02:06.757027+0000","last_active":"2024-09-21T19:04:30.932216+0000","last_peered":"2024-09-21T19:04:30.932216+0000","last_clean":"2024-09-21T19:04:30.932216+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:04:30.932216+0000","last_undegraded":"2024-09-21T19:04:30.932216+0000","last_fullsized":"2024-09-21T19:04:30.932216+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:02:06.756954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:06.756954+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"263'10","reported_seq":680,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930131+0000","last_change":"2024-09-21T19:02:46.144415+0000","last_active":"2024-09-21T19:04:30.930131+0000","last_peered":"2024-09-21T19:04:30.930131+0000","last_clean":"2024-09-21T19:04:30.930131+0000","last_became_active":"2024-09-21T19:02:40.887248+0000","last_became_peered":"2024-09-21T19:02:40.887248+0000","last_unstale":"2024-09-21T19:04:30.930131+0000","last_undegraded":"2024-09-21T19:04:30.930131+0000","last_fullsized":"2024-09-21T19:04:30.930131+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.939086+0000","last_change":"2024-09-21T19:04:24.782721+0000","last_active":"2024-09-21T19:04:30.939086+0000","last_peered":"2024-09-21T19:04:30.939086+0000","last_clean":"2024-09-21T19:04:30.939086+0000","last_became_active":"2024-09-21T19:03:09.618375+0000","last_became_peered":"2024-09-21T19:03:09.618375+0000","last_unstale":"2024-09-21T19:04:30.939086+0000","last_undegraded":"2024-09-21T19:04:30.939086+0000","last_fullsized":"2024-09-21T19:04:30.939086+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:26.072336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":460,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.940706+0000","last_change":"2024-09-21T19:02:40.882545+0000","last_active":"2024-09-21T19:04:30.940706+0000","last_peered":"2024-09-21T19:04:30.940706+0000","last_clean":"2024-09-21T19:04:30.940706+0000","last_became_active":"2024-09-21T19:02:40.882216+0000","last_became_peered":"2024-09-21T19:02:40.882216+0000","last_unstale":"2024-09-21T19:04:30.940706+0000","last_undegraded":"2024-09-21T19:04:30.940706+0000","last_fullsized":"2024-09-21T19:04:30.940706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.745928+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":542,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:31.523836+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:04:31.523836+0000","last_peered":"2024-09-21T19:04:31.523836+0000","last_clean":"2024-09-21T19:04:31.523836+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:04:31.523836+0000","last_undegraded":"2024-09-21T19:04:31.523836+0000","last_fullsized":"2024-09-21T19:04:31.523836+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:59.874264+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":67,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930597+0000","last_change":"2024-09-21T19:03:11.520901+0000","last_active":"2024-09-21T19:04:30.930597+0000","last_peered":"2024-09-21T19:04:30.930597+0000","last_clean":"2024-09-21T19:04:30.930597+0000","last_became_active":"2024-09-21T19:03:09.623017+0000","last_became_peered":"2024-09-21T19:03:09.623017+0000","last_unstale":"2024-09-21T19:04:30.930597+0000","last_undegraded":"2024-09-21T19:04:30.930597+0000","last_fullsized":"2024-09-21T19:04:30.930597+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.264605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":509,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931418+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:04:30.931418+0000","last_peered":"2024-09-21T19:04:30.931418+0000","last_clean":"2024-09-21T19:04:30.931418+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:04:30.931418+0000","last_undegraded":"2024-09-21T19:04:30.931418+0000","last_fullsized":"2024-09-21T19:04:30.931418+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":418,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932371+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:04:30.932371+0000","last_peered":"2024-09-21T19:04:30.932371+0000","last_clean":"2024-09-21T19:04:30.932371+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:04:30.932371+0000","last_undegraded":"2024-09-21T19:04:30.932371+0000","last_fullsized":"2024-09-21T19:04:30.932371+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:15.701573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":529,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930437+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:04:30.930437+0000","last_peered":"2024-09-21T19:04:30.930437+0000","last_clean":"2024-09-21T19:04:30.930437+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:04:30.930437+0000","last_undegraded":"2024-09-21T19:04:30.930437+0000","last_fullsized":"2024-09-21T19:04:30.930437+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.889409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":504,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.945955+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:04:30.945955+0000","last_peered":"2024-09-21T19:04:30.945955+0000","last_clean":"2024-09-21T19:04:30.945955+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:04:30.945955+0000","last_undegraded":"2024-09-21T19:04:30.945955+0000","last_fullsized":"2024-09-21T19:04:30.945955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.488303+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":523,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.927052+0000","last_change":"2024-09-21T19:03:44.187925+0000","last_active":"2024-09-21T19:04:30.927052+0000","last_peered":"2024-09-21T19:04:30.927052+0000","last_clean":"2024-09-21T19:04:30.927052+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:04:30.927052+0000","last_undegraded":"2024-09-21T19:04:30.927052+0000","last_fullsized":"2024-09-21T19:04:30.927052+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.242342+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":112,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931665+0000","last_change":"2024-09-21T19:03:14.187382+0000","last_active":"2024-09-21T19:04:30.931665+0000","last_peered":"2024-09-21T19:04:30.931665+0000","last_clean":"2024-09-21T19:04:30.931665+0000","last_became_active":"2024-09-21T19:03:09.624551+0000","last_became_peered":"2024-09-21T19:03:09.624551+0000","last_unstale":"2024-09-21T19:04:30.931665+0000","last_undegraded":"2024-09-21T19:04:30.931665+0000","last_fullsized":"2024-09-21T19:04:30.931665+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:03:14.187189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.187189+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":399,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933494+0000","last_change":"2024-09-21T19:02:15.401234+0000","last_active":"2024-09-21T19:04:30.933494+0000","last_peered":"2024-09-21T19:04:30.933494+0000","last_clean":"2024-09-21T19:04:30.933494+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:04:30.933494+0000","last_undegraded":"2024-09-21T19:04:30.933494+0000","last_fullsized":"2024-09-21T19:04:30.933494+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:15.401172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:15.401172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":658,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931814+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:04:30.931814+0000","last_peered":"2024-09-21T19:04:30.931814+0000","last_clean":"2024-09-21T19:04:30.931814+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:04:30.931814+0000","last_undegraded":"2024-09-21T19:04:30.931814+0000","last_fullsized":"2024-09-21T19:04:30.931814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.472526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":507,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.943905+0000","last_change":"2024-09-21T19:02:36.651427+0000","last_active":"2024-09-21T19:04:30.943905+0000","last_peered":"2024-09-21T19:04:30.943905+0000","last_clean":"2024-09-21T19:04:30.943905+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:04:30.943905+0000","last_undegraded":"2024-09-21T19:04:30.943905+0000","last_fullsized":"2024-09-21T19:04:30.943905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":538,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932971+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:04:30.932971+0000","last_peered":"2024-09-21T19:04:30.932971+0000","last_clean":"2024-09-21T19:04:30.932971+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:04:30.932971+0000","last_undegraded":"2024-09-21T19:04:30.932971+0000","last_fullsized":"2024-09-21T19:04:30.932971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.100625+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":510,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929898+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:04:30.929898+0000","last_peered":"2024-09-21T19:04:30.929898+0000","last_clean":"2024-09-21T19:04:30.929898+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:04:30.929898+0000","last_undegraded":"2024-09-21T19:04:30.929898+0000","last_fullsized":"2024-09-21T19:04:30.929898+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.236097+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"275'11","reported_seq":109,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942285+0000","last_change":"2024-09-21T19:03:11.527080+0000","last_active":"2024-09-21T19:04:30.942285+0000","last_peered":"2024-09-21T19:04:30.942285+0000","last_clean":"2024-09-21T19:04:30.942285+0000","last_became_active":"2024-09-21T19:03:09.624143+0000","last_became_peered":"2024-09-21T19:03:09.624143+0000","last_unstale":"2024-09-21T19:04:30.942285+0000","last_undegraded":"2024-09-21T19:04:30.942285+0000","last_fullsized":"2024-09-21T19:04:30.942285+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:11.527007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.527007+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.827620+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":407,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933419+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:04:30.933419+0000","last_peered":"2024-09-21T19:04:30.933419+0000","last_clean":"2024-09-21T19:04:30.933419+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:04:30.933419+0000","last_undegraded":"2024-09-21T19:04:30.933419+0000","last_fullsized":"2024-09-21T19:04:30.933419+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.717779+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":584,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942885+0000","last_change":"2024-09-21T19:04:29.685758+0000","last_active":"2024-09-21T19:04:30.942885+0000","last_peered":"2024-09-21T19:04:30.942885+0000","last_clean":"2024-09-21T19:04:30.942885+0000","last_became_active":"2024-09-21T19:02:40.880515+0000","last_became_peered":"2024-09-21T19:02:40.880515+0000","last_unstale":"2024-09-21T19:04:30.942885+0000","last_undegraded":"2024-09-21T19:04:30.942885+0000","last_fullsized":"2024-09-21T19:04:30.942885+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:04:29.685680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.685680+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:30.435080+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":530,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.944793+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:04:30.944793+0000","last_peered":"2024-09-21T19:04:30.944793+0000","last_clean":"2024-09-21T19:04:30.944793+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:04:30.944793+0000","last_undegraded":"2024-09-21T19:04:30.944793+0000","last_fullsized":"2024-09-21T19:04:30.944793+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:48.105596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":511,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.940785+0000","last_change":"2024-09-21T18:59:52.391075+0000","last_active":"2024-09-21T19:04:30.940785+0000","last_peered":"2024-09-21T19:04:30.940785+0000","last_clean":"2024-09-21T19:04:30.940785+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:04:30.940785+0000","last_undegraded":"2024-09-21T19:04:30.940785+0000","last_fullsized":"2024-09-21T19:04:30.940785+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:52.389908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:52.389908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":514,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930875+0000","last_change":"2024-09-21T19:02:35.435594+0000","last_active":"2024-09-21T19:04:30.930875+0000","last_peered":"2024-09-21T19:04:30.930875+0000","last_clean":"2024-09-21T19:04:30.930875+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:04:30.930875+0000","last_undegraded":"2024-09-21T19:04:30.930875+0000","last_fullsized":"2024-09-21T19:04:30.930875+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.435525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.435525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":559,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.941239+0000","last_change":"2024-09-21T19:02:40.879297+0000","last_active":"2024-09-21T19:04:30.941239+0000","last_peered":"2024-09-21T19:04:30.941239+0000","last_clean":"2024-09-21T19:04:30.941239+0000","last_became_active":"2024-09-21T19:02:40.878893+0000","last_became_peered":"2024-09-21T19:02:40.878893+0000","last_unstale":"2024-09-21T19:04:30.941239+0000","last_undegraded":"2024-09-21T19:04:30.941239+0000","last_fullsized":"2024-09-21T19:04:30.941239+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.762110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931776+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:04:30.931776+0000","last_peered":"2024-09-21T19:04:30.931776+0000","last_clean":"2024-09-21T19:04:30.931776+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:04:30.931776+0000","last_undegraded":"2024-09-21T19:04:30.931776+0000","last_fullsized":"2024-09-21T19:04:30.931776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.156153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930934+0000","last_change":"2024-09-21T19:02:36.480829+0000","last_active":"2024-09-21T19:04:30.930934+0000","last_peered":"2024-09-21T19:04:30.930934+0000","last_clean":"2024-09-21T19:04:30.930934+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:04:30.930934+0000","last_undegraded":"2024-09-21T19:04:30.930934+0000","last_fullsized":"2024-09-21T19:04:30.930934+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.480784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.480784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":407,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.932195+0000","last_change":"2024-09-21T19:02:19.440104+0000","last_active":"2024-09-21T19:04:30.932195+0000","last_peered":"2024-09-21T19:04:30.932195+0000","last_clean":"2024-09-21T19:04:30.932195+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:04:30.932195+0000","last_undegraded":"2024-09-21T19:04:30.932195+0000","last_fullsized":"2024-09-21T19:04:30.932195+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:19.440059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:19.440059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.686807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":550,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931192+0000","last_change":"2024-09-21T19:02:40.882171+0000","last_active":"2024-09-21T19:04:30.931192+0000","last_peered":"2024-09-21T19:04:30.931192+0000","last_clean":"2024-09-21T19:04:30.931192+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:04:30.931192+0000","last_undegraded":"2024-09-21T19:04:30.931192+0000","last_fullsized":"2024-09-21T19:04:30.931192+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.365192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":511,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929060+0000","last_change":"2024-09-21T18:59:12.577156+0000","last_active":"2024-09-21T19:04:30.929060+0000","last_peered":"2024-09-21T19:04:30.929060+0000","last_clean":"2024-09-21T19:04:30.929060+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:04:30.929060+0000","last_undegraded":"2024-09-21T19:04:30.929060+0000","last_fullsized":"2024-09-21T19:04:30.929060+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:12.577104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:12.577104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:32.379751+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":529,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929037+0000","last_change":"2024-09-21T19:02:48.126289+0000","last_active":"2024-09-21T19:04:30.929037+0000","last_peered":"2024-09-21T19:04:30.929037+0000","last_clean":"2024-09-21T19:04:30.929037+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:04:30.929037+0000","last_undegraded":"2024-09-21T19:04:30.929037+0000","last_fullsized":"2024-09-21T19:04:30.929037+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:48.126230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:48.126230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.930708+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:04:30.930708+0000","last_peered":"2024-09-21T19:04:30.930708+0000","last_clean":"2024-09-21T19:04:30.930708+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:04:30.930708+0000","last_undegraded":"2024-09-21T19:04:30.930708+0000","last_fullsized":"2024-09-21T19:04:30.930708+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:36.993219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":549,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931096+0000","last_change":"2024-09-21T19:02:59.672588+0000","last_active":"2024-09-21T19:04:30.931096+0000","last_peered":"2024-09-21T19:04:30.931096+0000","last_clean":"2024-09-21T19:04:30.931096+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:04:30.931096+0000","last_undegraded":"2024-09-21T19:04:30.931096+0000","last_fullsized":"2024-09-21T19:04:30.931096+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-21T19:02:41.535456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.535456+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":399,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933697+0000","last_change":"2024-09-21T19:03:55.627621+0000","last_active":"2024-09-21T19:04:30.933697+0000","last_peered":"2024-09-21T19:04:30.933697+0000","last_clean":"2024-09-21T19:04:30.933697+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:04:30.933697+0000","last_undegraded":"2024-09-21T19:04:30.933697+0000","last_fullsized":"2024-09-21T19:04:30.933697+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.004468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":438,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.938951+0000","last_change":"2024-09-21T19:02:40.881419+0000","last_active":"2024-09-21T19:04:30.938951+0000","last_peered":"2024-09-21T19:04:30.938951+0000","last_clean":"2024-09-21T19:04:30.938951+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:04:30.938951+0000","last_undegraded":"2024-09-21T19:04:30.938951+0000","last_fullsized":"2024-09-21T19:04:30.938951+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.789192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":506,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.929929+0000","last_change":"2024-09-21T19:02:36.128893+0000","last_active":"2024-09-21T19:04:30.929929+0000","last_peered":"2024-09-21T19:04:30.929929+0000","last_clean":"2024-09-21T19:04:30.929929+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:04:30.929929+0000","last_undegraded":"2024-09-21T19:04:30.929929+0000","last_fullsized":"2024-09-21T19:04:30.929929+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":499,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.942143+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:04:30.942143+0000","last_peered":"2024-09-21T19:04:30.942143+0000","last_clean":"2024-09-21T19:04:30.942143+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:04:30.942143+0000","last_undegraded":"2024-09-21T19:04:30.942143+0000","last_fullsized":"2024-09-21T19:04:30.942143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.485168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":542,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.931455+0000","last_change":"2024-09-21T19:03:18.435824+0000","last_active":"2024-09-21T19:04:30.931455+0000","last_peered":"2024-09-21T19:04:30.931455+0000","last_clean":"2024-09-21T19:04:30.931455+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:04:30.931455+0000","last_undegraded":"2024-09-21T19:04:30.931455+0000","last_fullsized":"2024-09-21T19:04:30.931455+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:18.435733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.435733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.735344+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":500,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.940035+0000","last_change":"2024-09-21T19:03:17.580525+0000","last_active":"2024-09-21T19:04:30.940035+0000","last_peered":"2024-09-21T19:04:30.940035+0000","last_clean":"2024-09-21T19:04:30.940035+0000","last_became_active":"2024-09-21T19:02:40.882966+0000","last_became_peered":"2024-09-21T19:02:40.882966+0000","last_unstale":"2024-09-21T19:04:30.940035+0000","last_undegraded":"2024-09-21T19:04:30.940035+0000","last_fullsized":"2024-09-21T19:04:30.940035+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:35.669525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":408,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-21T19:04:30.933325+0000","last_change":"2024-09-21T19:01:11.531925+0000","last_active":"2024-09-21T19:04:30.933325+0000","last_peered":"2024-09-21T19:04:30.933325+0000","last_clean":"2024-09-21T19:04:30.933325+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:04:30.933325+0000","last_undegraded":"2024-09-21T19:04:30.933325+0000","last_fullsized":"2024-09-21T19:04:30.933325+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.606767+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6440681496,"num_objects":98278,"num_object_clones":0,"num_object_copies":294834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157428,"num_write_kb":8182529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":97714176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19317600256,"data_stored":19317588040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79028,"ondisk_log_size":79028,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":673,"num_read_kb":668,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266829,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4006136,"kb_used_data":3587620,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":89739016,"statfs":{"total":95995035648,"available":91892752384,"internally_reserved":0,"allocated":3673722880,"data_stored":3673330470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464280,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1245992,"kb_used_data":1155672,"kb_used_omap":4,"kb_used_meta":90299,"kb_avail":92499160,"statfs":{"total":95995035648,"available":94719139840,"internally_reserved":0,"allocated":1183408128,"data_stored":1183027553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":92466852},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790101,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4073388,"kb_used_data":3572764,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":89671764,"statfs":{"total":95995035648,"available":91823886336,"internally_reserved":0,"allocated":3658510336,"data_stored":3658126143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019035,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1555236,"kb_used_data":1213076,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92189916,"statfs":{"total":95995035648,"available":94402473984,"internally_reserved":0,"allocated":1242189824,"data_stored":1241807140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182560,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3678348,"kb_used_data":3115256,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90066804,"statfs":{"total":95995035648,"available":92228407296,"internally_reserved":0,"allocated":3190022144,"data_stored":3189610573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":576387530},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411494,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1540848,"kb_used_data":1184800,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92204304,"statfs":{"total":95995035648,"available":94417207296,"internally_reserved":0,"allocated":1213235200,"data_stored":1212847442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4138824,"kb_used_data":3553652,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":89606328,"statfs":{"total":95995035648,"available":91756879872,"internally_reserved":0,"allocated":3638939648,"data_stored":3638531207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738544,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2099124,"kb_used_data":1576484,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91646028,"statfs":{"total":95995035648,"available":93845532672,"internally_reserved":0,"allocated":1614319616,"data_stored":1613927871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1606418432,"data_stored":1606418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3619688448,"data_stored":3619684376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1207566336,"data_stored":1207566336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3181903872,"data_stored":3181903872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1226899456,"data_stored":1226899456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3645444096,"data_stored":3645440024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1180499968,"data_stored":1180499968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3649179648,"data_stored":3649175576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:04:39.210 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T19:04:39.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:04:39.211 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T19:04:39.211 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T19:04:39.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:04:39.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[29384]: pgmap v560: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-21T19:04:39.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:39.249 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2211168852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:04:39.249 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[29384]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T19:04:39.249 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[39192]: pgmap v560: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-21T19:04:39.249 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:39.249 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2211168852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:04:39.249 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:38 smithi081 ceph-mon[39192]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T19:04:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:38 smithi104 ceph-mon[32438]: pgmap v560: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-21T19:04:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:38 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2211168852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:04:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:38 smithi104 ceph-mon[32438]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T19:04:39.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:39.306 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:39.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:39.454 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:39.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:39.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:39.706 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:39.737 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:39.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:39.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:04:39.868 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:39.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:39.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:39.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:40 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:40 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:40.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:40.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:40.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:40.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:40 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:40.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:40.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.402 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:40.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:40.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:40.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:40.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.667 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:40.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:40.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:40.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:40.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:40.972 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:40.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:41.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:41.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:41.149 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:41 smithi081 ceph-mon[39192]: pgmap v562: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-21T19:04:41.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:41 smithi081 ceph-mon[29384]: pgmap v562: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-21T19:04:41.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:41.242 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:41.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:41 smithi104 ceph-mon[32438]: pgmap v562: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-21T19:04:41.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:41.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:41.416 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:41.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:41.477 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:41.478 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:41.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:41.530 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:41.565 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:41.565 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:41.600 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:41.601 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:41.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:41.633 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:41.633 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:41.665 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:41.666 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:41.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:41.701 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:41.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:41.738 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:41.738 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:41.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:41.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:41.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:41.955 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:41.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:42.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/626098574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:04:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:04:42.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/626098574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:42.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:04:42.089 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:42.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.225 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:42.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:42.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:42.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:42.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:42.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/626098574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:04:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:04:42.591 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:04:42.593 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:04:42.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:42.741 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:42.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:04:42.772 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:42.874 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:42.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:43.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:43.049 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:43 smithi081 ceph-mon[29384]: pgmap v563: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 794 op/s 2024-09-21T19:04:43.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:43 smithi081 ceph-mon[29384]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T19:04:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:43 smithi081 ceph-mon[39192]: pgmap v563: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 794 op/s 2024-09-21T19:04:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:43 smithi081 ceph-mon[39192]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T19:04:43.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:43.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:43.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:43.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.470 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:43.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:43.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:43 smithi104 ceph-mon[32438]: pgmap v563: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 794 op/s 2024-09-21T19:04:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:43 smithi104 ceph-mon[32438]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T19:04:43.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:43.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:43.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.744 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:43.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:43.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:43.883 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:43.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:43.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:44.081 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:44.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:44.169 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:44.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:44.301 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:44.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:44 smithi081 ceph-mon[39192]: from='client.35927 v1:172.21.15.81:0/724667967' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:44 smithi081 ceph-mon[29384]: from='client.35927 v1:172.21.15.81:0/724667967' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:44.402 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:44.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:44.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:44 smithi104 ceph-mon[32438]: from='client.35927 v1:172.21.15.81:0/724667967' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:04:44.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:44.696 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:44.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:44.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:44.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:44.965 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:44.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:45.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:45.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:45.118 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-09-21T19:04:42.014839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5865937459,"num_objects":89698,"num_object_clones":0,"num_object_copies":269094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89698,"num_whiteouts":0,"num_read":1825,"num_read_kb":1577,"num_write":166929,"num_write_kb":8182574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1619,"num_bytes_recovered":97720406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80650,"ondisk_log_size":80650,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20570508,"kb_used_data":17122884,"kb_used_omap":15,"kb_used_meta":3447408,"kb_avail":729390708,"statfs":{"total":767960285184,"available":746896084992,"internally_reserved":0,"allocated":17533833216,"data_stored":17530619003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":3530146578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,9,11,8,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-575340544,"num_objects":-8779,"num_object_clones":0,"num_object_copies":-26337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8779,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8779,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.382453"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":410,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981278+0000","last_change":"2024-09-21T19:02:35.574023+0000","last_active":"2024-09-21T19:04:35.981278+0000","last_peered":"2024-09-21T19:04:35.981278+0000","last_clean":"2024-09-21T19:04:35.981278+0000","last_became_active":"2024-09-21T18:55:51.525167+0000","last_became_peered":"2024-09-21T18:55:51.525167+0000","last_unstale":"2024-09-21T19:04:35.981278+0000","last_undegraded":"2024-09-21T19:04:35.981278+0000","last_fullsized":"2024-09-21T19:04:35.981278+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.711076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":580,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.991632+0000","last_change":"2024-09-21T19:03:15.600063+0000","last_active":"2024-09-21T19:04:37.991632+0000","last_peered":"2024-09-21T19:04:37.991632+0000","last_clean":"2024-09-21T19:04:37.991632+0000","last_became_active":"2024-09-21T19:02:40.874205+0000","last_became_peered":"2024-09-21T19:02:40.874205+0000","last_unstale":"2024-09-21T19:04:37.991632+0000","last_undegraded":"2024-09-21T19:04:37.991632+0000","last_fullsized":"2024-09-21T19:04:37.991632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:15.600006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.600006+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.243933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":521,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.095595+0000","last_change":"2024-09-21T19:04:37.095595+0000","last_active":"2024-09-21T19:04:37.095595+0000","last_peered":"2024-09-21T19:04:37.095595+0000","last_clean":"2024-09-21T19:04:37.095595+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:04:37.095595+0000","last_undegraded":"2024-09-21T19:04:37.095595+0000","last_fullsized":"2024-09-21T19:04:37.095595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:00.810719+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":503,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992013+0000","last_change":"2024-09-21T19:01:48.626265+0000","last_active":"2024-09-21T19:04:37.992013+0000","last_peered":"2024-09-21T19:04:37.992013+0000","last_clean":"2024-09-21T19:04:37.992013+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:04:37.992013+0000","last_undegraded":"2024-09-21T19:04:37.992013+0000","last_fullsized":"2024-09-21T19:04:37.992013+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.626214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.626214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":529,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:36.586250+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:04:36.586250+0000","last_peered":"2024-09-21T19:04:36.586250+0000","last_clean":"2024-09-21T19:04:36.586250+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:04:36.586250+0000","last_undegraded":"2024-09-21T19:04:36.586250+0000","last_fullsized":"2024-09-21T19:04:36.586250+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:00.191299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":534,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970313+0000","last_change":"2024-09-21T19:02:47.515077+0000","last_active":"2024-09-21T19:04:35.970313+0000","last_peered":"2024-09-21T19:04:35.970313+0000","last_clean":"2024-09-21T19:04:35.970313+0000","last_became_active":"2024-09-21T19:02:40.879934+0000","last_became_peered":"2024-09-21T19:02:40.879934+0000","last_unstale":"2024-09-21T19:04:35.970313+0000","last_undegraded":"2024-09-21T19:04:35.970313+0000","last_fullsized":"2024-09-21T19:04:35.970313+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.514985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.514985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":599,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.971953+0000","last_change":"2024-09-21T19:03:16.432252+0000","last_active":"2024-09-21T19:04:35.971953+0000","last_peered":"2024-09-21T19:04:35.971953+0000","last_clean":"2024-09-21T19:04:35.971953+0000","last_became_active":"2024-09-21T18:55:51.529673+0000","last_became_peered":"2024-09-21T18:55:51.529673+0000","last_unstale":"2024-09-21T19:04:35.971953+0000","last_undegraded":"2024-09-21T19:04:35.971953+0000","last_fullsized":"2024-09-21T19:04:35.971953+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:16.432117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.432117+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":534,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.991626+0000","last_change":"2024-09-21T19:03:13.601474+0000","last_active":"2024-09-21T19:04:37.991626+0000","last_peered":"2024-09-21T19:04:37.991626+0000","last_clean":"2024-09-21T19:04:37.991626+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:04:37.991626+0000","last_undegraded":"2024-09-21T19:04:37.991626+0000","last_fullsized":"2024-09-21T19:04:37.991626+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:13.601407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.601407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":510,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992928+0000","last_change":"2024-09-21T19:03:10.612487+0000","last_active":"2024-09-21T19:04:37.992928+0000","last_peered":"2024-09-21T19:04:37.992928+0000","last_clean":"2024-09-21T19:04:37.992928+0000","last_became_active":"2024-09-21T19:03:09.619056+0000","last_became_peered":"2024-09-21T19:03:09.619056+0000","last_unstale":"2024-09-21T19:04:37.992928+0000","last_undegraded":"2024-09-21T19:04:37.992928+0000","last_fullsized":"2024-09-21T19:04:37.992928+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:10.612408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:10.612408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":586,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992362+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:04:37.992362+0000","last_peered":"2024-09-21T19:04:37.992362+0000","last_clean":"2024-09-21T19:04:37.992362+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:04:37.992362+0000","last_undegraded":"2024-09-21T19:04:37.992362+0000","last_fullsized":"2024-09-21T19:04:37.992362+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:35.240116+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":545,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.002170+0000","last_change":"2024-09-21T19:03:56.486775+0000","last_active":"2024-09-21T19:04:38.002170+0000","last_peered":"2024-09-21T19:04:38.002170+0000","last_clean":"2024-09-21T19:04:38.002170+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:04:38.002170+0000","last_undegraded":"2024-09-21T19:04:38.002170+0000","last_fullsized":"2024-09-21T19:04:38.002170+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:56.486496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.486496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.359373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":401,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981114+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:04:35.981114+0000","last_peered":"2024-09-21T19:04:35.981114+0000","last_clean":"2024-09-21T19:04:35.981114+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:04:35.981114+0000","last_undegraded":"2024-09-21T19:04:35.981114+0000","last_fullsized":"2024-09-21T19:04:35.981114+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:44.758167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":71,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.993096+0000","last_change":"2024-09-21T19:03:59.885217+0000","last_active":"2024-09-21T19:04:37.993096+0000","last_peered":"2024-09-21T19:04:37.993096+0000","last_clean":"2024-09-21T19:04:37.993096+0000","last_became_active":"2024-09-21T19:03:09.628920+0000","last_became_peered":"2024-09-21T19:03:09.628920+0000","last_unstale":"2024-09-21T19:04:37.993096+0000","last_undegraded":"2024-09-21T19:04:37.993096+0000","last_fullsized":"2024-09-21T19:04:37.993096+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.276502+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":524,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992202+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:04:37.992202+0000","last_peered":"2024-09-21T19:04:37.992202+0000","last_clean":"2024-09-21T19:04:37.992202+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:04:37.992202+0000","last_undegraded":"2024-09-21T19:04:37.992202+0000","last_fullsized":"2024-09-21T19:04:37.992202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.873852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":514,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999735+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:04:37.999735+0000","last_peered":"2024-09-21T19:04:37.999735+0000","last_clean":"2024-09-21T19:04:37.999735+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:04:37.999735+0000","last_undegraded":"2024-09-21T19:04:37.999735+0000","last_fullsized":"2024-09-21T19:04:37.999735+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:57.782768+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":69,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.977764+0000","last_change":"2024-09-21T19:03:16.368532+0000","last_active":"2024-09-21T19:04:35.977764+0000","last_peered":"2024-09-21T19:04:35.977764+0000","last_clean":"2024-09-21T19:04:35.977764+0000","last_became_active":"2024-09-21T19:03:09.627894+0000","last_became_peered":"2024-09-21T19:03:09.627894+0000","last_unstale":"2024-09-21T19:04:35.977764+0000","last_undegraded":"2024-09-21T19:04:35.977764+0000","last_fullsized":"2024-09-21T19:04:35.977764+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:16.368451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.368451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":531,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992162+0000","last_change":"2024-09-21T19:03:12.639309+0000","last_active":"2024-09-21T19:04:37.992162+0000","last_peered":"2024-09-21T19:04:37.992162+0000","last_clean":"2024-09-21T19:04:37.992162+0000","last_became_active":"2024-09-21T19:02:40.879452+0000","last_became_peered":"2024-09-21T19:02:40.879452+0000","last_unstale":"2024-09-21T19:04:37.992162+0000","last_undegraded":"2024-09-21T19:04:37.992162+0000","last_fullsized":"2024-09-21T19:04:37.992162+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:12.638987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:12.638987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":401,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981552+0000","last_change":"2024-09-21T19:01:16.593955+0000","last_active":"2024-09-21T19:04:35.981552+0000","last_peered":"2024-09-21T19:04:35.981552+0000","last_clean":"2024-09-21T19:04:35.981552+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:04:35.981552+0000","last_undegraded":"2024-09-21T19:04:35.981552+0000","last_fullsized":"2024-09-21T19:04:35.981552+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:16.593895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.593895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.577306+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":562,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.013297+0000","last_change":"2024-09-21T19:00:22.962922+0000","last_active":"2024-09-21T19:04:38.013297+0000","last_peered":"2024-09-21T19:04:38.013297+0000","last_clean":"2024-09-21T19:04:38.013297+0000","last_became_active":"2024-09-21T18:53:09.406869+0000","last_became_peered":"2024-09-21T18:53:09.406869+0000","last_unstale":"2024-09-21T19:04:38.013297+0000","last_undegraded":"2024-09-21T19:04:38.013297+0000","last_fullsized":"2024-09-21T19:04:38.013297+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-21T19:00:22.962840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:22.962840+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.400512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":535,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985337+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:04:35.985337+0000","last_peered":"2024-09-21T19:04:35.985337+0000","last_clean":"2024-09-21T19:04:35.985337+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:04:35.985337+0000","last_undegraded":"2024-09-21T19:04:35.985337+0000","last_fullsized":"2024-09-21T19:04:35.985337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.710160+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970956+0000","last_change":"2024-09-21T19:03:58.416244+0000","last_active":"2024-09-21T19:04:35.970956+0000","last_peered":"2024-09-21T19:04:35.970956+0000","last_clean":"2024-09-21T19:04:35.970956+0000","last_became_active":"2024-09-21T19:02:40.879991+0000","last_became_peered":"2024-09-21T19:02:40.879991+0000","last_unstale":"2024-09-21T19:04:35.970956+0000","last_undegraded":"2024-09-21T19:04:35.970956+0000","last_fullsized":"2024-09-21T19:04:35.970956+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:01.286671+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":539,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992494+0000","last_change":"2024-09-21T19:04:35.694635+0000","last_active":"2024-09-21T19:04:37.992494+0000","last_peered":"2024-09-21T19:04:37.992494+0000","last_clean":"2024-09-21T19:04:37.992494+0000","last_became_active":"2024-09-21T18:53:07.396716+0000","last_became_peered":"2024-09-21T18:53:07.396716+0000","last_unstale":"2024-09-21T19:04:37.992494+0000","last_undegraded":"2024-09-21T19:04:37.992494+0000","last_fullsized":"2024-09-21T19:04:37.992494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.857494+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":588,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.969994+0000","last_change":"2024-09-21T19:03:13.490415+0000","last_active":"2024-09-21T19:04:35.969994+0000","last_peered":"2024-09-21T19:04:35.969994+0000","last_clean":"2024-09-21T19:04:35.969994+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:04:35.969994+0000","last_undegraded":"2024-09-21T19:04:35.969994+0000","last_fullsized":"2024-09-21T19:04:35.969994+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:03:13.490332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.490332+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":515,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992840+0000","last_change":"2024-09-21T19:03:58.898230+0000","last_active":"2024-09-21T19:04:37.992840+0000","last_peered":"2024-09-21T19:04:37.992840+0000","last_clean":"2024-09-21T19:04:37.992840+0000","last_became_active":"2024-09-21T18:53:11.416346+0000","last_became_peered":"2024-09-21T18:53:11.416346+0000","last_unstale":"2024-09-21T19:04:37.992840+0000","last_undegraded":"2024-09-21T19:04:37.992840+0000","last_fullsized":"2024-09-21T19:04:37.992840+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.898154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.898154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.123418+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":527,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999932+0000","last_change":"2024-09-21T19:03:41.162782+0000","last_active":"2024-09-21T19:04:37.999932+0000","last_peered":"2024-09-21T19:04:37.999932+0000","last_clean":"2024-09-21T19:04:37.999932+0000","last_became_active":"2024-09-21T18:53:13.426427+0000","last_became_peered":"2024-09-21T18:53:13.426427+0000","last_unstale":"2024-09-21T19:04:37.999932+0000","last_undegraded":"2024-09-21T19:04:37.999932+0000","last_fullsized":"2024-09-21T19:04:37.999932+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:44.255552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":533,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.978045+0000","last_change":"2024-09-21T19:03:40.622949+0000","last_active":"2024-09-21T19:04:35.978045+0000","last_peered":"2024-09-21T19:04:35.978045+0000","last_clean":"2024-09-21T19:04:35.978045+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:04:35.978045+0000","last_undegraded":"2024-09-21T19:04:35.978045+0000","last_fullsized":"2024-09-21T19:04:35.978045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:03:40.622880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.622880+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:49.026818+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":437,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980701+0000","last_change":"2024-09-21T19:01:37.492543+0000","last_active":"2024-09-21T19:04:35.980701+0000","last_peered":"2024-09-21T19:04:35.980701+0000","last_clean":"2024-09-21T19:04:35.980701+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:04:35.980701+0000","last_undegraded":"2024-09-21T19:04:35.980701+0000","last_fullsized":"2024-09-21T19:04:35.980701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:01:37.492484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:37.492484+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.666040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"263'10","reported_seq":541,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005869+0000","last_change":"2024-09-21T19:02:40.879205+0000","last_active":"2024-09-21T19:04:38.005869+0000","last_peered":"2024-09-21T19:04:38.005869+0000","last_clean":"2024-09-21T19:04:38.005869+0000","last_became_active":"2024-09-21T19:02:40.878691+0000","last_became_peered":"2024-09-21T19:02:40.878691+0000","last_unstale":"2024-09-21T19:04:38.005869+0000","last_undegraded":"2024-09-21T19:04:38.005869+0000","last_fullsized":"2024-09-21T19:04:38.005869+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.734152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"299'10132","reported_seq":10308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.743813+0000","last_change":"2024-09-21T19:03:50.138454+0000","last_active":"2024-09-21T19:04:37.743813+0000","last_peered":"2024-09-21T19:04:37.743813+0000","last_clean":"2024-09-21T19:04:37.743813+0000","last_became_active":"2024-09-21T19:02:29.785679+0000","last_became_peered":"2024-09-21T19:02:29.785679+0000","last_unstale":"2024-09-21T19:04:37.743813+0000","last_undegraded":"2024-09-21T19:04:37.743813+0000","last_fullsized":"2024-09-21T19:04:37.743813+0000","mapping_epoch":254,"log_start":"285'5700","ondisk_log_start":"285'5700","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'6599","last_scrub_stamp":"2024-09-21T19:03:50.138203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.138203+0000","objects_scrubbed":6517,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:55.055653+0000","scrub_duration":2697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367919104,"num_objects":5614,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10132,"num_write_kb":503872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":80,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.014266+0000","last_change":"2024-09-21T19:04:32.512884+0000","last_active":"2024-09-21T19:04:38.014266+0000","last_peered":"2024-09-21T19:04:38.014266+0000","last_clean":"2024-09-21T19:04:38.014266+0000","last_became_active":"2024-09-21T19:03:09.622733+0000","last_became_peered":"2024-09-21T19:03:09.622733+0000","last_unstale":"2024-09-21T19:04:38.014266+0000","last_undegraded":"2024-09-21T19:04:38.014266+0000","last_fullsized":"2024-09-21T19:04:38.014266+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:32.512752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:32.512752+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.179266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":569,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.989648+0000","last_change":"2024-09-21T19:03:57.634402+0000","last_active":"2024-09-21T19:04:35.989648+0000","last_peered":"2024-09-21T19:04:35.989648+0000","last_clean":"2024-09-21T19:04:35.989648+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:04:35.989648+0000","last_undegraded":"2024-09-21T19:04:35.989648+0000","last_fullsized":"2024-09-21T19:04:35.989648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:57.634230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.634230+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:58.234423+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":523,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.971079+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:04:35.971079+0000","last_peered":"2024-09-21T19:04:35.971079+0000","last_clean":"2024-09-21T19:04:35.971079+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:04:35.971079+0000","last_undegraded":"2024-09-21T19:04:35.971079+0000","last_fullsized":"2024-09-21T19:04:35.971079+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.469739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":701,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992246+0000","last_change":"2024-09-21T19:03:53.851419+0000","last_active":"2024-09-21T19:04:37.992246+0000","last_peered":"2024-09-21T19:04:37.992246+0000","last_clean":"2024-09-21T19:04:37.992246+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:04:37.992246+0000","last_undegraded":"2024-09-21T19:04:37.992246+0000","last_fullsized":"2024-09-21T19:04:37.992246+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:55.131138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":193,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985566+0000","last_change":"2024-09-21T19:03:14.420119+0000","last_active":"2024-09-21T19:04:35.985566+0000","last_peered":"2024-09-21T19:04:35.985566+0000","last_clean":"2024-09-21T19:04:35.985566+0000","last_became_active":"2024-09-21T19:03:09.623771+0000","last_became_peered":"2024-09-21T19:03:09.623771+0000","last_unstale":"2024-09-21T19:04:35.985566+0000","last_undegraded":"2024-09-21T19:04:35.985566+0000","last_fullsized":"2024-09-21T19:04:35.985566+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:14.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"300'10591","reported_seq":10769,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:41.693116+0000","last_change":"2024-09-21T19:03:45.264553+0000","last_active":"2024-09-21T19:04:41.693116+0000","last_peered":"2024-09-21T19:04:41.693116+0000","last_clean":"2024-09-21T19:04:41.693116+0000","last_became_active":"2024-09-21T19:02:29.786797+0000","last_became_peered":"2024-09-21T19:02:29.786797+0000","last_unstale":"2024-09-21T19:04:41.693116+0000","last_undegraded":"2024-09-21T19:04:41.693116+0000","last_fullsized":"2024-09-21T19:04:41.693116+0000","mapping_epoch":254,"log_start":"285'6000","ondisk_log_start":"285'6000","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6439","last_scrub_stamp":"2024-09-21T19:03:45.264423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.264423+0000","objects_scrubbed":6363,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.021933+0000","scrub_duration":2070,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356778008,"num_objects":5445,"num_object_clones":0,"num_object_copies":16335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10591,"num_write_kb":513089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":299,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.993047+0000","last_change":"2024-09-21T19:02:45.696512+0000","last_active":"2024-09-21T19:04:37.993047+0000","last_peered":"2024-09-21T19:04:37.993047+0000","last_clean":"2024-09-21T19:04:37.993047+0000","last_became_active":"2024-09-21T19:02:40.874426+0000","last_became_peered":"2024-09-21T19:02:40.874426+0000","last_unstale":"2024-09-21T19:04:37.993047+0000","last_undegraded":"2024-09-21T19:04:37.993047+0000","last_fullsized":"2024-09-21T19:04:37.993047+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":611,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970796+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:04:35.970796+0000","last_peered":"2024-09-21T19:04:35.970796+0000","last_clean":"2024-09-21T19:04:35.970796+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:04:35.970796+0000","last_undegraded":"2024-09-21T19:04:35.970796+0000","last_fullsized":"2024-09-21T19:04:35.970796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.535087+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":537,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.979580+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:04:35.979580+0000","last_peered":"2024-09-21T19:04:35.979580+0000","last_clean":"2024-09-21T19:04:35.979580+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:04:35.979580+0000","last_undegraded":"2024-09-21T19:04:35.979580+0000","last_fullsized":"2024-09-21T19:04:35.979580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":524,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.991755+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:04:37.991755+0000","last_peered":"2024-09-21T19:04:37.991755+0000","last_clean":"2024-09-21T19:04:37.991755+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:04:37.991755+0000","last_undegraded":"2024-09-21T19:04:37.991755+0000","last_fullsized":"2024-09-21T19:04:37.991755+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:52.440101+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":504,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992270+0000","last_change":"2024-09-21T19:02:11.853722+0000","last_active":"2024-09-21T19:04:37.992270+0000","last_peered":"2024-09-21T19:04:37.992270+0000","last_clean":"2024-09-21T19:04:37.992270+0000","last_became_active":"2024-09-21T18:53:13.421841+0000","last_became_peered":"2024-09-21T18:53:13.421841+0000","last_unstale":"2024-09-21T19:04:37.992270+0000","last_undegraded":"2024-09-21T19:04:37.992270+0000","last_fullsized":"2024-09-21T19:04:37.992270+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:11.853678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:11.853678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"299'10286","reported_seq":10387,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.698006+0000","last_change":"2024-09-21T19:02:30.709345+0000","last_active":"2024-09-21T19:04:37.698006+0000","last_peered":"2024-09-21T19:04:37.698006+0000","last_clean":"2024-09-21T19:04:37.698006+0000","last_became_active":"2024-09-21T19:02:30.709045+0000","last_became_peered":"2024-09-21T19:02:30.709045+0000","last_unstale":"2024-09-21T19:04:37.698006+0000","last_undegraded":"2024-09-21T19:04:37.698006+0000","last_fullsized":"2024-09-21T19:04:37.698006+0000","mapping_epoch":254,"log_start":"285'5900","ondisk_log_start":"285'5900","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378142720,"num_objects":5770,"num_object_clones":0,"num_object_copies":17310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10286,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":283,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.974136+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:04:35.974136+0000","last_peered":"2024-09-21T19:04:35.974136+0000","last_clean":"2024-09-21T19:04:35.974136+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:04:35.974136+0000","last_undegraded":"2024-09-21T19:04:35.974136+0000","last_fullsized":"2024-09-21T19:04:35.974136+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:01.404232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":573,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.991835+0000","last_change":"2024-09-21T19:02:42.663946+0000","last_active":"2024-09-21T19:04:37.991835+0000","last_peered":"2024-09-21T19:04:37.991835+0000","last_clean":"2024-09-21T19:04:37.991835+0000","last_became_active":"2024-09-21T19:02:40.885348+0000","last_became_peered":"2024-09-21T19:02:40.885348+0000","last_unstale":"2024-09-21T19:04:37.991835+0000","last_undegraded":"2024-09-21T19:04:37.991835+0000","last_fullsized":"2024-09-21T19:04:37.991835+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-21T19:02:42.663859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:42.663859+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":414,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980979+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:04:35.980979+0000","last_peered":"2024-09-21T19:04:35.980979+0000","last_clean":"2024-09-21T19:04:35.980979+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:04:35.980979+0000","last_undegraded":"2024-09-21T19:04:35.980979+0000","last_fullsized":"2024-09-21T19:04:35.980979+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.439642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":654,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970668+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:04:35.970668+0000","last_peered":"2024-09-21T19:04:35.970668+0000","last_clean":"2024-09-21T19:04:35.970668+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:04:35.970668+0000","last_undegraded":"2024-09-21T19:04:35.970668+0000","last_fullsized":"2024-09-21T19:04:35.970668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:54.771825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":522,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999610+0000","last_change":"2024-09-21T19:04:35.316010+0000","last_active":"2024-09-21T19:04:37.999610+0000","last_peered":"2024-09-21T19:04:37.999610+0000","last_clean":"2024-09-21T19:04:37.999610+0000","last_became_active":"2024-09-21T19:02:40.886800+0000","last_became_peered":"2024-09-21T19:02:40.886800+0000","last_unstale":"2024-09-21T19:04:37.999610+0000","last_undegraded":"2024-09-21T19:04:37.999610+0000","last_fullsized":"2024-09-21T19:04:37.999610+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.315947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.315947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:47.374309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"300'10575","reported_seq":10751,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:41.697555+0000","last_change":"2024-09-21T19:03:46.619927+0000","last_active":"2024-09-21T19:04:41.697555+0000","last_peered":"2024-09-21T19:04:41.697555+0000","last_clean":"2024-09-21T19:04:41.697555+0000","last_became_active":"2024-09-21T19:02:29.787909+0000","last_became_peered":"2024-09-21T19:02:29.787909+0000","last_unstale":"2024-09-21T19:04:41.697555+0000","last_undegraded":"2024-09-21T19:04:41.697555+0000","last_fullsized":"2024-09-21T19:04:41.697555+0000","mapping_epoch":254,"log_start":"284'5600","ondisk_log_start":"284'5600","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6487","last_scrub_stamp":"2024-09-21T19:03:46.619847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:46.619847+0000","objects_scrubbed":6415,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.392510+0000","scrub_duration":2131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349110272,"num_objects":5327,"num_object_clones":0,"num_object_copies":15981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":508864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":328,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005962+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:04:38.005962+0000","last_peered":"2024-09-21T19:04:38.005962+0000","last_clean":"2024-09-21T19:04:38.005962+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:04:38.005962+0000","last_undegraded":"2024-09-21T19:04:38.005962+0000","last_fullsized":"2024-09-21T19:04:38.005962+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:23.574104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":548,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985190+0000","last_change":"2024-09-21T19:03:50.509414+0000","last_active":"2024-09-21T19:04:35.985190+0000","last_peered":"2024-09-21T19:04:35.985190+0000","last_clean":"2024-09-21T19:04:35.985190+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:04:35.985190+0000","last_undegraded":"2024-09-21T19:04:35.985190+0000","last_fullsized":"2024-09-21T19:04:35.985190+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:03:50.509275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.509275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:57.379074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":61,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970109+0000","last_change":"2024-09-21T19:03:09.624294+0000","last_active":"2024-09-21T19:04:35.970109+0000","last_peered":"2024-09-21T19:04:35.970109+0000","last_clean":"2024-09-21T19:04:35.970109+0000","last_became_active":"2024-09-21T19:03:09.623941+0000","last_became_peered":"2024-09-21T19:03:09.623941+0000","last_unstale":"2024-09-21T19:04:35.970109+0000","last_undegraded":"2024-09-21T19:04:35.970109+0000","last_fullsized":"2024-09-21T19:04:35.970109+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"263'12","reported_seq":516,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981500+0000","last_change":"2024-09-21T19:02:40.897242+0000","last_active":"2024-09-21T19:04:35.981500+0000","last_peered":"2024-09-21T19:04:35.981500+0000","last_clean":"2024-09-21T19:04:35.981500+0000","last_became_active":"2024-09-21T19:02:40.896856+0000","last_became_peered":"2024-09-21T19:02:40.896856+0000","last_unstale":"2024-09-21T19:04:35.981500+0000","last_undegraded":"2024-09-21T19:04:35.981500+0000","last_fullsized":"2024-09-21T19:04:35.981500+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:01:36.539314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:36.539314+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.638141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":402,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981533+0000","last_change":"2024-09-21T19:02:18.444667+0000","last_active":"2024-09-21T19:04:35.981533+0000","last_peered":"2024-09-21T19:04:35.981533+0000","last_clean":"2024-09-21T19:04:35.981533+0000","last_became_active":"2024-09-21T18:55:51.528779+0000","last_became_peered":"2024-09-21T18:55:51.528779+0000","last_unstale":"2024-09-21T19:04:35.981533+0000","last_undegraded":"2024-09-21T19:04:35.981533+0000","last_fullsized":"2024-09-21T19:04:35.981533+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.662660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"300'10896","reported_seq":10997,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:41.679343+0000","last_change":"2024-09-21T19:02:30.708018+0000","last_active":"2024-09-21T19:04:41.679343+0000","last_peered":"2024-09-21T19:04:41.679343+0000","last_clean":"2024-09-21T19:04:41.679343+0000","last_became_active":"2024-09-21T19:02:30.707708+0000","last_became_peered":"2024-09-21T19:02:30.707708+0000","last_unstale":"2024-09-21T19:04:41.679343+0000","last_undegraded":"2024-09-21T19:04:41.679343+0000","last_fullsized":"2024-09-21T19:04:41.679343+0000","mapping_epoch":254,"log_start":"286'6300","ondisk_log_start":"286'6300","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358744064,"num_objects":5474,"num_object_clones":0,"num_object_copies":16422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":523840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":306,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005950+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:04:38.005950+0000","last_peered":"2024-09-21T19:04:38.005950+0000","last_clean":"2024-09-21T19:04:38.005950+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:04:38.005950+0000","last_undegraded":"2024-09-21T19:04:38.005950+0000","last_fullsized":"2024-09-21T19:04:38.005950+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.034739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":443,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981588+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:04:35.981588+0000","last_peered":"2024-09-21T19:04:35.981588+0000","last_clean":"2024-09-21T19:04:35.981588+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:04:35.981588+0000","last_undegraded":"2024-09-21T19:04:35.981588+0000","last_fullsized":"2024-09-21T19:04:35.981588+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.511233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":522,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.988251+0000","last_change":"2024-09-21T19:00:56.467643+0000","last_active":"2024-09-21T19:04:35.988251+0000","last_peered":"2024-09-21T19:04:35.988251+0000","last_clean":"2024-09-21T19:04:35.988251+0000","last_became_active":"2024-09-21T18:53:07.399808+0000","last_became_peered":"2024-09-21T18:53:07.399808+0000","last_unstale":"2024-09-21T19:04:35.988251+0000","last_undegraded":"2024-09-21T19:04:35.988251+0000","last_fullsized":"2024-09-21T19:04:35.988251+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:03.696298+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":422,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005336+0000","last_change":"2024-09-21T19:02:40.882592+0000","last_active":"2024-09-21T19:04:38.005336+0000","last_peered":"2024-09-21T19:04:38.005336+0000","last_clean":"2024-09-21T19:04:38.005336+0000","last_became_active":"2024-09-21T19:02:40.882174+0000","last_became_peered":"2024-09-21T19:02:40.882174+0000","last_unstale":"2024-09-21T19:04:38.005336+0000","last_undegraded":"2024-09-21T19:04:38.005336+0000","last_fullsized":"2024-09-21T19:04:38.005336+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:18.736361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":521,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.977874+0000","last_change":"2024-09-21T19:03:51.466892+0000","last_active":"2024-09-21T19:04:35.977874+0000","last_peered":"2024-09-21T19:04:35.977874+0000","last_clean":"2024-09-21T19:04:35.977874+0000","last_became_active":"2024-09-21T19:02:40.882759+0000","last_became_peered":"2024-09-21T19:02:40.882759+0000","last_unstale":"2024-09-21T19:04:35.977874+0000","last_undegraded":"2024-09-21T19:04:35.977874+0000","last_fullsized":"2024-09-21T19:04:35.977874+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.466833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.466833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.333404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"299'10293","reported_seq":10470,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.155125+0000","last_change":"2024-09-21T19:03:51.831929+0000","last_active":"2024-09-21T19:04:37.155125+0000","last_peered":"2024-09-21T19:04:37.155125+0000","last_clean":"2024-09-21T19:04:37.155125+0000","last_became_active":"2024-09-21T19:02:29.787771+0000","last_became_peered":"2024-09-21T19:02:29.787771+0000","last_unstale":"2024-09-21T19:04:37.155125+0000","last_undegraded":"2024-09-21T19:04:37.155125+0000","last_fullsized":"2024-09-21T19:04:37.155125+0000","mapping_epoch":254,"log_start":"279'4600","ondisk_log_start":"279'4600","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6909","last_scrub_stamp":"2024-09-21T19:03:51.831669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.831669+0000","objects_scrubbed":6830,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.352296+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375455744,"num_objects":5729,"num_object_clones":0,"num_object_copies":17187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":512704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":255,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980425+0000","last_change":"2024-09-21T19:04:34.855964+0000","last_active":"2024-09-21T19:04:35.980425+0000","last_peered":"2024-09-21T19:04:35.980425+0000","last_clean":"2024-09-21T19:04:35.980425+0000","last_became_active":"2024-09-21T19:03:09.627416+0000","last_became_peered":"2024-09-21T19:03:09.627416+0000","last_unstale":"2024-09-21T19:04:35.980425+0000","last_undegraded":"2024-09-21T19:04:35.980425+0000","last_fullsized":"2024-09-21T19:04:35.980425+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:48.700091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"276'40","reported_seq":722,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.988125+0000","last_change":"2024-09-21T19:04:26.993917+0000","last_active":"2024-09-21T19:04:35.988125+0000","last_peered":"2024-09-21T19:04:35.988125+0000","last_clean":"2024-09-21T19:04:35.988125+0000","last_became_active":"2024-09-21T18:53:09.409882+0000","last_became_peered":"2024-09-21T18:53:09.409882+0000","last_unstale":"2024-09-21T19:04:35.988125+0000","last_undegraded":"2024-09-21T19:04:35.988125+0000","last_fullsized":"2024-09-21T19:04:35.988125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.205140+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":557,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992552+0000","last_change":"2024-09-21T19:02:43.617505+0000","last_active":"2024-09-21T19:04:37.992552+0000","last_peered":"2024-09-21T19:04:37.992552+0000","last_clean":"2024-09-21T19:04:37.992552+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:04:37.992552+0000","last_undegraded":"2024-09-21T19:04:37.992552+0000","last_fullsized":"2024-09-21T19:04:37.992552+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:43.617435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:43.617435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.861728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":91,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.986110+0000","last_change":"2024-09-21T19:03:15.425428+0000","last_active":"2024-09-21T19:04:35.986110+0000","last_peered":"2024-09-21T19:04:35.986110+0000","last_clean":"2024-09-21T19:04:35.986110+0000","last_became_active":"2024-09-21T19:03:09.623214+0000","last_became_peered":"2024-09-21T19:03:09.623214+0000","last_unstale":"2024-09-21T19:04:35.986110+0000","last_undegraded":"2024-09-21T19:04:35.986110+0000","last_fullsized":"2024-09-21T19:04:35.986110+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:03:15.425359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.425359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":402,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981029+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:04:35.981029+0000","last_peered":"2024-09-21T19:04:35.981029+0000","last_clean":"2024-09-21T19:04:35.981029+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:04:35.981029+0000","last_undegraded":"2024-09-21T19:04:35.981029+0000","last_fullsized":"2024-09-21T19:04:35.981029+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.712781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":503,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970085+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:04:35.970085+0000","last_peered":"2024-09-21T19:04:35.970085+0000","last_clean":"2024-09-21T19:04:35.970085+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:04:35.970085+0000","last_undegraded":"2024-09-21T19:04:35.970085+0000","last_fullsized":"2024-09-21T19:04:35.970085+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.494920+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"299'10298","reported_seq":10479,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.358519+0000","last_change":"2024-09-21T19:03:48.136769+0000","last_active":"2024-09-21T19:04:37.358519+0000","last_peered":"2024-09-21T19:04:37.358519+0000","last_clean":"2024-09-21T19:04:37.358519+0000","last_became_active":"2024-09-21T19:03:09.629373+0000","last_became_peered":"2024-09-21T19:03:09.629373+0000","last_unstale":"2024-09-21T19:04:37.358519+0000","last_undegraded":"2024-09-21T19:04:37.358519+0000","last_fullsized":"2024-09-21T19:04:37.358519+0000","mapping_epoch":274,"log_start":"284'5500","ondisk_log_start":"284'5500","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6614","last_scrub_stamp":"2024-09-21T19:03:48.136538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.136538+0000","objects_scrubbed":6508,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.921314+0000","scrub_duration":2592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380764160,"num_objects":5810,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":515456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":609,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999430+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:04:37.999430+0000","last_peered":"2024-09-21T19:04:37.999430+0000","last_clean":"2024-09-21T19:04:37.999430+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:04:37.999430+0000","last_undegraded":"2024-09-21T19:04:37.999430+0000","last_fullsized":"2024-09-21T19:04:37.999430+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":513,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.009280+0000","last_change":"2024-09-21T19:03:40.251914+0000","last_active":"2024-09-21T19:04:38.009280+0000","last_peered":"2024-09-21T19:04:38.009280+0000","last_clean":"2024-09-21T19:04:38.009280+0000","last_became_active":"2024-09-21T18:55:51.523826+0000","last_became_peered":"2024-09-21T18:55:51.523826+0000","last_unstale":"2024-09-21T19:04:38.009280+0000","last_undegraded":"2024-09-21T19:04:38.009280+0000","last_fullsized":"2024-09-21T19:04:38.009280+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:03:40.251822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.251822+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:43.197688+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":539,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.979246+0000","last_change":"2024-09-21T19:02:40.874431+0000","last_active":"2024-09-21T19:04:35.979246+0000","last_peered":"2024-09-21T19:04:35.979246+0000","last_clean":"2024-09-21T19:04:35.979246+0000","last_became_active":"2024-09-21T19:02:40.874003+0000","last_became_peered":"2024-09-21T19:02:40.874003+0000","last_unstale":"2024-09-21T19:04:35.979246+0000","last_undegraded":"2024-09-21T19:04:35.979246+0000","last_fullsized":"2024-09-21T19:04:35.979246+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.743742+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":563,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.008996+0000","last_change":"2024-09-21T19:00:21.025220+0000","last_active":"2024-09-21T19:04:38.008996+0000","last_peered":"2024-09-21T19:04:38.008996+0000","last_clean":"2024-09-21T19:04:38.008996+0000","last_became_active":"2024-09-21T18:55:51.524046+0000","last_became_peered":"2024-09-21T18:55:51.524046+0000","last_unstale":"2024-09-21T19:04:38.008996+0000","last_undegraded":"2024-09-21T19:04:38.008996+0000","last_fullsized":"2024-09-21T19:04:38.008996+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.413921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":532,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.010326+0000","last_change":"2024-09-21T19:04:23.530834+0000","last_active":"2024-09-21T19:04:38.010326+0000","last_peered":"2024-09-21T19:04:38.010326+0000","last_clean":"2024-09-21T19:04:38.010326+0000","last_became_active":"2024-09-21T18:53:11.413111+0000","last_became_peered":"2024-09-21T18:53:11.413111+0000","last_unstale":"2024-09-21T19:04:38.010326+0000","last_undegraded":"2024-09-21T19:04:38.010326+0000","last_fullsized":"2024-09-21T19:04:38.010326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:38.789330+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":526,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000203+0000","last_change":"2024-09-21T19:03:17.183599+0000","last_active":"2024-09-21T19:04:38.000203+0000","last_peered":"2024-09-21T19:04:38.000203+0000","last_clean":"2024-09-21T19:04:38.000203+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:04:38.000203+0000","last_undegraded":"2024-09-21T19:04:38.000203+0000","last_fullsized":"2024-09-21T19:04:38.000203+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:17.183521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.183521+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"299'10317","reported_seq":11453,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.149442+0000","last_change":"2024-09-21T19:02:46.916132+0000","last_active":"2024-09-21T19:04:37.149442+0000","last_peered":"2024-09-21T19:04:37.149442+0000","last_clean":"2024-09-21T19:04:37.149442+0000","last_became_active":"2024-09-21T19:02:46.913818+0000","last_became_peered":"2024-09-21T19:02:46.913818+0000","last_unstale":"2024-09-21T19:04:37.149442+0000","last_undegraded":"2024-09-21T19:04:37.149442+0000","last_fullsized":"2024-09-21T19:04:37.149442+0000","mapping_epoch":264,"log_start":"279'4600","ondisk_log_start":"279'4600","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380567552,"num_objects":5807,"num_object_clones":0,"num_object_copies":17421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":305,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985632+0000","last_change":"2024-09-21T19:03:56.613953+0000","last_active":"2024-09-21T19:04:35.985632+0000","last_peered":"2024-09-21T19:04:35.985632+0000","last_clean":"2024-09-21T19:04:35.985632+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:04:35.985632+0000","last_undegraded":"2024-09-21T19:04:35.985632+0000","last_fullsized":"2024-09-21T19:04:35.985632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:56.613898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.613898+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:00.158839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":442,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980793+0000","last_change":"2024-09-21T19:01:13.569543+0000","last_active":"2024-09-21T19:04:35.980793+0000","last_peered":"2024-09-21T19:04:35.980793+0000","last_clean":"2024-09-21T19:04:35.980793+0000","last_became_active":"2024-09-21T18:55:51.533955+0000","last_became_peered":"2024-09-21T18:55:51.533955+0000","last_unstale":"2024-09-21T19:04:35.980793+0000","last_undegraded":"2024-09-21T19:04:35.980793+0000","last_fullsized":"2024-09-21T19:04:35.980793+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:01:13.569464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.569464+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.601121+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":410,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:36.892132+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:04:36.892132+0000","last_peered":"2024-09-21T19:04:36.892132+0000","last_clean":"2024-09-21T19:04:36.892132+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:04:36.892132+0000","last_undegraded":"2024-09-21T19:04:36.892132+0000","last_fullsized":"2024-09-21T19:04:36.892132+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:56.727202+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1244,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:40.341848+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:04:40.341848+0000","last_peered":"2024-09-21T19:04:40.341848+0000","last_clean":"2024-09-21T19:04:40.341848+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:04:40.341848+0000","last_undegraded":"2024-09-21T19:04:40.341848+0000","last_fullsized":"2024-09-21T19:04:40.341848+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.975843+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":683,"num_read_kb":678,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":546,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.003219+0000","last_change":"2024-09-21T19:00:56.950003+0000","last_active":"2024-09-21T19:04:38.003219+0000","last_peered":"2024-09-21T19:04:38.003219+0000","last_clean":"2024-09-21T19:04:38.003219+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:04:38.003219+0000","last_undegraded":"2024-09-21T19:04:38.003219+0000","last_fullsized":"2024-09-21T19:04:38.003219+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.949931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.949931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.450029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.987655+0000","last_change":"2024-09-21T19:03:54.568908+0000","last_active":"2024-09-21T19:04:35.987655+0000","last_peered":"2024-09-21T19:04:35.987655+0000","last_clean":"2024-09-21T19:04:35.987655+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:04:35.987655+0000","last_undegraded":"2024-09-21T19:04:35.987655+0000","last_fullsized":"2024-09-21T19:04:35.987655+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:54.568845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:54.568845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.700759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"299'10322","reported_seq":10422,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.730016+0000","last_change":"2024-09-21T19:02:29.785323+0000","last_active":"2024-09-21T19:04:37.730016+0000","last_peered":"2024-09-21T19:04:37.730016+0000","last_clean":"2024-09-21T19:04:37.730016+0000","last_became_active":"2024-09-21T19:02:29.785050+0000","last_became_peered":"2024-09-21T19:02:29.785050+0000","last_unstale":"2024-09-21T19:04:37.730016+0000","last_undegraded":"2024-09-21T19:04:37.730016+0000","last_fullsized":"2024-09-21T19:04:37.730016+0000","mapping_epoch":254,"log_start":"285'5900","ondisk_log_start":"285'5900","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375652352,"num_objects":5732,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":651,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005710+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:04:38.005710+0000","last_peered":"2024-09-21T19:04:38.005710+0000","last_clean":"2024-09-21T19:04:38.005710+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:04:38.005710+0000","last_undegraded":"2024-09-21T19:04:38.005710+0000","last_fullsized":"2024-09-21T19:04:38.005710+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:56.838535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":526,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.973013+0000","last_change":"2024-09-21T19:02:40.877780+0000","last_active":"2024-09-21T19:04:35.973013+0000","last_peered":"2024-09-21T19:04:35.973013+0000","last_clean":"2024-09-21T19:04:35.973013+0000","last_became_active":"2024-09-21T19:02:40.877452+0000","last_became_peered":"2024-09-21T19:02:40.877452+0000","last_unstale":"2024-09-21T19:04:35.973013+0000","last_undegraded":"2024-09-21T19:04:35.973013+0000","last_fullsized":"2024-09-21T19:04:35.973013+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.441173+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":615,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999588+0000","last_change":"2024-09-21T19:04:28.281992+0000","last_active":"2024-09-21T19:04:37.999588+0000","last_peered":"2024-09-21T19:04:37.999588+0000","last_clean":"2024-09-21T19:04:37.999588+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:04:37.999588+0000","last_undegraded":"2024-09-21T19:04:37.999588+0000","last_fullsized":"2024-09-21T19:04:37.999588+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:28.281931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.281931+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.661197+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":523,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000118+0000","last_change":"2024-09-21T19:03:59.160104+0000","last_active":"2024-09-21T19:04:38.000118+0000","last_peered":"2024-09-21T19:04:38.000118+0000","last_clean":"2024-09-21T19:04:38.000118+0000","last_became_active":"2024-09-21T18:53:11.414291+0000","last_became_peered":"2024-09-21T18:53:11.414291+0000","last_unstale":"2024-09-21T19:04:38.000118+0000","last_undegraded":"2024-09-21T19:04:38.000118+0000","last_fullsized":"2024-09-21T19:04:38.000118+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.850193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":61,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970868+0000","last_change":"2024-09-21T19:03:09.625168+0000","last_active":"2024-09-21T19:04:35.970868+0000","last_peered":"2024-09-21T19:04:35.970868+0000","last_clean":"2024-09-21T19:04:35.970868+0000","last_became_active":"2024-09-21T19:03:09.624424+0000","last_became_peered":"2024-09-21T19:03:09.624424+0000","last_unstale":"2024-09-21T19:04:35.970868+0000","last_undegraded":"2024-09-21T19:04:35.970868+0000","last_fullsized":"2024-09-21T19:04:35.970868+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:38.459943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"300'10484","reported_seq":10588,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:41.679154+0000","last_change":"2024-09-21T19:02:29.785879+0000","last_active":"2024-09-21T19:04:41.679154+0000","last_peered":"2024-09-21T19:04:41.679154+0000","last_clean":"2024-09-21T19:04:41.679154+0000","last_became_active":"2024-09-21T19:02:29.785492+0000","last_became_peered":"2024-09-21T19:02:29.785492+0000","last_unstale":"2024-09-21T19:04:41.679154+0000","last_undegraded":"2024-09-21T19:04:41.679154+0000","last_fullsized":"2024-09-21T19:04:41.679154+0000","mapping_epoch":254,"log_start":"285'5900","ondisk_log_start":"285'5900","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.229667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348782592,"num_objects":5322,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10484,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":578,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992338+0000","last_change":"2024-09-21T19:03:18.675564+0000","last_active":"2024-09-21T19:04:37.992338+0000","last_peered":"2024-09-21T19:04:37.992338+0000","last_clean":"2024-09-21T19:04:37.992338+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:04:37.992338+0000","last_undegraded":"2024-09-21T19:04:37.992338+0000","last_fullsized":"2024-09-21T19:04:37.992338+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:18.675329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.675329+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:42.981301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":532,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.991974+0000","last_change":"2024-09-21T19:03:20.643868+0000","last_active":"2024-09-21T19:04:37.991974+0000","last_peered":"2024-09-21T19:04:37.991974+0000","last_clean":"2024-09-21T19:04:37.991974+0000","last_became_active":"2024-09-21T18:53:07.400001+0000","last_became_peered":"2024-09-21T18:53:07.400001+0000","last_unstale":"2024-09-21T19:04:37.991974+0000","last_undegraded":"2024-09-21T19:04:37.991974+0000","last_fullsized":"2024-09-21T19:04:37.991974+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.643800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.643800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":527,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.993117+0000","last_change":"2024-09-21T19:03:11.634133+0000","last_active":"2024-09-21T19:04:37.993117+0000","last_peered":"2024-09-21T19:04:37.993117+0000","last_clean":"2024-09-21T19:04:37.993117+0000","last_became_active":"2024-09-21T19:03:09.619609+0000","last_became_peered":"2024-09-21T19:03:09.619609+0000","last_unstale":"2024-09-21T19:04:37.993117+0000","last_undegraded":"2024-09-21T19:04:37.993117+0000","last_fullsized":"2024-09-21T19:04:37.993117+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":506,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.974235+0000","last_change":"2024-09-21T19:02:46.521753+0000","last_active":"2024-09-21T19:04:35.974235+0000","last_peered":"2024-09-21T19:04:35.974235+0000","last_clean":"2024-09-21T19:04:35.974235+0000","last_became_active":"2024-09-21T19:02:40.877238+0000","last_became_peered":"2024-09-21T19:02:40.877238+0000","last_unstale":"2024-09-21T19:04:35.974235+0000","last_undegraded":"2024-09-21T19:04:35.974235+0000","last_fullsized":"2024-09-21T19:04:35.974235+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"299'10207","reported_seq":11289,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.346645+0000","last_change":"2024-09-21T19:02:46.916256+0000","last_active":"2024-09-21T19:04:37.346645+0000","last_peered":"2024-09-21T19:04:37.346645+0000","last_clean":"2024-09-21T19:04:37.346645+0000","last_became_active":"2024-09-21T19:02:46.913875+0000","last_became_peered":"2024-09-21T19:02:46.913875+0000","last_unstale":"2024-09-21T19:04:37.346645+0000","last_undegraded":"2024-09-21T19:04:37.346645+0000","last_fullsized":"2024-09-21T19:04:37.346645+0000","mapping_epoch":264,"log_start":"283'5400","ondisk_log_start":"283'5400","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.725035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377683968,"num_objects":5763,"num_object_clones":0,"num_object_copies":17289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10207,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":31850496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":72,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.001710+0000","last_change":"2024-09-21T19:03:15.475191+0000","last_active":"2024-09-21T19:04:38.001710+0000","last_peered":"2024-09-21T19:04:38.001710+0000","last_clean":"2024-09-21T19:04:38.001710+0000","last_became_active":"2024-09-21T19:03:09.626413+0000","last_became_peered":"2024-09-21T19:03:09.626413+0000","last_unstale":"2024-09-21T19:04:38.001710+0000","last_undegraded":"2024-09-21T19:04:38.001710+0000","last_fullsized":"2024-09-21T19:04:38.001710+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:15.475112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.475112+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:43.597628+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":569,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992825+0000","last_change":"2024-09-21T19:02:07.716667+0000","last_active":"2024-09-21T19:04:37.992825+0000","last_peered":"2024-09-21T19:04:37.992825+0000","last_clean":"2024-09-21T19:04:37.992825+0000","last_became_active":"2024-09-21T18:53:09.406226+0000","last_became_peered":"2024-09-21T18:53:09.406226+0000","last_unstale":"2024-09-21T19:04:37.992825+0000","last_undegraded":"2024-09-21T19:04:37.992825+0000","last_fullsized":"2024-09-21T19:04:37.992825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-21T19:02:07.716607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:07.716607+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":444,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005576+0000","last_change":"2024-09-21T19:02:40.879761+0000","last_active":"2024-09-21T19:04:38.005576+0000","last_peered":"2024-09-21T19:04:38.005576+0000","last_clean":"2024-09-21T19:04:38.005576+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:04:38.005576+0000","last_undegraded":"2024-09-21T19:04:38.005576+0000","last_fullsized":"2024-09-21T19:04:38.005576+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:14.376671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:14.376671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.783406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":528,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.987743+0000","last_change":"2024-09-21T19:03:42.617727+0000","last_active":"2024-09-21T19:04:35.987743+0000","last_peered":"2024-09-21T19:04:35.987743+0000","last_clean":"2024-09-21T19:04:35.987743+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:04:35.987743+0000","last_undegraded":"2024-09-21T19:04:35.987743+0000","last_fullsized":"2024-09-21T19:04:35.987743+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.806920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":504,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.009412+0000","last_change":"2024-09-21T18:59:45.881615+0000","last_active":"2024-09-21T19:04:38.009412+0000","last_peered":"2024-09-21T19:04:38.009412+0000","last_clean":"2024-09-21T19:04:38.009412+0000","last_became_active":"2024-09-21T18:53:13.430534+0000","last_became_peered":"2024-09-21T18:53:13.430534+0000","last_unstale":"2024-09-21T19:04:38.009412+0000","last_undegraded":"2024-09-21T19:04:38.009412+0000","last_fullsized":"2024-09-21T19:04:38.009412+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.397993+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"299'10246","reported_seq":11338,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.727163+0000","last_change":"2024-09-21T19:02:46.953650+0000","last_active":"2024-09-21T19:04:37.727163+0000","last_peered":"2024-09-21T19:04:37.727163+0000","last_clean":"2024-09-21T19:04:37.727163+0000","last_became_active":"2024-09-21T19:02:46.951397+0000","last_became_peered":"2024-09-21T19:02:46.951397+0000","last_unstale":"2024-09-21T19:04:37.727163+0000","last_undegraded":"2024-09-21T19:04:37.727163+0000","last_fullsized":"2024-09-21T19:04:37.727163+0000","mapping_epoch":264,"log_start":"285'5800","ondisk_log_start":"285'5800","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369754112,"num_objects":5642,"num_object_clones":0,"num_object_copies":16926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10246,"num_write_kb":508416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":32243712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":609,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000181+0000","last_change":"2024-09-21T19:03:48.127190+0000","last_active":"2024-09-21T19:04:38.000181+0000","last_peered":"2024-09-21T19:04:38.000181+0000","last_clean":"2024-09-21T19:04:38.000181+0000","last_became_active":"2024-09-21T18:58:40.916310+0000","last_became_peered":"2024-09-21T18:58:40.916310+0000","last_unstale":"2024-09-21T19:04:38.000181+0000","last_undegraded":"2024-09-21T19:04:38.000181+0000","last_fullsized":"2024-09-21T19:04:38.000181+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:48.127135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.127135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.997742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":553,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992918+0000","last_change":"2024-09-21T19:02:40.883840+0000","last_active":"2024-09-21T19:04:37.992918+0000","last_peered":"2024-09-21T19:04:37.992918+0000","last_clean":"2024-09-21T19:04:37.992918+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:04:37.992918+0000","last_undegraded":"2024-09-21T19:04:37.992918+0000","last_fullsized":"2024-09-21T19:04:37.992918+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.862643+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":551,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.974045+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:04:35.974045+0000","last_peered":"2024-09-21T19:04:35.974045+0000","last_clean":"2024-09-21T19:04:35.974045+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:04:35.974045+0000","last_undegraded":"2024-09-21T19:04:35.974045+0000","last_fullsized":"2024-09-21T19:04:35.974045+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":671,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.008763+0000","last_change":"2024-09-21T19:00:20.037953+0000","last_active":"2024-09-21T19:04:38.008763+0000","last_peered":"2024-09-21T19:04:38.008763+0000","last_clean":"2024-09-21T19:04:38.008763+0000","last_became_active":"2024-09-21T18:53:11.413206+0000","last_became_peered":"2024-09-21T18:53:11.413206+0000","last_unstale":"2024-09-21T19:04:38.008763+0000","last_undegraded":"2024-09-21T19:04:38.008763+0000","last_fullsized":"2024-09-21T19:04:38.008763+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.454335+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":504,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992071+0000","last_change":"2024-09-21T18:59:19.920415+0000","last_active":"2024-09-21T19:04:37.992071+0000","last_peered":"2024-09-21T19:04:37.992071+0000","last_clean":"2024-09-21T19:04:37.992071+0000","last_became_active":"2024-09-21T18:53:13.427913+0000","last_became_peered":"2024-09-21T18:53:13.427913+0000","last_unstale":"2024-09-21T19:04:37.992071+0000","last_undegraded":"2024-09-21T19:04:37.992071+0000","last_fullsized":"2024-09-21T19:04:37.992071+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:19.920340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:19.920340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.831345+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"300'10756","reported_seq":10937,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:41.868915+0000","last_change":"2024-09-21T19:04:32.782606+0000","last_active":"2024-09-21T19:04:41.868915+0000","last_peered":"2024-09-21T19:04:41.868915+0000","last_clean":"2024-09-21T19:04:41.868915+0000","last_became_active":"2024-09-21T19:03:09.623358+0000","last_became_peered":"2024-09-21T19:03:09.623358+0000","last_unstale":"2024-09-21T19:04:41.868915+0000","last_undegraded":"2024-09-21T19:04:41.868915+0000","last_fullsized":"2024-09-21T19:04:41.868915+0000","mapping_epoch":274,"log_start":"276'4300","ondisk_log_start":"276'4300","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'10031","last_scrub_stamp":"2024-09-21T19:04:32.782482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:04:32.782482+0000","objects_scrubbed":6176,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.623369+0000","scrub_duration":2207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350617600,"num_objects":5350,"num_object_clones":0,"num_object_copies":16050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10756,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":412,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981731+0000","last_change":"2024-09-21T19:03:52.722880+0000","last_active":"2024-09-21T19:04:35.981731+0000","last_peered":"2024-09-21T19:04:35.981731+0000","last_clean":"2024-09-21T19:04:35.981731+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:04:35.981731+0000","last_undegraded":"2024-09-21T19:04:35.981731+0000","last_fullsized":"2024-09-21T19:04:35.981731+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:52.722797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.722797+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.206303+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.974796+0000","last_change":"2024-09-21T19:03:09.630384+0000","last_active":"2024-09-21T19:04:35.974796+0000","last_peered":"2024-09-21T19:04:35.974796+0000","last_clean":"2024-09-21T19:04:35.974796+0000","last_became_active":"2024-09-21T19:03:09.629753+0000","last_became_peered":"2024-09-21T19:03:09.629753+0000","last_unstale":"2024-09-21T19:04:35.974796+0000","last_undegraded":"2024-09-21T19:04:35.974796+0000","last_fullsized":"2024-09-21T19:04:35.974796+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":516,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.014494+0000","last_change":"2024-09-21T19:03:57.305837+0000","last_active":"2024-09-21T19:04:38.014494+0000","last_peered":"2024-09-21T19:04:38.014494+0000","last_clean":"2024-09-21T19:04:38.014494+0000","last_became_active":"2024-09-21T18:53:11.415379+0000","last_became_peered":"2024-09-21T18:53:11.415379+0000","last_unstale":"2024-09-21T19:04:38.014494+0000","last_undegraded":"2024-09-21T19:04:38.014494+0000","last_fullsized":"2024-09-21T19:04:38.014494+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:26.798148+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":394,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980731+0000","last_change":"2024-09-21T18:57:59.994302+0000","last_active":"2024-09-21T19:04:35.980731+0000","last_peered":"2024-09-21T19:04:35.980731+0000","last_clean":"2024-09-21T19:04:35.980731+0000","last_became_active":"2024-09-21T18:55:51.540221+0000","last_became_peered":"2024-09-21T18:55:51.540221+0000","last_unstale":"2024-09-21T19:04:35.980731+0000","last_undegraded":"2024-09-21T19:04:35.980731+0000","last_fullsized":"2024-09-21T19:04:35.980731+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:59.994227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:59.994227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.597338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"300'10656","reported_seq":10839,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:41.672758+0000","last_change":"2024-09-21T19:03:47.989682+0000","last_active":"2024-09-21T19:04:41.672758+0000","last_peered":"2024-09-21T19:04:41.672758+0000","last_clean":"2024-09-21T19:04:41.672758+0000","last_became_active":"2024-09-21T19:03:09.625334+0000","last_became_peered":"2024-09-21T19:03:09.625334+0000","last_unstale":"2024-09-21T19:04:41.672758+0000","last_undegraded":"2024-09-21T19:04:41.672758+0000","last_fullsized":"2024-09-21T19:04:41.672758+0000","mapping_epoch":274,"log_start":"286'6100","ondisk_log_start":"286'6100","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6556","last_scrub_stamp":"2024-09-21T19:03:47.989483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.989483+0000","objects_scrubbed":6505,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:51.168208+0000","scrub_duration":1845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352976896,"num_objects":5386,"num_object_clones":0,"num_object_copies":16158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10656,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":463,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005517+0000","last_change":"2024-09-21T19:02:40.878888+0000","last_active":"2024-09-21T19:04:38.005517+0000","last_peered":"2024-09-21T19:04:38.005517+0000","last_clean":"2024-09-21T19:04:38.005517+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:04:38.005517+0000","last_undegraded":"2024-09-21T19:04:38.005517+0000","last_fullsized":"2024-09-21T19:04:38.005517+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.891844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":542,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000728+0000","last_change":"2024-09-21T19:04:29.259617+0000","last_active":"2024-09-21T19:04:38.000728+0000","last_peered":"2024-09-21T19:04:38.000728+0000","last_clean":"2024-09-21T19:04:38.000728+0000","last_became_active":"2024-09-21T19:02:40.888713+0000","last_became_peered":"2024-09-21T19:02:40.888713+0000","last_unstale":"2024-09-21T19:04:38.000728+0000","last_undegraded":"2024-09-21T19:04:38.000728+0000","last_fullsized":"2024-09-21T19:04:38.000728+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:57.089123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":538,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.009138+0000","last_change":"2024-09-21T19:02:47.357147+0000","last_active":"2024-09-21T19:04:38.009138+0000","last_peered":"2024-09-21T19:04:38.009138+0000","last_clean":"2024-09-21T19:04:38.009138+0000","last_became_active":"2024-09-21T18:55:51.523645+0000","last_became_peered":"2024-09-21T18:55:51.523645+0000","last_unstale":"2024-09-21T19:04:38.009138+0000","last_undegraded":"2024-09-21T19:04:38.009138+0000","last_fullsized":"2024-09-21T19:04:38.009138+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.448219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":503,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999435+0000","last_change":"2024-09-21T18:57:20.122598+0000","last_active":"2024-09-21T19:04:37.999435+0000","last_peered":"2024-09-21T19:04:37.999435+0000","last_clean":"2024-09-21T19:04:37.999435+0000","last_became_active":"2024-09-21T18:53:13.428496+0000","last_became_peered":"2024-09-21T18:53:13.428496+0000","last_unstale":"2024-09-21T19:04:37.999435+0000","last_undegraded":"2024-09-21T19:04:37.999435+0000","last_fullsized":"2024-09-21T19:04:37.999435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.183675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"299'10060","reported_seq":10160,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.381718+0000","last_change":"2024-09-21T19:02:30.708593+0000","last_active":"2024-09-21T19:04:37.381718+0000","last_peered":"2024-09-21T19:04:37.381718+0000","last_clean":"2024-09-21T19:04:37.381718+0000","last_became_active":"2024-09-21T19:02:30.708207+0000","last_became_peered":"2024-09-21T19:02:30.708207+0000","last_unstale":"2024-09-21T19:04:37.381718+0000","last_undegraded":"2024-09-21T19:04:37.381718+0000","last_fullsized":"2024-09-21T19:04:37.381718+0000","mapping_epoch":254,"log_start":"283'5200","ondisk_log_start":"283'5200","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.754754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372375552,"num_objects":5682,"num_object_clones":0,"num_object_copies":17046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10060,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":571,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.991922+0000","last_change":"2024-09-21T19:02:04.675992+0000","last_active":"2024-09-21T19:04:37.991922+0000","last_peered":"2024-09-21T19:04:37.991922+0000","last_clean":"2024-09-21T19:04:37.991922+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:04:37.991922+0000","last_undegraded":"2024-09-21T19:04:37.991922+0000","last_fullsized":"2024-09-21T19:04:37.991922+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:02:04.675911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:04.675911+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":539,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000663+0000","last_change":"2024-09-21T19:03:47.143498+0000","last_active":"2024-09-21T19:04:38.000663+0000","last_peered":"2024-09-21T19:04:38.000663+0000","last_clean":"2024-09-21T19:04:38.000663+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:04:38.000663+0000","last_undegraded":"2024-09-21T19:04:38.000663+0000","last_fullsized":"2024-09-21T19:04:38.000663+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:47.143439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.143439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:52.903583+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":538,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.008601+0000","last_change":"2024-09-21T19:01:48.464855+0000","last_active":"2024-09-21T19:04:38.008601+0000","last_peered":"2024-09-21T19:04:38.008601+0000","last_clean":"2024-09-21T19:04:38.008601+0000","last_became_active":"2024-09-21T18:55:51.524107+0000","last_became_peered":"2024-09-21T18:55:51.524107+0000","last_unstale":"2024-09-21T19:04:38.008601+0000","last_undegraded":"2024-09-21T19:04:38.008601+0000","last_fullsized":"2024-09-21T19:04:38.008601+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.370827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":503,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.973383+0000","last_change":"2024-09-21T19:03:20.446083+0000","last_active":"2024-09-21T19:04:35.973383+0000","last_peered":"2024-09-21T19:04:35.973383+0000","last_clean":"2024-09-21T19:04:35.973383+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:04:35.973383+0000","last_undegraded":"2024-09-21T19:04:35.973383+0000","last_fullsized":"2024-09-21T19:04:35.973383+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.446014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.446014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"299'10088","reported_seq":10189,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.176664+0000","last_change":"2024-09-21T19:02:29.787669+0000","last_active":"2024-09-21T19:04:37.176664+0000","last_peered":"2024-09-21T19:04:37.176664+0000","last_clean":"2024-09-21T19:04:37.176664+0000","last_became_active":"2024-09-21T19:02:29.787259+0000","last_became_peered":"2024-09-21T19:02:29.787259+0000","last_unstale":"2024-09-21T19:04:37.176664+0000","last_undegraded":"2024-09-21T19:04:37.176664+0000","last_fullsized":"2024-09-21T19:04:37.176664+0000","mapping_epoch":254,"log_start":"277'4400","ondisk_log_start":"277'4400","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:39.635993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370016256,"num_objects":5646,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":134,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999298+0000","last_change":"2024-09-21T19:03:56.136641+0000","last_active":"2024-09-21T19:04:37.999298+0000","last_peered":"2024-09-21T19:04:37.999298+0000","last_clean":"2024-09-21T19:04:37.999298+0000","last_became_active":"2024-09-21T19:03:09.619439+0000","last_became_peered":"2024-09-21T19:03:09.619439+0000","last_unstale":"2024-09-21T19:04:37.999298+0000","last_undegraded":"2024-09-21T19:04:37.999298+0000","last_fullsized":"2024-09-21T19:04:37.999298+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:03.843546+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":553,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.002835+0000","last_change":"2024-09-21T19:02:06.757027+0000","last_active":"2024-09-21T19:04:38.002835+0000","last_peered":"2024-09-21T19:04:38.002835+0000","last_clean":"2024-09-21T19:04:38.002835+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:04:38.002835+0000","last_undegraded":"2024-09-21T19:04:38.002835+0000","last_fullsized":"2024-09-21T19:04:38.002835+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:02:06.756954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:06.756954+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"263'10","reported_seq":686,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000034+0000","last_change":"2024-09-21T19:02:46.144415+0000","last_active":"2024-09-21T19:04:38.000034+0000","last_peered":"2024-09-21T19:04:38.000034+0000","last_clean":"2024-09-21T19:04:38.000034+0000","last_became_active":"2024-09-21T19:02:40.887248+0000","last_became_peered":"2024-09-21T19:02:40.887248+0000","last_unstale":"2024-09-21T19:04:38.000034+0000","last_undegraded":"2024-09-21T19:04:38.000034+0000","last_fullsized":"2024-09-21T19:04:38.000034+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":537,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005416+0000","last_change":"2024-09-21T19:04:24.782721+0000","last_active":"2024-09-21T19:04:38.005416+0000","last_peered":"2024-09-21T19:04:38.005416+0000","last_clean":"2024-09-21T19:04:38.005416+0000","last_became_active":"2024-09-21T19:03:09.618375+0000","last_became_peered":"2024-09-21T19:03:09.618375+0000","last_unstale":"2024-09-21T19:04:38.005416+0000","last_undegraded":"2024-09-21T19:04:38.005416+0000","last_fullsized":"2024-09-21T19:04:38.005416+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:26.072336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":464,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005161+0000","last_change":"2024-09-21T19:02:40.882545+0000","last_active":"2024-09-21T19:04:38.005161+0000","last_peered":"2024-09-21T19:04:38.005161+0000","last_clean":"2024-09-21T19:04:38.005161+0000","last_became_active":"2024-09-21T19:02:40.882216+0000","last_became_peered":"2024-09-21T19:02:40.882216+0000","last_unstale":"2024-09-21T19:04:38.005161+0000","last_undegraded":"2024-09-21T19:04:38.005161+0000","last_fullsized":"2024-09-21T19:04:38.005161+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:04.745928+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":544,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.979414+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:04:35.979414+0000","last_peered":"2024-09-21T19:04:35.979414+0000","last_clean":"2024-09-21T19:04:35.979414+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:04:35.979414+0000","last_undegraded":"2024-09-21T19:04:35.979414+0000","last_fullsized":"2024-09-21T19:04:35.979414+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:59.874264+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":69,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.969798+0000","last_change":"2024-09-21T19:03:11.520901+0000","last_active":"2024-09-21T19:04:35.969798+0000","last_peered":"2024-09-21T19:04:35.969798+0000","last_clean":"2024-09-21T19:04:35.969798+0000","last_became_active":"2024-09-21T19:03:09.623017+0000","last_became_peered":"2024-09-21T19:03:09.623017+0000","last_unstale":"2024-09-21T19:04:35.969798+0000","last_undegraded":"2024-09-21T19:04:35.969798+0000","last_fullsized":"2024-09-21T19:04:35.969798+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.264605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":513,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000108+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:04:38.000108+0000","last_peered":"2024-09-21T19:04:38.000108+0000","last_clean":"2024-09-21T19:04:38.000108+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:04:38.000108+0000","last_undegraded":"2024-09-21T19:04:38.000108+0000","last_fullsized":"2024-09-21T19:04:38.000108+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":420,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980890+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:04:35.980890+0000","last_peered":"2024-09-21T19:04:35.980890+0000","last_clean":"2024-09-21T19:04:35.980890+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:04:35.980890+0000","last_undegraded":"2024-09-21T19:04:35.980890+0000","last_fullsized":"2024-09-21T19:04:35.980890+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:15.701573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":533,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000372+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:04:38.000372+0000","last_peered":"2024-09-21T19:04:38.000372+0000","last_clean":"2024-09-21T19:04:38.000372+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:04:38.000372+0000","last_undegraded":"2024-09-21T19:04:38.000372+0000","last_fullsized":"2024-09-21T19:04:38.000372+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.889409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":506,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.971038+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:04:35.971038+0000","last_peered":"2024-09-21T19:04:35.971038+0000","last_clean":"2024-09-21T19:04:35.971038+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:04:35.971038+0000","last_undegraded":"2024-09-21T19:04:35.971038+0000","last_fullsized":"2024-09-21T19:04:35.971038+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.488303+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":527,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.012953+0000","last_change":"2024-09-21T19:03:44.187925+0000","last_active":"2024-09-21T19:04:38.012953+0000","last_peered":"2024-09-21T19:04:38.012953+0000","last_clean":"2024-09-21T19:04:38.012953+0000","last_became_active":"2024-09-21T18:53:13.422860+0000","last_became_peered":"2024-09-21T18:53:13.422860+0000","last_unstale":"2024-09-21T19:04:38.012953+0000","last_undegraded":"2024-09-21T19:04:38.012953+0000","last_fullsized":"2024-09-21T19:04:38.012953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.242342+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":116,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000447+0000","last_change":"2024-09-21T19:03:14.187382+0000","last_active":"2024-09-21T19:04:38.000447+0000","last_peered":"2024-09-21T19:04:38.000447+0000","last_clean":"2024-09-21T19:04:38.000447+0000","last_became_active":"2024-09-21T19:03:09.624551+0000","last_became_peered":"2024-09-21T19:03:09.624551+0000","last_unstale":"2024-09-21T19:04:38.000447+0000","last_undegraded":"2024-09-21T19:04:38.000447+0000","last_fullsized":"2024-09-21T19:04:38.000447+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:03:14.187189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.187189+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":409,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.980811+0000","last_change":"2024-09-21T19:04:33.875553+0000","last_active":"2024-09-21T19:04:35.980811+0000","last_peered":"2024-09-21T19:04:35.980811+0000","last_clean":"2024-09-21T19:04:35.980811+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:04:35.980811+0000","last_undegraded":"2024-09-21T19:04:35.980811+0000","last_fullsized":"2024-09-21T19:04:35.980811+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.875457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.875457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.453777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":661,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.974350+0000","last_change":"2024-09-21T19:00:19.852100+0000","last_active":"2024-09-21T19:04:35.974350+0000","last_peered":"2024-09-21T19:04:35.974350+0000","last_clean":"2024-09-21T19:04:35.974350+0000","last_became_active":"2024-09-21T18:53:11.414166+0000","last_became_peered":"2024-09-21T18:53:11.414166+0000","last_unstale":"2024-09-21T19:04:35.974350+0000","last_undegraded":"2024-09-21T19:04:35.974350+0000","last_fullsized":"2024-09-21T19:04:35.974350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.472526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":511,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.993124+0000","last_change":"2024-09-21T19:02:36.651427+0000","last_active":"2024-09-21T19:04:37.993124+0000","last_peered":"2024-09-21T19:04:37.993124+0000","last_clean":"2024-09-21T19:04:37.993124+0000","last_became_active":"2024-09-21T18:53:13.423551+0000","last_became_peered":"2024-09-21T18:53:13.423551+0000","last_unstale":"2024-09-21T19:04:37.993124+0000","last_undegraded":"2024-09-21T19:04:37.993124+0000","last_fullsized":"2024-09-21T19:04:37.993124+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":540,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.974604+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:04:35.974604+0000","last_peered":"2024-09-21T19:04:35.974604+0000","last_clean":"2024-09-21T19:04:35.974604+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:04:35.974604+0000","last_undegraded":"2024-09-21T19:04:35.974604+0000","last_fullsized":"2024-09-21T19:04:35.974604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.100625+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":514,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999861+0000","last_change":"2024-09-21T18:57:16.078610+0000","last_active":"2024-09-21T19:04:37.999861+0000","last_peered":"2024-09-21T19:04:37.999861+0000","last_clean":"2024-09-21T19:04:37.999861+0000","last_became_active":"2024-09-21T18:53:07.402094+0000","last_became_peered":"2024-09-21T18:53:07.402094+0000","last_unstale":"2024-09-21T19:04:37.999861+0000","last_undegraded":"2024-09-21T19:04:37.999861+0000","last_fullsized":"2024-09-21T19:04:37.999861+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.236097+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"275'11","reported_seq":112,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985224+0000","last_change":"2024-09-21T19:03:11.527080+0000","last_active":"2024-09-21T19:04:35.985224+0000","last_peered":"2024-09-21T19:04:35.985224+0000","last_clean":"2024-09-21T19:04:35.985224+0000","last_became_active":"2024-09-21T19:03:09.624143+0000","last_became_peered":"2024-09-21T19:03:09.624143+0000","last_unstale":"2024-09-21T19:04:35.985224+0000","last_undegraded":"2024-09-21T19:04:35.985224+0000","last_fullsized":"2024-09-21T19:04:35.985224+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:11.527007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.527007+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.827620+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":409,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981177+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:04:35.981177+0000","last_peered":"2024-09-21T19:04:35.981177+0000","last_clean":"2024-09-21T19:04:35.981177+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:04:35.981177+0000","last_undegraded":"2024-09-21T19:04:35.981177+0000","last_fullsized":"2024-09-21T19:04:35.981177+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.717779+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":588,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.992286+0000","last_change":"2024-09-21T19:04:29.685758+0000","last_active":"2024-09-21T19:04:37.992286+0000","last_peered":"2024-09-21T19:04:37.992286+0000","last_clean":"2024-09-21T19:04:37.992286+0000","last_became_active":"2024-09-21T19:02:40.880515+0000","last_became_peered":"2024-09-21T19:02:40.880515+0000","last_unstale":"2024-09-21T19:04:37.992286+0000","last_undegraded":"2024-09-21T19:04:37.992286+0000","last_fullsized":"2024-09-21T19:04:37.992286+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:04:29.685680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.685680+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:30.435080+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":534,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.993181+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:04:37.993181+0000","last_peered":"2024-09-21T19:04:37.993181+0000","last_clean":"2024-09-21T19:04:37.993181+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:04:37.993181+0000","last_undegraded":"2024-09-21T19:04:37.993181+0000","last_fullsized":"2024-09-21T19:04:37.993181+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:48.105596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":521,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985044+0000","last_change":"2024-09-21T19:04:33.010279+0000","last_active":"2024-09-21T19:04:35.985044+0000","last_peered":"2024-09-21T19:04:35.985044+0000","last_clean":"2024-09-21T19:04:35.985044+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:04:35.985044+0000","last_undegraded":"2024-09-21T19:04:35.985044+0000","last_fullsized":"2024-09-21T19:04:35.985044+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.490106+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":516,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.971414+0000","last_change":"2024-09-21T19:02:35.435594+0000","last_active":"2024-09-21T19:04:35.971414+0000","last_peered":"2024-09-21T19:04:35.971414+0000","last_clean":"2024-09-21T19:04:35.971414+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:04:35.971414+0000","last_undegraded":"2024-09-21T19:04:35.971414+0000","last_fullsized":"2024-09-21T19:04:35.971414+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.435525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.435525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":561,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985335+0000","last_change":"2024-09-21T19:02:40.879297+0000","last_active":"2024-09-21T19:04:35.985335+0000","last_peered":"2024-09-21T19:04:35.985335+0000","last_clean":"2024-09-21T19:04:35.985335+0000","last_became_active":"2024-09-21T19:02:40.878893+0000","last_became_peered":"2024-09-21T19:02:40.878893+0000","last_unstale":"2024-09-21T19:04:35.985335+0000","last_undegraded":"2024-09-21T19:04:35.985335+0000","last_fullsized":"2024-09-21T19:04:35.985335+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.762110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":530,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.000568+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:04:38.000568+0000","last_peered":"2024-09-21T19:04:38.000568+0000","last_clean":"2024-09-21T19:04:38.000568+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:04:38.000568+0000","last_undegraded":"2024-09-21T19:04:38.000568+0000","last_fullsized":"2024-09-21T19:04:38.000568+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.156153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":528,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.973465+0000","last_change":"2024-09-21T19:02:36.480829+0000","last_active":"2024-09-21T19:04:35.973465+0000","last_peered":"2024-09-21T19:04:35.973465+0000","last_clean":"2024-09-21T19:04:35.973465+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:04:35.973465+0000","last_undegraded":"2024-09-21T19:04:35.973465+0000","last_fullsized":"2024-09-21T19:04:35.973465+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.480784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.480784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":409,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981624+0000","last_change":"2024-09-21T19:02:19.440104+0000","last_active":"2024-09-21T19:04:35.981624+0000","last_peered":"2024-09-21T19:04:35.981624+0000","last_clean":"2024-09-21T19:04:35.981624+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:04:35.981624+0000","last_undegraded":"2024-09-21T19:04:35.981624+0000","last_fullsized":"2024-09-21T19:04:35.981624+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:19.440059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:19.440059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:37.686807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":552,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.971746+0000","last_change":"2024-09-21T19:02:40.882171+0000","last_active":"2024-09-21T19:04:35.971746+0000","last_peered":"2024-09-21T19:04:35.971746+0000","last_clean":"2024-09-21T19:04:35.971746+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:04:35.971746+0000","last_undegraded":"2024-09-21T19:04:35.971746+0000","last_fullsized":"2024-09-21T19:04:35.971746+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:02.365192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":523,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.002894+0000","last_change":"2024-09-21T19:04:32.517465+0000","last_active":"2024-09-21T19:04:38.002894+0000","last_peered":"2024-09-21T19:04:38.002894+0000","last_clean":"2024-09-21T19:04:38.002894+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:04:38.002894+0000","last_undegraded":"2024-09-21T19:04:38.002894+0000","last_fullsized":"2024-09-21T19:04:38.002894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:32.517394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:32.517394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:32.564881+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":533,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999194+0000","last_change":"2024-09-21T19:02:48.126289+0000","last_active":"2024-09-21T19:04:37.999194+0000","last_peered":"2024-09-21T19:04:37.999194+0000","last_clean":"2024-09-21T19:04:37.999194+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:04:37.999194+0000","last_undegraded":"2024-09-21T19:04:37.999194+0000","last_fullsized":"2024-09-21T19:04:37.999194+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:48.126230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:48.126230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":535,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.973951+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:04:35.973951+0000","last_peered":"2024-09-21T19:04:35.973951+0000","last_clean":"2024-09-21T19:04:35.973951+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:04:35.973951+0000","last_undegraded":"2024-09-21T19:04:35.973951+0000","last_fullsized":"2024-09-21T19:04:35.973951+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:36.993219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":551,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.970713+0000","last_change":"2024-09-21T19:02:59.672588+0000","last_active":"2024-09-21T19:04:35.970713+0000","last_peered":"2024-09-21T19:04:35.970713+0000","last_clean":"2024-09-21T19:04:35.970713+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:04:35.970713+0000","last_undegraded":"2024-09-21T19:04:35.970713+0000","last_fullsized":"2024-09-21T19:04:35.970713+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-21T19:02:41.535456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.535456+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":401,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981097+0000","last_change":"2024-09-21T19:03:55.627621+0000","last_active":"2024-09-21T19:04:35.981097+0000","last_peered":"2024-09-21T19:04:35.981097+0000","last_clean":"2024-09-21T19:04:35.981097+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:04:35.981097+0000","last_undegraded":"2024-09-21T19:04:35.981097+0000","last_fullsized":"2024-09-21T19:04:35.981097+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.004468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":442,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005183+0000","last_change":"2024-09-21T19:02:40.881419+0000","last_active":"2024-09-21T19:04:38.005183+0000","last_peered":"2024-09-21T19:04:38.005183+0000","last_clean":"2024-09-21T19:04:38.005183+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:04:38.005183+0000","last_undegraded":"2024-09-21T19:04:38.005183+0000","last_fullsized":"2024-09-21T19:04:38.005183+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.789192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":510,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:37.999647+0000","last_change":"2024-09-21T19:02:36.128893+0000","last_active":"2024-09-21T19:04:37.999647+0000","last_peered":"2024-09-21T19:04:37.999647+0000","last_clean":"2024-09-21T19:04:37.999647+0000","last_became_active":"2024-09-21T18:53:13.422734+0000","last_became_peered":"2024-09-21T18:53:13.422734+0000","last_unstale":"2024-09-21T19:04:37.999647+0000","last_undegraded":"2024-09-21T19:04:37.999647+0000","last_fullsized":"2024-09-21T19:04:37.999647+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":501,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.985076+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:04:35.985076+0000","last_peered":"2024-09-21T19:04:35.985076+0000","last_clean":"2024-09-21T19:04:35.985076+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:04:35.985076+0000","last_undegraded":"2024-09-21T19:04:35.985076+0000","last_fullsized":"2024-09-21T19:04:35.985076+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":544,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.971006+0000","last_change":"2024-09-21T19:03:18.435824+0000","last_active":"2024-09-21T19:04:35.971006+0000","last_peered":"2024-09-21T19:04:35.971006+0000","last_clean":"2024-09-21T19:04:35.971006+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:04:35.971006+0000","last_undegraded":"2024-09-21T19:04:35.971006+0000","last_fullsized":"2024-09-21T19:04:35.971006+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:18.435733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.435733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:04:36.735344+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":504,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T19:04:38.005897+0000","last_change":"2024-09-21T19:03:17.580525+0000","last_active":"2024-09-21T19:04:38.005897+0000","last_peered":"2024-09-21T19:04:38.005897+0000","last_clean":"2024-09-21T19:04:38.005897+0000","last_became_active":"2024-09-21T19:02:40.882966+0000","last_became_peered":"2024-09-21T19:02:40.882966+0000","last_unstale":"2024-09-21T19:04:38.005897+0000","last_undegraded":"2024-09-21T19:04:38.005897+0000","last_fullsized":"2024-09-21T19:04:38.005897+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":410,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T19:04:35.981063+0000","last_change":"2024-09-21T19:01:11.531925+0000","last_active":"2024-09-21T19:04:35.981063+0000","last_peered":"2024-09-21T19:04:35.981063+0000","last_clean":"2024-09-21T19:04:35.981063+0000","last_became_active":"2024-09-21T18:55:51.545247+0000","last_became_peered":"2024-09-21T18:55:51.545247+0000","last_unstale":"2024-09-21T19:04:35.981063+0000","last_undegraded":"2024-09-21T19:04:35.981063+0000","last_fullsized":"2024-09-21T19:04:35.981063+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.606767+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5865340952,"num_objects":89499,"num_object_clones":0,"num_object_copies":268497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166207,"num_write_kb":8182529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":97714176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17438552064,"data_stored":17438539848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79007,"ondisk_log_size":79007,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":3190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":683,"num_read_kb":678,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266831,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3592076,"kb_used_data":3127676,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":90153076,"statfs":{"total":95995035648,"available":92316749824,"internally_reserved":0,"allocated":3202740224,"data_stored":3202337116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464281,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1174848,"kb_used_data":1080368,"kb_used_omap":4,"kb_used_meta":94459,"kb_avail":92570304,"statfs":{"total":95995035648,"available":94791991296,"internally_reserved":0,"allocated":1106296832,"data_stored":1105905559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":96726692},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790103,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3614212,"kb_used_data":3117940,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":90130940,"statfs":{"total":95995035648,"available":92294082560,"internally_reserved":0,"allocated":3192770560,"data_stored":3192375669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019037,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1394632,"kb_used_data":1056632,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92350520,"statfs":{"total":95995035648,"available":94566932480,"internally_reserved":0,"allocated":1081991168,"data_stored":1081592412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182561,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3491924,"kb_used_data":2915140,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":90253228,"statfs":{"total":95995035648,"available":92419305472,"internally_reserved":0,"allocated":2985103360,"data_stored":2984686465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411496,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1396040,"kb_used_data":1031672,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92349112,"statfs":{"total":95995035648,"available":94565490688,"internally_reserved":0,"allocated":1056432128,"data_stored":1056033672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3916372,"kb_used_data":3317376,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":89828780,"statfs":{"total":95995035648,"available":91984670720,"internally_reserved":0,"allocated":3396993024,"data_stored":3396579259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1990404,"kb_used_data":1476080,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91754748,"statfs":{"total":95995035648,"available":93956861952,"internally_reserved":0,"allocated":1511505920,"data_stored":1511108851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1504378880,"data_stored":1504378880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3379761152,"data_stored":3379757080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1052114944,"data_stored":1052114944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2973499392,"data_stored":2973499392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1068826624,"data_stored":1068826624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3175354368,"data_stored":3175350296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1104150528,"data_stored":1104150528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3180466176,"data_stored":3180462104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:04:45.126 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-21T19:04:45.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:45.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:45.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:45 smithi081 ceph-mon[39192]: pgmap v565: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-21T19:04:45.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:45 smithi081 ceph-mon[29384]: pgmap v565: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-21T19:04:45.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:45.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:45.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:45 smithi104 ceph-mon[32438]: pgmap v565: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-21T19:04:45.527 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:45.530 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:45.586 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:04:45.586 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:04:45.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:45.665 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:45.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:45.803 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:45.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:45.945 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:45.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:46.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:46.083 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:46.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:04:46.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:46.208 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:46.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:46.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:46.352 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:46.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.498 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:46.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:46.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.642 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:46.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:46.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:46.775 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:46.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:46.916 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:46.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:46.998 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:04:47.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:47.069 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:04:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:47.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:47.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:04:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:04:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[39192]: pgmap v566: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 991 op/s 2024-09-21T19:04:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[39192]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1910370341' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[39192]: from='client.35933 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[29384]: pgmap v566: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 991 op/s 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[29384]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1910370341' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-mon[29384]: from='client.35933 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:47.217 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:47.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.361 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:47.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:47.389 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:47.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:47.481 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:04:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:04:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:04:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.500 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:47.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:47.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:47 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:04:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:47 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:04:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:47 smithi104 ceph-mon[32438]: pgmap v566: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 991 op/s 2024-09-21T19:04:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:47 smithi104 ceph-mon[32438]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T19:04:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1910370341' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:47 smithi104 ceph-mon[32438]: from='client.35933 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:04:47.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:47.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:47.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:47.774 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:47.845 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:47.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:47.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:48.010 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:04:48.010 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:04:48.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:48.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:48.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:48 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:48.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:48 smithi081 ceph-mon[39192]: from='client.35933 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:48.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:48 smithi081 ceph-mon[39192]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T19:04:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:48 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:48 smithi081 ceph-mon[29384]: from='client.35933 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:48 smithi081 ceph-mon[29384]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T19:04:48.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.201 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:48.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:48.245 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:48 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:48.245 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:48 smithi104 ceph-mon[32438]: from='client.35933 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:04:48.245 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:48 smithi104 ceph-mon[32438]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T19:04:48.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:48.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:48.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:48.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:04:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:04:48.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:48.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:48.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:48.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:48.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:48.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:48.998 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:48.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:49.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:04:49.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:49.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:49.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:49.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:49.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:49.472 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:49.472 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:49.472 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:49.473 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:49.474 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:49.474 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:49.474 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:49.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:49 smithi081 ceph-mon[39192]: pgmap v569: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-09-21T19:04:49.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/40529282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:49.475 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:49 smithi081 ceph-mon[29384]: pgmap v569: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-09-21T19:04:49.475 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/40529282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:49 smithi104 ceph-mon[32438]: pgmap v569: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-09-21T19:04:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/40529282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:49.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:49.511 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.540 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:49.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:49.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:49.671 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:49.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.797 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:49.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:49.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:04:49.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:49.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:49.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:50.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:50.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:50.127 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:04:50.127 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:04:50.127 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T19:04:50.127 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T19:04:50.127 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T19:04:50.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T19:04:50.156 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:04:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:04:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:04:50.157 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:04:50.157 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:04:50.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:50.227 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:50 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:50.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:50 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:50.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.357 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:50.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:50.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:50.495 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:50.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:50 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:04:50.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:50.727 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:50.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:04:50.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:50.899 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:50.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:50.943 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:51.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:51.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:51.170 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:51.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:51.301 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:51 smithi104 ceph-mon[32438]: pgmap v570: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-21T19:04:51.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.440 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:51.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:51.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:51 smithi081 ceph-mon[29384]: pgmap v570: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-21T19:04:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:51 smithi081 ceph-mon[39192]: pgmap v570: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-21T19:04:51.576 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:51.599 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:04:51.599 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "" 2024-09-21T19:04:51.600 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:04:51.609 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:51.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.715 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:51.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:51.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:51.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:51.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:51.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:52.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:52.118 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:52.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:52.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:52.257 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:52.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:52.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:52.391 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:52.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:52.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:52.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:52.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:52.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:52.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:52.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:52.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:52.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:52.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:52 smithi104 ceph-mon[32438]: pgmap v571: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 802 op/s 2024-09-21T19:04:52.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:52.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:52.859 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:04:52.859 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:04:52.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:04:52.860 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:04:52.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.860 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102355 7.4 7:2a73ca6f:::benchmark_data_smithi081_73391_object102354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:04:52.860 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:04:52.860 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 572.52960134600005, 2024-09-21T19:04:52.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16875653400000001, 2024-09-21T19:04:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102355 2024-09-21T19:04:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325045+0000", 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325049+0000", 2024-09-21T19:04:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.754e-06 2024-09-21T19:04:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325092+0000", 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3551999999999999e-05 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325112+0000", 2024-09-21T19:04:52.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9306000000000001e-05 2024-09-21T19:04:52.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325122+0000", 2024-09-21T19:04:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7960000000000007e-06 2024-09-21T19:04:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325298+0000", 2024-09-21T19:04:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000176014 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325430+0000", 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013203999999999999 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:04:52.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.325589+0000", 2024-09-21T19:04:52.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000159486 2024-09-21T19:04:52.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.328322+0000", 2024-09-21T19:04:52.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002732656 2024-09-21T19:04:52.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329357+0000", 2024-09-21T19:04:52.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010357400000000001 2024-09-21T19:04:52.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493698+0000", 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16434100300000001 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493758+0000", 2024-09-21T19:04:52.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9515999999999997e-05 2024-09-21T19:04:52.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.493802+0000", 2024-09-21T19:04:52.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3671000000000003e-05 2024-09-21T19:04:52.873 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102361 7.4 7:2b82beac:::benchmark_data_smithi081_73391_object102360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:04:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 572.52476101699995, 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.166344565, 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102361 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329885+0000", 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329887+0000", 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.545e-06 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329943+0000", 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5818e-05 2024-09-21T19:04:52.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329952+0000", 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6729999999999998e-06 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.329956+0000", 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8509999999999998e-06 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330052+0000", 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5827000000000003e-05 2024-09-21T19:04:52.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330254+0000", 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020186800000000001 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.330415+0000", 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016062699999999999 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338473+0000", 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080582750000000002 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341282+0000", 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028086040000000001 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496118+0000", 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15483660299999999 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496188+0000", 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9626e-05 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496230+0000", 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2247999999999999e-05 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102367 7.4 7:2bc3d420:::benchmark_data_smithi081_73391_object102366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:04:52.880 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:04:52.881 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 572.52195985799995, 2024-09-21T19:04:52.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16373927799999999, 2024-09-21T19:04:52.881 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.881 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.881 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102367 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332687+0000", 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332689+0000", 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0810000000000001e-06 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332717+0000", 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8388999999999999e-05 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332731+0000", 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4158999999999999e-05 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332739+0000", 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1159999999999999e-06 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332800+0000", 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0343999999999997e-05 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.332928+0000", 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128341 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333068+0000", 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013955200000000001 2024-09-21T19:04:52.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338482+0000", 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054145959999999998 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341320+0000", 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028377950000000002 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496287+0000", 2024-09-21T19:04:52.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15496721699999999 2024-09-21T19:04:52.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496382+0000", 2024-09-21T19:04:52.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4477999999999999e-05 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496426+0000", 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4209999999999999e-05 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.888 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102368 7.4 7:29e74c0a:::benchmark_data_smithi081_73391_object102367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 572.52109907900001, 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16300020900000001, 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.889 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102368 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333547+0000", 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333549+0000", 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1930000000000001e-06 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333748+0000", 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019946800000000001 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333759+0000", 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0645000000000001e-05 2024-09-21T19:04:52.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.333766+0000", 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0219999999999999e-06 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335818+0000", 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002052356 2024-09-21T19:04:52.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.335940+0000", 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012176999999999999 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.336078+0000", 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137639 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341538+0000", 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054607579999999996 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341553+0000", 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4975e-05 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496464+0000", 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.154911199 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496506+0000", 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1530999999999999e-05 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496548+0000", 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1653000000000003e-05 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.895 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102370 7.4 7:261d1db0:::benchmark_data_smithi081_73391_object102369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 572.51729931399996, 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15954671000000001, 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102370 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337347+0000", 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337352+0000", 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1070000000000004e-06 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337487+0000", 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134325 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337498+0000", 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1063999999999999e-05 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.337505+0000", 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3159999999999999e-06 2024-09-21T19:04:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338628+0000", 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001123298 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338755+0000", 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127059 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.338910+0000", 2024-09-21T19:04:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000154213 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.341560+0000", 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026500429999999999 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.342134+0000", 2024-09-21T19:04:52.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057463499999999999 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496815+0000", 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15468104099999999 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496862+0000", 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7197999999999998e-05 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496894+0000", 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1411000000000001e-05 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:102384 7.1 7:8c91c1ab:::benchmark_data_smithi081_73391_object102383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 572.50958893500001, 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.151452372, 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.902 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 102384 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345058+0000", 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345059+0000", 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.714e-06 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345113+0000", 2024-09-21T19:04:52.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3814999999999999e-05 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345121+0000", 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2479999999999996e-06 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345126+0000", 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9309999999999999e-06 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345176+0000", 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9310000000000001e-05 2024-09-21T19:04:52.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345271+0000", 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4955000000000001e-05 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.345358+0000", 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7319999999999997e-05 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.347395+0000", 2024-09-21T19:04:52.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020371719999999999 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.349879+0000", 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002483519 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496421+0000", 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14654283400000001 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496473+0000", 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1610000000000002e-05 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:20.496510+0000", 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6943999999999997e-05 2024-09-21T19:04:52.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242080 7.1 7:892cd020:::benchmark_data_smithi081_73391_object103081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 423.34893778000003, 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16669837000000001, 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.909 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242080 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505709+0000", 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505710+0000", 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5400000000000001e-06 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505716+0000", 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3170000000000001e-06 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505725+0000", 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7899999999999994e-06 2024-09-21T19:04:52.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505732+0000", 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9789999999999996e-06 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505867+0000", 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134523 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.505964+0000", 2024-09-21T19:04:52.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6680000000000003e-05 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.506050+0000", 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6395000000000002e-05 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.509085+0000", 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030351779999999999 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.509739+0000", 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00065418699999999998 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672337+0000", 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16259812500000001 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672376+0000", 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8399999999999998e-05 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672407+0000", 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1256e-05 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242082 7.1 7:83ef7caf:::benchmark_data_smithi081_73391_object103083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:04:52.915 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 423.344038619, 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16217166099999999, 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242082 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510608+0000", 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510609+0000", 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.345e-06 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510614+0000", 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3030000000000001e-06 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510623+0000", 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3559999999999999e-06 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510630+0000", 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9430000000000003e-06 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510673+0000", 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3180999999999999e-05 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510747+0000", 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4388999999999995e-05 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.510818+0000", 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0288000000000002e-05 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515225+0000", 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044072699999999996 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515269+0000", 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4165000000000002e-05 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672707+0000", 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15743800899999999 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672742+0000", 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4878e-05 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.672780+0000", 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7533999999999997e-05 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.921 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:242083 7.1 7:8f0bc0f0:::benchmark_data_smithi081_73391_object103084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 423.34268939999998, 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18721689799999999, 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 242083 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511957+0000", 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511958+0000", 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7300000000000004e-07 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511962+0000", 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9199999999999997e-06 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511973+0000", 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0577000000000001e-05 2024-09-21T19:04:52.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.511980+0000", 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5310000000000004e-06 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512698+0000", 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071787200000000004 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512808+0000", 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110412 2024-09-21T19:04:52.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.512929+0000", 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012087999999999999 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.515461+0000", 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002532191 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.516028+0000", 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056677599999999996 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699096+0000", 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18306810000000001 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699133+0000", 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6837e-05 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:57:49.699174+0000", 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0828999999999998e-05 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.929 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235896 8.22 8:45adf4b4:::benchmark_data_smithi081_105538_object103664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 179.179506637, 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039943074, 2024-09-21T19:04:52.930 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235896 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:04:52.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675143+0000", 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.729e-06 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675150+0000", 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6820000000000003e-06 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675168+0000", 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7875000000000001e-05 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675178+0000", 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5610000000000004e-06 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731748+0000", 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056570691999999999 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731754+0000", 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.976e-06 2024-09-21T19:04:52.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.700981+0000", 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96922663099999995 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701110+0000", 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864500000000001 2024-09-21T19:04:52.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:04:52.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701320+0000", 2024-09-21T19:04:52.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021059200000000001 2024-09-21T19:04:52.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702216+0000", 2024-09-21T19:04:52.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089577400000000001 2024-09-21T19:04:52.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702245+0000", 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8608000000000001e-05 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.714979+0000", 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012734354999999999 2024-09-21T19:04:52.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715038+0000", 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8588999999999998e-05 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715083+0000", 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5364999999999997e-05 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.940 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235898 8.2 8:4337194e:::benchmark_data_smithi081_105538_object103666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 179.17945325900001, 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0181600850000001, 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.941 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235898 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:04:52.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:04:52.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675194+0000", 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1699999999999997e-07 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675197+0000", 2024-09-21T19:04:52.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T19:04:52.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675203+0000", 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2369999999999997e-06 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675208+0000", 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9989999999999999e-06 2024-09-21T19:04:52.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731592+0000", 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056383961000000003 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731604+0000", 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2278e-05 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685684+0000", 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95407945800000005 2024-09-21T19:04:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685861+0000", 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017756400000000001 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.686082+0000", 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022110200000000001 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.687697+0000", 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016145230000000001 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.690382+0000", 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026848850000000001 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693261+0000", 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879291 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693314+0000", 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2845e-05 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693353+0000", 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9369000000000002e-05 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.950 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237679 8.1b 8:deadaa1e:::benchmark_data_smithi081_105538_object105447:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 177.17000097900001, 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0964987129999999, 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237679 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:04:52.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684647+0000", 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.153e-06 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684651+0000", 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3370000000000001e-06 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684660+0000", 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6470000000000004e-06 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684667+0000", 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684783+0000", 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011565600000000001 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684791+0000", 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7809999999999994e-06 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691384+0000", 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0065922899999999 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691392+0000", 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.636e-06 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781063+0000", 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0896703569999999 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781144+0000", 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1546999999999997e-05 2024-09-21T19:04:52.955 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237686 8.1b 8:df3678b5:::benchmark_data_smithi081_105538_object105454:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 177.169961922, 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.173105171, 2024-09-21T19:04:52.956 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237686 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:04:52.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1299999999999999e-07 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684688+0000", 2024-09-21T19:04:52.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5830000000000001e-06 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684693+0000", 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5910000000000003e-06 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684699+0000", 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684822+0000", 2024-09-21T19:04:52.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012324499999999999 2024-09-21T19:04:52.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684830+0000", 2024-09-21T19:04:52.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6750000000000002e-06 2024-09-21T19:04:52.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691420+0000", 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006590262 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691426+0000", 2024-09-21T19:04:52.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4559999999999997e-06 2024-09-21T19:04:52.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781412+0000", 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.089985687 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for missing object", 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781478+0000", 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6272000000000005e-05 2024-09-21T19:04:52.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857711+0000", 2024-09-21T19:04:52.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076232366999999995 2024-09-21T19:04:52.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857790+0000", 2024-09-21T19:04:52.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9029999999999994e-05 2024-09-21T19:04:52.967 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.967 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.967 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237687 8.1b 8:dc29a777:::benchmark_data_smithi081_105538_object105455:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 177.16994158399999, 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1710659049999999, 2024-09-21T19:04:52.968 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237687 2024-09-21T19:04:52.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:04:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:04:52.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4899999999999995e-07 2024-09-21T19:04:52.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684708+0000", 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1339999999999998e-06 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684713+0000", 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9749999999999998e-06 2024-09-21T19:04:52.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684734+0000", 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1206e-05 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684861+0000", 2024-09-21T19:04:52.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012699799999999999 2024-09-21T19:04:52.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684870+0000", 2024-09-21T19:04:52.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9770000000000006e-06 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691453+0000", 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006583418 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691459+0000", 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2580000000000001e-06 2024-09-21T19:04:52.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781524+0000", 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0900641010000001 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781643+0000", 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119735 2024-09-21T19:04:52.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781902+0000", 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025915999999999998 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784476+0000", 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025738229999999998 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784488+0000", 2024-09-21T19:04:52.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1401e-05 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784513+0000", 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5462000000000001e-05 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855646+0000", 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071132476 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855686+0000", 2024-09-21T19:04:52.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0157999999999997e-05 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855771+0000", 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5074000000000006e-05 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237708 8.1b 8:dbdbeb85:::benchmark_data_smithi081_105538_object105476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:04:52.980 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 177.150253988, 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.151669386, 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237708 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704394+0000", 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.855e-06 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704399+0000", 2024-09-21T19:04:52.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9699999999999998e-06 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704413+0000", 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3560000000000001e-05 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704421+0000", 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6399999999999997e-06 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704473+0000", 2024-09-21T19:04:52.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.27e-05 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704483+0000", 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4520000000000003e-06 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691484+0000", 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98700145800000005 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691491+0000", 2024-09-21T19:04:52.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782825+0000", 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0913340810000001 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782935+0000", 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110647 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.783135+0000", 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020000399999999999 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784491+0000", 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013554319999999999 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784543+0000", 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2609999999999999e-05 2024-09-21T19:04:52.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784566+0000", 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2945000000000001e-05 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855871+0000", 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071304949000000006 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855929+0000", 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7924999999999999e-05 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.856062+0000", 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013279700000000001 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.989 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237711 8.1b 8:dd96cf97:::benchmark_data_smithi081_105538_object105479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 177.148054401, 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1506975869999998, 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:04:52.990 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237711 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706594+0000", 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9099999999999999e-06 2024-09-21T19:04:52.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706599+0000", 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0259999999999998e-06 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706618+0000", 2024-09-21T19:04:52.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8499e-05 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706633+0000", 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5931999999999999e-05 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706751+0000", 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117395 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706762+0000", 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1316e-05 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691770+0000", 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.985007619 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691776+0000", 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5490000000000003e-06 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784627+0000", 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0928504649999999 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784699+0000", 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2539000000000005e-05 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784817+0000", 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118029 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785805+0000", 2024-09-21T19:04:52.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098732000000000008 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785853+0000", 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8554000000000003e-05 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:52.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785945+0000", 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1835000000000004e-05 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857195+0000", 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071249589000000002 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.000 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857226+0000", 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1705999999999998e-05 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857290+0000", 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3304000000000002e-05 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.001 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2026 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94695648315648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 175.58225659499999, 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46135494399999999, 2024-09-21T19:04:53.002 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2026 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:53.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:04:53.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-09-21T19:04:53.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:53.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9019999999999994e-06 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272421+0000", 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7499999999999998e-05 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0607e-05 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272504+0000", 2024-09-21T19:04:53.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2151999999999996e-05 2024-09-21T19:04:53.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:53.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272515+0000", 2024-09-21T19:04:53.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1241e-05 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702573+0000", 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.430058259 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:53.009 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702717+0000", 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143931 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.733745+0000", 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031027852000000002 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2027 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:04:53.010 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 175.58218889599999, 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.464594917, 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2027 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:04:53.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4000000000000001e-07 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272462+0000", 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4199999999999999e-06 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272469+0000", 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272474+0000", 2024-09-21T19:04:53.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4249999999999997e-06 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272539+0000", 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5066000000000003e-05 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272549+0000", 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8709999999999992e-06 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704278+0000", 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43172896500000002 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704401+0000", 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012265999999999999 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.737053+0000", 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.032651612000000003 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2036 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 175.300828956, 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15105808700000001, 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2036 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:53.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:04:53.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0399999999999996e-07 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553821+0000", 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6769999999999999e-06 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553827+0000", 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553831+0000", 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6550000000000003e-06 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553873+0000", 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1241e-05 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553879+0000", 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5429999999999999e-06 2024-09-21T19:04:53.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704657+0000", 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150777833 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704730+0000", 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3398000000000004e-05 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704876+0000", 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014533600000000001 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76067 9.c e280/254 9:3b8d2f8b:::benchmark_data_smithi081_131274_object76066:head v 280'4754, mlcod=280'4748)", 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 89.509260952000005, 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23228328300000001, 2024-09-21T19:04:53.021 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345387+0000", 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5460000000000001e-06 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570553+0000", 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.22516550099999999 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570563+0000", 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0163000000000001e-05 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570569+0000", 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7660000000000004e-06 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570644+0000", 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5710999999999994e-05 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570678+0000", 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3608000000000001e-05 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577637+0000", 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069593909999999997 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577669+0000", 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1597000000000003e-05 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:53.025 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:04:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.043 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:53.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:52 smithi081 ceph-mon[39192]: pgmap v571: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 802 op/s 2024-09-21T19:04:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:52 smithi081 ceph-mon[29384]: pgmap v571: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 802 op/s 2024-09-21T19:04:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:04:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:04:53.144 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:53.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:53.276 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:53.308 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:53.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:53.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:53.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:53.695 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:53.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:53.826 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:53.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:53.965 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:53.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:54.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:54.097 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:54.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:54.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:04:54.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:54.254 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:54.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:54.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:54.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:54.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:54.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:54.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:54.582 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:54.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:54.672 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:54.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:54.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:54.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:54 smithi081 ceph-mon[29384]: pgmap v572: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-21T19:04:54.897 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:54 smithi081 ceph-mon[39192]: pgmap v572: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-21T19:04:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:54.927 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:54.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:55.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:54 smithi104 ceph-mon[32438]: pgmap v572: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-21T19:04:55.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:55.069 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:55.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:55.196 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:55.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.321 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:55.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:186224 9.f 9:facbcc26:::benchmark_data_smithi081_131274_object58369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:55.260770+0000", 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.105991771, 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106013335, 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:55.377 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 186224 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.260770+0000", 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.260770+0000", 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.260771+0000", 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.235e-06 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.260775+0000", 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9929999999999997e-06 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.260785+0000", 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6099999999999995e-06 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.260790+0000", 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4029999999999998e-06 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.261810+0000", 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010197889999999999 2024-09-21T19:04:55.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.261889+0000", 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8969999999999998e-05 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.261958+0000", 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8841e-05 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:55.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.263663+0000", 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00170465 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.267508+0000", 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003845621 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.382 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:186233 9.f 9:fed79735:::benchmark_data_smithi081_131274_object58378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:55.269913+0000", 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.096848792000000003, 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096922495999999997, 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 186233 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:55.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269913+0000", 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269913+0000", 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269915+0000", 2024-09-21T19:04:55.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4580000000000001e-06 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269919+0000", 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4900000000000002e-06 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269928+0000", 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3400000000000004e-06 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269934+0000", 2024-09-21T19:04:55.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2809999999999999e-06 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.269983+0000", 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9753999999999998e-05 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.270073+0000", 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9277000000000006e-05 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.270147+0000", 2024-09-21T19:04:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3837999999999993e-05 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.272453+0000", 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023067529999999999 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.273114+0000", 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00066077699999999998 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:186228 9.a e303/274 9:52a20141:::benchmark_data_smithi081_131274_object58373:head v 303'11679, mlcod=303'11677)", 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:55.366585+0000", 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00017680300000000001, 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029664600000000002, 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366585+0000", 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366585+0000", 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366587+0000", 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3429999999999998e-06 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366598+0000", 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0752e-05 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366616+0000", 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8165e-05 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366626+0000", 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1980000000000007e-06 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366681+0000", 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5892e-05 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:04:55.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:55.366703+0000", 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1331e-05 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.391 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:04:55.394 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:04:55.394 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 3 2024-09-21T19:04:55.394 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:04:55.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:55.454 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:55.556 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:55.585 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:55.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:55.711 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:04:55.731 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:55.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:55.866 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:55.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:55.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.007 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:56.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/723111791' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:56.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:55 smithi104 ceph-mon[32438]: from='client.35942 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:56.028 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/723111791' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:56.028 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:55 smithi081 ceph-mon[39192]: from='client.35942 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:56.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/723111791' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:56.029 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:55 smithi081 ceph-mon[29384]: from='client.35942 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:04:56.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:56.132 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:04:56.140 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:56.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:56.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:04:56.276 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:56.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:56.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:56.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:56.527 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:56.557 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:56.659 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:56.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:56.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.862 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:56.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:56.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:56.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:56 smithi104 ceph-mon[32438]: pgmap v573: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T19:04:56.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:56 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:56.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:56 smithi104 ceph-mon[32438]: from='client.35942 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:04:56.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:56 smithi104 ceph-mon[32438]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T19:04:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:56.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:56.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[29384]: pgmap v573: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[29384]: from='client.35942 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[29384]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[39192]: pgmap v573: 158 pgs: 158 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[39192]: from='client.35942 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:04:57.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:56 smithi081 ceph-mon[39192]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T19:04:57.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:04:57.119 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:04:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:57.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:57.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:57.269 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:57.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:57.337 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:04:57.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:04:57.337 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T19:04:57.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd out 2 2024-09-21T19:04:57.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:57.372 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:57.426 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:57.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:57.518 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:04:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:04:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:04:57.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:57.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:57.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:57.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:57.725 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:57.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:57.770 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:57 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:04:57.770 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:57 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:04:57.770 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:57 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:04:57.770 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:57 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:04:57.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:57.861 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:57.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:57.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:04:57.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:04:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:58.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:04:58.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:57 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:04:58.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:57 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:04:58.083 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:04:58.084 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:04:58.095 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:04:58.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:04:58.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:04:58.118 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:04:58.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:58.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:04:58.152 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:04:58.152 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:04:58.184 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:04:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:04:58.220 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:04:58.220 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:04:58.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:58.253 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:04:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:04:58.254 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:04:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.289 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:58.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:58.290 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:04:58.290 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:04:58.324 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:04:58.324 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:04:58.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:58.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:58.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:04:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:04:58.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:58.568 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:58.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.703 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:58.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:58.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:58.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:58 smithi104 ceph-mon[32438]: pgmap v575: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-21T19:04:58.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:58 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4026273062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:58.853 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:58.862 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:04:58.862 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:04:58.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:04:58.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:58.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:04:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:58 smithi081 ceph-mon[39192]: pgmap v575: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-21T19:04:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:58 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4026273062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:59.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:58 smithi081 ceph-mon[29384]: pgmap v575: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-21T19:04:59.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4026273062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:04:59.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:59.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:59.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:04:59.264 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:04:59.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:04:59.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:04:59.395 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:04:59.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:04:59.523 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:04:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:04:59.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T19:04:59.327+0000 7ff1af1de640 -1 mon.a@0(leader).osd e304 definitely_dead 0 2024-09-21T19:04:59.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:59.687 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:59.712 INFO:teuthology.orchestra.run.smithi081.stderr:marked out osd.2. 2024-09-21T19:04:59.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:04:59.817 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:04:59.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:04:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:04:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:04:59.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:04:59.959 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: 4.1 scrub starts 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: 4.1 scrub ok 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: 6.11 scrub starts 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: 6.11 scrub ok 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1186292047' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-21T19:05:00.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:04:59 smithi104 ceph-mon[32438]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-21T19:05:00.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:00.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: 4.1 scrub starts 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: 4.1 scrub ok 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: 6.11 scrub starts 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: 6.11 scrub ok 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1186292047' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[29384]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: 4.1 scrub starts 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: 4.1 scrub ok 2024-09-21T19:05:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: 6.11 scrub starts 2024-09-21T19:05:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: 6.11 scrub ok 2024-09-21T19:05:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1186292047' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-21T19:05:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:04:59 smithi081 ceph-mon[39192]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-21T19:05:00.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:00.189 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:00.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:00.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:00.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.434 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:00.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:00.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:00.556 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:00.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:00.680 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:00.722 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:05:00.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:00.810 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:00.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:00.936 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:00.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: pgmap v576: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: 3.1d scrub starts 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: 3.1d scrub ok 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1186292047' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: osdmap e305: 8 total, 8 up, 7 in 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: 4.15 scrub starts 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: 4.15 scrub ok 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2392626491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:01.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:00 smithi104 ceph-mon[32438]: from='client.35954 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:01.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:01.060 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: pgmap v576: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: 3.1d scrub starts 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: 3.1d scrub ok 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1186292047' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: osdmap e305: 8 total, 8 up, 7 in 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: 4.15 scrub starts 2024-09-21T19:05:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: 4.15 scrub ok 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2392626491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[39192]: from='client.35954 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: pgmap v576: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: 3.1d scrub starts 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: 3.1d scrub ok 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1186292047' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: osdmap e305: 8 total, 8 up, 7 in 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: 4.15 scrub starts 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: 4.15 scrub ok 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2392626491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:00 smithi081 ceph-mon[29384]: from='client.35954 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:01.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:01.186 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:01.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:01.416 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:01.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:01.543 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:01.568 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:01.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:05:01.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:05:01.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:01.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:01.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:01.819 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:01.905 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:05:01.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:01.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: osdmap e306: 8 total, 8 up, 7 in 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:05:02.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:01 smithi104 ceph-mon[32438]: osdmap e307: 8 total, 8 up, 7 in 2024-09-21T19:05:02.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: osdmap e306: 8 total, 8 up, 7 in 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:05:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[39192]: osdmap e307: 8 total, 8 up, 7 in 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: osdmap e306: 8 total, 8 up, 7 in 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:05:02.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:01 smithi081 ceph-mon[29384]: osdmap e307: 8 total, 8 up, 7 in 2024-09-21T19:05:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:02.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:02.231 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:02.333 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:02.367 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:02.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:02.487 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:05:02.487 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:05:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.519 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:02.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:02.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:02.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:02.762 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:02.801 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:02.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:02.938 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:02.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: pgmap v579: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 54 B/s, 2 objects/s recovering 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 304'11804 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 304'11976 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 304'12101 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 6.9 scrub starts 2024-09-21T19:05:03.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:02 smithi104 ceph-mon[32438]: 6.9 scrub ok 2024-09-21T19:05:03.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: pgmap v579: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 54 B/s, 2 objects/s recovering 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 304'11804 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 304'11976 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 304'12101 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 6.9 scrub starts 2024-09-21T19:05:03.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[39192]: 6.9 scrub ok 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: pgmap v579: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 54 B/s, 2 objects/s recovering 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 304'11804 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 304'11976 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 9.6 starting backfill to osd.6 from (0'0,0'0] MAX to 304'12101 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 6.9 scrub starts 2024-09-21T19:05:03.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:02 smithi081 ceph-mon[29384]: 6.9 scrub ok 2024-09-21T19:05:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:03.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:03.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.216 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:03.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:03.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:03.318 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:03.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:05:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:03.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:03.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:03.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:03.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.643 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:03.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:03.717 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:05:03.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:03.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:03.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:03.929 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:03.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:03.994 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:05:03.995 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:05:03.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:03 smithi081 ceph-mon[29384]: 9.f scrub starts 2024-09-21T19:05:03.995 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:03 smithi081 ceph-mon[39192]: 9.f scrub starts 2024-09-21T19:05:04.003 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:03 smithi104 ceph-mon[32438]: 9.f scrub starts 2024-09-21T19:05:04.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:04.034 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:05:04.035 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:05:04.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:04.065 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:05:04.065 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:05:04.097 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:05:04.098 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:05:04.132 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:05:04.132 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:05:04.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:04.165 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:05:04.165 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:05:04.199 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:05:04.199 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:05:04.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:04.231 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:05:04.232 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:05:04.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:04.337 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:05:04.337 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:05:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:05:04.338 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:05:04.338 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:05:04.346 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:04.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:04.484 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:04.586 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:04.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:04.751 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:04.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:04.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:04.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:04 smithi104 ceph-mon[32438]: 9.f scrub ok 2024-09-21T19:05:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:04 smithi104 ceph-mon[32438]: pgmap v581: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 887 op/s; 11524/190311 objects misplaced (6.055%); 2.7 MiB/s, 53 objects/s recovering 2024-09-21T19:05:05.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/560396672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:05.029 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:05.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:05.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:04 smithi081 ceph-mon[39192]: 9.f scrub ok 2024-09-21T19:05:05.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:04 smithi081 ceph-mon[39192]: pgmap v581: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 887 op/s; 11524/190311 objects misplaced (6.055%); 2.7 MiB/s, 53 objects/s recovering 2024-09-21T19:05:05.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/560396672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:05.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:04 smithi081 ceph-mon[29384]: 9.f scrub ok 2024-09-21T19:05:05.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:04 smithi081 ceph-mon[29384]: pgmap v581: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 887 op/s; 11524/190311 objects misplaced (6.055%); 2.7 MiB/s, 53 objects/s recovering 2024-09-21T19:05:05.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/560396672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:05.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:05.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:05:05.195 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:05.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:05.329 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:05.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:05.462 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:05.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:05:05.564 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:05.606 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:05.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:05.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:05.874 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:05.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:05.899 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:05:05.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: osdmap e308: 8 total, 8 up, 7 in 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: 6.e scrub starts 2024-09-21T19:05:06.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:05 smithi104 ceph-mon[32438]: 6.e scrub ok 2024-09-21T19:05:06.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: osdmap e308: 8 total, 8 up, 7 in 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: 6.e scrub starts 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[39192]: 6.e scrub ok 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: osdmap e308: 8 total, 8 up, 7 in 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: 6.e scrub starts 2024-09-21T19:05:06.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:05 smithi081 ceph-mon[29384]: 6.e scrub ok 2024-09-21T19:05:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:06.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.174 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:06.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:06.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:06.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:06.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.442 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:06.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114519 7.6 7:6b5e3486:::benchmark_data_smithi081_73391_object114518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:05:06.462 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:05:06.463 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 575.74655654900005, 2024-09-21T19:05:06.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12712907900000001, 2024-09-21T19:05:06.463 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.463 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.463 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:05:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114519 2024-09-21T19:05:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:05:06.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709106+0000", 2024-09-21T19:05:06.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709107+0000", 2024-09-21T19:05:06.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.66e-06 2024-09-21T19:05:06.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709157+0000", 2024-09-21T19:05:06.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9911999999999999e-05 2024-09-21T19:05:06.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709171+0000", 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3947e-05 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709179+0000", 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3529999999999997e-06 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716464+0000", 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072858719999999997 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716590+0000", 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012554999999999999 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.716718+0000", 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012810499999999999 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723181+0000", 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006462944 2024-09-21T19:05:06.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723217+0000", 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5593000000000003e-05 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836081+0000", 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11286423700000001 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836188+0000", 2024-09-21T19:05:06.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107315 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836235+0000", 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6591000000000002e-05 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.34742.0:114521 7.6 7:6fe397a7:::benchmark_data_smithi081_73391_object114520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 575.74636199999998, 2024-09-21T19:05:06.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12709793799999999, 2024-09-21T19:05:06.473 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.473 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.473 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.473 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.473 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.34742", 2024-09-21T19:05:06.473 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114521 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709300+0000", 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709301+0000", 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6199999999999996e-07 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709354+0000", 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2936999999999998e-05 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709361+0000", 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.097e-06 2024-09-21T19:05:06.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.709367+0000", 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6550000000000001e-06 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719193+0000", 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098261459999999991 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.719453+0000", 2024-09-21T19:05:06.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026041699999999998 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.720410+0000", 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00095699200000000004 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723227+0000", 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002816888 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.723832+0000", 2024-09-21T19:05:06.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060487799999999999 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836280+0000", 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11244789099999999 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836362+0000", 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1967999999999999e-05 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T18:55:30.836398+0000", 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6207000000000001e-05 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237756 8.22 8.9e11cee2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:05:06.479 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 190.717598119, 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94932720900000001, 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.480 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237756 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738066+0000", 2024-09-21T19:05:06.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6950000000000001e-06 2024-09-21T19:05:06.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738071+0000", 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1390000000000003e-06 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738083+0000", 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1609e-05 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738088+0000", 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3000000000000001e-06 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738133+0000", 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5189999999999999e-05 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738139+0000", 2024-09-21T19:05:06.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5860000000000004e-06 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.687381+0000", 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94924230799999998 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.687391+0000", 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0382e-05 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2028 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94695648323712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 189.183272386, 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43766471099999998, 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.487 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2028 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5080000000000001e-06 2024-09-21T19:05:06.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:05:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8600000000000005e-06 2024-09-21T19:05:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272420+0000", 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6631999999999999e-05 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1506e-05 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272509+0000", 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7401e-05 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272519+0000", 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0308e-05 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.701593+0000", 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.429073594 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.701753+0000", 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000160717 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.710055+0000", 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083011850000000009 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.493 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2038 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94695648323712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 188.90196773900001, 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14878480499999999, 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2038 2024-09-21T19:05:06.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553698+0000", 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0649999999999999e-06 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553706+0000", 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3359999999999995e-06 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553727+0000", 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0899000000000001e-05 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553737+0000", 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0056e-05 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553819+0000", 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2121999999999997e-05 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553828+0000", 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6789999999999994e-06 2024-09-21T19:05:06.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702052+0000", 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.148224774 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702124+0000", 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1186000000000006e-05 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702479+0000", 2024-09-21T19:05:06.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000355688 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:239931 8.17 8:ec94a649:::benchmark_data_smithi081_105538_object107699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 186.909423258, 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13502588600000001, 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.500 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239931 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:05:06.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546241+0000", 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9300000000000002e-06 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546246+0000", 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3970000000000002e-06 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546258+0000", 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546265+0000", 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.99e-06 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546441+0000", 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017672099999999999 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546556+0000", 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114626 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546806+0000", 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025021199999999998 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.549068+0000", 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002261833 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681108+0000", 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13204001600000001 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681166+0000", 2024-09-21T19:05:06.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8137999999999998e-05 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681265+0000", 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8869000000000005e-05 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:239934 8.17 8:e9fd7e5d:::benchmark_data_smithi081_105538_object107702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:59.547680+0000", 2024-09-21T19:05:06.506 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 186.907982772, 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.133722849, 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239934 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547680+0000", 2024-09-21T19:05:06.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547680+0000", 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547681+0000", 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.564e-06 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547686+0000", 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6920000000000001e-06 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547699+0000", 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2733e-05 2024-09-21T19:05:06.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547706+0000", 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0670000000000003e-06 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.547877+0000", 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017096900000000001 2024-09-21T19:05:06.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.548004+0000", 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127663 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.548231+0000", 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022652999999999999 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.549173+0000", 2024-09-21T19:05:06.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094251700000000003 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681298+0000", 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.132124518 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681347+0000", 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8982e-05 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681402+0000", 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5614e-05 2024-09-21T19:05:06.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95331 9.7 9:e0eb634d:::benchmark_data_smithi081_131274_object95330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.920524+0000", 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 87.535138376999996, 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15383042099999999, 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.513 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95331 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920524+0000", 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920524+0000", 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920525+0000", 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3400000000000001e-06 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920589+0000", 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3471000000000003e-05 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920597+0000", 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8339999999999999e-06 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920601+0000", 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6809999999999997e-06 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920638+0000", 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6457e-05 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920730+0000", 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2324999999999994e-05 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.921080+0000", 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034981600000000003 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926003+0000", 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049228919999999999 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926023+0000", 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9816000000000001e-05 2024-09-21T19:05:06.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074262+0000", 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14823952100000001 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074313+0000", 2024-09-21T19:05:06.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1267999999999998e-05 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074354+0000", 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1e-05 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95332 9.7 9:e12c8675:::benchmark_data_smithi081_131274_object95331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.922873+0000", 2024-09-21T19:05:06.519 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 87.532789152999996, 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15164102199999999, 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95332 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922873+0000", 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922873+0000", 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922875+0000", 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.418e-06 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922926+0000", 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1678999999999998e-05 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922938+0000", 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1871999999999999e-05 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922946+0000", 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4399999999999999e-06 2024-09-21T19:05:06.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.923421+0000", 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047575800000000001 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.923549+0000", 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012808399999999999 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:05:06.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.923689+0000", 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013961999999999999 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926108+0000", 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00241873 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926417+0000", 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000308843 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074412+0000", 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.147995024 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074471+0000", 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9035e-05 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074514+0000", 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3519000000000001e-05 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.525 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95346 9.7 9:e6facc78:::benchmark_data_smithi081_131274_object95345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.932556+0000", 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 87.523106302000002, 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.142115833, 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95346 2024-09-21T19:05:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932556+0000", 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932556+0000", 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932558+0000", 2024-09-21T19:05:06.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.705e-06 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932617+0000", 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9277000000000002e-05 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932629+0000", 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.238e-05 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932637+0000", 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1980000000000004e-06 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932693+0000", 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6020999999999998e-05 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932830+0000", 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013747999999999999 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932997+0000", 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016660700000000001 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.935340+0000", 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023430180000000001 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.936494+0000", 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011541240000000001 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074569+0000", 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.138075219 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074627+0000", 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7645999999999997e-05 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074672+0000", 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5157999999999997e-05 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.531 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95350 9.d 9:b86b8700:::benchmark_data_smithi081_131274_object95349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.937661+0000", 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 87.518001131999995, 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.136252596, 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.532 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95350 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937661+0000", 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937661+0000", 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937663+0000", 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7120000000000001e-06 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937721+0000", 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7759e-05 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937732+0000", 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1389000000000001e-05 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937738+0000", 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.02e-06 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937795+0000", 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7093999999999999e-05 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937930+0000", 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013523499999999999 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.938083+0000", 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000152292 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.941028+0000", 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029454379999999999 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.941051+0000", 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3169000000000001e-05 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.073785+0000", 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13273373499999999 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.073863+0000", 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7625000000000003e-05 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.073914+0000", 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1128000000000002e-05 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.537 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:99332 9.e 9:70a87165:::benchmark_data_smithi081_131274_object99331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:42.493627+0000", 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.962035295000007, 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.125358253, 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99332 2024-09-21T19:05:06.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493627+0000", 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493627+0000", 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493629+0000", 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9489999999999999e-06 2024-09-21T19:05:06.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493680+0000", 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1202000000000002e-05 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493693+0000", 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2573e-05 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493700+0000", 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8759999999999999e-06 2024-09-21T19:05:06.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493748+0000", 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8695e-05 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493871+0000", 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012307799999999999 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.494004+0000", 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013299899999999999 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.498251+0000", 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042463730000000003 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.498527+0000", 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027607199999999998 2024-09-21T19:05:06.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.618896+0000", 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12036952400000001 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.618949+0000", 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3050999999999997e-05 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.618985+0000", 2024-09-21T19:05:06.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5861000000000001e-05 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:99333 9.e 9:7323a8c5:::benchmark_data_smithi081_131274_object99332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:42.493935+0000", 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 83.961727053999994, 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12588122500000001, 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.544 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 99333 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493935+0000", 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493935+0000", 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493937+0000", 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4610000000000001e-06 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493973+0000", 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6114e-05 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493984+0000", 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0705e-05 2024-09-21T19:05:06.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.493990+0000", 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8349999999999998e-06 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.496176+0000", 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021860820000000002 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.496279+0000", 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102225 2024-09-21T19:05:06.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.496375+0000", 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6401000000000001e-05 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.498397+0000", 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020218380000000002 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.498664+0000", 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026727199999999998 2024-09-21T19:05:06.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.619732+0000", 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12106787300000001 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.619777+0000", 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4495e-05 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:42.619816+0000", 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9923999999999999e-05 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:103695 9.3 9:c209ad05:::benchmark_data_smithi081_131274_object103694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:46.417474+0000", 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 80.038187851000004, 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.124373574, 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.550 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 103695 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417474+0000", 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417474+0000", 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417476+0000", 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8199999999999999e-06 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417516+0000", 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9509999999999999e-05 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417532+0000", 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6657e-05 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417540+0000", 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.289e-06 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417731+0000", 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019144600000000001 2024-09-21T19:05:06.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.417877+0000", 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014626999999999999 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.418039+0000", 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000161989 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.422091+0000", 2024-09-21T19:05:06.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040513820000000001 2024-09-21T19:05:06.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:06.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.422152+0000", 2024-09-21T19:05:06.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1556999999999994e-05 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.541750+0000", 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.119597447 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.541806+0000", 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6192e-05 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:46.541848+0000", 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2014999999999997e-05 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:120206 9.e 9:79176c90:::benchmark_data_smithi081_131274_object120205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:00.883836+0000", 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 65.571826732000005, 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15420758500000001, 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.558 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 120206 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883836+0000", 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883836+0000", 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883837+0000", 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2079999999999999e-06 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883868+0000", 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1656000000000003e-05 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883879+0000", 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0866999999999999e-05 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883887+0000", 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7389999999999999e-06 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883943+0000", 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6378000000000001e-05 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.884055+0000", 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011174 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.884194+0000", 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013911800000000001 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.888029+0000", 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038351430000000001 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.888089+0000", 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9212999999999997e-05 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.037963+0000", 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14987436300000001 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.038005+0000", 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2434000000000001e-05 2024-09-21T19:05:06.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.038043+0000", 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7725999999999998e-05 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:120212 9.d 9:b677527e:::benchmark_data_smithi081_131274_object120211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:00.889552+0000", 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 65.566110144999996, 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15145771499999999, 2024-09-21T19:05:06.564 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 120212 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889552+0000", 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889552+0000", 2024-09-21T19:05:06.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889554+0000", 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5510000000000001e-06 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889582+0000", 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7877000000000001e-05 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889592+0000", 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.059e-05 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889599+0000", 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2010000000000002e-06 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889706+0000", 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010678700000000001 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889854+0000", 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014771200000000001 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.890031+0000", 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017752499999999999 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.893604+0000", 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035727060000000001 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.893635+0000", 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1044000000000003e-05 2024-09-21T19:05:06.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040917+0000", 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14728138399999999 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040971+0000", 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4277000000000002e-05 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.041010+0000", 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9060999999999998e-05 2024-09-21T19:05:06.571 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:120227 9.3 9:ce04ec7c:::benchmark_data_smithi081_131274_object120226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:00.902473+0000", 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 65.553188798999997, 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13844619799999999, 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.572 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 120227 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902473+0000", 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902473+0000", 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902475+0000", 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.699e-06 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902526+0000", 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0787e-05 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902544+0000", 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7873999999999999e-05 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902551+0000", 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9859999999999997e-06 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902590+0000", 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9415000000000001e-05 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902697+0000", 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106425 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902822+0000", 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012568799999999999 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.904654+0000", 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018317310000000001 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.905187+0000", 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053294099999999999 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040812+0000", 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13562534100000001 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040878+0000", 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5909999999999997e-05 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040920+0000", 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1400999999999999e-05 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:124012 9.d 9:bd9ecacc:::benchmark_data_smithi081_131274_object124011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:04.069557+0000", 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 62.386105233999999, 2024-09-21T19:05:06.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13071865899999999, 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 124012 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069557+0000", 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069557+0000", 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069558+0000", 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1570000000000001e-06 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069607+0000", 2024-09-21T19:05:06.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8505999999999999e-05 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069614+0000", 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2830000000000004e-06 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069619+0000", 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6179999999999997e-06 2024-09-21T19:05:06.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.075675+0000", 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060567010000000003 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.075763+0000", 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7931999999999998e-05 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.075892+0000", 2024-09-21T19:05:06.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012853100000000001 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.081115+0000", 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005223648 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.081156+0000", 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0836e-05 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.200205+0000", 2024-09-21T19:05:06.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.119048388 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.200242+0000", 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7721e-05 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.200276+0000", 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3337999999999998e-05 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:124014 9.d 9:bc42e58d:::benchmark_data_smithi081_131274_object124013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:04.069626+0000", 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 62.386036335999997, 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13146592300000001, 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 124014 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069626+0000", 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069626+0000", 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069627+0000", 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-07 2024-09-21T19:05:06.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069673+0000", 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6298000000000003e-05 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069678+0000", 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9099999999999996e-06 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.069681+0000", 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3809999999999999e-06 2024-09-21T19:05:06.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.078470+0000", 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0087883449999999995 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.078594+0000", 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124401 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.078727+0000", 2024-09-21T19:05:06.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133268 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.081328+0000", 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026007970000000002 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.081362+0000", 2024-09-21T19:05:06.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3685e-05 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.201008+0000", 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11964590999999999 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.201051+0000", 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.333e-05 2024-09-21T19:05:06.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:04.201092+0000", 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.091e-05 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:181792 9.3 9:cd9b7558:::benchmark_data_smithi081_131274_object53937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:51.554266+0000", 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 14.901396427, 2024-09-21T19:05:06.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12273445500000001, 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 181792 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.554266+0000", 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.554266+0000", 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.554267+0000", 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.172e-06 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.554272+0000", 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5619999999999997e-06 2024-09-21T19:05:06.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.554281+0000", 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0969999999999995e-06 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.554288+0000", 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0670000000000003e-06 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:06.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.559294+0000", 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050066950000000002 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.559446+0000", 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015186799999999999 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.560041+0000", 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059494699999999997 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.563635+0000", 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035940690000000001 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.670407+0000", 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10677175899999999 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.676919+0000", 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0065117769999999998 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.676959+0000", 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0163000000000002e-05 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:51.677000+0000", 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1279000000000002e-05 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:06.597 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:05:06.627 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:06.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.757 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:06.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:06.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:06.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:06.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:06.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:07.013 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[29384]: pgmap v583: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 517 B/s rd, 0 B/s wr, 634 op/s; 11524/190311 objects misplaced (6.055%); 2.7 MiB/s, 53 objects/s recovering 2024-09-21T19:05:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T19:05:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T19:05:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:07.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[39192]: pgmap v583: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 517 B/s rd, 0 B/s wr, 634 op/s; 11524/190311 objects misplaced (6.055%); 2.7 MiB/s, 53 objects/s recovering 2024-09-21T19:05:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T19:05:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T19:05:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:07.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:07.149 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:07.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:06 smithi104 ceph-mon[32438]: pgmap v583: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 517 B/s rd, 0 B/s wr, 634 op/s; 11524/190311 objects misplaced (6.055%); 2.7 MiB/s, 53 objects/s recovering 2024-09-21T19:05:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:06 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T19:05:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:06 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T19:05:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:07.271 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:07.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:07.399 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:07.451 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:05:07.451 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:05:07.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:05:07.475 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:05:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:05:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:05:07.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:07.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:07.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:07.662 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:07.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:05:07.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:07.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:07 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T19:05:07.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:07 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T19:05:07.835 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:07.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:07.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:07.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:08.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:07 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T19:05:08.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:07 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T19:05:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:07 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T19:05:08.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:07 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T19:05:08.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.143 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:08.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:08.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:08.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:08.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:08.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:05:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:05:08.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:08.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.682 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:08.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:08.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:05:08.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:08.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:08.849 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:05:08.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:08.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:08 smithi104 ceph-mon[32438]: pgmap v584: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 9672/177123 objects misplaced (5.461%); 13 MiB/s, 224 objects/s recovering 2024-09-21T19:05:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:08 smithi104 ceph-mon[32438]: 6.2 scrub starts 2024-09-21T19:05:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:08 smithi104 ceph-mon[32438]: 6.2 scrub ok 2024-09-21T19:05:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/185093437' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:08 smithi104 ceph-mon[32438]: from='client.35966 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:09.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[29384]: pgmap v584: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 9672/177123 objects misplaced (5.461%); 13 MiB/s, 224 objects/s recovering 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[29384]: 6.2 scrub starts 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[29384]: 6.2 scrub ok 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/185093437' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[29384]: from='client.35966 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[39192]: pgmap v584: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 9672/177123 objects misplaced (5.461%); 13 MiB/s, 224 objects/s recovering 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[39192]: 6.2 scrub starts 2024-09-21T19:05:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[39192]: 6.2 scrub ok 2024-09-21T19:05:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/185093437' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:08 smithi081 ceph-mon[39192]: from='client.35966 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:09.104 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:05:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.105 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:09.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:09.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:09.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:09.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:09.378 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:09.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:09.521 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:09.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:09.659 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:09.683 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:05:09.683 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:05:09.683 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T19:05:09.683 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:05:09.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:09.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:09.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:09.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:09.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:09 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:05:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:09 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:05:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:09 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:09 smithi104 ceph-mon[32438]: from='client.35966 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:05:10.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:09 smithi104 ceph-mon[32438]: osdmap e309: 8 total, 8 up, 7 in 2024-09-21T19:05:10.042 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[39192]: from='client.35966 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[39192]: osdmap e309: 8 total, 8 up, 7 in 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:05:10.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:05:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[29384]: from='client.35966 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:05:10.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:09 smithi081 ceph-mon[29384]: osdmap e309: 8 total, 8 up, 7 in 2024-09-21T19:05:10.077 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:10.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.214 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:10.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:10.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:10.365 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:10.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:10.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:10.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:10.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:05:10.645 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:10.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:10.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:10.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:10.930 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:10.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:11.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:10 smithi104 ceph-mon[32438]: pgmap v586: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s; 8713/173010 objects misplaced (5.036%); 17 MiB/s, 275 objects/s recovering 2024-09-21T19:05:11.021 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:05:11.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:11.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:10 smithi081 ceph-mon[29384]: pgmap v586: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s; 8713/173010 objects misplaced (5.036%); 17 MiB/s, 275 objects/s recovering 2024-09-21T19:05:11.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:10 smithi081 ceph-mon[39192]: pgmap v586: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s; 8713/173010 objects misplaced (5.036%); 17 MiB/s, 275 objects/s recovering 2024-09-21T19:05:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:11.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:11.202 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.247 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:11.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:11.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:11.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:11.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.543 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:11.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:11.607 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:11.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:11.647 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:05:11.647 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:05:11.647 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:05:11.647 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:05:11.647 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:05:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:11.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.839 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:11.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:11.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:11.973 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:11.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:12.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:12.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:12.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:12.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.383 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:12.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:12.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:05:12.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:12.533 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:12.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:12.639 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:05:12.648 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:05:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:12.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:12.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.842 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:12.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:12.887 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:05:12.942 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:12.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:13.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:12 smithi104 ceph-mon[32438]: pgmap v587: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 720 op/s; 8713/173010 objects misplaced (5.036%); 15 MiB/s, 234 objects/s recovering 2024-09-21T19:05:13.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:12 smithi081 ceph-mon[39192]: pgmap v587: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 720 op/s; 8713/173010 objects misplaced (5.036%); 15 MiB/s, 234 objects/s recovering 2024-09-21T19:05:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:12 smithi081 ceph-mon[29384]: pgmap v587: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 720 op/s; 8713/173010 objects misplaced (5.036%); 15 MiB/s, 234 objects/s recovering 2024-09-21T19:05:13.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:13.119 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:13.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:13.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:13.259 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:13.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:13.318 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:05:13.318 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:05:13.355 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:05:13.355 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:05:13.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:13.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:05:13.387 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:05:13.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:13.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:13.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:13.419 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:05:13.419 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:05:13.453 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:05:13.454 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:05:13.489 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:05:13.489 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:05:13.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:11398 9.b e263/262 9:daeaad5e:::benchmark_data_smithi081_131274_object11397:head v 263'734, mlcod=259'204)", 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:43.408908+0000", 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 150.094904571, 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019368351999999998, 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.508 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408908+0000", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408908+0000", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408910+0000", 2024-09-21T19:05:13.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.187e-06 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.408946+0000", 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6644999999999999e-05 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425482+0000", 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016535840999999999 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425494+0000", 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1590000000000001e-05 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425948+0000", 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00045490199999999998 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.425993+0000", 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4421e-05 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.428248+0000", 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022553310000000002 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:43.428277+0000", 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8435000000000001e-05 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:16081 9.0 e263/262 9:0e88a00b:::benchmark_data_smithi081_131274_object16080:head v 263'995, mlcod=260'458)", 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:45.259344+0000", 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 148.24446844600001, 2024-09-21T19:05:13.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020181035999999999, 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259344+0000", 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259344+0000", 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.259345+0000", 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7899999999999997e-07 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278567+0000", 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019222132999999999 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278579+0000", 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1671999999999999e-05 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278585+0000", 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.302e-06 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278676+0000", 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0228000000000004e-05 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278705+0000", 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9516e-05 2024-09-21T19:05:13.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.279478+0000", 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00077334500000000004 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.279525+0000", 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7061000000000002e-05 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:16102 9.0 e263/262 9:05db5d84:::benchmark_data_smithi081_131274_object16101:head v 263'996, mlcod=260'458)", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:45.278601+0000", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 148.22521140500001, 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021694821999999999, 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278601+0000", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278601+0000", 2024-09-21T19:05:13.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.278602+0000", 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7000000000000004e-07 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297657+0000", 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019054588000000001 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297677+0000", 2024-09-21T19:05:13.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9950999999999999e-05 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297684+0000", 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.165e-06 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297763+0000", 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8876999999999997e-05 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.297784+0000", 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1348e-05 2024-09-21T19:05:13.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.300252+0000", 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024677879999999998 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:45.300296+0000", 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4335000000000002e-05 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.520 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20170 9.1 e265/264 9:855477f7:::benchmark_data_smithi081_131274_object20169:head v 265'1249, mlcod=265'1247)", 2024-09-21T19:05:13.521 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.017697+0000", 2024-09-21T19:05:13.521 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.48611603800001, 2024-09-21T19:05:13.521 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.061766164999999998, 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.017697+0000", 2024-09-21T19:05:13.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.017697+0000", 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.017698+0000", 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4810000000000001e-06 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078263+0000", 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060564918000000002 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078271+0000", 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6260000000000003e-06 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078276+0000", 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7210000000000002e-06 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078342+0000", 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6116000000000002e-05 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.078361+0000", 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9527000000000001e-05 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.079387+0000", 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010253180000000001 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.079463+0000", 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6458000000000006e-05 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:20383 9.b e265/264 9:d078fbde:::benchmark_data_smithi081_131274_object20382:head v 265'1289, mlcod=265'1286)", 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:02:48.159973+0000", 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 145.343839465, 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020680995000000001, 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.526 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159973+0000", 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159973+0000", 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.159974+0000", 2024-09-21T19:05:13.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6499999999999999e-07 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179511+0000", 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019537171999999998 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179517+0000", 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9699999999999996e-06 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179523+0000", 2024-09-21T19:05:13.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0379999999999996e-06 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179592+0000", 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8739999999999996e-05 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.179609+0000", 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6538999999999999e-05 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.180619+0000", 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010108389999999999 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:02:48.180654+0000", 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5031999999999999e-05 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:45973 9.1 e271/264 9:8e01c6ff:::benchmark_data_smithi081_131274_object45972:head v 271'2817, mlcod=271'2813)", 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:00.871645+0000", 2024-09-21T19:05:13.530 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 132.63216752400001, 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022307799, 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.871645+0000", 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.871645+0000", 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.871646+0000", 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1699999999999998e-07 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.890826+0000", 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019180413 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.890846+0000", 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.961e-05 2024-09-21T19:05:13.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.890855+0000", 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9709999999999993e-06 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.891035+0000", 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018034099999999999 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.891058+0000", 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2971000000000001e-05 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.893633+0000", 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025748469999999999 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:00.893953+0000", 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032002899999999999 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:51179 9.0 e272/264 9:0fe09c03:::benchmark_data_smithi081_131274_object51178:head v 272'3158, mlcod=272'3155)", 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:04.051554+0000", 2024-09-21T19:05:13.534 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 129.45225868099999, 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022420020999999998, 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051554+0000", 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051554+0000", 2024-09-21T19:05:13.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.051555+0000", 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.074e-06 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070810+0000", 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019254934000000001 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070846+0000", 2024-09-21T19:05:13.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5941e-05 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070873+0000", 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6699e-05 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070949+0000", 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5819e-05 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.070971+0000", 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2586999999999999e-05 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.073850+0000", 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028786390000000001 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.073974+0000", 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012432799999999999 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:51187 9.0 e272/264 9:0207673d:::benchmark_data_smithi081_131274_object51186:head v 272'3160, mlcod=272'3155)", 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:04.071039+0000", 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 129.43277367300001, 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022849504, 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.071039+0000", 2024-09-21T19:05:13.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.071039+0000", 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.071040+0000", 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.13e-07 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090220+0000", 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019180064 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090239+0000", 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8753000000000002e-05 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090247+0000", 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8070000000000006e-06 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090310+0000", 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2716000000000001e-05 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.090332+0000", 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1603e-05 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.093595+0000", 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032630070000000001 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:04.093889+0000", 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029394099999999998 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:65785 9.b e276/264 9:d56336b3:::benchmark_data_smithi081_131274_object65784:head v 276'4179, mlcod=276'4177)", 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:15.083433+0000", 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 118.420379916, 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022701378000000001, 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.083433+0000", 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.083433+0000", 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.083434+0000", 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5e-06 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102055+0000", 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018620384 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102066+0000", 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0659000000000001e-05 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102071+0000", 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.135e-06 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102144+0000", 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3219000000000005e-05 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.102177+0000", 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2728999999999998e-05 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.105996+0000", 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038189399999999998 2024-09-21T19:05:13.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:15.106134+0000", 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013881200000000001 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76679 9.b e280/264 9:d2675948:::benchmark_data_smithi081_131274_object76678:head v 280'4854, mlcod=280'4851)", 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.879754+0000", 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.624058719, 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023429959, 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.879754+0000", 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.879754+0000", 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.879755+0000", 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5900000000000003e-07 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898779+0000", 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019024946000000001 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898799+0000", 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9938000000000001e-05 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898809+0000", 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3489999999999998e-06 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898983+0000", 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000174558 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.899011+0000", 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7339e-05 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.903155+0000", 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041439129999999999 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.903184+0000", 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9556999999999998e-05 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76716 9.b e280/264 9:def1090e:::benchmark_data_smithi081_131274_object76715:head v 280'4856, mlcod=280'4852)", 2024-09-21T19:05:13.551 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.898926+0000", 2024-09-21T19:05:13.552 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.6048864, 2024-09-21T19:05:13.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021674242, 2024-09-21T19:05:13.552 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.552 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.552 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.552 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898926+0000", 2024-09-21T19:05:13.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898926+0000", 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.898927+0000", 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8400000000000004e-07 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918136+0000", 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019208920000000001 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918147+0000", 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0685e-05 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918153+0000", 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6900000000000003e-06 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918208+0000", 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5034000000000002e-05 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.918228+0000", 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9646000000000001e-05 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.920521+0000", 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022929339999999999 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.920601+0000", 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9748999999999998e-05 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:79529 9.b e281/264 9:dc47ee77:::benchmark_data_smithi081_131274_object79528:head v 281'5036, mlcod=281'5034)", 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:26.186342+0000", 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 107.317471008, 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019501747, 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.557 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.186342+0000", 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.186342+0000", 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.186344+0000", 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1880000000000001e-06 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.203193+0000", 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016848742 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.203212+0000", 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9621000000000001e-05 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.203221+0000", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0699999999999996e-06 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.203317+0000", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5625999999999996e-05 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.203339+0000", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2333000000000001e-05 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.205786+0000", 2024-09-21T19:05:13.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002447023 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:26.205844+0000", 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7144e-05 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:97639 9.1 e286/264 9:8e166448:::benchmark_data_smithi081_131274_object97638:head v 286'6007, mlcod=286'6002)", 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:41.038978+0000", 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 92.464835180999998, 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020542767999999999, 2024-09-21T19:05:13.561 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.038978+0000", 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.038978+0000", 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.038979+0000", 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4789999999999999e-06 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057032+0000", 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018053311999999998 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057051+0000", 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8379000000000001e-05 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057059+0000", 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6090000000000001e-06 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057174+0000", 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011408000000000001 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057193+0000", 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9714000000000001e-05 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.059455+0000", 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002261426 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.059520+0000", 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5769e-05 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.566 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:97645 9.1 e286/264 9:8f947c90:::benchmark_data_smithi081_131274_object97644:head v 286'6008, mlcod=286'6005)", 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:41.057092+0000", 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 92.446720638000002, 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023371237999999999, 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057092+0000", 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057092+0000", 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.057093+0000", 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1000000000000001e-06 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.076171+0000", 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019077187999999998 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.076200+0000", 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9116999999999999e-05 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.076207+0000", 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5650000000000004e-06 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.076282+0000", 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4989999999999999e-05 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.076301+0000", 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9279999999999998e-05 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.080423+0000", 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041216250000000003 2024-09-21T19:05:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:41.080463+0000", 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0373000000000001e-05 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:105707 9.1 e289/264 9:80477c51:::benchmark_data_smithi081_131274_object105706:head v 289'6493, mlcod=289'6489)", 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:48.491352+0000", 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 85.012460500000003, 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.025498287000000001, 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.571 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.491352+0000", 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.491352+0000", 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.491353+0000", 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6400000000000002e-07 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.510574+0000", 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019221205000000002 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.510592+0000", 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7538e-05 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.510609+0000", 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6880000000000001e-05 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.510749+0000", 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014025100000000001 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.510767+0000", 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8479999999999999e-05 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.516786+0000", 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060190499999999997 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.516851+0000", 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4219000000000003e-05 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:105721 9.0 e289/264 9:0e67de0c:::benchmark_data_smithi081_131274_object105720:head v 289'6557, mlcod=289'6555)", 2024-09-21T19:05:13.575 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:48.494292+0000", 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 85.009521109000005, 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026708012999999999, 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.494292+0000", 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.494292+0000", 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.494293+0000", 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3179999999999999e-06 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.518108+0000", 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023814744999999998 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.518119+0000", 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.166e-05 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.518127+0000", 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5580000000000003e-06 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.518179+0000", 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2401999999999997e-05 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.518195+0000", 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5787999999999999e-05 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.520970+0000", 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027746749999999999 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:48.521000+0000", 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9867000000000001e-05 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.579 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:122185 9.1 e292/264 9:861447ba:::benchmark_data_smithi081_131274_object122184:head v 292'7567, mlcod=292'7564)", 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:02.525093+0000", 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.978720233000004, 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.025649941999999998, 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.525093+0000", 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.525093+0000", 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.525093+0000", 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6899999999999996e-07 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.544095+0000", 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019001364999999999 2024-09-21T19:05:13.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.544111+0000", 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6585000000000001e-05 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.544121+0000", 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1989999999999995e-06 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.544228+0000", 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107784 2024-09-21T19:05:13.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.544250+0000", 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1489999999999999e-05 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.550618+0000", 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063684040000000003 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:02.550743+0000", 2024-09-21T19:05:13.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012434600000000001 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:126831 9.b e293/264 9:d9404454:::benchmark_data_smithi081_131274_object126830:head v 293'7985, mlcod=293'7982)", 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:06.465148+0000", 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.038665210999994, 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024473007000000001, 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.465148+0000", 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.465148+0000", 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.465149+0000", 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.437e-06 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484674+0000", 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019524835000000001 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484681+0000", 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.939e-06 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484686+0000", 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2619999999999998e-06 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484746+0000", 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0189999999999998e-05 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484763+0000", 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6617999999999999e-05 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.489525+0000", 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047624399999999997 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.489621+0000", 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5285999999999995e-05 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:126837 9.0 e293/264 9:091bae88:::benchmark_data_smithi081_131274_object126836:head v 293'7919, mlcod=293'7916)", 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:06.465367+0000", 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 67.038445574999997, 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024190069000000002, 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.588 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.465367+0000", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.465367+0000", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.465368+0000", 2024-09-21T19:05:13.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.68e-07 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484316+0000", 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018947991000000001 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484352+0000", 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6463e-05 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484406+0000", 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.376e-05 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484587+0000", 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018083500000000001 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.484632+0000", 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5287000000000002e-05 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.489502+0000", 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048693520000000004 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:06.489557+0000", 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5713e-05 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:183100 9.b e303/264 9:d43d90a7:::benchmark_data_smithi081_131274_object55245:head v 303'11509, mlcod=303'11506)", 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:52.702949+0000", 2024-09-21T19:05:13.592 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.800863531000001, 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018808172000000001, 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.702949+0000", 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.702949+0000", 2024-09-21T19:05:13.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.702951+0000", 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.494e-06 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.702957+0000", 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5320000000000003e-06 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:05:13.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.702971+0000", 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3193999999999999e-05 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.702977+0000", 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3090000000000001e-06 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.703030+0000", 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3046e-05 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.703055+0000", 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5006000000000001e-05 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.721598+0000", 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018542949 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:52.721758+0000", 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015964199999999999 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:05:13.597 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:05:13.597 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:05:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:13.597 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:13.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:13.598 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:05:13.598 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:05:13.598 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:05:13.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:13.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:13.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:13.920 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:13.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:13.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:14.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:13 smithi081 ceph-mon[29384]: from='client.16095 v1:172.21.15.81:0/1851973942' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:05:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2022771935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:14.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:13 smithi081 ceph-mon[39192]: from='client.16095 v1:172.21.15.81:0/1851973942' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:05:14.056 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2022771935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:14.070 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:14.106 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-09-21T19:05:11.654763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3766950451,"num_objects":57670,"num_object_clones":0,"num_object_copies":173010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8713,"num_objects_unfound":0,"num_objects_dirty":57670,"num_whiteouts":0,"num_read":1848,"num_read_kb":1600,"num_write":198957,"num_write_kb":8182574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3819,"num_bytes_recovered":238035169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71217,"ondisk_log_size":71217,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13669704,"kb_used_data":10483724,"kb_used_omap":13,"kb_used_meta":3185778,"kb_avail":642546360,"statfs":{"total":671965249536,"available":657967472640,"internally_reserved":0,"allocated":10735333376,"data_stored":10732502525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14003,"internal_metadata":3262237005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,4,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-377946112,"num_objects":-5767,"num_object_clones":0,"num_object_copies":-17301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2811,"num_objects_unfound":0,"num_objects_dirty":-5767,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1881,"num_bytes_recovered":123273216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005778"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":435,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790221+0000","last_change":"2024-09-21T19:05:00.733974+0000","last_active":"2024-09-21T19:05:04.790221+0000","last_peered":"2024-09-21T19:05:04.790221+0000","last_clean":"2024-09-21T19:05:04.790221+0000","last_became_active":"2024-09-21T19:05:00.733612+0000","last_became_peered":"2024-09-21T19:05:00.733612+0000","last_unstale":"2024-09-21T19:05:04.790221+0000","last_undegraded":"2024-09-21T19:05:04.790221+0000","last_fullsized":"2024-09-21T19:05:04.790221+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":615,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822116+0000","last_change":"2024-09-21T19:05:00.811766+0000","last_active":"2024-09-21T19:05:08.822116+0000","last_peered":"2024-09-21T19:05:08.822116+0000","last_clean":"2024-09-21T19:05:08.822116+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:05:08.822116+0000","last_undegraded":"2024-09-21T19:05:08.822116+0000","last_fullsized":"2024-09-21T19:05:08.822116+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:15.600006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.600006+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":539,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784805+0000","last_change":"2024-09-21T19:04:37.095595+0000","last_active":"2024-09-21T19:05:04.784805+0000","last_peered":"2024-09-21T19:05:04.784805+0000","last_clean":"2024-09-21T19:05:04.784805+0000","last_became_active":"2024-09-21T18:53:11.410247+0000","last_became_peered":"2024-09-21T18:53:11.410247+0000","last_unstale":"2024-09-21T19:05:04.784805+0000","last_undegraded":"2024-09-21T19:05:04.784805+0000","last_fullsized":"2024-09-21T19:05:04.784805+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:00.810719+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":521,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822214+0000","last_change":"2024-09-21T19:01:48.626265+0000","last_active":"2024-09-21T19:05:08.822214+0000","last_peered":"2024-09-21T19:05:08.822214+0000","last_clean":"2024-09-21T19:05:08.822214+0000","last_became_active":"2024-09-21T18:53:13.428023+0000","last_became_peered":"2024-09-21T18:53:13.428023+0000","last_unstale":"2024-09-21T19:05:08.822214+0000","last_undegraded":"2024-09-21T19:05:08.822214+0000","last_fullsized":"2024-09-21T19:05:08.822214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.626214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.626214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:38.664442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":547,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.778952+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:05:04.778952+0000","last_peered":"2024-09-21T19:05:04.778952+0000","last_clean":"2024-09-21T19:05:04.778952+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:05:04.778952+0000","last_undegraded":"2024-09-21T19:05:04.778952+0000","last_fullsized":"2024-09-21T19:05:04.778952+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:00.191299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":559,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779861+0000","last_change":"2024-09-21T19:05:00.734482+0000","last_active":"2024-09-21T19:05:04.779861+0000","last_peered":"2024-09-21T19:05:04.779861+0000","last_clean":"2024-09-21T19:05:04.779861+0000","last_became_active":"2024-09-21T19:05:00.733775+0000","last_became_peered":"2024-09-21T19:05:00.733775+0000","last_unstale":"2024-09-21T19:05:04.779861+0000","last_undegraded":"2024-09-21T19:05:04.779861+0000","last_fullsized":"2024-09-21T19:05:04.779861+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.514985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.514985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":640,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779895+0000","last_change":"2024-09-21T19:05:00.809792+0000","last_active":"2024-09-21T19:05:04.779895+0000","last_peered":"2024-09-21T19:05:04.779895+0000","last_clean":"2024-09-21T19:05:04.779895+0000","last_became_active":"2024-09-21T19:05:00.733835+0000","last_became_peered":"2024-09-21T19:05:00.733835+0000","last_unstale":"2024-09-21T19:05:04.779895+0000","last_undegraded":"2024-09-21T19:05:04.779895+0000","last_fullsized":"2024-09-21T19:05:04.779895+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:16.432117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.432117+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":560,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822029+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:05:08.822029+0000","last_peered":"2024-09-21T19:05:08.822029+0000","last_clean":"2024-09-21T19:05:08.822029+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:05:08.822029+0000","last_undegraded":"2024-09-21T19:05:08.822029+0000","last_fullsized":"2024-09-21T19:05:08.822029+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:22.556515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":536,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823585+0000","last_change":"2024-09-21T19:04:59.659018+0000","last_active":"2024-09-21T19:05:08.823585+0000","last_peered":"2024-09-21T19:05:08.823585+0000","last_clean":"2024-09-21T19:05:08.823585+0000","last_became_active":"2024-09-21T19:03:09.619056+0000","last_became_peered":"2024-09-21T19:03:09.619056+0000","last_unstale":"2024-09-21T19:05:08.823585+0000","last_undegraded":"2024-09-21T19:05:08.823585+0000","last_fullsized":"2024-09-21T19:05:08.823585+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:22.731230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":604,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822499+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:05:08.822499+0000","last_peered":"2024-09-21T19:05:08.822499+0000","last_clean":"2024-09-21T19:05:08.822499+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:05:08.822499+0000","last_undegraded":"2024-09-21T19:05:08.822499+0000","last_fullsized":"2024-09-21T19:05:08.822499+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:35.240116+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":561,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.774110+0000","last_change":"2024-09-21T19:03:56.486775+0000","last_active":"2024-09-21T19:05:04.774110+0000","last_peered":"2024-09-21T19:05:04.774110+0000","last_clean":"2024-09-21T19:05:04.774110+0000","last_became_active":"2024-09-21T18:55:51.524404+0000","last_became_peered":"2024-09-21T18:55:51.524404+0000","last_unstale":"2024-09-21T19:05:04.774110+0000","last_undegraded":"2024-09-21T19:05:04.774110+0000","last_fullsized":"2024-09-21T19:05:04.774110+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:56.486496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.486496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:05.359373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":419,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788270+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:05:04.788270+0000","last_peered":"2024-09-21T19:05:04.788270+0000","last_clean":"2024-09-21T19:05:04.788270+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:05:04.788270+0000","last_undegraded":"2024-09-21T19:05:04.788270+0000","last_fullsized":"2024-09-21T19:05:04.788270+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:44.758167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":89,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823872+0000","last_change":"2024-09-21T19:03:59.885217+0000","last_active":"2024-09-21T19:05:08.823872+0000","last_peered":"2024-09-21T19:05:08.823872+0000","last_clean":"2024-09-21T19:05:08.823872+0000","last_became_active":"2024-09-21T19:03:09.628920+0000","last_became_peered":"2024-09-21T19:03:09.628920+0000","last_unstale":"2024-09-21T19:05:08.823872+0000","last_undegraded":"2024-09-21T19:05:08.823872+0000","last_fullsized":"2024-09-21T19:05:08.823872+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":542,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822843+0000","last_change":"2024-09-21T18:59:14.926158+0000","last_active":"2024-09-21T19:05:08.822843+0000","last_peered":"2024-09-21T19:05:08.822843+0000","last_clean":"2024-09-21T19:05:08.822843+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:05:08.822843+0000","last_undegraded":"2024-09-21T19:05:08.822843+0000","last_fullsized":"2024-09-21T19:05:08.822843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T18:59:14.926076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.926076+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.873852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":530,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781405+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:05:04.781405+0000","last_peered":"2024-09-21T19:05:04.781405+0000","last_clean":"2024-09-21T19:05:04.781405+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:05:04.781405+0000","last_undegraded":"2024-09-21T19:05:04.781405+0000","last_fullsized":"2024-09-21T19:05:04.781405+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":87,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784636+0000","last_change":"2024-09-21T19:03:16.368532+0000","last_active":"2024-09-21T19:05:04.784636+0000","last_peered":"2024-09-21T19:05:04.784636+0000","last_clean":"2024-09-21T19:05:04.784636+0000","last_became_active":"2024-09-21T19:03:09.627894+0000","last_became_peered":"2024-09-21T19:03:09.627894+0000","last_unstale":"2024-09-21T19:05:04.784636+0000","last_undegraded":"2024-09-21T19:05:04.784636+0000","last_fullsized":"2024-09-21T19:05:04.784636+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:16.368451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.368451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.108082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":549,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822913+0000","last_change":"2024-09-21T19:03:12.639309+0000","last_active":"2024-09-21T19:05:08.822913+0000","last_peered":"2024-09-21T19:05:08.822913+0000","last_clean":"2024-09-21T19:05:08.822913+0000","last_became_active":"2024-09-21T19:02:40.879452+0000","last_became_peered":"2024-09-21T19:02:40.879452+0000","last_unstale":"2024-09-21T19:05:08.822913+0000","last_undegraded":"2024-09-21T19:05:08.822913+0000","last_fullsized":"2024-09-21T19:05:08.822913+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:12.638987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:12.638987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":419,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788860+0000","last_change":"2024-09-21T19:01:16.593955+0000","last_active":"2024-09-21T19:05:04.788860+0000","last_peered":"2024-09-21T19:05:04.788860+0000","last_clean":"2024-09-21T19:05:04.788860+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:05:04.788860+0000","last_undegraded":"2024-09-21T19:05:04.788860+0000","last_fullsized":"2024-09-21T19:05:04.788860+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:16.593895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:16.593895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.577306+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":553,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788540+0000","last_change":"2024-09-21T19:05:01.273452+0000","last_active":"2024-09-21T19:05:04.788540+0000","last_peered":"2024-09-21T19:05:04.788540+0000","last_clean":"2024-09-21T19:05:04.788540+0000","last_became_active":"2024-09-21T19:05:00.800308+0000","last_became_peered":"2024-09-21T19:05:00.800308+0000","last_unstale":"2024-09-21T19:05:04.788540+0000","last_undegraded":"2024-09-21T19:05:04.788540+0000","last_fullsized":"2024-09-21T19:05:04.788540+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:05:01.273381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.273381+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:09.067368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":553,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790492+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:05:04.790492+0000","last_peered":"2024-09-21T19:05:04.790492+0000","last_clean":"2024-09-21T19:05:04.790492+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:05:04.790492+0000","last_undegraded":"2024-09-21T19:05:04.790492+0000","last_fullsized":"2024-09-21T19:05:04.790492+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":532,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780430+0000","last_change":"2024-09-21T19:03:58.416244+0000","last_active":"2024-09-21T19:05:04.780430+0000","last_peered":"2024-09-21T19:05:04.780430+0000","last_clean":"2024-09-21T19:05:04.780430+0000","last_became_active":"2024-09-21T19:02:40.879991+0000","last_became_peered":"2024-09-21T19:02:40.879991+0000","last_unstale":"2024-09-21T19:05:04.780430+0000","last_undegraded":"2024-09-21T19:05:04.780430+0000","last_fullsized":"2024-09-21T19:05:04.780430+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":564,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822641+0000","last_change":"2024-09-21T19:05:00.733715+0000","last_active":"2024-09-21T19:05:08.822641+0000","last_peered":"2024-09-21T19:05:08.822641+0000","last_clean":"2024-09-21T19:05:08.822641+0000","last_became_active":"2024-09-21T19:05:00.733360+0000","last_became_peered":"2024-09-21T19:05:00.733360+0000","last_unstale":"2024-09-21T19:05:08.822641+0000","last_undegraded":"2024-09-21T19:05:08.822641+0000","last_fullsized":"2024-09-21T19:05:08.822641+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:39.858692+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":606,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779180+0000","last_change":"2024-09-21T19:03:13.490415+0000","last_active":"2024-09-21T19:05:04.779180+0000","last_peered":"2024-09-21T19:05:04.779180+0000","last_clean":"2024-09-21T19:05:04.779180+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:05:04.779180+0000","last_undegraded":"2024-09-21T19:05:04.779180+0000","last_fullsized":"2024-09-21T19:05:04.779180+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:03:13.490332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:13.490332+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":540,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823485+0000","last_change":"2024-09-21T19:05:00.731935+0000","last_active":"2024-09-21T19:05:08.823485+0000","last_peered":"2024-09-21T19:05:08.823485+0000","last_clean":"2024-09-21T19:05:08.823485+0000","last_became_active":"2024-09-21T19:05:00.731614+0000","last_became_peered":"2024-09-21T19:05:00.731614+0000","last_unstale":"2024-09-21T19:05:08.823485+0000","last_undegraded":"2024-09-21T19:05:08.823485+0000","last_fullsized":"2024-09-21T19:05:08.823485+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.898154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.898154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":554,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781414+0000","last_change":"2024-09-21T19:05:00.801961+0000","last_active":"2024-09-21T19:05:04.781414+0000","last_peered":"2024-09-21T19:05:04.781414+0000","last_clean":"2024-09-21T19:05:04.781414+0000","last_became_active":"2024-09-21T19:05:00.732877+0000","last_became_peered":"2024-09-21T19:05:00.732877+0000","last_unstale":"2024-09-21T19:05:04.781414+0000","last_undegraded":"2024-09-21T19:05:04.781414+0000","last_fullsized":"2024-09-21T19:05:04.781414+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":551,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784875+0000","last_change":"2024-09-21T19:03:40.622949+0000","last_active":"2024-09-21T19:05:04.784875+0000","last_peered":"2024-09-21T19:05:04.784875+0000","last_clean":"2024-09-21T19:05:04.784875+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:05:04.784875+0000","last_undegraded":"2024-09-21T19:05:04.784875+0000","last_fullsized":"2024-09-21T19:05:04.784875+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:03:40.622880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.622880+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:50.116199+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":455,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789087+0000","last_change":"2024-09-21T19:01:37.492543+0000","last_active":"2024-09-21T19:05:04.789087+0000","last_peered":"2024-09-21T19:05:04.789087+0000","last_clean":"2024-09-21T19:05:04.789087+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:05:04.789087+0000","last_undegraded":"2024-09-21T19:05:04.789087+0000","last_fullsized":"2024-09-21T19:05:04.789087+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T19:01:37.492484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:37.492484+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.666040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"263'10","reported_seq":566,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.810989+0000","last_change":"2024-09-21T19:02:40.879205+0000","last_active":"2024-09-21T19:05:08.810989+0000","last_peered":"2024-09-21T19:05:08.810989+0000","last_clean":"2024-09-21T19:05:08.810989+0000","last_became_active":"2024-09-21T19:02:40.878691+0000","last_became_peered":"2024-09-21T19:02:40.878691+0000","last_unstale":"2024-09-21T19:05:08.810989+0000","last_undegraded":"2024-09-21T19:05:08.810989+0000","last_fullsized":"2024-09-21T19:05:08.810989+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:35.362716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:35.362716+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.734152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"308'12223","reported_seq":13902,"reported_epoch":308,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:05:07.774147+0000","last_change":"2024-09-21T19:05:01.934729+0000","last_active":"2024-09-21T19:05:07.774147+0000","last_peered":"2024-09-21T19:05:07.774147+0000","last_clean":"2024-09-21T19:04:59.702380+0000","last_became_active":"2024-09-21T19:05:01.892717+0000","last_became_peered":"2024-09-21T19:05:01.892717+0000","last_unstale":"2024-09-21T19:05:07.774147+0000","last_undegraded":"2024-09-21T19:05:07.774147+0000","last_fullsized":"2024-09-21T19:05:07.774147+0000","mapping_epoch":306,"log_start":"293'8038","ondisk_log_start":"293'8038","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'6599","last_scrub_stamp":"2024-09-21T19:03:50.138203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:50.138203+0000","objects_scrubbed":6517,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230883328,"num_objects":3523,"num_object_clones":0,"num_object_copies":10569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2814,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":503872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":48496640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":3523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":110,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.808808+0000","last_change":"2024-09-21T19:05:00.831597+0000","last_active":"2024-09-21T19:05:08.808808+0000","last_peered":"2024-09-21T19:05:08.808808+0000","last_clean":"2024-09-21T19:05:08.808808+0000","last_became_active":"2024-09-21T19:05:00.733527+0000","last_became_peered":"2024-09-21T19:05:00.733527+0000","last_unstale":"2024-09-21T19:05:08.808808+0000","last_undegraded":"2024-09-21T19:05:08.808808+0000","last_fullsized":"2024-09-21T19:05:08.808808+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:00.831540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.831540+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:28.069246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":595,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.794662+0000","last_change":"2024-09-21T19:05:00.237787+0000","last_active":"2024-09-21T19:05:04.794662+0000","last_peered":"2024-09-21T19:05:04.794662+0000","last_clean":"2024-09-21T19:05:04.794662+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:05:04.794662+0000","last_undegraded":"2024-09-21T19:05:04.794662+0000","last_fullsized":"2024-09-21T19:05:04.794662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:05:00.237724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.237724+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:13.445487+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":541,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.783043+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:05:04.783043+0000","last_peered":"2024-09-21T19:05:04.783043+0000","last_clean":"2024-09-21T19:05:04.783043+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:05:04.783043+0000","last_undegraded":"2024-09-21T19:05:04.783043+0000","last_fullsized":"2024-09-21T19:05:04.783043+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.469739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"114'10","reported_seq":725,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822997+0000","last_change":"2024-09-21T19:03:53.851419+0000","last_active":"2024-09-21T19:05:08.822997+0000","last_peered":"2024-09-21T19:05:08.822997+0000","last_clean":"2024-09-21T19:05:08.822997+0000","last_became_active":"2024-09-21T18:55:51.521895+0000","last_became_peered":"2024-09-21T18:55:51.521895+0000","last_unstale":"2024-09-21T19:05:08.822997+0000","last_undegraded":"2024-09-21T19:05:08.822997+0000","last_fullsized":"2024-09-21T19:05:08.822997+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":211,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790395+0000","last_change":"2024-09-21T19:03:14.420119+0000","last_active":"2024-09-21T19:05:04.790395+0000","last_peered":"2024-09-21T19:05:04.790395+0000","last_clean":"2024-09-21T19:05:04.790395+0000","last_became_active":"2024-09-21T19:03:09.623771+0000","last_became_peered":"2024-09-21T19:03:09.623771+0000","last_unstale":"2024-09-21T19:05:04.790395+0000","last_undegraded":"2024-09-21T19:05:04.790395+0000","last_fullsized":"2024-09-21T19:05:04.790395+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:14.420050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:14.420050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.093704+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"308'12398","reported_seq":12593,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:06.707413+0000","last_change":"2024-09-21T19:03:45.264553+0000","last_active":"2024-09-21T19:05:06.707413+0000","last_peered":"2024-09-21T19:05:06.707413+0000","last_clean":"2024-09-21T19:05:06.707413+0000","last_became_active":"2024-09-21T19:02:29.786797+0000","last_became_peered":"2024-09-21T19:02:29.786797+0000","last_unstale":"2024-09-21T19:05:06.707413+0000","last_undegraded":"2024-09-21T19:05:06.707413+0000","last_fullsized":"2024-09-21T19:05:06.707413+0000","mapping_epoch":254,"log_start":"293'8093","ondisk_log_start":"293'8093","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6439","last_scrub_stamp":"2024-09-21T19:03:45.264423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.264423+0000","objects_scrubbed":6363,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2070,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238354456,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12398,"num_write_kb":513089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":325,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823670+0000","last_change":"2024-09-21T19:05:00.737049+0000","last_active":"2024-09-21T19:05:08.823670+0000","last_peered":"2024-09-21T19:05:08.823670+0000","last_clean":"2024-09-21T19:05:08.823670+0000","last_became_active":"2024-09-21T19:05:00.736694+0000","last_became_peered":"2024-09-21T19:05:00.736694+0000","last_unstale":"2024-09-21T19:05:08.823670+0000","last_undegraded":"2024-09-21T19:05:08.823670+0000","last_fullsized":"2024-09-21T19:05:08.823670+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":629,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.782030+0000","last_change":"2024-09-21T18:59:57.951960+0000","last_active":"2024-09-21T19:05:04.782030+0000","last_peered":"2024-09-21T19:05:04.782030+0000","last_clean":"2024-09-21T19:05:04.782030+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:05:04.782030+0000","last_undegraded":"2024-09-21T19:05:04.782030+0000","last_fullsized":"2024-09-21T19:05:04.782030+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-21T18:59:57.951881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:57.951881+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.535087+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":555,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780932+0000","last_change":"2024-09-21T19:01:07.875861+0000","last_active":"2024-09-21T19:05:04.780932+0000","last_peered":"2024-09-21T19:05:04.780932+0000","last_clean":"2024-09-21T19:05:04.780932+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:05:04.780932+0000","last_undegraded":"2024-09-21T19:05:04.780932+0000","last_fullsized":"2024-09-21T19:05:04.780932+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:07.875798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.875798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.587977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":542,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822033+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:05:08.822033+0000","last_peered":"2024-09-21T19:05:08.822033+0000","last_clean":"2024-09-21T19:05:08.822033+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:05:08.822033+0000","last_undegraded":"2024-09-21T19:05:08.822033+0000","last_fullsized":"2024-09-21T19:05:08.822033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":537,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822450+0000","last_change":"2024-09-21T19:05:07.741683+0000","last_active":"2024-09-21T19:05:08.822450+0000","last_peered":"2024-09-21T19:05:08.822450+0000","last_clean":"2024-09-21T19:05:08.822450+0000","last_became_active":"2024-09-21T19:05:00.732448+0000","last_became_peered":"2024-09-21T19:05:00.732448+0000","last_unstale":"2024-09-21T19:05:08.822450+0000","last_undegraded":"2024-09-21T19:05:08.822450+0000","last_fullsized":"2024-09-21T19:05:08.822450+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:07.741620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:07.741620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:31.179965+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"308'12448","reported_seq":12568,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.762234+0000","last_change":"2024-09-21T19:02:30.709345+0000","last_active":"2024-09-21T19:05:07.762234+0000","last_peered":"2024-09-21T19:05:07.762234+0000","last_clean":"2024-09-21T19:05:07.762234+0000","last_became_active":"2024-09-21T19:02:30.709045+0000","last_became_peered":"2024-09-21T19:02:30.709045+0000","last_unstale":"2024-09-21T19:05:07.762234+0000","last_undegraded":"2024-09-21T19:05:07.762234+0000","last_fullsized":"2024-09-21T19:05:07.762234+0000","mapping_epoch":254,"log_start":"294'8252","ondisk_log_start":"294'8252","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4196,"log_dups_size":0,"ondisk_log_size":4196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:51.663529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":302,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781025+0000","last_change":"2024-09-21T18:59:46.950064+0000","last_active":"2024-09-21T19:05:04.781025+0000","last_peered":"2024-09-21T19:05:04.781025+0000","last_clean":"2024-09-21T19:05:04.781025+0000","last_became_active":"2024-09-21T18:58:40.528652+0000","last_became_peered":"2024-09-21T18:58:40.528652+0000","last_unstale":"2024-09-21T19:05:04.781025+0000","last_undegraded":"2024-09-21T19:05:04.781025+0000","last_fullsized":"2024-09-21T19:05:04.781025+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T18:59:46.950000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T18:59:46.950000+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":599,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822107+0000","last_change":"2024-09-21T19:04:58.633928+0000","last_active":"2024-09-21T19:05:08.822107+0000","last_peered":"2024-09-21T19:05:08.822107+0000","last_clean":"2024-09-21T19:05:08.822107+0000","last_became_active":"2024-09-21T19:02:40.885348+0000","last_became_peered":"2024-09-21T19:02:40.885348+0000","last_unstale":"2024-09-21T19:05:08.822107+0000","last_undegraded":"2024-09-21T19:05:08.822107+0000","last_fullsized":"2024-09-21T19:05:08.822107+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:09.610314+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":432,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789687+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:05:04.789687+0000","last_peered":"2024-09-21T19:05:04.789687+0000","last_clean":"2024-09-21T19:05:04.789687+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:05:04.789687+0000","last_undegraded":"2024-09-21T19:05:04.789687+0000","last_fullsized":"2024-09-21T19:05:04.789687+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.439642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":678,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781893+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:05:04.781893+0000","last_peered":"2024-09-21T19:05:04.781893+0000","last_clean":"2024-09-21T19:05:04.781893+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:05:04.781893+0000","last_undegraded":"2024-09-21T19:05:04.781893+0000","last_fullsized":"2024-09-21T19:05:04.781893+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:54.771825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":545,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780852+0000","last_change":"2024-09-21T19:05:00.728835+0000","last_active":"2024-09-21T19:05:04.780852+0000","last_peered":"2024-09-21T19:05:04.780852+0000","last_clean":"2024-09-21T19:05:04.780852+0000","last_became_active":"2024-09-21T19:05:00.728115+0000","last_became_peered":"2024-09-21T19:05:00.728115+0000","last_unstale":"2024-09-21T19:05:04.780852+0000","last_undegraded":"2024-09-21T19:05:04.780852+0000","last_fullsized":"2024-09-21T19:05:04.780852+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.315947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.315947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:36.241376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"308'12304","reported_seq":12545,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:06.686261+0000","last_change":"2024-09-21T19:05:03.409759+0000","last_active":"2024-09-21T19:05:06.686261+0000","last_peered":"2024-09-21T19:05:06.686261+0000","last_clean":"2024-09-21T19:05:06.686261+0000","last_became_active":"2024-09-21T19:02:29.787909+0000","last_became_peered":"2024-09-21T19:02:29.787909+0000","last_unstale":"2024-09-21T19:05:06.686261+0000","last_undegraded":"2024-09-21T19:05:06.686261+0000","last_fullsized":"2024-09-21T19:05:06.686261+0000","mapping_epoch":254,"log_start":"292'7760","ondisk_log_start":"292'7760","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'12064","last_scrub_stamp":"2024-09-21T19:05:03.409666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:05:03.409666+0000","objects_scrubbed":3886,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:30.743547+0000","scrub_duration":1261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235798528,"num_objects":3598,"num_object_clones":0,"num_object_copies":10794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12304,"num_write_kb":508864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":347,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.811153+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:05:08.811153+0000","last_peered":"2024-09-21T19:05:08.811153+0000","last_clean":"2024-09-21T19:05:08.811153+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:05:08.811153+0000","last_undegraded":"2024-09-21T19:05:08.811153+0000","last_fullsized":"2024-09-21T19:05:08.811153+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:23.574104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":574,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784099+0000","last_change":"2024-09-21T19:04:59.218575+0000","last_active":"2024-09-21T19:05:04.784099+0000","last_peered":"2024-09-21T19:05:04.784099+0000","last_clean":"2024-09-21T19:05:04.784099+0000","last_became_active":"2024-09-21T18:53:09.413675+0000","last_became_peered":"2024-09-21T18:53:09.413675+0000","last_unstale":"2024-09-21T19:05:04.784099+0000","last_undegraded":"2024-09-21T19:05:04.784099+0000","last_fullsized":"2024-09-21T19:05:04.784099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:11.825975+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779285+0000","last_change":"2024-09-21T19:05:00.796384+0000","last_active":"2024-09-21T19:05:04.779285+0000","last_peered":"2024-09-21T19:05:04.779285+0000","last_clean":"2024-09-21T19:05:04.779285+0000","last_became_active":"2024-09-21T19:05:00.796023+0000","last_became_peered":"2024-09-21T19:05:00.796023+0000","last_unstale":"2024-09-21T19:05:04.779285+0000","last_undegraded":"2024-09-21T19:05:04.779285+0000","last_fullsized":"2024-09-21T19:05:04.779285+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"263'12","reported_seq":540,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788776+0000","last_change":"2024-09-21T19:02:40.897242+0000","last_active":"2024-09-21T19:05:04.788776+0000","last_peered":"2024-09-21T19:05:04.788776+0000","last_clean":"2024-09-21T19:05:04.788776+0000","last_became_active":"2024-09-21T19:02:40.896856+0000","last_became_peered":"2024-09-21T19:02:40.896856+0000","last_unstale":"2024-09-21T19:05:04.788776+0000","last_undegraded":"2024-09-21T19:05:04.788776+0000","last_fullsized":"2024-09-21T19:05:04.788776+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:01:36.539314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:36.539314+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.638141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":427,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788409+0000","last_change":"2024-09-21T19:05:00.796617+0000","last_active":"2024-09-21T19:05:04.788409+0000","last_peered":"2024-09-21T19:05:04.788409+0000","last_clean":"2024-09-21T19:05:04.788409+0000","last_became_active":"2024-09-21T19:05:00.796281+0000","last_became_peered":"2024-09-21T19:05:00.796281+0000","last_unstale":"2024-09-21T19:05:04.788409+0000","last_undegraded":"2024-09-21T19:05:04.788409+0000","last_fullsized":"2024-09-21T19:05:04.788409+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"308'12722","reported_seq":12842,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:06.783055+0000","last_change":"2024-09-21T19:02:30.708018+0000","last_active":"2024-09-21T19:05:06.783055+0000","last_peered":"2024-09-21T19:05:06.783055+0000","last_clean":"2024-09-21T19:05:06.783055+0000","last_became_active":"2024-09-21T19:02:30.707708+0000","last_became_peered":"2024-09-21T19:02:30.707708+0000","last_unstale":"2024-09-21T19:05:06.783055+0000","last_undegraded":"2024-09-21T19:05:06.783055+0000","last_fullsized":"2024-09-21T19:05:06.783055+0000","mapping_epoch":254,"log_start":"294'8433","ondisk_log_start":"294'8433","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:45.173441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239075328,"num_objects":3648,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12722,"num_write_kb":523840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":325,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.811991+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:05:08.811991+0000","last_peered":"2024-09-21T19:05:08.811991+0000","last_clean":"2024-09-21T19:05:08.811991+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:05:08.811991+0000","last_undegraded":"2024-09-21T19:05:08.811991+0000","last_fullsized":"2024-09-21T19:05:08.811991+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":461,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788941+0000","last_change":"2024-09-21T19:01:00.400955+0000","last_active":"2024-09-21T19:05:04.788941+0000","last_peered":"2024-09-21T19:05:04.788941+0000","last_clean":"2024-09-21T19:05:04.788941+0000","last_became_active":"2024-09-21T18:55:51.521208+0000","last_became_peered":"2024-09-21T18:55:51.521208+0000","last_unstale":"2024-09-21T19:05:04.788941+0000","last_undegraded":"2024-09-21T19:05:04.788941+0000","last_fullsized":"2024-09-21T19:05:04.788941+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:01:00.400874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:00.400874+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.511233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":547,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788928+0000","last_change":"2024-09-21T19:05:00.802884+0000","last_active":"2024-09-21T19:05:04.788928+0000","last_peered":"2024-09-21T19:05:04.788928+0000","last_clean":"2024-09-21T19:05:04.788928+0000","last_became_active":"2024-09-21T19:05:00.801046+0000","last_became_peered":"2024-09-21T19:05:00.801046+0000","last_unstale":"2024-09-21T19:05:04.788928+0000","last_undegraded":"2024-09-21T19:05:04.788928+0000","last_fullsized":"2024-09-21T19:05:04.788928+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:56.467566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:56.467566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":441,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.808895+0000","last_change":"2024-09-21T19:02:40.882592+0000","last_active":"2024-09-21T19:05:08.808895+0000","last_peered":"2024-09-21T19:05:08.808895+0000","last_clean":"2024-09-21T19:05:08.808895+0000","last_became_active":"2024-09-21T19:02:40.882174+0000","last_became_peered":"2024-09-21T19:02:40.882174+0000","last_unstale":"2024-09-21T19:05:08.808895+0000","last_undegraded":"2024-09-21T19:05:08.808895+0000","last_fullsized":"2024-09-21T19:05:08.808895+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:34.408936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:34.408936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:18.736361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784732+0000","last_change":"2024-09-21T19:05:00.799806+0000","last_active":"2024-09-21T19:05:04.784732+0000","last_peered":"2024-09-21T19:05:04.784732+0000","last_clean":"2024-09-21T19:05:04.784732+0000","last_became_active":"2024-09-21T19:05:00.799499+0000","last_became_peered":"2024-09-21T19:05:00.799499+0000","last_unstale":"2024-09-21T19:05:04.784732+0000","last_undegraded":"2024-09-21T19:05:04.784732+0000","last_fullsized":"2024-09-21T19:05:04.784732+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.466833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.466833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.569229+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"308'12491","reported_seq":12687,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.162938+0000","last_change":"2024-09-21T19:03:51.831929+0000","last_active":"2024-09-21T19:05:07.162938+0000","last_peered":"2024-09-21T19:05:07.162938+0000","last_clean":"2024-09-21T19:05:07.162938+0000","last_became_active":"2024-09-21T19:02:29.787771+0000","last_became_peered":"2024-09-21T19:02:29.787771+0000","last_unstale":"2024-09-21T19:05:07.162938+0000","last_undegraded":"2024-09-21T19:05:07.162938+0000","last_fullsized":"2024-09-21T19:05:07.162938+0000","mapping_epoch":254,"log_start":"293'7802","ondisk_log_start":"293'7802","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6909","last_scrub_stamp":"2024-09-21T19:03:51.831669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.831669+0000","objects_scrubbed":6830,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.352296+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231407616,"num_objects":3531,"num_object_clones":0,"num_object_copies":10593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12491,"num_write_kb":512704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":280,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.787879+0000","last_change":"2024-09-21T19:05:00.789813+0000","last_active":"2024-09-21T19:05:04.787879+0000","last_peered":"2024-09-21T19:05:04.787879+0000","last_clean":"2024-09-21T19:05:04.787879+0000","last_became_active":"2024-09-21T19:05:00.733832+0000","last_became_peered":"2024-09-21T19:05:00.733832+0000","last_unstale":"2024-09-21T19:05:04.787879+0000","last_undegraded":"2024-09-21T19:05:04.787879+0000","last_fullsized":"2024-09-21T19:05:04.787879+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:03.383036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"306'41","reported_seq":776,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788790+0000","last_change":"2024-09-21T19:05:00.815224+0000","last_active":"2024-09-21T19:05:04.788790+0000","last_peered":"2024-09-21T19:05:04.788790+0000","last_clean":"2024-09-21T19:05:04.788790+0000","last_became_active":"2024-09-21T19:05:00.793785+0000","last_became_peered":"2024-09-21T19:05:00.793785+0000","last_unstale":"2024-09-21T19:05:04.788790+0000","last_undegraded":"2024-09-21T19:05:04.788790+0000","last_fullsized":"2024-09-21T19:05:04.788790+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:50.925755+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":575,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823111+0000","last_change":"2024-09-21T19:02:43.617505+0000","last_active":"2024-09-21T19:05:08.823111+0000","last_peered":"2024-09-21T19:05:08.823111+0000","last_clean":"2024-09-21T19:05:08.823111+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:05:08.823111+0000","last_undegraded":"2024-09-21T19:05:08.823111+0000","last_fullsized":"2024-09-21T19:05:08.823111+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:43.617435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:43.617435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":109,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784232+0000","last_change":"2024-09-21T19:03:15.425428+0000","last_active":"2024-09-21T19:05:04.784232+0000","last_peered":"2024-09-21T19:05:04.784232+0000","last_clean":"2024-09-21T19:05:04.784232+0000","last_became_active":"2024-09-21T19:03:09.623214+0000","last_became_peered":"2024-09-21T19:03:09.623214+0000","last_unstale":"2024-09-21T19:05:04.784232+0000","last_undegraded":"2024-09-21T19:05:04.784232+0000","last_fullsized":"2024-09-21T19:05:04.784232+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:03:15.425359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.425359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:44.099427+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":420,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788930+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:05:04.788930+0000","last_peered":"2024-09-21T19:05:04.788930+0000","last_clean":"2024-09-21T19:05:04.788930+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:05:04.788930+0000","last_undegraded":"2024-09-21T19:05:04.788930+0000","last_fullsized":"2024-09-21T19:05:04.788930+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.712781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":521,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779575+0000","last_change":"2024-09-21T18:59:51.959368+0000","last_active":"2024-09-21T19:05:04.779575+0000","last_peered":"2024-09-21T19:05:04.779575+0000","last_clean":"2024-09-21T19:05:04.779575+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:05:04.779575+0000","last_undegraded":"2024-09-21T19:05:04.779575+0000","last_fullsized":"2024-09-21T19:05:04.779575+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:51.959284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:51.959284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.494920+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"308'12465","reported_seq":12664,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.378556+0000","last_change":"2024-09-21T19:03:48.136769+0000","last_active":"2024-09-21T19:05:07.378556+0000","last_peered":"2024-09-21T19:05:07.378556+0000","last_clean":"2024-09-21T19:05:07.378556+0000","last_became_active":"2024-09-21T19:03:09.629373+0000","last_became_peered":"2024-09-21T19:03:09.629373+0000","last_unstale":"2024-09-21T19:05:07.378556+0000","last_undegraded":"2024-09-21T19:05:07.378556+0000","last_fullsized":"2024-09-21T19:05:07.378556+0000","mapping_epoch":274,"log_start":"294'8292","ondisk_log_start":"294'8292","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6614","last_scrub_stamp":"2024-09-21T19:03:48.136538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.136538+0000","objects_scrubbed":6508,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.921314+0000","scrub_duration":2592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238747648,"num_objects":3643,"num_object_clones":0,"num_object_copies":10929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12465,"num_write_kb":515456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":626,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780738+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:05:04.780738+0000","last_peered":"2024-09-21T19:05:04.780738+0000","last_clean":"2024-09-21T19:05:04.780738+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:05:04.780738+0000","last_undegraded":"2024-09-21T19:05:04.780738+0000","last_fullsized":"2024-09-21T19:05:04.780738+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:42.214762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":28,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.787379+0000","last_change":"2024-09-21T19:05:00.813894+0000","last_active":"2024-09-21T19:05:04.787379+0000","last_peered":"2024-09-21T19:05:04.787379+0000","last_clean":"2024-09-21T19:05:04.787379+0000","last_became_active":"2024-09-21T19:05:00.797845+0000","last_became_peered":"2024-09-21T19:05:00.797845+0000","last_unstale":"2024-09-21T19:05:04.787379+0000","last_undegraded":"2024-09-21T19:05:04.787379+0000","last_fullsized":"2024-09-21T19:05:04.787379+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:03:40.251822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.251822+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":568,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.785867+0000","last_change":"2024-09-21T19:05:00.816318+0000","last_active":"2024-09-21T19:05:04.785867+0000","last_peered":"2024-09-21T19:05:04.785867+0000","last_clean":"2024-09-21T19:05:04.785867+0000","last_became_active":"2024-09-21T19:05:00.800025+0000","last_became_peered":"2024-09-21T19:05:00.800025+0000","last_unstale":"2024-09-21T19:05:04.785867+0000","last_undegraded":"2024-09-21T19:05:04.785867+0000","last_fullsized":"2024-09-21T19:05:04.785867+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:07.496391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:07.496391+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":34,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779984+0000","last_change":"2024-09-21T19:05:00.803985+0000","last_active":"2024-09-21T19:05:04.779984+0000","last_peered":"2024-09-21T19:05:04.779984+0000","last_clean":"2024-09-21T19:05:04.779984+0000","last_became_active":"2024-09-21T19:05:00.728451+0000","last_became_peered":"2024-09-21T19:05:00.728451+0000","last_unstale":"2024-09-21T19:05:04.779984+0000","last_undegraded":"2024-09-21T19:05:04.779984+0000","last_fullsized":"2024-09-21T19:05:04.779984+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:00:21.025139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:00:21.025139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":17,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781345+0000","last_change":"2024-09-21T19:05:00.731047+0000","last_active":"2024-09-21T19:05:04.781345+0000","last_peered":"2024-09-21T19:05:04.781345+0000","last_clean":"2024-09-21T19:05:04.781345+0000","last_became_active":"2024-09-21T19:05:00.730696+0000","last_became_peered":"2024-09-21T19:05:00.730696+0000","last_unstale":"2024-09-21T19:05:04.781345+0000","last_undegraded":"2024-09-21T19:05:04.781345+0000","last_fullsized":"2024-09-21T19:05:04.781345+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:25.684313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":542,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781909+0000","last_change":"2024-09-21T19:03:17.183599+0000","last_active":"2024-09-21T19:05:04.781909+0000","last_peered":"2024-09-21T19:05:04.781909+0000","last_clean":"2024-09-21T19:05:04.781909+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:05:04.781909+0000","last_undegraded":"2024-09-21T19:05:04.781909+0000","last_fullsized":"2024-09-21T19:05:04.781909+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:17.183521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.183521+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"308'12532","reported_seq":13687,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.179071+0000","last_change":"2024-09-21T19:02:46.916132+0000","last_active":"2024-09-21T19:05:07.179071+0000","last_peered":"2024-09-21T19:05:07.179071+0000","last_clean":"2024-09-21T19:05:07.179071+0000","last_became_active":"2024-09-21T19:02:46.913818+0000","last_became_peered":"2024-09-21T19:02:46.913818+0000","last_unstale":"2024-09-21T19:05:07.179071+0000","last_undegraded":"2024-09-21T19:05:07.179071+0000","last_fullsized":"2024-09-21T19:05:07.179071+0000","mapping_epoch":264,"log_start":"292'7745","ondisk_log_start":"292'7745","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:39.842017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":324,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788004+0000","last_change":"2024-09-21T19:03:56.613953+0000","last_active":"2024-09-21T19:05:04.788004+0000","last_peered":"2024-09-21T19:05:04.788004+0000","last_clean":"2024-09-21T19:05:04.788004+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:05:04.788004+0000","last_undegraded":"2024-09-21T19:05:04.788004+0000","last_fullsized":"2024-09-21T19:05:04.788004+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:56.613898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.613898+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":481,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789495+0000","last_change":"2024-09-21T19:05:00.812125+0000","last_active":"2024-09-21T19:05:04.789495+0000","last_peered":"2024-09-21T19:05:04.789495+0000","last_clean":"2024-09-21T19:05:04.789495+0000","last_became_active":"2024-09-21T19:05:00.789913+0000","last_became_peered":"2024-09-21T19:05:00.789913+0000","last_unstale":"2024-09-21T19:05:04.789495+0000","last_undegraded":"2024-09-21T19:05:04.789495+0000","last_fullsized":"2024-09-21T19:05:04.789495+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:01:13.569464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:13.569464+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":428,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788178+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:05:04.788178+0000","last_peered":"2024-09-21T19:05:04.788178+0000","last_clean":"2024-09-21T19:05:04.788178+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:05:04.788178+0000","last_undegraded":"2024-09-21T19:05:04.788178+0000","last_fullsized":"2024-09-21T19:05:04.788178+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:56.727202+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1285,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:05.384902+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:05:05.384902+0000","last_peered":"2024-09-21T19:05:05.384902+0000","last_clean":"2024-09-21T19:05:05.384902+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:05:05.384902+0000","last_undegraded":"2024-09-21T19:05:05.384902+0000","last_fullsized":"2024-09-21T19:05:05.384902+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:20.975843+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":708,"num_read_kb":703,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":570,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:06.242424+0000","last_change":"2024-09-21T19:05:06.242424+0000","last_active":"2024-09-21T19:05:06.242424+0000","last_peered":"2024-09-21T19:05:06.242424+0000","last_clean":"2024-09-21T19:05:06.242424+0000","last_became_active":"2024-09-21T18:55:51.524096+0000","last_became_peered":"2024-09-21T18:55:51.524096+0000","last_unstale":"2024-09-21T19:05:06.242424+0000","last_undegraded":"2024-09-21T19:05:06.242424+0000","last_fullsized":"2024-09-21T19:05:06.242424+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:20.726779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788355+0000","last_change":"2024-09-21T19:03:54.568908+0000","last_active":"2024-09-21T19:05:04.788355+0000","last_peered":"2024-09-21T19:05:04.788355+0000","last_clean":"2024-09-21T19:05:04.788355+0000","last_became_active":"2024-09-21T18:53:13.431702+0000","last_became_peered":"2024-09-21T18:53:13.431702+0000","last_unstale":"2024-09-21T19:05:04.788355+0000","last_undegraded":"2024-09-21T19:05:04.788355+0000","last_fullsized":"2024-09-21T19:05:04.788355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:54.568845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:54.568845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:07.700759+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"308'12446","reported_seq":12566,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.771009+0000","last_change":"2024-09-21T19:02:29.785323+0000","last_active":"2024-09-21T19:05:07.771009+0000","last_peered":"2024-09-21T19:05:07.771009+0000","last_clean":"2024-09-21T19:05:07.771009+0000","last_became_active":"2024-09-21T19:02:29.785050+0000","last_became_peered":"2024-09-21T19:02:29.785050+0000","last_unstale":"2024-09-21T19:05:07.771009+0000","last_undegraded":"2024-09-21T19:05:07.771009+0000","last_fullsized":"2024-09-21T19:05:07.771009+0000","mapping_epoch":254,"log_start":"294'8264","ondisk_log_start":"294'8264","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12446,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":676,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.809784+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:05:08.809784+0000","last_peered":"2024-09-21T19:05:08.809784+0000","last_clean":"2024-09-21T19:05:08.809784+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:05:08.809784+0000","last_undegraded":"2024-09-21T19:05:08.809784+0000","last_fullsized":"2024-09-21T19:05:08.809784+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:56.838535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781209+0000","last_change":"2024-09-21T19:02:40.877780+0000","last_active":"2024-09-21T19:05:04.781209+0000","last_peered":"2024-09-21T19:05:04.781209+0000","last_clean":"2024-09-21T19:05:04.781209+0000","last_became_active":"2024-09-21T19:02:40.877452+0000","last_became_peered":"2024-09-21T19:02:40.877452+0000","last_unstale":"2024-09-21T19:05:04.781209+0000","last_undegraded":"2024-09-21T19:05:04.781209+0000","last_fullsized":"2024-09-21T19:05:04.781209+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:50.993837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:50.993837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:13.441173+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":632,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780856+0000","last_change":"2024-09-21T19:04:28.281992+0000","last_active":"2024-09-21T19:05:04.780856+0000","last_peered":"2024-09-21T19:05:04.780856+0000","last_clean":"2024-09-21T19:05:04.780856+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:05:04.780856+0000","last_undegraded":"2024-09-21T19:05:04.780856+0000","last_fullsized":"2024-09-21T19:05:04.780856+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:28.281931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.281931+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.661197+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781500+0000","last_change":"2024-09-21T19:05:00.791296+0000","last_active":"2024-09-21T19:05:04.781500+0000","last_peered":"2024-09-21T19:05:04.781500+0000","last_clean":"2024-09-21T19:05:04.781500+0000","last_became_active":"2024-09-21T19:05:00.790778+0000","last_became_peered":"2024-09-21T19:05:00.790778+0000","last_unstale":"2024-09-21T19:05:04.781500+0000","last_undegraded":"2024-09-21T19:05:04.781500+0000","last_fullsized":"2024-09-21T19:05:04.781500+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:12.528320+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":79,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780222+0000","last_change":"2024-09-21T19:03:09.625168+0000","last_active":"2024-09-21T19:05:04.780222+0000","last_peered":"2024-09-21T19:05:04.780222+0000","last_clean":"2024-09-21T19:05:04.780222+0000","last_became_active":"2024-09-21T19:03:09.624424+0000","last_became_peered":"2024-09-21T19:03:09.624424+0000","last_unstale":"2024-09-21T19:05:04.780222+0000","last_undegraded":"2024-09-21T19:05:04.780222+0000","last_fullsized":"2024-09-21T19:05:04.780222+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:49.562265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"308'12269","reported_seq":12389,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:06.710129+0000","last_change":"2024-09-21T19:02:29.785879+0000","last_active":"2024-09-21T19:05:06.710129+0000","last_peered":"2024-09-21T19:05:06.710129+0000","last_clean":"2024-09-21T19:05:06.710129+0000","last_became_active":"2024-09-21T19:02:29.785492+0000","last_became_peered":"2024-09-21T19:02:29.785492+0000","last_unstale":"2024-09-21T19:05:06.710129+0000","last_undegraded":"2024-09-21T19:05:06.710129+0000","last_fullsized":"2024-09-21T19:05:06.710129+0000","mapping_epoch":254,"log_start":"293'7956","ondisk_log_start":"293'7956","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231800832,"num_objects":3537,"num_object_clones":0,"num_object_copies":10611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12269,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":596,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823096+0000","last_change":"2024-09-21T19:03:18.675564+0000","last_active":"2024-09-21T19:05:08.823096+0000","last_peered":"2024-09-21T19:05:08.823096+0000","last_clean":"2024-09-21T19:05:08.823096+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:05:08.823096+0000","last_undegraded":"2024-09-21T19:05:08.823096+0000","last_fullsized":"2024-09-21T19:05:08.823096+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:18.675329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.675329+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":557,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822282+0000","last_change":"2024-09-21T19:05:00.726513+0000","last_active":"2024-09-21T19:05:08.822282+0000","last_peered":"2024-09-21T19:05:08.822282+0000","last_clean":"2024-09-21T19:05:08.822282+0000","last_became_active":"2024-09-21T19:05:00.725554+0000","last_became_peered":"2024-09-21T19:05:00.725554+0000","last_unstale":"2024-09-21T19:05:08.822282+0000","last_undegraded":"2024-09-21T19:05:08.822282+0000","last_fullsized":"2024-09-21T19:05:08.822282+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.643800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.643800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":545,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823360+0000","last_change":"2024-09-21T19:03:11.634133+0000","last_active":"2024-09-21T19:05:08.823360+0000","last_peered":"2024-09-21T19:05:08.823360+0000","last_clean":"2024-09-21T19:05:08.823360+0000","last_became_active":"2024-09-21T19:03:09.619609+0000","last_became_peered":"2024-09-21T19:03:09.619609+0000","last_unstale":"2024-09-21T19:05:08.823360+0000","last_undegraded":"2024-09-21T19:05:08.823360+0000","last_fullsized":"2024-09-21T19:05:08.823360+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":524,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781110+0000","last_change":"2024-09-21T19:02:46.521753+0000","last_active":"2024-09-21T19:05:04.781110+0000","last_peered":"2024-09-21T19:05:04.781110+0000","last_clean":"2024-09-21T19:05:04.781110+0000","last_became_active":"2024-09-21T19:02:40.877238+0000","last_became_peered":"2024-09-21T19:02:40.877238+0000","last_unstale":"2024-09-21T19:05:04.781110+0000","last_undegraded":"2024-09-21T19:05:04.781110+0000","last_fullsized":"2024-09-21T19:05:04.781110+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"308'12357","reported_seq":14741,"reported_epoch":308,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:05:07.338117+0000","last_change":"2024-09-21T19:05:02.009864+0000","last_active":"2024-09-21T19:05:07.338117+0000","last_peered":"2024-09-21T19:05:07.338117+0000","last_clean":"2024-09-21T19:04:59.698329+0000","last_became_active":"2024-09-21T19:05:01.934938+0000","last_became_peered":"2024-09-21T19:05:01.934938+0000","last_unstale":"2024-09-21T19:05:07.338117+0000","last_undegraded":"2024-09-21T19:05:07.338117+0000","last_fullsized":"2024-09-21T19:05:07.338117+0000","mapping_epoch":306,"log_start":"293'8110","ondisk_log_start":"293'8110","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236781568,"num_objects":3613,"num_object_clones":0,"num_object_copies":10839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3009,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":511040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1125,"num_bytes_recovered":73728000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,2],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2,6","objects":3613}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":30,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781692+0000","last_change":"2024-09-21T19:05:01.593667+0000","last_active":"2024-09-21T19:05:04.781692+0000","last_peered":"2024-09-21T19:05:04.781692+0000","last_clean":"2024-09-21T19:05:04.781692+0000","last_became_active":"2024-09-21T19:05:00.796323+0000","last_became_peered":"2024-09-21T19:05:00.796323+0000","last_unstale":"2024-09-21T19:05:04.781692+0000","last_undegraded":"2024-09-21T19:05:04.781692+0000","last_fullsized":"2024-09-21T19:05:04.781692+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:09.661413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":620,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823467+0000","last_change":"2024-09-21T19:05:05.756208+0000","last_active":"2024-09-21T19:05:08.823467+0000","last_peered":"2024-09-21T19:05:08.823467+0000","last_clean":"2024-09-21T19:05:08.823467+0000","last_became_active":"2024-09-21T19:05:00.730130+0000","last_became_peered":"2024-09-21T19:05:00.730130+0000","last_unstale":"2024-09-21T19:05:08.823467+0000","last_undegraded":"2024-09-21T19:05:08.823467+0000","last_fullsized":"2024-09-21T19:05:08.823467+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:05:05.756153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:05.756153+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:33.805007+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":463,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.809254+0000","last_change":"2024-09-21T19:02:40.879761+0000","last_active":"2024-09-21T19:05:08.809254+0000","last_peered":"2024-09-21T19:05:08.809254+0000","last_clean":"2024-09-21T19:05:08.809254+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:05:08.809254+0000","last_undegraded":"2024-09-21T19:05:08.809254+0000","last_fullsized":"2024-09-21T19:05:08.809254+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:14.376671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:14.376671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.783406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788443+0000","last_change":"2024-09-21T19:03:42.617727+0000","last_active":"2024-09-21T19:05:04.788443+0000","last_peered":"2024-09-21T19:05:04.788443+0000","last_clean":"2024-09-21T19:05:04.788443+0000","last_became_active":"2024-09-21T18:55:51.535080+0000","last_became_peered":"2024-09-21T18:55:51.535080+0000","last_unstale":"2024-09-21T19:05:04.788443+0000","last_undegraded":"2024-09-21T19:05:04.788443+0000","last_fullsized":"2024-09-21T19:05:04.788443+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:11.806920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780062+0000","last_change":"2024-09-21T19:05:00.731897+0000","last_active":"2024-09-21T19:05:04.780062+0000","last_peered":"2024-09-21T19:05:04.780062+0000","last_clean":"2024-09-21T19:05:04.780062+0000","last_became_active":"2024-09-21T19:05:00.731556+0000","last_became_peered":"2024-09-21T19:05:00.731556+0000","last_unstale":"2024-09-21T19:05:04.780062+0000","last_undegraded":"2024-09-21T19:05:04.780062+0000","last_fullsized":"2024-09-21T19:05:04.780062+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:45.880385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:59:45.880385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"308'12248","reported_seq":13361,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.755194+0000","last_change":"2024-09-21T19:02:46.953650+0000","last_active":"2024-09-21T19:05:07.755194+0000","last_peered":"2024-09-21T19:05:07.755194+0000","last_clean":"2024-09-21T19:05:07.755194+0000","last_became_active":"2024-09-21T19:02:46.951397+0000","last_became_peered":"2024-09-21T19:02:46.951397+0000","last_unstale":"2024-09-21T19:05:07.755194+0000","last_undegraded":"2024-09-21T19:05:07.755194+0000","last_fullsized":"2024-09-21T19:05:07.755194+0000","mapping_epoch":264,"log_start":"293'7991","ondisk_log_start":"293'7991","created":254,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:52.660293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238551040,"num_objects":3640,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12248,"num_write_kb":508416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":32243712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":632,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780930+0000","last_change":"2024-09-21T19:05:00.733833+0000","last_active":"2024-09-21T19:05:04.780930+0000","last_peered":"2024-09-21T19:05:04.780930+0000","last_clean":"2024-09-21T19:05:04.780930+0000","last_became_active":"2024-09-21T19:05:00.733328+0000","last_became_peered":"2024-09-21T19:05:00.733328+0000","last_unstale":"2024-09-21T19:05:04.780930+0000","last_undegraded":"2024-09-21T19:05:04.780930+0000","last_fullsized":"2024-09-21T19:05:04.780930+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:48.127135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:48.127135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":571,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823574+0000","last_change":"2024-09-21T19:02:40.883840+0000","last_active":"2024-09-21T19:05:08.823574+0000","last_peered":"2024-09-21T19:05:08.823574+0000","last_clean":"2024-09-21T19:05:08.823574+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:05:08.823574+0000","last_undegraded":"2024-09-21T19:05:08.823574+0000","last_fullsized":"2024-09-21T19:05:08.823574+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-21T18:59:53.699556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T18:59:53.699556+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.862643+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":569,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779357+0000","last_change":"2024-09-21T19:01:03.984566+0000","last_active":"2024-09-21T19:05:04.779357+0000","last_peered":"2024-09-21T19:05:04.779357+0000","last_clean":"2024-09-21T19:05:04.779357+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:05:04.779357+0000","last_undegraded":"2024-09-21T19:05:04.779357+0000","last_fullsized":"2024-09-21T19:05:04.779357+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:01:03.984423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:01:03.984423+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:41.593624+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"306'9","reported_seq":49,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781992+0000","last_change":"2024-09-21T19:05:00.797943+0000","last_active":"2024-09-21T19:05:04.781992+0000","last_peered":"2024-09-21T19:05:04.781992+0000","last_clean":"2024-09-21T19:05:04.781992+0000","last_became_active":"2024-09-21T19:05:00.736651+0000","last_became_peered":"2024-09-21T19:05:00.736651+0000","last_unstale":"2024-09-21T19:05:04.781992+0000","last_undegraded":"2024-09-21T19:05:04.781992+0000","last_fullsized":"2024-09-21T19:05:04.781992+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:20.037881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:20.037881+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":537,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822833+0000","last_change":"2024-09-21T19:05:04.777379+0000","last_active":"2024-09-21T19:05:08.822833+0000","last_peered":"2024-09-21T19:05:08.822833+0000","last_clean":"2024-09-21T19:05:08.822833+0000","last_became_active":"2024-09-21T19:05:00.734846+0000","last_became_peered":"2024-09-21T19:05:00.734846+0000","last_unstale":"2024-09-21T19:05:08.822833+0000","last_undegraded":"2024-09-21T19:05:08.822833+0000","last_fullsized":"2024-09-21T19:05:08.822833+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:04.777314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:04.777314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:06.238149+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"308'12502","reported_seq":14228,"reported_epoch":308,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:05:07.457297+0000","last_change":"2024-09-21T19:05:01.844747+0000","last_active":"2024-09-21T19:05:07.457297+0000","last_peered":"2024-09-21T19:05:07.457297+0000","last_clean":"2024-09-21T19:04:59.703922+0000","last_became_active":"2024-09-21T19:05:01.797089+0000","last_became_peered":"2024-09-21T19:05:01.797089+0000","last_unstale":"2024-09-21T19:05:07.457297+0000","last_undegraded":"2024-09-21T19:05:07.457297+0000","last_fullsized":"2024-09-21T19:05:07.457297+0000","mapping_epoch":306,"log_start":"294'8335","ondisk_log_start":"294'8335","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'10031","last_scrub_stamp":"2024-09-21T19:04:32.782482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:04:32.782482+0000","objects_scrubbed":6176,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236191744,"num_objects":3604,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2890,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12502,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":3604}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":431,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790395+0000","last_change":"2024-09-21T19:03:52.722880+0000","last_active":"2024-09-21T19:05:04.790395+0000","last_peered":"2024-09-21T19:05:04.790395+0000","last_clean":"2024-09-21T19:05:04.790395+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:05:04.790395+0000","last_undegraded":"2024-09-21T19:05:04.790395+0000","last_fullsized":"2024-09-21T19:05:04.790395+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:03:52.722797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.722797+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.206303+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":527,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.782389+0000","last_change":"2024-09-21T19:03:09.630384+0000","last_active":"2024-09-21T19:05:04.782389+0000","last_peered":"2024-09-21T19:05:04.782389+0000","last_clean":"2024-09-21T19:05:04.782389+0000","last_became_active":"2024-09-21T19:03:09.629753+0000","last_became_peered":"2024-09-21T19:03:09.629753+0000","last_unstale":"2024-09-21T19:05:04.782389+0000","last_undegraded":"2024-09-21T19:05:04.782389+0000","last_fullsized":"2024-09-21T19:05:04.782389+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":19,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822297+0000","last_change":"2024-09-21T19:05:00.732161+0000","last_active":"2024-09-21T19:05:08.822297+0000","last_peered":"2024-09-21T19:05:08.822297+0000","last_clean":"2024-09-21T19:05:08.822297+0000","last_became_active":"2024-09-21T19:05:00.731837+0000","last_became_peered":"2024-09-21T19:05:00.731837+0000","last_unstale":"2024-09-21T19:05:08.822297+0000","last_undegraded":"2024-09-21T19:05:08.822297+0000","last_fullsized":"2024-09-21T19:05:08.822297+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:09.563414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":427,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788098+0000","last_change":"2024-09-21T19:05:01.901347+0000","last_active":"2024-09-21T19:05:04.788098+0000","last_peered":"2024-09-21T19:05:04.788098+0000","last_clean":"2024-09-21T19:05:04.788098+0000","last_became_active":"2024-09-21T19:05:00.734219+0000","last_became_peered":"2024-09-21T19:05:00.734219+0000","last_unstale":"2024-09-21T19:05:04.788098+0000","last_undegraded":"2024-09-21T19:05:04.788098+0000","last_fullsized":"2024-09-21T19:05:04.788098+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:14.095602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"308'12379","reported_seq":12578,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:06.784762+0000","last_change":"2024-09-21T19:03:47.989682+0000","last_active":"2024-09-21T19:05:06.784762+0000","last_peered":"2024-09-21T19:05:06.784762+0000","last_clean":"2024-09-21T19:05:06.784762+0000","last_became_active":"2024-09-21T19:03:09.625334+0000","last_became_peered":"2024-09-21T19:03:09.625334+0000","last_unstale":"2024-09-21T19:05:06.784762+0000","last_undegraded":"2024-09-21T19:05:06.784762+0000","last_fullsized":"2024-09-21T19:05:06.784762+0000","mapping_epoch":274,"log_start":"293'8112","ondisk_log_start":"293'8112","created":254,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'6556","last_scrub_stamp":"2024-09-21T19:03:47.989483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.989483+0000","objects_scrubbed":6505,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240058368,"num_objects":3663,"num_object_clones":0,"num_object_copies":10989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":482,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.809121+0000","last_change":"2024-09-21T19:02:40.878888+0000","last_active":"2024-09-21T19:05:08.809121+0000","last_peered":"2024-09-21T19:05:08.809121+0000","last_clean":"2024-09-21T19:05:08.809121+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:05:08.809121+0000","last_undegraded":"2024-09-21T19:05:08.809121+0000","last_fullsized":"2024-09-21T19:05:08.809121+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-21T19:00:23.328085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:23.328085+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:14.891844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":558,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.783567+0000","last_change":"2024-09-21T19:04:29.259617+0000","last_active":"2024-09-21T19:05:04.783567+0000","last_peered":"2024-09-21T19:05:04.783567+0000","last_clean":"2024-09-21T19:05:04.783567+0000","last_became_active":"2024-09-21T19:02:40.888713+0000","last_became_peered":"2024-09-21T19:02:40.888713+0000","last_unstale":"2024-09-21T19:05:04.783567+0000","last_undegraded":"2024-09-21T19:05:04.783567+0000","last_fullsized":"2024-09-21T19:05:04.783567+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:57.089123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":145,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788643+0000","last_change":"2024-09-21T19:05:00.790698+0000","last_active":"2024-09-21T19:05:04.788643+0000","last_peered":"2024-09-21T19:05:04.788643+0000","last_clean":"2024-09-21T19:05:04.788643+0000","last_became_active":"2024-09-21T19:05:00.789388+0000","last_became_peered":"2024-09-21T19:05:00.789388+0000","last_unstale":"2024-09-21T19:05:04.788643+0000","last_undegraded":"2024-09-21T19:05:04.788643+0000","last_fullsized":"2024-09-21T19:05:04.788643+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":526,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780709+0000","last_change":"2024-09-21T19:05:00.732771+0000","last_active":"2024-09-21T19:05:04.780709+0000","last_peered":"2024-09-21T19:05:04.780709+0000","last_clean":"2024-09-21T19:05:04.780709+0000","last_became_active":"2024-09-21T19:05:00.732463+0000","last_became_peered":"2024-09-21T19:05:00.732463+0000","last_unstale":"2024-09-21T19:05:04.780709+0000","last_undegraded":"2024-09-21T19:05:04.780709+0000","last_fullsized":"2024-09-21T19:05:04.780709+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:20.122545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T18:57:20.122545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"308'12180","reported_seq":12300,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.374875+0000","last_change":"2024-09-21T19:02:30.708593+0000","last_active":"2024-09-21T19:05:07.374875+0000","last_peered":"2024-09-21T19:05:07.374875+0000","last_clean":"2024-09-21T19:05:07.374875+0000","last_became_active":"2024-09-21T19:02:30.708207+0000","last_became_peered":"2024-09-21T19:02:30.708207+0000","last_unstale":"2024-09-21T19:05:07.374875+0000","last_undegraded":"2024-09-21T19:05:07.374875+0000","last_fullsized":"2024-09-21T19:05:07.374875+0000","mapping_epoch":254,"log_start":"293'7991","ondisk_log_start":"293'7991","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:17.754754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233439232,"num_objects":3562,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12180,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":589,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822292+0000","last_change":"2024-09-21T19:02:04.675992+0000","last_active":"2024-09-21T19:05:08.822292+0000","last_peered":"2024-09-21T19:05:08.822292+0000","last_clean":"2024-09-21T19:05:08.822292+0000","last_became_active":"2024-09-21T18:54:36.753332+0000","last_became_peered":"2024-09-21T18:54:36.753332+0000","last_unstale":"2024-09-21T19:05:08.822292+0000","last_undegraded":"2024-09-21T19:05:08.822292+0000","last_fullsized":"2024-09-21T19:05:08.822292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-21T19:02:04.675911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:04.675911+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:41.693589+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":555,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781814+0000","last_change":"2024-09-21T19:03:47.143498+0000","last_active":"2024-09-21T19:05:04.781814+0000","last_peered":"2024-09-21T19:05:04.781814+0000","last_clean":"2024-09-21T19:05:04.781814+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:05:04.781814+0000","last_undegraded":"2024-09-21T19:05:04.781814+0000","last_fullsized":"2024-09-21T19:05:04.781814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:47.143439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:47.143439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":153,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.787836+0000","last_change":"2024-09-21T19:05:00.799338+0000","last_active":"2024-09-21T19:05:04.787836+0000","last_peered":"2024-09-21T19:05:04.787836+0000","last_clean":"2024-09-21T19:05:04.787836+0000","last_became_active":"2024-09-21T19:05:00.798540+0000","last_became_peered":"2024-09-21T19:05:00.798540+0000","last_unstale":"2024-09-21T19:05:04.787836+0000","last_undegraded":"2024-09-21T19:05:04.787836+0000","last_fullsized":"2024-09-21T19:05:04.787836+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":521,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779054+0000","last_change":"2024-09-21T19:03:20.446083+0000","last_active":"2024-09-21T19:05:04.779054+0000","last_peered":"2024-09-21T19:05:04.779054+0000","last_clean":"2024-09-21T19:05:04.779054+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:05:04.779054+0000","last_undegraded":"2024-09-21T19:05:04.779054+0000","last_fullsized":"2024-09-21T19:05:04.779054+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:20.446014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:20.446014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"308'12271","reported_seq":12391,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:07.173637+0000","last_change":"2024-09-21T19:02:29.787669+0000","last_active":"2024-09-21T19:05:07.173637+0000","last_peered":"2024-09-21T19:05:07.173637+0000","last_clean":"2024-09-21T19:05:07.173637+0000","last_became_active":"2024-09-21T19:02:29.787259+0000","last_became_peered":"2024-09-21T19:02:29.787259+0000","last_unstale":"2024-09-21T19:05:07.173637+0000","last_undegraded":"2024-09-21T19:05:07.173637+0000","last_fullsized":"2024-09-21T19:05:07.173637+0000","mapping_epoch":254,"log_start":"292'7490","ondisk_log_start":"292'7490","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:02:28.762973+0000","last_clean_scrub_stamp":"2024-09-21T19:02:28.762973+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:42.877228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226951168,"num_objects":3463,"num_object_clones":0,"num_object_copies":10389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12271,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":150,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780606+0000","last_change":"2024-09-21T19:03:56.136641+0000","last_active":"2024-09-21T19:05:04.780606+0000","last_peered":"2024-09-21T19:05:04.780606+0000","last_clean":"2024-09-21T19:05:04.780606+0000","last_became_active":"2024-09-21T19:03:09.619439+0000","last_became_peered":"2024-09-21T19:03:09.619439+0000","last_unstale":"2024-09-21T19:05:04.780606+0000","last_undegraded":"2024-09-21T19:05:04.780606+0000","last_fullsized":"2024-09-21T19:05:04.780606+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":569,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781525+0000","last_change":"2024-09-21T19:02:06.757027+0000","last_active":"2024-09-21T19:05:04.781525+0000","last_peered":"2024-09-21T19:05:04.781525+0000","last_clean":"2024-09-21T19:05:04.781525+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:05:04.781525+0000","last_undegraded":"2024-09-21T19:05:04.781525+0000","last_fullsized":"2024-09-21T19:05:04.781525+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:02:06.756954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:06.756954+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:38.273019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"263'10","reported_seq":707,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780797+0000","last_change":"2024-09-21T19:02:46.144415+0000","last_active":"2024-09-21T19:05:04.780797+0000","last_peered":"2024-09-21T19:05:04.780797+0000","last_clean":"2024-09-21T19:05:04.780797+0000","last_became_active":"2024-09-21T19:02:40.887248+0000","last_became_peered":"2024-09-21T19:02:40.887248+0000","last_unstale":"2024-09-21T19:05:04.780797+0000","last_undegraded":"2024-09-21T19:05:04.780797+0000","last_fullsized":"2024-09-21T19:05:04.780797+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:47.243633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":555,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.808968+0000","last_change":"2024-09-21T19:04:24.782721+0000","last_active":"2024-09-21T19:05:08.808968+0000","last_peered":"2024-09-21T19:05:08.808968+0000","last_clean":"2024-09-21T19:05:08.808968+0000","last_became_active":"2024-09-21T19:03:09.618375+0000","last_became_peered":"2024-09-21T19:03:09.618375+0000","last_unstale":"2024-09-21T19:05:08.808968+0000","last_undegraded":"2024-09-21T19:05:08.808968+0000","last_fullsized":"2024-09-21T19:05:08.808968+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:26.072336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":483,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.809446+0000","last_change":"2024-09-21T19:02:40.882545+0000","last_active":"2024-09-21T19:05:08.809446+0000","last_peered":"2024-09-21T19:05:08.809446+0000","last_clean":"2024-09-21T19:05:08.809446+0000","last_became_active":"2024-09-21T19:02:40.882216+0000","last_became_peered":"2024-09-21T19:02:40.882216+0000","last_unstale":"2024-09-21T19:05:08.809446+0000","last_undegraded":"2024-09-21T19:05:08.809446+0000","last_fullsized":"2024-09-21T19:05:08.809446+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:14.180359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:59:14.180359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":562,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780784+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:05:04.780784+0000","last_peered":"2024-09-21T19:05:04.780784+0000","last_clean":"2024-09-21T19:05:04.780784+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:05:04.780784+0000","last_undegraded":"2024-09-21T19:05:04.780784+0000","last_fullsized":"2024-09-21T19:05:04.780784+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:59.874264+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779494+0000","last_change":"2024-09-21T19:03:11.520901+0000","last_active":"2024-09-21T19:05:04.779494+0000","last_peered":"2024-09-21T19:05:04.779494+0000","last_clean":"2024-09-21T19:05:04.779494+0000","last_became_active":"2024-09-21T19:03:09.623017+0000","last_became_peered":"2024-09-21T19:03:09.623017+0000","last_unstale":"2024-09-21T19:05:04.779494+0000","last_undegraded":"2024-09-21T19:05:04.779494+0000","last_fullsized":"2024-09-21T19:05:04.779494+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":529,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781828+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:05:04.781828+0000","last_peered":"2024-09-21T19:05:04.781828+0000","last_clean":"2024-09-21T19:05:04.781828+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:05:04.781828+0000","last_undegraded":"2024-09-21T19:05:04.781828+0000","last_fullsized":"2024-09-21T19:05:04.781828+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.216393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":438,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789599+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:05:04.789599+0000","last_peered":"2024-09-21T19:05:04.789599+0000","last_clean":"2024-09-21T19:05:04.789599+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:05:04.789599+0000","last_undegraded":"2024-09-21T19:05:04.789599+0000","last_fullsized":"2024-09-21T19:05:04.789599+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:15.701573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":550,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781049+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:05:04.781049+0000","last_peered":"2024-09-21T19:05:04.781049+0000","last_clean":"2024-09-21T19:05:04.781049+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:05:04.781049+0000","last_undegraded":"2024-09-21T19:05:04.781049+0000","last_fullsized":"2024-09-21T19:05:04.781049+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:08.889409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":524,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780531+0000","last_change":"2024-09-21T18:59:56.004177+0000","last_active":"2024-09-21T19:05:04.780531+0000","last_peered":"2024-09-21T19:05:04.780531+0000","last_clean":"2024-09-21T19:05:04.780531+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:05:04.780531+0000","last_undegraded":"2024-09-21T19:05:04.780531+0000","last_fullsized":"2024-09-21T19:05:04.780531+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:56.004101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:56.004101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.488303+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":19,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823809+0000","last_change":"2024-09-21T19:05:00.791156+0000","last_active":"2024-09-21T19:05:08.823809+0000","last_peered":"2024-09-21T19:05:08.823809+0000","last_clean":"2024-09-21T19:05:08.823809+0000","last_became_active":"2024-09-21T19:05:00.790818+0000","last_became_peered":"2024-09-21T19:05:00.790818+0000","last_unstale":"2024-09-21T19:05:08.823809+0000","last_undegraded":"2024-09-21T19:05:08.823809+0000","last_fullsized":"2024-09-21T19:05:08.823809+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.782155+0000","last_change":"2024-09-21T19:05:00.125178+0000","last_active":"2024-09-21T19:05:04.782155+0000","last_peered":"2024-09-21T19:05:04.782155+0000","last_clean":"2024-09-21T19:05:04.782155+0000","last_became_active":"2024-09-21T19:03:09.624551+0000","last_became_peered":"2024-09-21T19:03:09.624551+0000","last_unstale":"2024-09-21T19:05:04.782155+0000","last_undegraded":"2024-09-21T19:05:04.782155+0000","last_fullsized":"2024-09-21T19:05:04.782155+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:28.722804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":427,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.788322+0000","last_change":"2024-09-21T19:04:33.875553+0000","last_active":"2024-09-21T19:05:04.788322+0000","last_peered":"2024-09-21T19:05:04.788322+0000","last_clean":"2024-09-21T19:05:04.788322+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:05:04.788322+0000","last_undegraded":"2024-09-21T19:05:04.788322+0000","last_fullsized":"2024-09-21T19:05:04.788322+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.875457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.875457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.453777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"306'9","reported_seq":698,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781230+0000","last_change":"2024-09-21T19:05:00.796428+0000","last_active":"2024-09-21T19:05:04.781230+0000","last_peered":"2024-09-21T19:05:04.781230+0000","last_clean":"2024-09-21T19:05:04.781230+0000","last_became_active":"2024-09-21T19:05:00.729962+0000","last_became_peered":"2024-09-21T19:05:00.729962+0000","last_unstale":"2024-09-21T19:05:04.781230+0000","last_undegraded":"2024-09-21T19:05:04.781230+0000","last_fullsized":"2024-09-21T19:05:04.781230+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":536,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823304+0000","last_change":"2024-09-21T19:05:00.791367+0000","last_active":"2024-09-21T19:05:08.823304+0000","last_peered":"2024-09-21T19:05:08.823304+0000","last_clean":"2024-09-21T19:05:08.823304+0000","last_became_active":"2024-09-21T19:05:00.790905+0000","last_became_peered":"2024-09-21T19:05:00.790905+0000","last_unstale":"2024-09-21T19:05:08.823304+0000","last_undegraded":"2024-09-21T19:05:08.823304+0000","last_fullsized":"2024-09-21T19:05:08.823304+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":559,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.782286+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:05:04.782286+0000","last_peered":"2024-09-21T19:05:04.782286+0000","last_clean":"2024-09-21T19:05:04.782286+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:05:04.782286+0000","last_undegraded":"2024-09-21T19:05:04.782286+0000","last_fullsized":"2024-09-21T19:05:04.782286+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:43.100625+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":538,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781191+0000","last_change":"2024-09-21T19:05:00.728352+0000","last_active":"2024-09-21T19:05:04.781191+0000","last_peered":"2024-09-21T19:05:04.781191+0000","last_clean":"2024-09-21T19:05:04.781191+0000","last_became_active":"2024-09-21T19:05:00.727201+0000","last_became_peered":"2024-09-21T19:05:00.727201+0000","last_unstale":"2024-09-21T19:05:04.781191+0000","last_undegraded":"2024-09-21T19:05:04.781191+0000","last_fullsized":"2024-09-21T19:05:04.781191+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:57:16.078547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T18:57:16.078547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"275'11","reported_seq":136,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790284+0000","last_change":"2024-09-21T19:03:11.527080+0000","last_active":"2024-09-21T19:05:04.790284+0000","last_peered":"2024-09-21T19:05:04.790284+0000","last_clean":"2024-09-21T19:05:04.790284+0000","last_became_active":"2024-09-21T19:03:09.624143+0000","last_became_peered":"2024-09-21T19:03:09.624143+0000","last_unstale":"2024-09-21T19:05:04.790284+0000","last_undegraded":"2024-09-21T19:05:04.790284+0000","last_fullsized":"2024-09-21T19:05:04.790284+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:11.527007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.527007+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.827620+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":427,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789845+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:05:04.789845+0000","last_peered":"2024-09-21T19:05:04.789845+0000","last_clean":"2024-09-21T19:05:04.789845+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:05:04.789845+0000","last_undegraded":"2024-09-21T19:05:04.789845+0000","last_fullsized":"2024-09-21T19:05:04.789845+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:19.717779+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":626,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.822929+0000","last_change":"2024-09-21T19:05:00.807869+0000","last_active":"2024-09-21T19:05:08.822929+0000","last_peered":"2024-09-21T19:05:08.822929+0000","last_clean":"2024-09-21T19:05:08.822929+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:05:08.822929+0000","last_undegraded":"2024-09-21T19:05:08.822929+0000","last_fullsized":"2024-09-21T19:05:08.822929+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:04:29.685680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.685680+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:31.292414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":553,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.823935+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:05:08.823935+0000","last_peered":"2024-09-21T19:05:08.823935+0000","last_clean":"2024-09-21T19:05:08.823935+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:05:08.823935+0000","last_undegraded":"2024-09-21T19:05:08.823935+0000","last_fullsized":"2024-09-21T19:05:08.823935+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:48.105596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":540,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784020+0000","last_change":"2024-09-21T19:04:33.010279+0000","last_active":"2024-09-21T19:05:04.784020+0000","last_peered":"2024-09-21T19:05:04.784020+0000","last_clean":"2024-09-21T19:05:04.784020+0000","last_became_active":"2024-09-21T18:53:11.413104+0000","last_became_peered":"2024-09-21T18:53:11.413104+0000","last_unstale":"2024-09-21T19:05:04.784020+0000","last_undegraded":"2024-09-21T19:05:04.784020+0000","last_fullsized":"2024-09-21T19:05:04.784020+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:46.490106+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":543,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.778847+0000","last_change":"2024-09-21T19:04:58.663684+0000","last_active":"2024-09-21T19:05:04.778847+0000","last_peered":"2024-09-21T19:05:04.778847+0000","last_clean":"2024-09-21T19:05:04.778847+0000","last_became_active":"2024-09-21T18:55:51.516200+0000","last_became_peered":"2024-09-21T18:55:51.516200+0000","last_unstale":"2024-09-21T19:05:04.778847+0000","last_undegraded":"2024-09-21T19:05:04.778847+0000","last_fullsized":"2024-09-21T19:05:04.778847+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:25.305309+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":597,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.784165+0000","last_change":"2024-09-21T19:05:00.814228+0000","last_active":"2024-09-21T19:05:04.784165+0000","last_peered":"2024-09-21T19:05:04.784165+0000","last_clean":"2024-09-21T19:05:04.784165+0000","last_became_active":"2024-09-21T19:05:00.797773+0000","last_became_peered":"2024-09-21T19:05:00.797773+0000","last_unstale":"2024-09-21T19:05:04.784165+0000","last_undegraded":"2024-09-21T19:05:04.784165+0000","last_fullsized":"2024-09-21T19:05:04.784165+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:01:11.541310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.541310+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.782234+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:05:04.782234+0000","last_peered":"2024-09-21T19:05:04.782234+0000","last_clean":"2024-09-21T19:05:04.782234+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:05:04.782234+0000","last_undegraded":"2024-09-21T19:05:04.782234+0000","last_fullsized":"2024-09-21T19:05:04.782234+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:21.156153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":547,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779132+0000","last_change":"2024-09-21T19:02:36.480829+0000","last_active":"2024-09-21T19:05:04.779132+0000","last_peered":"2024-09-21T19:05:04.779132+0000","last_clean":"2024-09-21T19:05:04.779132+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:05:04.779132+0000","last_undegraded":"2024-09-21T19:05:04.779132+0000","last_fullsized":"2024-09-21T19:05:04.779132+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.480784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.480784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:54.679479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":427,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789084+0000","last_change":"2024-09-21T19:02:19.440104+0000","last_active":"2024-09-21T19:05:04.789084+0000","last_peered":"2024-09-21T19:05:04.789084+0000","last_clean":"2024-09-21T19:05:04.789084+0000","last_became_active":"2024-09-21T18:55:51.536042+0000","last_became_peered":"2024-09-21T18:55:51.536042+0000","last_unstale":"2024-09-21T19:05:04.789084+0000","last_undegraded":"2024-09-21T19:05:04.789084+0000","last_fullsized":"2024-09-21T19:05:04.789084+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:19.440059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:19.440059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:40.873594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":571,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779609+0000","last_change":"2024-09-21T19:02:40.882171+0000","last_active":"2024-09-21T19:05:04.779609+0000","last_peered":"2024-09-21T19:05:04.779609+0000","last_clean":"2024-09-21T19:05:04.779609+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:05:04.779609+0000","last_undegraded":"2024-09-21T19:05:04.779609+0000","last_fullsized":"2024-09-21T19:05:04.779609+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-21T19:00:00.939483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.939483+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":539,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.774608+0000","last_change":"2024-09-21T19:04:32.517465+0000","last_active":"2024-09-21T19:05:04.774608+0000","last_peered":"2024-09-21T19:05:04.774608+0000","last_clean":"2024-09-21T19:05:04.774608+0000","last_became_active":"2024-09-21T18:53:07.402170+0000","last_became_peered":"2024-09-21T18:53:07.402170+0000","last_unstale":"2024-09-21T19:05:04.774608+0000","last_undegraded":"2024-09-21T19:05:04.774608+0000","last_fullsized":"2024-09-21T19:05:04.774608+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:32.517394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:32.517394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:32.564881+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":558,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.780507+0000","last_change":"2024-09-21T19:04:57.155848+0000","last_active":"2024-09-21T19:05:04.780507+0000","last_peered":"2024-09-21T19:05:04.780507+0000","last_clean":"2024-09-21T19:05:04.780507+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:05:04.780507+0000","last_undegraded":"2024-09-21T19:05:04.780507+0000","last_fullsized":"2024-09-21T19:05:04.780507+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:57.155788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:57.155788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:06:06.790988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":554,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.779270+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:05:04.779270+0000","last_peered":"2024-09-21T19:05:04.779270+0000","last_clean":"2024-09-21T19:05:04.779270+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:05:04.779270+0000","last_undegraded":"2024-09-21T19:05:04.779270+0000","last_fullsized":"2024-09-21T19:05:04.779270+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:36.993219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":570,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781945+0000","last_change":"2024-09-21T19:02:59.672588+0000","last_active":"2024-09-21T19:05:04.781945+0000","last_peered":"2024-09-21T19:05:04.781945+0000","last_clean":"2024-09-21T19:05:04.781945+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:05:04.781945+0000","last_undegraded":"2024-09-21T19:05:04.781945+0000","last_fullsized":"2024-09-21T19:05:04.781945+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-21T19:02:41.535456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.535456+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":419,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790111+0000","last_change":"2024-09-21T19:03:55.627621+0000","last_active":"2024-09-21T19:05:04.790111+0000","last_peered":"2024-09-21T19:05:04.790111+0000","last_clean":"2024-09-21T19:05:04.790111+0000","last_became_active":"2024-09-21T18:55:51.528805+0000","last_became_peered":"2024-09-21T18:55:51.528805+0000","last_unstale":"2024-09-21T19:05:04.790111+0000","last_undegraded":"2024-09-21T19:05:04.790111+0000","last_fullsized":"2024-09-21T19:05:04.790111+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:10.004468+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":461,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.808685+0000","last_change":"2024-09-21T19:02:40.881419+0000","last_active":"2024-09-21T19:05:08.808685+0000","last_peered":"2024-09-21T19:05:08.808685+0000","last_clean":"2024-09-21T19:05:08.808685+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:05:08.808685+0000","last_undegraded":"2024-09-21T19:05:08.808685+0000","last_fullsized":"2024-09-21T19:05:08.808685+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:59:17.117512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T18:59:17.117512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.781026+0000","last_change":"2024-09-21T19:05:00.733098+0000","last_active":"2024-09-21T19:05:04.781026+0000","last_peered":"2024-09-21T19:05:04.781026+0000","last_clean":"2024-09-21T19:05:04.781026+0000","last_became_active":"2024-09-21T19:05:00.732734+0000","last_became_peered":"2024-09-21T19:05:00.732734+0000","last_unstale":"2024-09-21T19:05:04.781026+0000","last_undegraded":"2024-09-21T19:05:04.781026+0000","last_fullsized":"2024-09-21T19:05:04.781026+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":519,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.790149+0000","last_change":"2024-09-21T19:00:00.427554+0000","last_active":"2024-09-21T19:05:04.790149+0000","last_peered":"2024-09-21T19:05:04.790149+0000","last_clean":"2024-09-21T19:05:04.790149+0000","last_became_active":"2024-09-21T18:53:13.427178+0000","last_became_peered":"2024-09-21T18:53:13.427178+0000","last_unstale":"2024-09-21T19:05:04.790149+0000","last_undegraded":"2024-09-21T19:05:04.790149+0000","last_fullsized":"2024-09-21T19:05:04.790149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:00.427483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:00:00.427483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:41.084976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":562,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.782965+0000","last_change":"2024-09-21T19:03:18.435824+0000","last_active":"2024-09-21T19:05:04.782965+0000","last_peered":"2024-09-21T19:05:04.782965+0000","last_clean":"2024-09-21T19:05:04.782965+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:05:04.782965+0000","last_undegraded":"2024-09-21T19:05:04.782965+0000","last_fullsized":"2024-09-21T19:05:04.782965+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:18.435733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:18.435733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":523,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-09-21T19:05:08.811074+0000","last_change":"2024-09-21T19:03:17.580525+0000","last_active":"2024-09-21T19:05:08.811074+0000","last_peered":"2024-09-21T19:05:08.811074+0000","last_clean":"2024-09-21T19:05:08.811074+0000","last_became_active":"2024-09-21T19:02:40.882966+0000","last_became_peered":"2024-09-21T19:02:40.882966+0000","last_unstale":"2024-09-21T19:05:08.811074+0000","last_undegraded":"2024-09-21T19:05:08.811074+0000","last_fullsized":"2024-09-21T19:05:08.811074+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:05:38.872443+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":435,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T19:05:04.789778+0000","last_change":"2024-09-21T19:05:00.790758+0000","last_active":"2024-09-21T19:05:04.789778+0000","last_peered":"2024-09-21T19:05:04.789778+0000","last_clean":"2024-09-21T19:05:04.789778+0000","last_became_active":"2024-09-21T19:05:00.790195+0000","last_became_peered":"2024-09-21T19:05:00.790195+0000","last_unstale":"2024-09-21T19:05:04.789778+0000","last_undegraded":"2024-09-21T19:05:04.789778+0000","last_fullsized":"2024-09-21T19:05:04.789778+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:11.531855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:11.531855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3766353944,"num_objects":57471,"num_object_clones":0,"num_object_copies":172413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8713,"num_objects_unfound":0,"num_objects_dirty":57471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198235,"num_write_kb":8182529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3632,"num_bytes_recovered":238026752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11367165952,"data_stored":11367153736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69571,"ondisk_log_size":69571,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":4538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":682,"ondisk_log_size":682,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":708,"num_read_kb":703,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266836,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2622760,"kb_used_data":2144536,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91122392,"statfs":{"total":95995035648,"available":93309329408,"internally_reserved":0,"allocated":2196004864,"data_stored":2195612407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":489683402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464287,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":851076,"kb_used_data":734384,"kb_used_omap":4,"kb_used_meta":116667,"kb_avail":92894076,"statfs":{"total":95995035648,"available":95123533824,"internally_reserved":0,"allocated":752009216,"data_stored":751599668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":119467684},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790108,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2623912,"kb_used_data":2101204,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91121240,"statfs":{"total":95995035648,"available":93308149760,"internally_reserved":0,"allocated":2151632896,"data_stored":2151244670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019043,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994612,"kb_used_data":648288,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92750540,"statfs":{"total":95995035648,"available":94976552960,"internally_reserved":0,"allocated":663846912,"data_stored":663453445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182567,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2404992,"kb_used_data":1836528,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":91340160,"statfs":{"total":95995035648,"available":93532323840,"internally_reserved":0,"allocated":1880604672,"data_stored":1880189361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":582089162},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2665484,"kb_used_data":2066492,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91079668,"statfs":{"total":95995035648,"available":93265580032,"internally_reserved":0,"allocated":2116087808,"data_stored":2115659390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506868,"kb_used_data":952292,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92238284,"statfs":{"total":95995035648,"available":94452002816,"internally_reserved":0,"allocated":975147008,"data_stored":974743584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":971046912,"data_stored":971046912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2114785280,"data_stored":2114781208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":708575232,"data_stored":708575232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1874132992,"data_stored":1874132992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":649396224,"data_stored":649396224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2130907136,"data_stored":2130903064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":750780416,"data_stored":750780416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2167541760,"data_stored":2167537688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:05:14.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:14.208 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:14.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:13 smithi104 ceph-mon[32438]: from='client.16095 v1:172.21.15.81:0/1851973942' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:05:14.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2022771935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:14.310 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:14.342 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:14.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.479 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:14.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:14.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:14.598 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:05:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.631 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:14.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:14.699 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:14.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:14.809 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:14.852 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:14 smithi081 ceph-mon[39192]: pgmap v588: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 6239/156468 objects misplaced (3.987%); 25 MiB/s, 402 objects/s recovering 2024-09-21T19:05:14.852 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:14 smithi081 ceph-mon[29384]: pgmap v588: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 6239/156468 objects misplaced (3.987%); 25 MiB/s, 402 objects/s recovering 2024-09-21T19:05:14.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:14.944 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:14.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:15.045 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:15.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:15.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.211 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:15.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:14 smithi104 ceph-mon[32438]: pgmap v588: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 6239/156468 objects misplaced (3.987%); 25 MiB/s, 402 objects/s recovering 2024-09-21T19:05:15.313 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:15.358 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:15.446 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:15.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.527 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:15.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:15.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.658 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:15.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:15.762 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:15.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:15.846 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:05:15.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:15.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:15 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2398494604' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:15.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:15 smithi081 ceph-mon[29384]: from='client.35984 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:15.918 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:15 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2398494604' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:15.918 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:15 smithi081 ceph-mon[39192]: from='client.35984 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:15.927 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:15.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:16.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:16.068 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:16.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:16.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:16.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:15 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2398494604' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:16.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:15 smithi104 ceph-mon[32438]: from='client.35984 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:16.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:16.343 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:16.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:16.484 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:16.586 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:16.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:16.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:05:16.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:16.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:16.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:16.885 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:16.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:05:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:05:16.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[29384]: pgmap v589: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 6239/156468 objects misplaced (3.987%); 22 MiB/s, 358 objects/s recovering 2024-09-21T19:05:16.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:16.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[29384]: from='client.35984 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:16.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[29384]: osdmap e310: 8 total, 8 up, 7 in 2024-09-21T19:05:16.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[39192]: pgmap v589: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 6239/156468 objects misplaced (3.987%); 22 MiB/s, 358 objects/s recovering 2024-09-21T19:05:16.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:16.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[39192]: from='client.35984 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:16.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:16 smithi081 ceph-mon[39192]: osdmap e310: 8 total, 8 up, 7 in 2024-09-21T19:05:16.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:17.034 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:17.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:17.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:16 smithi104 ceph-mon[32438]: pgmap v589: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 6239/156468 objects misplaced (3.987%); 22 MiB/s, 358 objects/s recovering 2024-09-21T19:05:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:16 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:16 smithi104 ceph-mon[32438]: from='client.35984 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:16 smithi104 ceph-mon[32438]: osdmap e310: 8 total, 8 up, 7 in 2024-09-21T19:05:17.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:17.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:17.315 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:05:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:05:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:05:17.414 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:17.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:17.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.583 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:17.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:17.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:17.727 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:17.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:17.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:05:17.855 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:17.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:17.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:18.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.136 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:18.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:18.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:18.268 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:18.369 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:18.409 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:18.511 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:18.542 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:18.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:18.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:05:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:05:18.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:18.801 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:18.811 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:05:18.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:18.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:18.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:19.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:19.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:19.114 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:05:19.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:05:19.114 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T19:05:19.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T19:05:19.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:18 smithi081 ceph-mon[29384]: pgmap v591: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s; 4661/143082 objects misplaced (3.258%); 20 MiB/s, 325 objects/s recovering 2024-09-21T19:05:19.151 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:18 smithi081 ceph-mon[39192]: pgmap v591: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s; 4661/143082 objects misplaced (3.258%); 20 MiB/s, 325 objects/s recovering 2024-09-21T19:05:19.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:19.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:18 smithi104 ceph-mon[32438]: pgmap v591: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s; 4661/143082 objects misplaced (3.258%); 20 MiB/s, 325 objects/s recovering 2024-09-21T19:05:19.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:19.415 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:05:19.416 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:05:19.432 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:19.451 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:05:19.451 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:05:19.451 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:19.458 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:19.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:19.491 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:05:19.492 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:05:19.529 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:19.532 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:05:19.532 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:05:19.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:19.566 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:05:19.566 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:05:19.604 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:05:19.604 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:05:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:19.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:19.640 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:05:19.640 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:05:19.681 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:05:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:05:19.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:19.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:19.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:19.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:19.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:19.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:20.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:20.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:20.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3311813110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:20.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:20.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3311813110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:20.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3311813110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:20.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:20.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:20.448 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:20.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:20.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:20.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:20.722 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:20.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:20.859 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:20.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:20.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:20.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:21.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:21.120 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:21.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.256 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:21.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:20 smithi104 ceph-mon[32438]: pgmap v592: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.15k op/s; 3801/138546 objects misplaced (2.743%); 22 MiB/s, 353 objects/s recovering 2024-09-21T19:05:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:20 smithi081 ceph-mon[29384]: pgmap v592: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.15k op/s; 3801/138546 objects misplaced (2.743%); 22 MiB/s, 353 objects/s recovering 2024-09-21T19:05:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:20 smithi081 ceph-mon[39192]: pgmap v592: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.15k op/s; 3801/138546 objects misplaced (2.743%); 22 MiB/s, 353 objects/s recovering 2024-09-21T19:05:21.357 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.391 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:21.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:21.492 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:21.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:21.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:21.784 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:21.804 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:21.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:21.944 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:21.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:22.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.086 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:22.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:22.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:22.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:22.394 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:22.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:22.532 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:22.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:22.645 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:05:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:05:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:05:22.645 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:05:22.645 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:05:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.664 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:22.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:22.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:22.802 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:22.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:22.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: pgmap v593: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.15k op/s; 3801/138546 objects misplaced (2.743%); 22 MiB/s, 353 objects/s recovering 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:05:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:23.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:23.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: pgmap v593: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.15k op/s; 3801/138546 objects misplaced (2.743%); 22 MiB/s, 353 objects/s recovering 2024-09-21T19:05:23.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:23.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:23.042 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: pgmap v593: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.15k op/s; 3801/138546 objects misplaced (2.743%); 22 MiB/s, 353 objects/s recovering 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:05:23.043 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:05:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:23.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:23.199 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:23.301 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:23.332 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:23.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:23.472 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:23.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:05:23.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.628 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:23.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:23.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:23.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:23.860 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:23.910 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:23.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:24.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.048 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:24.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:24.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:24.204 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:24.286 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:24.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.353 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:24.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:24.370 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:24.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:24.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:24.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:24.628 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:24.724 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:05:24.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:24.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:24.905 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:24.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:25.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:24 smithi104 ceph-mon[32438]: pgmap v594: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.20k op/s; 1612/120558 objects misplaced (1.337%); 23 MiB/s, 367 objects/s recovering 2024-09-21T19:05:25.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/360184088' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:05:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:25.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:24 smithi081 ceph-mon[39192]: pgmap v594: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.20k op/s; 1612/120558 objects misplaced (1.337%); 23 MiB/s, 367 objects/s recovering 2024-09-21T19:05:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/360184088' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:05:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:24 smithi081 ceph-mon[29384]: pgmap v594: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.20k op/s; 1612/120558 objects misplaced (1.337%); 23 MiB/s, 367 objects/s recovering 2024-09-21T19:05:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/360184088' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:05:25.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:25.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:25.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:25.329 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:25.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:25.469 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:25.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:25.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:25.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:25.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:25.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:25.866 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:25.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:25.991 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:25.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:26.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:25 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:26.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/360184088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:05:26.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:25 smithi104 ceph-mon[32438]: osdmap e311: 8 total, 8 up, 7 in 2024-09-21T19:05:26.019 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:05:26.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:25 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:26.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/360184088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:05:26.045 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:25 smithi081 ceph-mon[39192]: osdmap e311: 8 total, 8 up, 7 in 2024-09-21T19:05:26.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:25 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:26.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/360184088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:05:26.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:25 smithi081 ceph-mon[29384]: osdmap e311: 8 total, 8 up, 7 in 2024-09-21T19:05:26.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:26.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:26.241 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:26.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:26.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:26.490 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:26.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:26.614 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:26.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:26.737 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:26.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:26.868 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:26.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:26.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:26.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:27.036 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:05:27.036 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:05:27.036 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[29384]: pgmap v596: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s; 1612/120558 objects misplaced (1.337%); 23 MiB/s, 374 objects/s recovering 2024-09-21T19:05:27.036 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[29384]: osdmap e312: 8 total, 8 up, 7 in 2024-09-21T19:05:27.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:05:27.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:05:27.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[39192]: pgmap v596: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s; 1612/120558 objects misplaced (1.337%); 23 MiB/s, 374 objects/s recovering 2024-09-21T19:05:27.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:26 smithi081 ceph-mon[39192]: osdmap e312: 8 total, 8 up, 7 in 2024-09-21T19:05:27.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:27.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:27.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:27.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:26 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:05:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:26 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:05:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:26 smithi104 ceph-mon[32438]: pgmap v596: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s; 1612/120558 objects misplaced (1.337%); 23 MiB/s, 374 objects/s recovering 2024-09-21T19:05:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:26 smithi104 ceph-mon[32438]: osdmap e312: 8 total, 8 up, 7 in 2024-09-21T19:05:27.308 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:05:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:05:27.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:27.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:27.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:27.483 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:27.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:27.607 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:27.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:27.730 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:27.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:27.853 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:27.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:27.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:28.053 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:05:28.053 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 313, 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T19:05:26.757243+0000", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T19:02:39.856486+0000", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T19:04:59.701094+0000", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5804032, 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "nodeep-scrub", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T19:05:28.054 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 19, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 9, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T19:05:28.055 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.056 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 11, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 11, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 11, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 11, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 11, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "170", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "168", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.059 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.1700000762939453, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.1700000762939453, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:05:28.061 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.062 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T19:05:28.063 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.064 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.065 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.067 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6700000762939453, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6700000762939453, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.75, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:05:28.068 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.069 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.070 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.071 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.072 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.073 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.9900000095367432, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.9900000095367432, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.75, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.75, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.074 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.075 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:05:28.076 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.077 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.078 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.079 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.2400000095367432, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.2400000095367432, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:05:28.080 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.081 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.082 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.083 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.084 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:05:28.085 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.4900000095367432, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.4900000095367432, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.309999942779541, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.309999942779541, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.086 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.087 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.088 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.089 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.090 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.091 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.2400000095367432, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.2400000095367432, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T19:05:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 9, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_2", 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T19:02:27.858840+0000", 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:05:28.093 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 16, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 16, 2024-09-21T19:05:28.094 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 16, 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 16, 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 16, 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:05:28.095 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "258", 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:05:28.096 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:05:28.097 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:05:28.098 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.9900000095367432, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.9900000095367432, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.75, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.75, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:05:28.099 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 305, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.100 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:05:28.101 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.102 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 312, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.104 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T19:05:28.105 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.106 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T19:05:28.107 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 0, 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 0, 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.108 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 274, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.109 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T19:05:28.110 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T19:05:28.111 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T19:05:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 312, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.113 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.114 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T19:05:28.116 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.117 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 305, 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.118 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.119 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T19:05:28.120 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.121 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 0.16912841796875, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 37, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 261, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.122 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2569144231 2024-09-21T19:05:28.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/2569144231", 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/2569144231", 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/2569144231", 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/2569144231", 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:05:28.125 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 45, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 238, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 261, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 305, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 239, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.127 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/2795996229", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/2795996229", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/2795996229", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/2795996229", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 305, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:05:28.130 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.131 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T19:05:28.133 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:05:25.567577+0000", 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:04:41.449456+0000", 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.135 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 65536, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:04:35.620036+0000", 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.136 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:04:45.396280+0000", 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:04:23.667414+0000", 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:04:21.769950+0000", 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.138 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:05:04.661648+0000", 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 238 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:05:04.440723+0000", 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.139 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.1d", 2024-09-21T19:05:28.140 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [], 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:05:28.141 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T19:05:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T19:05:28.143 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T19:05:28.144 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T19:05:28.144 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T19:05:28.144 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T19:05:28.144 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T19:05:28.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:28.144 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:05:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[29384]: osdmap e313: 8 total, 8 up, 7 in 2024-09-21T19:05:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[29384]: 5.1e scrub starts 2024-09-21T19:05:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[29384]: 5.1e scrub ok 2024-09-21T19:05:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2639239939' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:05:28.145 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[39192]: osdmap e313: 8 total, 8 up, 7 in 2024-09-21T19:05:28.145 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[39192]: 5.1e scrub starts 2024-09-21T19:05:28.145 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[39192]: 5.1e scrub ok 2024-09-21T19:05:28.145 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:27 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2639239939' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:05:28.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:28.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:27 smithi104 ceph-mon[32438]: osdmap e313: 8 total, 8 up, 7 in 2024-09-21T19:05:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:27 smithi104 ceph-mon[32438]: 5.1e scrub starts 2024-09-21T19:05:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:27 smithi104 ceph-mon[32438]: 5.1e scrub ok 2024-09-21T19:05:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:27 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2639239939' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:05:28.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:28.312 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:28.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:28.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:28.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:28.583 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:28.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:28.720 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:28.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:05:28.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.857 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:28.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:28.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:28.992 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:28.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[29384]: pgmap v599: 158 pgs: 2 peering, 1 active+remapped+backfilling, 155 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s; 365/110241 objects misplaced (0.331%); 31 MiB/s, 500 objects/s recovering 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[29384]: 3.10 scrub starts 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[29384]: 3.10 scrub ok 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[29384]: osdmap e314: 8 total, 8 up, 7 in 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T19:05:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T19:05:29.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[39192]: pgmap v599: 158 pgs: 2 peering, 1 active+remapped+backfilling, 155 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s; 365/110241 objects misplaced (0.331%); 31 MiB/s, 500 objects/s recovering 2024-09-21T19:05:29.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[39192]: 3.10 scrub starts 2024-09-21T19:05:29.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[39192]: 3.10 scrub ok 2024-09-21T19:05:29.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:28 smithi081 ceph-mon[39192]: osdmap e314: 8 total, 8 up, 7 in 2024-09-21T19:05:29.094 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:29.126 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:29.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T19:05:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T19:05:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-mon[32438]: pgmap v599: 158 pgs: 2 peering, 1 active+remapped+backfilling, 155 active+clean; 2.2 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s; 365/110241 objects misplaced (0.331%); 31 MiB/s, 500 objects/s recovering 2024-09-21T19:05:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-mon[32438]: 3.10 scrub starts 2024-09-21T19:05:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-mon[32438]: 3.10 scrub ok 2024-09-21T19:05:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:28 smithi104 ceph-mon[32438]: osdmap e314: 8 total, 8 up, 7 in 2024-09-21T19:05:29.267 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:29.369 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:29.409 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:29.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:29.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:29.686 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:29.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:29.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:29.953 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:29.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:30.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:30.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[39192]: 3.4 scrub starts 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[39192]: 3.4 scrub ok 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[39192]: 1.4 scrub starts 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[39192]: 1.4 scrub ok 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[29384]: 3.4 scrub starts 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[29384]: 3.4 scrub ok 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[29384]: 1.4 scrub starts 2024-09-21T19:05:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:29 smithi081 ceph-mon[29384]: 1.4 scrub ok 2024-09-21T19:05:30.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:30.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.224 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:30.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:29 smithi104 ceph-mon[32438]: 3.4 scrub starts 2024-09-21T19:05:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:29 smithi104 ceph-mon[32438]: 3.4 scrub ok 2024-09-21T19:05:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:29 smithi104 ceph-mon[32438]: 1.4 scrub starts 2024-09-21T19:05:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:29 smithi104 ceph-mon[32438]: 1.4 scrub ok 2024-09-21T19:05:30.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:30.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:30.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.501 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:30.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:30.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:30.635 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:30.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:30.771 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:30.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:30.908 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:30.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:31.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:31.042 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:31.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[29384]: pgmap v601: 158 pgs: 2 peering, 156 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 903 op/s; 19 MiB/s, 303 objects/s recovering 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[29384]: osdmap e315: 8 total, 8 up, 7 in 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[39192]: pgmap v601: 158 pgs: 2 peering, 156 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 903 op/s; 19 MiB/s, 303 objects/s recovering 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T19:05:31.063 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T19:05:31.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:30 smithi081 ceph-mon[39192]: osdmap e315: 8 total, 8 up, 7 in 2024-09-21T19:05:31.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:31.175 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:30 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:05:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:30 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:05:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:30 smithi104 ceph-mon[32438]: pgmap v601: 158 pgs: 2 peering, 156 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 903 op/s; 19 MiB/s, 303 objects/s recovering 2024-09-21T19:05:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:30 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T19:05:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:30 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T19:05:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:30 smithi104 ceph-mon[32438]: osdmap e315: 8 total, 8 up, 7 in 2024-09-21T19:05:31.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:31.316 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:31.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.453 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:31.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:31.554 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:31.596 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:31.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:31.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:31.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:31.998 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:31.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[29384]: 4.11 scrub starts 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[29384]: 4.11 scrub ok 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[39192]: 4.11 scrub starts 2024-09-21T19:05:32.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:31 smithi081 ceph-mon[39192]: 4.11 scrub ok 2024-09-21T19:05:32.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:32.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:32.232 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:31 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:05:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:31 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:05:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:31 smithi104 ceph-mon[32438]: 4.11 scrub starts 2024-09-21T19:05:32.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:31 smithi104 ceph-mon[32438]: 4.11 scrub ok 2024-09-21T19:05:32.263 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:32.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:32.399 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:32.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.524 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:32.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:32.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:32.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:32.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:32.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:32.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:32.998 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216350 7.6 7:693ca6d8:::benchmark_data_smithi081_73391_object77351:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.47573987599998, 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2616681970000001, 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:32.999 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216350 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518162+0000", 2024-09-21T19:05:33.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518164+0000", 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.593e-06 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518169+0000", 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6689999999999997e-06 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518176+0000", 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8180000000000004e-06 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518182+0000", 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2549999999999997e-06 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518270+0000", 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8738000000000003e-05 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518279+0000", 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2660000000000001e-06 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779775+0000", 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.26149613 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.779830+0000", 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5727999999999999e-05 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.003 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216360 7.6 7:6a035ea2:::benchmark_data_smithi081_73391_object77361:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.47571382199999, 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2839865960000001, 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.004 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216360 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518188+0000", 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518189+0000", 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5400000000000002e-07 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518191+0000", 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518196+0000", 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.622e-06 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518200+0000", 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6190000000000002e-06 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518299+0000", 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8587999999999999e-05 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.518304+0000", 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6210000000000001e-06 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780077+0000", 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2617725909999999 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780187+0000", 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110469 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802102+0000", 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021914330999999999 2024-09-21T19:05:33.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802175+0000", 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3116999999999998e-05 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216394 7.6 7:6d24eb15:::benchmark_data_smithi081_73391_object77395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.43488981600001, 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2297531310000001, 2024-09-21T19:05:33.009 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216394 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559012+0000", 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559014+0000", 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0889999999999998e-06 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559020+0000", 2024-09-21T19:05:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1669999999999998e-06 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559034+0000", 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4385e-05 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.559041+0000", 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.238e-06 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570552+0000", 2024-09-21T19:05:33.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011510347000000001 2024-09-21T19:05:33.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.570561+0000", 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6209999999999999e-06 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780222+0000", 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2096608149999999 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780290+0000", 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7988e-05 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.780361+0000", 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1239e-05 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788612+0000", 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082509909999999992 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788622+0000", 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0159999999999999e-05 2024-09-21T19:05:33.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788671+0000", 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8939e-05 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788709+0000", 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7295999999999997e-05 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788765+0000", 2024-09-21T19:05:33.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6855999999999999e-05 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216429 7.6 7:6000252a:::benchmark_data_smithi081_73391_object77430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.37742615299999, 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1724721330000001, 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.017 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216429 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616476+0000", 2024-09-21T19:05:33.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616478+0000", 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616484+0000", 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616495+0000", 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1134999999999999e-05 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616502+0000", 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8569999999999997e-06 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616558+0000", 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6023999999999998e-05 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.616567+0000", 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8980000000000002e-06 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782338+0000", 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.165771565 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782505+0000", 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166558 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.782607+0000", 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102357 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788626+0000", 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006018921 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788801+0000", 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000174852 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788853+0000", 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2006000000000003e-05 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788898+0000", 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4962000000000001e-05 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788948+0000", 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0006999999999997e-05 2024-09-21T19:05:33.023 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216455 7.6 7:68c88556:::benchmark_data_smithi081_73391_object77456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.348927377, 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.144373474, 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.024 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216455 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644975+0000", 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644977+0000", 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5500000000000001e-06 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644983+0000", 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0360000000000003e-06 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.644995+0000", 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2006e-05 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645003+0000", 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9459999999999998e-06 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645059+0000", 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5337999999999998e-05 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.645068+0000", 2024-09-21T19:05:33.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4320000000000005e-06 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.784935+0000", 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1398670499999999 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785006+0000", 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0840999999999995e-05 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.785084+0000", 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8228999999999999e-05 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.788630+0000", 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035455730000000002 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789144+0000", 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00051452099999999999 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789199+0000", 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4484000000000002e-05 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789251+0000", 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2633000000000002e-05 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789348+0000", 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6835000000000004e-05 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.030 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216459 7.6 7:66d9de63:::benchmark_data_smithi081_73391_object77460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.34433857599998, 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1586775789999999, 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216459 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.031 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649564+0000", 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649566+0000", 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0540000000000002e-06 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649586+0000", 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0472e-05 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649616+0000", 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9848e-05 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649637+0000", 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1114000000000001e-05 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649894+0000", 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025646600000000001 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.649902+0000", 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4589999999999998e-06 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789632+0000", 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1397303350000001 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789788+0000", 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155852 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.789900+0000", 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111909 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802209+0000", 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012309334 2024-09-21T19:05:33.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802251+0000", 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1421000000000002e-05 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808109+0000", 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058583660000000003 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808145+0000", 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5374999999999999e-05 2024-09-21T19:05:33.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808241+0000", 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6574000000000001e-05 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216478 7.6 7:69b4062c:::benchmark_data_smithi081_73391_object77479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.31995196999998, 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.137844316, 2024-09-21T19:05:33.037 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216478 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673950+0000", 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.673953+0000", 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3910000000000001e-06 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674028+0000", 2024-09-21T19:05:33.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5197000000000005e-05 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674184+0000", 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156129 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674193+0000", 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0410000000000003e-06 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674323+0000", 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013050400000000001 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.674368+0000", 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4211000000000001e-05 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792228+0000", 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.117859843 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792333+0000", 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105216 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.792492+0000", 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015956400000000001 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802291+0000", 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097984360000000006 2024-09-21T19:05:33.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802325+0000", 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4235e-05 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811655+0000", 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093300610000000006 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811704+0000", 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8520999999999998e-05 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811795+0000", 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0966999999999998e-05 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216483 7.6 7:679f3a63:::benchmark_data_smithi081_73391_object77484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.313424027, 2024-09-21T19:05:33.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1314383640000001, 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216483 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680478+0000", 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680480+0000", 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3010000000000001e-06 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680487+0000", 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8090000000000004e-06 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680532+0000", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4369999999999997e-05 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680540+0000", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5969999999999992e-06 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680600+0000", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0275000000000001e-05 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.680610+0000", 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6090000000000007e-06 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798740+0000", 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1181301029999999 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798848+0000", 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010757 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.798947+0000", 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9349e-05 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802432+0000", 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034852860000000002 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808361+0000", 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059282689999999999 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811802+0000", 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003440869 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811840+0000", 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8197e-05 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811916+0000", 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6760000000000004e-05 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.050 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216512 7.6 7:6f163cff:::benchmark_data_smithi081_73391_object77513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.27908473500003, 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0972389849999999, 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216512 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714817+0000", 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714820+0000", 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2450000000000001e-06 2024-09-21T19:05:33.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714826+0000", 2024-09-21T19:05:33.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1859999999999997e-06 2024-09-21T19:05:33.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714841+0000", 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5288999999999999e-05 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714850+0000", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6540000000000005e-06 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714963+0000", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112893 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.714971+0000", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7120000000000006e-06 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802465+0000", 2024-09-21T19:05:33.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0874939400000001 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802564+0000", 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.904e-05 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.802667+0000", 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103101 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808099+0000", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054315889999999997 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808411+0000", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00031148299999999999 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811922+0000", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035117199999999999 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.811958+0000", 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5766999999999998e-05 2024-09-21T19:05:33.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812056+0000", 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8366e-05 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216545 7.6 7:6b69332e:::benchmark_data_smithi081_73391_object77546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.24071322399999, 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0617420390000001, 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.058 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216545 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753189+0000", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753191+0000", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7680000000000001e-06 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753196+0000", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1410000000000004e-06 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753207+0000", 2024-09-21T19:05:33.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1143e-05 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.753214+0000", 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.917e-06 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754192+0000", 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097769299999999996 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.754200+0000", 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9400000000000002e-06 2024-09-21T19:05:33.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.805892+0000", 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0516927620000001 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806005+0000", 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113143 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.806125+0000", 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119331 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808476+0000", 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023510919999999999 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812065+0000", 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035891429999999999 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814743+0000", 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026784489999999998 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814839+0000", 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5358000000000004e-05 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814931+0000", 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2158999999999996e-05 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216552 7.6 7:63b154c8:::benchmark_data_smithi081_73391_object77553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.23186111799998, 2024-09-21T19:05:33.064 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.060947284, 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216552 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.065 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762041+0000", 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762043+0000", 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9889999999999999e-06 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762056+0000", 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3441999999999999e-05 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762068+0000", 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-05 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762075+0000", 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1169999999999998e-06 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762135+0000", 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9759000000000002e-05 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.762143+0000", 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808532+0000", 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0463883469999999 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808647+0000", 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011510600000000001 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.808757+0000", 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109849 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814943+0000", 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006186558 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.814980+0000", 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6729e-05 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822841+0000", 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078607529999999998 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822895+0000", 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4282e-05 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.822988+0000", 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3422000000000001e-05 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.070 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.34742.0:216555 7.6 7:6310baba:::benchmark_data_smithi081_73391_object77556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 493.23053527899998, 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0695222289999999, 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.34742", 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/1043106151", 2024-09-21T19:05:33.071 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 216555 2024-09-21T19:05:33.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.072 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:05:33.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.073 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763367+0000", 2024-09-21T19:05:33.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763369+0000", 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.635e-06 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763374+0000", 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.101e-06 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763503+0000", 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000129276 2024-09-21T19:05:33.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763511+0000", 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5019999999999992e-06 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763815+0000", 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030339199999999998 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:19.763823+0000", 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7600000000000002e-06 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812093+0000", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0482702930000001 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812199+0000", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010665100000000001 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.812307+0000", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107206 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819747+0000", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074399499999999999 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.819790+0000", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3297999999999998e-05 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832684+0000", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012894012 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832778+0000", 2024-09-21T19:05:33.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4456999999999994e-05 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T18:57:20.832889+0000", 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110696 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235880 8.1e 8:7f5b8aa9:::benchmark_data_smithi081_105538_object103648:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.319884242, 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.088566871, 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.078 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235880 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7300000000000001e-07 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674021+0000", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1380000000000001e-06 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674025+0000", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3499999999999999e-06 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674029+0000", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1690000000000002e-06 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842647+0000", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16861802400000001 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842654+0000", 2024-09-21T19:05:33.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2509999999999997e-06 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758247+0000", 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91559294899999999 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758376+0000", 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012917699999999999 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762523+0000", 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041469219999999999 2024-09-21T19:05:33.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762585+0000", 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1418e-05 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235892 8.c 8:34d14ac3:::benchmark_data_smithi081_105538_object103660:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 219.31986883799999, 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0724834350000001, 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.083 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235892 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674033+0000", 2024-09-21T19:05:33.084 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674034+0000", 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.27e-07 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674036+0000", 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1840000000000002e-06 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674040+0000", 2024-09-21T19:05:33.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0249999999999996e-06 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674044+0000", 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7160000000000001e-06 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.868789+0000", 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19474559399999999 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.868798+0000", 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5029999999999997e-06 2024-09-21T19:05:33.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.720825+0000", 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.85202745899999999 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.720942+0000", 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117217 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:05:33.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.721225+0000", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028250900000000002 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746281+0000", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025056334 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746375+0000", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4040000000000001e-05 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746416+0000", 2024-09-21T19:05:33.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0538999999999999e-05 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746478+0000", 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2225999999999997e-05 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746517+0000", 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8761999999999999e-05 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.089 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237693 8.21 8:86cd0139:::benchmark_data_smithi081_105538_object105461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 217.29446004900001, 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.19476636, 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237693 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.090 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699444+0000", 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999999999999999e-06 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.091 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699448+0000", 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9110000000000003e-06 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699456+0000", 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3259999999999996e-06 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699471+0000", 2024-09-21T19:05:33.092 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4681000000000001e-05 2024-09-21T19:05:33.093 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:32 smithi081 ceph-mon[29384]: pgmap v603: 158 pgs: 2 peering, 156 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 866 B/s rd, 0 B/s wr, 755 op/s; 16 MiB/s, 254 objects/s recovering 2024-09-21T19:05:33.093 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:32 smithi081 ceph-mon[39192]: pgmap v603: 158 pgs: 2 peering, 156 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 866 B/s rd, 0 B/s wr, 755 op/s; 16 MiB/s, 254 objects/s recovering 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700689+0000", 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012186180000000001 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700697+0000", 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4739999999999999e-06 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.094 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698487+0000", 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.99779048800000003 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698497+0000", 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1120000000000002e-06 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889014+0000", 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1905176980000001 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.095 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889174+0000", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000159545 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889403+0000", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022944299999999999 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891432+0000", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002028448 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891637+0000", 2024-09-21T19:05:33.096 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020514200000000001 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891727+0000", 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0705000000000007e-05 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.893997+0000", 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002269802 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894105+0000", 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107367 2024-09-21T19:05:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894208+0000", 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001038 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237741 8.21 8:86a63524:::benchmark_data_smithi081_105538_object105509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 217.27121941999999, 2024-09-21T19:05:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1717911829999998, 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237741 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722684+0000", 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0270000000000001e-06 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722688+0000", 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7929999999999999e-06 2024-09-21T19:05:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722697+0000", 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1290000000000002e-06 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722703+0000", 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6200000000000001e-06 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722891+0000", 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018788600000000001 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722902+0000", 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0541e-05 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702948+0000", 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98004672199999998 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702955+0000", 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0549999999999996e-06 2024-09-21T19:05:33.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890451+0000", 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1874960109999999 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890572+0000", 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012139800000000001 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890731+0000", 2024-09-21T19:05:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015913599999999999 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891711+0000", 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097963900000000003 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892154+0000", 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044374699999999998 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892258+0000", 2024-09-21T19:05:33.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010389400000000001 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894257+0000", 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019987550000000001 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894360+0000", 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010312200000000001 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894474+0000", 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113708 2024-09-21T19:05:33.105 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237675 8.1c 8:3e931efd:::benchmark_data_smithi081_105538_object105443:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.300551297, 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.207533752, 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.106 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237675 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693353+0000", 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9609999999999999e-06 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693358+0000", 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6030000000000004e-06 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694353+0000", 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00099412400000000009 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694362+0000", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3500000000000003e-06 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698188+0000", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038259919999999998 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698196+0000", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7100000000000007e-06 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900800+0000", 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2026040019999999 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900885+0000", 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5010000000000001e-05 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237706 8.1c 8:39f9b21e:::benchmark_data_smithi081_105538_object105474:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:33.110 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.29953027400001, 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2136628949999999, 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237706 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694373+0000", 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7300000000000005e-07 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694377+0000", 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5800000000000002e-06 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694385+0000", 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3030000000000002e-06 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694390+0000", 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4140000000000002e-06 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698219+0000", 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038285289999999998 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698225+0000", 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4269999999999996e-06 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901135+0000", 2024-09-21T19:05:33.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2029100589999999 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901255+0000", 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120221 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901527+0000", 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000272045 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907437+0000", 2024-09-21T19:05:33.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059092579999999997 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907688+0000", 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025114199999999999 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907731+0000", 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3180000000000003e-05 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907774+0000", 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3256000000000001e-05 2024-09-21T19:05:33.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907856+0000", 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1582000000000006e-05 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907917+0000", 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0930000000000001e-05 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908035+0000", 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118096 2024-09-21T19:05:33.117 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237735 8.1c 8:3aa9ee71:::benchmark_data_smithi081_105538_object105503:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.29950569100001, 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2197030149999999, 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.118 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237735 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694397+0000", 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7000000000000005e-07 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694400+0000", 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7470000000000001e-06 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694405+0000", 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6230000000000002e-06 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694410+0000", 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.972e-06 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698247+0000", 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038369789999999999 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698253+0000", 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9390000000000002e-06 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.904941+0000", 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.206687686 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905096+0000", 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155236 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905336+0000", 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024011 2024-09-21T19:05:33.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907815+0000", 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024790139999999999 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908076+0000", 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026105799999999999 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908089+0000", 2024-09-21T19:05:33.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2847000000000001e-05 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908126+0000", 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7177e-05 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913917+0000", 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057903410000000001 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913979+0000", 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2253000000000002e-05 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.914099+0000", 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120463 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237756 8.22 8:47738879:::benchmark_data_smithi081_105538_object105524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 216.299485941, 2024-09-21T19:05:33.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.228195653, 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237756 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694417+0000", 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7199999999999999e-07 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694419+0000", 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.447e-06 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694425+0000", 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4689999999999996e-06 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694430+0000", 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9099999999999996e-06 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701670+0000", 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0072406739999999999 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701679+0000", 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908648+0000", 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2069698309999999 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908761+0000", 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011227500000000001 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,3,4,5,7", 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908968+0000", 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020707700000000001 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912465+0000", 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003497207 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912477+0000", 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2086000000000001e-05 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913031+0000", 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055408400000000004 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913427+0000", 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039556599999999997 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913488+0000", 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1543000000000004e-05 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922372+0000", 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008883574 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922440+0000", 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7909000000000003e-05 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922612+0000", 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017206799999999999 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.133 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:33.133 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:05:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:33.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:32 smithi104 ceph-mon[32438]: pgmap v603: 158 pgs: 2 peering, 156 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 866 B/s rd, 0 B/s wr, 755 op/s; 16 MiB/s, 254 objects/s recovering 2024-09-21T19:05:33.286 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:33.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:33.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.557 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:33.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:33.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:33.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:33.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:33.833 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:33.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:33.972 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:33.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[29384]: 6.6 scrub starts 2024-09-21T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[29384]: 6.6 scrub ok 2024-09-21T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T19:05:34.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[39192]: 6.6 scrub starts 2024-09-21T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[39192]: 6.6 scrub ok 2024-09-21T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:33 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T19:05:34.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:34.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:34.241 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:33 smithi104 ceph-mon[32438]: 6.6 scrub starts 2024-09-21T19:05:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:33 smithi104 ceph-mon[32438]: 6.6 scrub ok 2024-09-21T19:05:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:33 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T19:05:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:33 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T19:05:34.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:34.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:05:34.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:34.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.510 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:34.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:34.611 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:34.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:34.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:34.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:34.925 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:34.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:34.950 INFO:tasks.thrashosds.thrasher:j is {'epoch': 313, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T19:05:26.757243+0000', 'last_up_change': '2024-09-21T19:02:39.856486+0000', 'last_in_change': '2024-09-21T19:04:59.701094+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '168', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-21T19:02:27.858840+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '258', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.16912841796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 2569144231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 2569144231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 2569144231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 2569144231}]}, 'public_addr': '172.21.15.104:6804/2569144231', 'cluster_addr': '172.21.15.104:6805/2569144231', 'heartbeat_back_addr': '172.21.15.104:6807/2569144231', 'heartbeat_front_addr': '172.21.15.104:6806/2569144231', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 45, 'last_clean_end': 238, 'up_from': 261, 'up_thru': 305, 'down_at': 239, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 2795996229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 2795996229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 2795996229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 2795996229}]}, 'public_addr': '172.21.15.104:6808/2795996229', 'cluster_addr': '172.21.15.104:6809/2795996229', 'heartbeat_back_addr': '172.21.15.104:6811/2795996229', 'heartbeat_front_addr': '172.21.15.104:6810/2795996229', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 305, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:05:25.567577+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:04:41.449456+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T19:04:35.620036+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:04:45.396280+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:04:23.667414+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:04:21.769950+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:05:04.661648+0000', 'dead_epoch': 238}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:05:04.440723+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 4, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T19:05:34.950 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T19:05:35.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: 3.16 scrub starts 2024-09-21T19:05:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: 3.16 scrub ok 2024-09-21T19:05:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: pgmap v604: 158 pgs: 158 active+clean; 1.8 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.57k op/s; 14 MiB/s, 217 objects/s recovering 2024-09-21T19:05:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: 5.11 scrub starts 2024-09-21T19:05:35.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: 5.11 scrub ok 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: 3.16 scrub starts 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: 3.16 scrub ok 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: pgmap v604: 158 pgs: 158 active+clean; 1.8 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.57k op/s; 14 MiB/s, 217 objects/s recovering 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: 5.11 scrub starts 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: 5.11 scrub ok 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T19:05:35.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:34 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T19:05:35.071 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:35.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:35.208 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: 3.16 scrub starts 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: 3.16 scrub ok 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: pgmap v604: 158 pgs: 158 active+clean; 1.8 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.57k op/s; 14 MiB/s, 217 objects/s recovering 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: 5.11 scrub starts 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: 5.11 scrub ok 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T19:05:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:34 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T19:05:35.310 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:35.340 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:35.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:35.475 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:35.577 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:35.613 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:35.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:35.748 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:35.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:35.883 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:35.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:35.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:05:35.985 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:36.012 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:36.088 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:36.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:36.141 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:36.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:36.276 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:36.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:36.415 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:36.516 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.550 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:36.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:36.686 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:36.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:36.819 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:36.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:36.955 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:36.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:37.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: pgmap v605: 158 pgs: 158 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 964 op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-21T19:05:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T19:05:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]} : dispatch 2024-09-21T19:05:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]} : dispatch 2024-09-21T19:05:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]} : dispatch 2024-09-21T19:05:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: pgmap v605: 158 pgs: 158 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 964 op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T19:05:37.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]} : dispatch 2024-09-21T19:05:37.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]} : dispatch 2024-09-21T19:05:37.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]} : dispatch 2024-09-21T19:05:37.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:36 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.098 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:37.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:37.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:37.231 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: pgmap v605: 158 pgs: 158 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 964 op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]} : dispatch 2024-09-21T19:05:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]} : dispatch 2024-09-21T19:05:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]} : dispatch 2024-09-21T19:05:37.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:36 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:37.303 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:05:37.303 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:05:37.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:37.337 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:05:37.337 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:05:37.337 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:05:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:05:37.365 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:37.367 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:05:37.367 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:05:37.399 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:05:37.399 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:05:37.431 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:05:37.431 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:05:37.465 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:05:37.466 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:05:37.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:37.495 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:05:37.496 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:05:37.498 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:37.530 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:05:37.530 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:05:37.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:37.633 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:37.734 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:37.772 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:37.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:37.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:38.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: 4.0 scrub starts 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: 4.0 scrub ok 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]}]': finished 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]}]': finished 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]': finished 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]': finished 2024-09-21T19:05:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: osdmap e316: 8 total, 8 up, 7 in 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: 3.13 scrub starts 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: 3.13 scrub ok 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1577277261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[39192]: osdmap e317: 8 total, 8 up, 7 in 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: 4.0 scrub starts 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: 4.0 scrub ok 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T19:05:38.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]}]': finished 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]}]': finished 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]': finished 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]': finished 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: osdmap e316: 8 total, 8 up, 7 in 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: 3.13 scrub starts 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: 3.13 scrub ok 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1577277261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:38.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:37 smithi081 ceph-mon[29384]: osdmap e317: 8 total, 8 up, 7 in 2024-09-21T19:05:38.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:38.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: 4.0 scrub starts 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: 4.0 scrub ok 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.9", "id": [7, 3]}]': finished 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.c", "id": [1, 6]}]': finished 2024-09-21T19:05:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]': finished 2024-09-21T19:05:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]': finished 2024-09-21T19:05:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: osdmap e316: 8 total, 8 up, 7 in 2024-09-21T19:05:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: 3.13 scrub starts 2024-09-21T19:05:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: 3.13 scrub ok 2024-09-21T19:05:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1577277261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:37 smithi104 ceph-mon[32438]: osdmap e317: 8 total, 8 up, 7 in 2024-09-21T19:05:38.280 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:38.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:38.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:38.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:38.548 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:38.558 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:38.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:38.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:38.730 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:38.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:05:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:05:38.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:38.860 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:05:38.865 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:38.967 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:38.994 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:38.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:39.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.130 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:39.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[39192]: pgmap v607: 158 pgs: 1 peering, 157 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/909167609' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[29384]: pgmap v607: 158 pgs: 1 peering, 157 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-21T19:05:39.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/909167609' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:39.231 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:38 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T19:05:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:38 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T19:05:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:38 smithi104 ceph-mon[32438]: pgmap v607: 158 pgs: 1 peering, 157 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-21T19:05:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/909167609' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.265 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:39.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:39.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.401 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:39.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:39.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:39.538 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:39.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:39.676 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:39.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:39.814 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:39.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:39.950 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:05:39.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:05:39.951 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-21T19:05:39.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd in 2 2024-09-21T19:05:39.955 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:05:39.956 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/909167609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: osdmap e318: 8 total, 8 up, 7 in 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: 6.8 scrub starts 2024-09-21T19:05:39.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[29384]: 6.8 scrub ok 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/909167609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: osdmap e318: 8 total, 8 up, 7 in 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: 6.8 scrub starts 2024-09-21T19:05:39.957 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:39 smithi081 ceph-mon[39192]: 6.8 scrub ok 2024-09-21T19:05:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:39.960 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:39.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:40.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:40.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:40.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/909167609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: osdmap e318: 8 total, 8 up, 7 in 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: 6.8 scrub starts 2024-09-21T19:05:40.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:39 smithi104 ceph-mon[32438]: 6.8 scrub ok 2024-09-21T19:05:40.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:40.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:40.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:40.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:40.368 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:40.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:40.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:40.512 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:40.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:40.613 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:05:40.614 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:40.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:40.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:40.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:05:40.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:40.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:40.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:40.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:05:40.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:05:40.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: pgmap v610: 158 pgs: 2 peering, 156 active+clean; 1.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:05:40.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:40.976 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: pgmap v610: 158 pgs: 2 peering, 156 active+clean; 1.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: 4.a scrub starts 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: 4.a scrub ok 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[29384]: osdmap e319: 8 total, 8 up, 7 in 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:05:40.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: pgmap v610: 158 pgs: 2 peering, 156 active+clean; 1.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: 4.a scrub starts 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: 4.a scrub ok 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:05:40.978 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:40 smithi081 ceph-mon[39192]: osdmap e319: 8 total, 8 up, 7 in 2024-09-21T19:05:41.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:41.132 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:41.163 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:05:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:05:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: 4.a scrub starts 2024-09-21T19:05:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: 4.a scrub ok 2024-09-21T19:05:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:05:41.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:40 smithi104 ceph-mon[32438]: osdmap e319: 8 total, 8 up, 7 in 2024-09-21T19:05:41.265 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:41.299 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:41.401 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:41.435 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:41.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:41.605 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:41.618 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:41.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:41.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:05:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:41.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.860 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:41.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:41.863 INFO:teuthology.orchestra.run.smithi081.stderr:marked in osd.2. 2024-09-21T19:05:41.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:41.982 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:41.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: 6.1a scrub starts 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: 6.1a scrub ok 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T19:05:42.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/679358025' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T19:05:41.834+0000 7ff1af1de640 -1 mon.a@0(leader).osd e319 definitely_dead 0 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: 6.1a scrub starts 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: 6.1a scrub ok 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T19:05:42.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/679358025' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-21T19:05:42.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:42.105 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:42.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:42.227 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:42.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T19:05:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T19:05:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: 6.1a scrub starts 2024-09-21T19:05:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: 6.1a scrub ok 2024-09-21T19:05:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T19:05:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T19:05:42.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/679358025' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-21T19:05:42.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:42.350 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:42.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:42.473 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:42.575 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:42.595 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:42.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.724 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:42.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:42.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:42.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:42.972 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:42.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:43.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:43.095 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:43.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:43.218 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:42 smithi104 ceph-mon[32438]: pgmap v612: 158 pgs: 2 peering, 156 active+clean; 1.3 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:05:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/679358025' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-21T19:05:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:42 smithi104 ceph-mon[32438]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T19:05:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:42 smithi104 ceph-mon[32438]: 5.2 scrub starts 2024-09-21T19:05:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:42 smithi104 ceph-mon[32438]: 5.2 scrub ok 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[29384]: pgmap v612: 158 pgs: 2 peering, 156 active+clean; 1.3 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/679358025' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[29384]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[29384]: 5.2 scrub starts 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[29384]: 5.2 scrub ok 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[39192]: pgmap v612: 158 pgs: 2 peering, 156 active+clean; 1.3 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/679358025' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[39192]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[39192]: 5.2 scrub starts 2024-09-21T19:05:43.275 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:42 smithi081 ceph-mon[39192]: 5.2 scrub ok 2024-09-21T19:05:43.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.345 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:43.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:43.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:43.467 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:43.569 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:43.590 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:43.692 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:43.713 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:43.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:43.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:43.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:43.959 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:43.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:44.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:44.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:44.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:44.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T19:05:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T19:05:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: 4.9 scrub starts 2024-09-21T19:05:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: 4.9 scrub ok 2024-09-21T19:05:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: 3.2 scrub starts 2024-09-21T19:05:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: 3.2 scrub ok 2024-09-21T19:05:44.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:43 smithi104 ceph-mon[32438]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T19:05:44.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:44.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: 4.9 scrub starts 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: 4.9 scrub ok 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: 3.2 scrub starts 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: 3.2 scrub ok 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[39192]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: 4.9 scrub starts 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: 4.9 scrub ok 2024-09-21T19:05:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: 3.2 scrub starts 2024-09-21T19:05:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: 3.2 scrub ok 2024-09-21T19:05:44.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:43 smithi081 ceph-mon[29384]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T19:05:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.335 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:44.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:44.436 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:44.459 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:44.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:44.584 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:05:44.584 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:05:44.600 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:44.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:44.723 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:44.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:44.856 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:44.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:44.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:44.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:45.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:45.108 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:45.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.246 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:45.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: pgmap v615: 158 pgs: 29 peering, 129 active+clean; 1015 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 31 KiB/s wr, 1.09k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 9.0 starting backfill to osd.2 from (297'9510,312'13689] MIN to 319'14975 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 9.6 starting backfill to osd.2 from (297'9635,312'13869] MIN to 319'15168 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: 9.2 starting backfill to osd.2 from (297'9238,311'13499] MIN to 319'14805 2024-09-21T19:05:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:44 smithi104 ceph-mon[32438]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: pgmap v615: 158 pgs: 29 peering, 129 active+clean; 1015 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 31 KiB/s wr, 1.09k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 9.0 starting backfill to osd.2 from (297'9510,312'13689] MIN to 319'14975 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 9.6 starting backfill to osd.2 from (297'9635,312'13869] MIN to 319'15168 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: 9.2 starting backfill to osd.2 from (297'9238,311'13499] MIN to 319'14805 2024-09-21T19:05:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[29384]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: pgmap v615: 158 pgs: 29 peering, 129 active+clean; 1015 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 31 KiB/s wr, 1.09k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 9.0 starting backfill to osd.2 from (297'9510,312'13689] MIN to 319'14975 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 9.6 starting backfill to osd.2 from (297'9635,312'13869] MIN to 319'15168 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: 9.2 starting backfill to osd.2 from (297'9238,311'13499] MIN to 319'14805 2024-09-21T19:05:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:44 smithi081 ceph-mon[39192]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T19:05:45.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:45.378 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:45.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:45.511 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:45.613 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:45.651 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:45.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:45.784 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:45.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:45.916 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:45.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:46.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:46.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:05:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.078 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:46.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:46.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:46.203 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:46.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[39192]: 5.3 scrub starts 2024-09-21T19:05:46.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[39192]: 5.3 scrub ok 2024-09-21T19:05:46.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T19:05:46.250 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T19:05:46.250 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[29384]: 5.3 scrub starts 2024-09-21T19:05:46.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[29384]: 5.3 scrub ok 2024-09-21T19:05:46.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T19:05:46.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:45 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T19:05:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:45 smithi104 ceph-mon[32438]: 5.3 scrub starts 2024-09-21T19:05:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:45 smithi104 ceph-mon[32438]: 5.3 scrub ok 2024-09-21T19:05:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:45 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T19:05:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:45 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T19:05:46.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:46.429 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:46.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:46.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:46.741 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-21T19:05:46.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:46.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:46.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:46.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:46.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[39192]: pgmap v617: 158 pgs: 29 peering, 129 active+clean; 1015 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 25 KiB/s wr, 902 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T19:05:46.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T19:05:46.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[29384]: pgmap v617: 158 pgs: 29 peering, 129 active+clean; 1015 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 25 KiB/s wr, 902 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-09-21T19:05:46.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T19:05:46.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:46 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T19:05:46.993 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:47.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:47.065 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:05:47.065 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:05:47.099 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:47.104 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:05:47.105 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:05:47.136 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:05:47.137 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:05:47.148 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:47.157 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:05:47.157 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:05:47.175 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:05:47.176 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:05:47.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:47.204 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:05:47.204 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:05:47.228 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:05:47.229 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:05:47.229 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:05:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:05:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.233 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:47.248 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:05:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:05:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:46 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T19:05:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:46 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T19:05:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:46 smithi104 ceph-mon[32438]: pgmap v617: 158 pgs: 29 peering, 129 active+clean; 1015 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 25 KiB/s wr, 902 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-09-21T19:05:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:46 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T19:05:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:46 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T19:05:47.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:47.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:47.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:47.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:47.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.641 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:47.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:47.766 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:47.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:47.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:48.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:48.029 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:48.131 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:48.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:48.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:47 smithi104 ceph-mon[32438]: 9.b scrub starts 2024-09-21T19:05:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:47 smithi104 ceph-mon[32438]: 9.b scrub ok 2024-09-21T19:05:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2794771078' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:48.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:48.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-mon[29384]: 9.b scrub starts 2024-09-21T19:05:48.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-mon[29384]: 9.b scrub ok 2024-09-21T19:05:48.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2794771078' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:48.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-mon[39192]: 9.b scrub starts 2024-09-21T19:05:48.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-mon[39192]: 9.b scrub ok 2024-09-21T19:05:48.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2794771078' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:48.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.403 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:48.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:48.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:48.525 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:48.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.654 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:48.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:48.755 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:48.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:05:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:05:48.781 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:48.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:48.906 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:48.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:49.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:49.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:49.167 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:48 smithi104 ceph-mon[32438]: pgmap v618: 158 pgs: 2 active+clean+remapped, 5 peering, 151 active+clean; 825 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s; 3.6 MiB/s, 62 objects/s recovering 2024-09-21T19:05:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:48 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T19:05:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:48 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:05:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:48 smithi104 ceph-mon[32438]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T19:05:49.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:49.289 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[39192]: pgmap v618: 158 pgs: 2 active+clean+remapped, 5 peering, 151 active+clean; 825 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s; 3.6 MiB/s, 62 objects/s recovering 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[39192]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[29384]: pgmap v618: 158 pgs: 2 active+clean+remapped, 5 peering, 151 active+clean; 825 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s; 3.6 MiB/s, 62 objects/s recovering 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:05:49.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:48 smithi081 ceph-mon[29384]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T19:05:49.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:49.424 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:49.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:49.546 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:49.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:49.692 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:49.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:49.828 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:49.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:49.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:50.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.104 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:50.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:50.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:50.235 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:50.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:49 smithi104 ceph-mon[32438]: 5.17 scrub starts 2024-09-21T19:05:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:49 smithi104 ceph-mon[32438]: 5.17 scrub ok 2024-09-21T19:05:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:49 smithi104 ceph-mon[32438]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T19:05:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:49 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T19:05:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:49 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T19:05:50.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:50.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[29384]: 5.17 scrub starts 2024-09-21T19:05:50.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[29384]: 5.17 scrub ok 2024-09-21T19:05:50.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[29384]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T19:05:50.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T19:05:50.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T19:05:50.286 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:50.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[39192]: 5.17 scrub starts 2024-09-21T19:05:50.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[39192]: 5.17 scrub ok 2024-09-21T19:05:50.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[39192]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T19:05:50.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T19:05:50.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:49 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T19:05:50.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.407 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:50.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:50.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:50.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:05:50.662 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235882 8.21 8:87f52933:::benchmark_data_smithi081_105538_object103650:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 236.97499345899999, 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0591238139999999, 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:50.663 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235882 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676578+0000", 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4799999999999997e-07 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676581+0000", 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0920000000000002e-06 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676588+0000", 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3090000000000001e-06 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676594+0000", 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1589999999999998e-06 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.795011+0000", 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.118417017 2024-09-21T19:05:50.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.795019+0000", 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0039999999999999e-06 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.735295+0000", 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.94027655099999996 2024-09-21T19:05:50.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.735701+0000", 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040593399999999999 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235888 8.a 8:559d0eda:::benchmark_data_smithi081_105538_object103656:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:05:50.668 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 236.97495996999999, 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.075440073, 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235888 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:05:50.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676614+0000", 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3970000000000001e-06 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676619+0000", 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2859999999999999e-06 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676624+0000", 2024-09-21T19:05:50.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.04e-06 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679419+0000", 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027943460000000001 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679424+0000", 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.215e-06 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876061+0000", 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19663730400000001 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876071+0000", 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6500000000000008e-06 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.751965+0000", 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87589449500000005 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752051+0000", 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5783000000000002e-05 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235893 8.a 8:56f22e24:::benchmark_data_smithi081_105538_object103661:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 236.97493915499999, 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1078101929999999, 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.674 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235893 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:05:50.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2e-07 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676634+0000", 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9439999999999999e-06 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676640+0000", 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6949999999999997e-06 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676644+0000", 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6739999999999996e-06 2024-09-21T19:05:50.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679477+0000", 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028324510000000002 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting_for_map not empty", 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679482+0000", 2024-09-21T19:05:50.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7180000000000004e-06 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876097+0000", 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19661515199999999 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876105+0000", 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.524e-06 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752411+0000", 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87630529599999996 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752544+0000", 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013372500000000001 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784273+0000", 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031728734000000001 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784442+0000", 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016886000000000001 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.683 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237696 8.1a 8:5de8ae1c:::benchmark_data_smithi081_105538_object105464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 233.95870560099999, 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2294143280000001, 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:50.684 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237696 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692868+0000", 2024-09-21T19:05:50.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3070000000000001e-06 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692875+0000", 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9779999999999999e-06 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692894+0000", 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9734000000000001e-05 2024-09-21T19:05:50.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692906+0000", 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1218e-05 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693046+0000", 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140288 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693056+0000", 2024-09-21T19:05:50.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0366999999999999e-05 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.918656+0000", 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2255999339999999 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.918766+0000", 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109454 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.919037+0000", 2024-09-21T19:05:50.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027156700000000001 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921749+0000", 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027122779999999998 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921789+0000", 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9567000000000003e-05 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921942+0000", 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000152612 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922003+0000", 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1637999999999995e-05 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922094+0000", 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0926000000000003e-05 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922144+0000", 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0142000000000002e-05 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922280+0000", 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013531799999999999 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237743 8.1a 8:5f844437:::benchmark_data_smithi081_105538_object105511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 233.95864153700001, 2024-09-21T19:05:50.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2389039639999999, 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237743 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692930+0000", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4099999999999998e-07 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692933+0000", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.371e-06 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692940+0000", 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4679999999999997e-06 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692945+0000", 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5600000000000001e-06 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693080+0000", 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000134232 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693085+0000", 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.57e-06 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922497+0000", 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.229411636 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922649+0000", 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015161100000000001 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:05:50.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922904+0000", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025589599999999998 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931394+0000", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084900319999999998 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931456+0000", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1847000000000006e-05 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931520+0000", 2024-09-21T19:05:50.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3540999999999994e-05 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931559+0000", 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9418000000000001e-05 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931652+0000", 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3154999999999998e-05 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931710+0000", 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7414999999999998e-05 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931833+0000", 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123471 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237758 8.1a 8:5b1e9ef3:::benchmark_data_smithi081_105538_object105526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:05:50.699 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 233.95861840699999, 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2525915910000001, 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237758 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692953+0000", 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7999999999999996e-07 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692955+0000", 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3410000000000001e-06 2024-09-21T19:05:50.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692961+0000", 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2209999999999996e-06 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692965+0000", 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7990000000000001e-06 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693105+0000", 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013992799999999999 2024-09-21T19:05:50.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693111+0000", 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924108+0000", 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.230996792 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924214+0000", 2024-09-21T19:05:50.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106278 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924401+0000", 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018696000000000001 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931404+0000", 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0070028060000000003 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931875+0000", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00047151400000000003 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931917+0000", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1340000000000001e-05 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.932034+0000", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011719500000000001 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945070+0000", 2024-09-21T19:05:50.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013035573 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945131+0000", 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1702e-05 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945544+0000", 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041286200000000001 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220084 9.0 9:0e4fd42a:::benchmark_data_smithi081_131274_object92229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.793318+0000", 2024-09-21T19:05:50.707 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.858252693000001, 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.122923267, 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220084 2024-09-21T19:05:50.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793318+0000", 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793318+0000", 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793320+0000", 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9300000000000002e-06 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793327+0000", 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5740000000000001e-06 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793341+0000", 2024-09-21T19:05:50.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.398e-05 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793349+0000", 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.225e-06 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793432+0000", 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2929000000000005e-05 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793440+0000", 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5669999999999995e-06 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913658+0000", 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1202174709999999 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913729+0000", 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1544000000000005e-05 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:05:50.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913801+0000", 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1413000000000002e-05 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916062+0000", 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022611839999999999 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916133+0000", 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0709000000000004e-05 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916149+0000", 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6439e-05 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916194+0000", 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4857000000000001e-05 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916241+0000", 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7444999999999997e-05 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.715 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220086 9.6 9:64ade3dc:::benchmark_data_smithi081_131274_object92231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.795252+0000", 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.856318567999999, 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.12467045, 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.716 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220086 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795252+0000", 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795252+0000", 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795254+0000", 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.734e-06 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795259+0000", 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2950000000000001e-06 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795272+0000", 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2602000000000001e-05 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795279+0000", 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1219999999999997e-06 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795338+0000", 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8900000000000002e-05 2024-09-21T19:05:50.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795347+0000", 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8899999999999996e-06 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913320+0000", 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1179732060000001 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913417+0000", 2024-09-21T19:05:50.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7256000000000005e-05 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913550+0000", 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013275799999999999 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.915789+0000", 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002238982 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919819+0000", 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040299159999999997 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919867+0000", 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7783e-05 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919897+0000", 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0576e-05 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919923+0000", 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5429999999999999e-05 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220106 9.0 9:05263109:::benchmark_data_smithi081_131274_object92251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.814872+0000", 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.836698825999999, 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1053909749999999, 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.723 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.724 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220106 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814872+0000", 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814872+0000", 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814874+0000", 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0949999999999998e-06 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814880+0000", 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4210000000000003e-06 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814896+0000", 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6657999999999999e-05 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814904+0000", 2024-09-21T19:05:50.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6850000000000001e-06 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814964+0000", 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9774000000000001e-05 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814972+0000", 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0860000000000002e-06 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916287+0000", 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.101315327 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916386+0000", 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9087999999999997e-05 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916506+0000", 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120146 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920079+0000", 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003572464 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920168+0000", 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9536999999999993e-05 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920183+0000", 2024-09-21T19:05:50.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4664999999999999e-05 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920224+0000", 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1270999999999998e-05 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920263+0000", 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8757999999999997e-05 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220112 9.0 9:03ea8174:::benchmark_data_smithi081_131274_object92257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.816780+0000", 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.834791024000001, 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1074493400000001, 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220112 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.732 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816780+0000", 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816780+0000", 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816781+0000", 2024-09-21T19:05:50.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4750000000000001e-06 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816786+0000", 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0239999999999997e-06 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816799+0000", 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2174000000000001e-05 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816806+0000", 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7260000000000002e-06 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816860+0000", 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4076000000000002e-05 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816866+0000", 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2240000000000003e-06 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918333+0000", 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.101467937 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918437+0000", 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103133 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918573+0000", 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013647799999999999 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920339+0000", 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001766297 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922149+0000", 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018096519999999999 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924130+0000", 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019811120000000002 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924185+0000", 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4701999999999999e-05 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924229+0000", 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4329999999999997e-05 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.738 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220117 9.6 9:6f3a98df:::benchmark_data_smithi081_131274_object92262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.822735+0000", 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.828835623, 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0975038029999999, 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220117 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822735+0000", 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822735+0000", 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822737+0000", 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.871e-06 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822743+0000", 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9680000000000003e-06 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822756+0000", 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2984e-05 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822763+0000", 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9199999999999998e-06 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822822+0000", 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9385000000000001e-05 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822830+0000", 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5909999999999998e-06 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.915826+0000", 2024-09-21T19:05:50.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.092996222 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.915926+0000", 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9735999999999995e-05 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916035+0000", 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108657 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919964+0000", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039297070000000002 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920037+0000", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2786000000000004e-05 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920150+0000", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112744 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920201+0000", 2024-09-21T19:05:50.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0902000000000002e-05 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920239+0000", 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8330000000000001e-05 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220122 9.0 9:0c6321ee:::benchmark_data_smithi081_131274_object92267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.825986+0000", 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.825584677999998, 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.100532968, 2024-09-21T19:05:50.745 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220122 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825986+0000", 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825986+0000", 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825988+0000", 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7069999999999999e-06 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825993+0000", 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4500000000000003e-06 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826006+0000", 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2561999999999999e-05 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826013+0000", 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0600000000000002e-06 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826061+0000", 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8186999999999998e-05 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826067+0000", 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.434e-06 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920392+0000", 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.094324946 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920507+0000", 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114973 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920619+0000", 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112753 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924297+0000", 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036775319999999998 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926375+0000", 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020784319999999998 2024-09-21T19:05:50.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926420+0000", 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4790999999999999e-05 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926476+0000", 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5685e-05 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926519+0000", 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3455999999999999e-05 2024-09-21T19:05:50.751 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220136 9.6 9:62065d13:::benchmark_data_smithi081_131274_object92281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.837800+0000", 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.813771288000002, 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.084514252, 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.752 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220136 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837800+0000", 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837800+0000", 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837801+0000", 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5319999999999999e-06 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837806+0000", 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7949999999999998e-06 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837817+0000", 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1245e-05 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837824+0000", 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7499999999999997e-06 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.838416+0000", 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000591809 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.838425+0000", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9409999999999996e-06 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918115+0000", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.079690332 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918208+0000", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2885000000000003e-05 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918278+0000", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0358999999999995e-05 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922124+0000", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038462309999999999 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922174+0000", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9570999999999997e-05 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922234+0000", 2024-09-21T19:05:50.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9478999999999998e-05 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922273+0000", 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9539999999999998e-05 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922314+0000", 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0782999999999999e-05 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220141 9.6 9:67bf545e:::benchmark_data_smithi081_131274_object92286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.841044+0000", 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.810527302000001, 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0901363319999999, 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220141 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841044+0000", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841044+0000", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841045+0000", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.607e-06 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841050+0000", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.792e-06 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841060+0000", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0179e-05 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841068+0000", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7190000000000001e-06 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841120+0000", 2024-09-21T19:05:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1857999999999997e-05 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841128+0000", 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0949999999999996e-06 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920269+0000", 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0791413379999999 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920385+0000", 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011542799999999999 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920500+0000", 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115887 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924337+0000", 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003836665 2024-09-21T19:05:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931030+0000", 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066925429999999996 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931084+0000", 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3983000000000001e-05 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931141+0000", 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7275999999999998e-05 2024-09-21T19:05:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931180+0000", 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8961999999999997e-05 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220162 9.6 9:6dc944fb:::benchmark_data_smithi081_131274_object92307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.858423+0000", 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.793147888, 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0732035870000001, 2024-09-21T19:05:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220162 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858423+0000", 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858423+0000", 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858425+0000", 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0219999999999999e-06 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858431+0000", 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.575e-06 2024-09-21T19:05:50.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858442+0000", 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1494999999999999e-05 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858449+0000", 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4089999999999997e-06 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858503+0000", 2024-09-21T19:05:50.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3995000000000001e-05 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858509+0000", 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3750000000000002e-06 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922344+0000", 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0638351479999999 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922455+0000", 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111475 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922580+0000", 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012425500000000001 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.930959+0000", 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0083796870000000002 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931314+0000", 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035444600000000001 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931328+0000", 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4002000000000001e-05 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931589+0000", 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026142500000000003 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931627+0000", 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7277999999999998e-05 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220172 9.6 9:63b6decf:::benchmark_data_smithi081_131274_object92317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.864693+0000", 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.786877565000001, 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0673737990000001, 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.772 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220172 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864693+0000", 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864693+0000", 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864695+0000", 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.257e-06 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864699+0000", 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4610000000000001e-06 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864706+0000", 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.374e-06 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864716+0000", 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9059999999999997e-06 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864769+0000", 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3052e-05 2024-09-21T19:05:50.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864775+0000", 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7189999999999998e-06 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924367+0000", 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0595922900000001 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924469+0000", 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010186100000000001 2024-09-21T19:05:50.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924573+0000", 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010414800000000001 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931014+0000", 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064408640000000001 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931637+0000", 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062255100000000005 2024-09-21T19:05:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931969+0000", 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00033202399999999999 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932028+0000", 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8981000000000003e-05 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932067+0000", 2024-09-21T19:05:50.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9310999999999997e-05 2024-09-21T19:05:50.779 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.779 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.779 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220174 9.6 9:6fbae686:::benchmark_data_smithi081_131274_object92319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.870009+0000", 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.781562378, 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.065893177, 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220174 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.780 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870009+0000", 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870009+0000", 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870010+0000", 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.232e-06 2024-09-21T19:05:50.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870014+0000", 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1300000000000003e-06 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870022+0000", 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5169999999999999e-06 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870028+0000", 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1150000000000002e-06 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870081+0000", 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3016000000000002e-05 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870086+0000", 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4049999999999999e-06 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926686+0000", 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.056600395 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926781+0000", 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5039999999999998e-05 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926889+0000", 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010791899999999999 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931641+0000", 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047520599999999998 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931778+0000", 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013652599999999999 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935798+0000", 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040200610000000001 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935853+0000", 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4672000000000001e-05 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935902+0000", 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9088999999999997e-05 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220193 9.6 9:68b62631:::benchmark_data_smithi081_131274_object92338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.883802+0000", 2024-09-21T19:05:50.786 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.767769303000001, 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.052301138, 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220193 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883802+0000", 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883802+0000", 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883803+0000", 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.556e-06 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883808+0000", 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.746e-06 2024-09-21T19:05:50.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883820+0000", 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2238999999999999e-05 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883828+0000", 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.413e-06 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883882+0000", 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4839000000000001e-05 2024-09-21T19:05:50.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883888+0000", 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6110000000000001e-06 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.928928+0000", 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.045039823 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.929033+0000", 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010493300000000001 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.929098+0000", 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5024999999999995e-05 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932076+0000", 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029780090000000002 2024-09-21T19:05:50.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932105+0000", 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.932e-05 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935964+0000", 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038585859999999998 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.936061+0000", 2024-09-21T19:05:50.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7151000000000005e-05 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.936103+0000", 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1887000000000001e-05 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220203 9.6 9:6d933243:::benchmark_data_smithi081_131274_object92348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.894230+0000", 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 23.757340766999999, 2024-09-21T19:05:50.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.048930103, 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220203 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894230+0000", 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894230+0000", 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894231+0000", 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.158e-06 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894235+0000", 2024-09-21T19:05:50.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1540000000000004e-06 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894245+0000", 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7559999999999994e-06 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894251+0000", 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9039999999999997e-06 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894295+0000", 2024-09-21T19:05:50.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4288e-05 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894302+0000", 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6420000000000001e-06 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.933209+0000", 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0389072500000001 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.933303+0000", 2024-09-21T19:05:50.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3661999999999999e-05 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.933574+0000", 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027142600000000001 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.939924+0000", 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063491440000000001 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943005+0000", 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030819480000000002 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943077+0000", 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1402999999999994e-05 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943122+0000", 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4982999999999999e-05 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943160+0000", 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8384999999999999e-05 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:243954 9.0 9:05c846c6:::benchmark_data_smithi081_131274_object116099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.906685+0000", 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 2.74488566, 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.046879836, 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:50.800 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243954 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906685+0000", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906685+0000", 2024-09-21T19:05:50.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906686+0000", 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8999999999999996e-07 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906688+0000", 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5679999999999998e-06 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:50.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906695+0000", 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5409999999999997e-06 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906700+0000", 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5929999999999999e-06 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906752+0000", 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2562999999999998e-05 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:50.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906867+0000", 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011485900000000001 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,6", 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906974+0000", 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106817 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.910354+0000", 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003380699 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:50.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.911367+0000", 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001013 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.913393+0000", 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020258680000000001 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.921236+0000", 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078429820000000001 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.921249+0000", 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3012e-05 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.953475+0000", 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0322257130000001 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.953565+0000", 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0031000000000006e-05 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.806 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.807 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:50.807 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:50.807 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:05:50.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:50.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:50.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:50.987 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:50.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:51.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:51.124 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:51.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:50 smithi104 ceph-mon[32438]: pgmap v621: 158 pgs: 3 active+clean+remapped, 155 active+clean; 775 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 648 op/s; 5.6 MiB/s, 97 objects/s recovering 2024-09-21T19:05:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1912805482' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:50 smithi104 ceph-mon[32438]: from='client.36023 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:51.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:50 smithi081 ceph-mon[29384]: pgmap v621: 158 pgs: 3 active+clean+remapped, 155 active+clean; 775 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 648 op/s; 5.6 MiB/s, 97 objects/s recovering 2024-09-21T19:05:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1912805482' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:50 smithi081 ceph-mon[29384]: from='client.36023 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:51.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:50 smithi081 ceph-mon[39192]: pgmap v621: 158 pgs: 3 active+clean+remapped, 155 active+clean; 775 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 648 op/s; 5.6 MiB/s, 97 objects/s recovering 2024-09-21T19:05:51.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1912805482' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:51.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:50 smithi081 ceph-mon[39192]: from='client.36023 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:05:51.365 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.404 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:51.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:51.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:51.537 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:51.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:51.678 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:05:51.678 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:51.742 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:05:51.742 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:05:51.742 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T19:05:51.742 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T19:05:51.742 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T19:05:51.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T19:05:51.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:51.812 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:51.913 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:51.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:52.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:52.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:52.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:52.200 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[29384]: 3.f scrub starts 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[29384]: 3.f scrub ok 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[29384]: from='client.36023 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[29384]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[39192]: 3.f scrub starts 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[39192]: 3.f scrub ok 2024-09-21T19:05:52.201 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:52.202 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[39192]: from='client.36023 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:05:52.202 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[39192]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T19:05:52.202 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:51 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:51 smithi104 ceph-mon[32438]: 3.f scrub starts 2024-09-21T19:05:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:51 smithi104 ceph-mon[32438]: 3.f scrub ok 2024-09-21T19:05:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:51 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:05:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:51 smithi104 ceph-mon[32438]: from='client.36023 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:05:52.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:51 smithi104 ceph-mon[32438]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T19:05:52.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:51 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:05:52.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:52.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:52.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:52.507 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:52.609 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:52.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:52.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:52.918 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:52.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:53.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:53.055 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:53.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:52 smithi081 ceph-mon[39192]: 6.10 scrub starts 2024-09-21T19:05:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:52 smithi081 ceph-mon[39192]: 6.10 scrub ok 2024-09-21T19:05:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:52 smithi081 ceph-mon[39192]: pgmap v623: 158 pgs: 3 active+clean+remapped, 155 active+clean; 775 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 648 op/s; 5.6 MiB/s, 97 objects/s recovering 2024-09-21T19:05:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:52 smithi081 ceph-mon[29384]: 6.10 scrub starts 2024-09-21T19:05:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:52 smithi081 ceph-mon[29384]: 6.10 scrub ok 2024-09-21T19:05:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:52 smithi081 ceph-mon[29384]: pgmap v623: 158 pgs: 3 active+clean+remapped, 155 active+clean; 775 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 648 op/s; 5.6 MiB/s, 97 objects/s recovering 2024-09-21T19:05:53.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:53.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:53.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:52 smithi104 ceph-mon[32438]: 6.10 scrub starts 2024-09-21T19:05:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:52 smithi104 ceph-mon[32438]: 6.10 scrub ok 2024-09-21T19:05:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:52 smithi104 ceph-mon[32438]: pgmap v623: 158 pgs: 3 active+clean+remapped, 155 active+clean; 775 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 648 op/s; 5.6 MiB/s, 97 objects/s recovering 2024-09-21T19:05:53.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:53.328 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:53.429 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:53.466 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:53.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:53.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:53.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:05:53.650 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:53.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:53.741 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:53.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:53.874 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:53.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:53.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:54.050 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:54.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.194 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:54.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:54.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:05:54.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:54.312 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:05:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "" 2024-09-21T19:05:54.312 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:05:54.318 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:54.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:54.448 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:54.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:54.577 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:54.604 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:54.706 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:54.743 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:54.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:54.882 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:54.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:54.984 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:55.014 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:55.116 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:55.150 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:55.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:05:55.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:54 smithi081 ceph-mon[29384]: pgmap v624: 158 pgs: 158 active+clean; 453 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 994 op/s; 2.1 MiB/s, 34 objects/s recovering 2024-09-21T19:05:55.228 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:54 smithi081 ceph-mon[39192]: pgmap v624: 158 pgs: 158 active+clean; 453 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 994 op/s; 2.1 MiB/s, 34 objects/s recovering 2024-09-21T19:05:55.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:55.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:54 smithi104 ceph-mon[32438]: pgmap v624: 158 pgs: 158 active+clean; 453 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 994 op/s; 2.1 MiB/s, 34 objects/s recovering 2024-09-21T19:05:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:55.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:55.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:55.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:55.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:55.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:55.693 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:55.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:55.930 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:55.969 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:55.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:56.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:56.102 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:56.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:56.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:56.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:05:56.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:05:56.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.522 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:56.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:56.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:56.664 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:56.764 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:05:56.764 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:05:56.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.765 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:254437 9.6 9:6d5577fb:::benchmark_data_smithi081_131274_object126582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T19:05:56.765 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:56.742303+0000", 2024-09-21T19:05:56.765 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.016515810999999998, 2024-09-21T19:05:56.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016549805000000001, 2024-09-21T19:05:56.765 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:56.766 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:56.766 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:05:56.766 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:56.766 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:56.766 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 254437 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742303+0000", 2024-09-21T19:05:56.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:56.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.768 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:56.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742303+0000", 2024-09-21T19:05:56.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:56.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742305+0000", 2024-09-21T19:05:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3609999999999999e-06 2024-09-21T19:05:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742311+0000", 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1460000000000001e-06 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:56.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742328+0000", 2024-09-21T19:05:56.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7204e-05 2024-09-21T19:05:56.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:56.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742337+0000", 2024-09-21T19:05:56.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9250000000000001e-06 2024-09-21T19:05:56.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:56.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742550+0000", 2024-09-21T19:05:56.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021312100000000001 2024-09-21T19:05:56.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.745873+0000", 2024-09-21T19:05:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003322415 2024-09-21T19:05:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:05:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.746314+0000", 2024-09-21T19:05:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044081099999999999 2024-09-21T19:05:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.753305+0000", 2024-09-21T19:05:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0069912840000000004 2024-09-21T19:05:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.753351+0000", 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5772999999999997e-05 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.776 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:254438 9.6 9:678d74de:::benchmark_data_smithi081_131274_object126583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T19:05:56.777 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:56.742356+0000", 2024-09-21T19:05:56.777 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.016462564999999998, 2024-09-21T19:05:56.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016572910999999999, 2024-09-21T19:05:56.777 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:05:56.777 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:05:56.777 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:05:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:05:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:05:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:05:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 254438 2024-09-21T19:05:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:05:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:05:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742356+0000", 2024-09-21T19:05:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:05:56.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742356+0000", 2024-09-21T19:05:56.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:05:56.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742357+0000", 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9500000000000002e-07 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742360+0000", 2024-09-21T19:05:56.781 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3179999999999998e-06 2024-09-21T19:05:56.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.782 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.782 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:05:56.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742367+0000", 2024-09-21T19:05:56.782 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6699999999999997e-06 2024-09-21T19:05:56.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.742372+0000", 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6300000000000003e-06 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:05:56.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.750374+0000", 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0080015759999999998 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.750501+0000", 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012725599999999999 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.784 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.750608+0000", 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106551 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.754114+0000", 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035068339999999999 2024-09-21T19:05:56.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:56.755073+0000", 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00095803900000000005 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:05:56.786 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:05:56.787 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 2 2024-09-21T19:05:56.787 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:05:56.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.835 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:56.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:56.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:56.960 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:56.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:57.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:57.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:56 smithi081 ceph-mon[39192]: pgmap v625: 158 pgs: 158 active+clean; 453 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 659 B/s rd, 0 B/s wr, 665 op/s 2024-09-21T19:05:57.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:56 smithi081 ceph-mon[29384]: pgmap v625: 158 pgs: 158 active+clean; 453 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 659 B/s rd, 0 B/s wr, 665 op/s 2024-09-21T19:05:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.100 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:57.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:57.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.231 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:57.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:57.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:56 smithi104 ceph-mon[32438]: pgmap v625: 158 pgs: 158 active+clean; 453 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 659 B/s rd, 0 B/s wr, 665 op/s 2024-09-21T19:05:57.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:57.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:57.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:57.477 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:05:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:05:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:05:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:57.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:57.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:57.646 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:05:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:05:57.684 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:05:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:05:57.718 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:05:57.718 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:05:57.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:57.740 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Removed 127852 objects 2024-09-21T19:05:57.740 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Clean up completed and total clean up time :109.32 2024-09-21T19:05:57.746 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:05:57.746 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:05:57.754 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:57.762 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:05:57.762 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:05:57.779 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:05:57.779 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:05:57.795 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:05:57.795 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:05:57.812 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:05:57.812 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:05:57.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:57.878 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:57.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:57.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:58.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2463458023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:58.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2463458023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:58.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:58.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:58.252 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:05:58.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2463458023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:05:58.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:58.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:58.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:58.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:58.494 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:58.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:05:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:05:58.695 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T19:05:58.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T19:05:58.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.742 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:58.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:58.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:05:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:05:58.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:58.865 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:58.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:05:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:58.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:05:59.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:59.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:05:59.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:59.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:58 smithi104 ceph-mon[32438]: pgmap v626: 158 pgs: 158 active+clean; 141 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T19:05:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:58 smithi081 ceph-mon[29384]: pgmap v626: 158 pgs: 158 active+clean; 141 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T19:05:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:58 smithi081 ceph-mon[39192]: pgmap v626: 158 pgs: 158 active+clean; 141 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T19:05:59.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:05:59.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:05:59.443 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:59.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:05:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:05:59.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:59.610 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:59.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:05:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:05:59.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:05:59.859 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:05:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:05:59.924 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:05:59.924 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:05:59.930 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:05:59.953 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:05:59.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:05:59.968 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:59.968 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:05:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2366589721' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:59.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:05:59.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:05:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2366589721' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:05:59.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:05:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:05:59.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:00.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:00.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:00.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:00.231 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:00.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:05:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2366589721' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:00.333 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:00.356 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:00.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:00.481 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:00.583 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:00.605 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:00.707 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:00.731 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:00.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:00.858 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:00.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:00.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:00.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:01.085 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.108 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:01.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:01.174 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:06:01.174 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:06:01.174 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T19:06:01.174 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T19:06:01.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5 2024-09-21T19:06:01.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:01.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:00 smithi104 ceph-mon[32438]: pgmap v627: 158 pgs: 158 active+clean; 37 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T19:06:01.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:00 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:01.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2366589721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:01.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:00 smithi104 ceph-mon[32438]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T19:06:01.239 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:01.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[29384]: pgmap v627: 158 pgs: 158 active+clean; 37 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2366589721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[29384]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[39192]: pgmap v627: 158 pgs: 158 active+clean; 37 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2366589721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:00 smithi081 ceph-mon[39192]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T19:06:01.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:01.368 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:01.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:01.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:06:01.477 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:01 smithi104 systemd[1]: Stopping Ceph osd.5 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:06:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.496 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:01.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:01.597 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:01.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:01.720 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:01.742 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:01.756 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[41800]: 2024-09-21T19:06:01.474+0000 7ff24a7a0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:06:01.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[41800]: 2024-09-21T19:06:01.474+0000 7ff24a7a0640 -1 osd.5 326 *** Got signal Terminated *** 2024-09-21T19:06:01.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[41800]: 2024-09-21T19:06:01.474+0000 7ff24a7a0640 -1 osd.5 326 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:06:01.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:01.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:01.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:01.995 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:01.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:02.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:02.118 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:02.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:02.242 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:02.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:01 smithi104 ceph-mon[32438]: osd.5 marked itself down and dead 2024-09-21T19:06:02.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:01 smithi081 ceph-mon[29384]: osd.5 marked itself down and dead 2024-09-21T19:06:02.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:01 smithi081 ceph-mon[39192]: osd.5 marked itself down and dead 2024-09-21T19:06:02.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:02.366 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:02.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:02.489 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:02.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:02.612 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:02.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:02.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:02.810 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:02 smithi104 podman[117560]: 2024-09-21 19:06:02.555847349 +0000 UTC m=+1.194663421 container died 1ebea6206675b3cc9b5955cc0ac3a52fe4197eff004bd09aaa3a12118ff609ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T19:06:02.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:02.857 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:02.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:02.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:03.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:03.110 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:03.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:06:03.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:03.220 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:02 smithi104 podman[117560]: 2024-09-21 19:06:02.983054183 +0000 UTC m=+1.621870260 container cleanup 1ebea6206675b3cc9b5955cc0ac3a52fe4197eff004bd09aaa3a12118ff609ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T19:06:03.220 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:02 smithi104 bash[117560]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5 2024-09-21T19:06:03.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:02 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:06:03.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:02 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:06:03.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:02 smithi104 ceph-mon[32438]: pgmap v629: 158 pgs: 158 active+clean; 37 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T19:06:03.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:02 smithi104 ceph-mon[32438]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T19:06:03.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:02 smithi104 ceph-mon[32438]: osdmap e327: 8 total, 7 up, 8 in 2024-09-21T19:06:03.233 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[29384]: pgmap v629: 158 pgs: 158 active+clean; 37 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[29384]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[29384]: osdmap e327: 8 total, 7 up, 8 in 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[39192]: pgmap v629: 158 pgs: 158 active+clean; 37 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[39192]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T19:06:03.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:02 smithi081 ceph-mon[39192]: osdmap e327: 8 total, 7 up, 8 in 2024-09-21T19:06:03.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:03.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:03.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:03.482 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:03.506 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:03 smithi104 podman[117644]: 2024-09-21 19:06:03.218258181 +0000 UTC m=+0.658519772 container remove 1ebea6206675b3cc9b5955cc0ac3a52fe4197eff004bd09aaa3a12118ff609ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T19:06:03.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:03.615 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:03.652 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:03.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:03.728 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:03.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:03.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:03.932 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:03 smithi104 podman[117783]: 2024-09-21 19:06:03.553469466 +0000 UTC m=+0.016079385 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:06:03.932 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:03 smithi104 podman[117783]: 2024-09-21 19:06:03.66927493 +0000 UTC m=+0.131884830 container create c511cf50bc922bd77fb7a3f998e04cecb69c7e526ce8b8d74dd5babe1b7d9e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T19:06:03.932 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:03 smithi104 podman[117783]: 2024-09-21 19:06:03.930561978 +0000 UTC m=+0.393171895 container init c511cf50bc922bd77fb7a3f998e04cecb69c7e526ce8b8d74dd5babe1b7d9e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43) 2024-09-21T19:06:03.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_2' removed 2024-09-21T19:06:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:03.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:04.019 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[39192]: osdmap e328: 8 total, 7 up, 8 in 2024-09-21T19:06:04.020 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3512153728' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:06:04.020 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:06:04.020 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:06:04.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[29384]: osdmap e328: 8 total, 7 up, 8 in 2024-09-21T19:06:04.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3512153728' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:06:04.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:06:04.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:03 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:06:04.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.101 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:04.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:03 smithi104 ceph-mon[32438]: osdmap e328: 8 total, 7 up, 8 in 2024-09-21T19:06:04.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3512153728' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:06:04.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:03 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:06:04.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:03 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:06:04.200 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:03 smithi104 podman[117783]: 2024-09-21 19:06:03.934098021 +0000 UTC m=+0.396707920 container start c511cf50bc922bd77fb7a3f998e04cecb69c7e526ce8b8d74dd5babe1b7d9e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T19:06:04.200 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:04 smithi104 podman[117783]: 2024-09-21 19:06:04.172627182 +0000 UTC m=+0.635237083 container attach c511cf50bc922bd77fb7a3f998e04cecb69c7e526ce8b8d74dd5babe1b7d9e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True) 2024-09-21T19:06:04.200 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:04 smithi104 podman[117783]: 2024-09-21 19:06:04.172891664 +0000 UTC m=+0.635501570 container died c511cf50bc922bd77fb7a3f998e04cecb69c7e526ce8b8d74dd5babe1b7d9e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T19:06:04.202 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:04.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:04.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:04.346 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:04.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:04.469 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:04.540 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:04.540 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:04.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:04.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:06:04.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:04.714 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:04.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T19:06:04.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:04.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:06:04.831 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T19:06:04.831 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-21T19:06:04.831 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T19:06:04.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.5 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-21T19:06:04.839 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:04.941 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:04.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:05.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:05.066 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T19:06:05.085 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:05.092 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:05.093 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:05.100 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:05.100 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:05.117 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:05.117 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:05.133 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:05.133 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:05.148 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:05.149 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:05.149 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:05.165 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:05.166 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:05.181 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:05.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:05.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:04 smithi104 ceph-mon[32438]: pgmap v632: 158 pgs: 35 peering, 1 stale+active+clean, 122 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 376 op/s 2024-09-21T19:06:05.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:04 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T19:06:05.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3512153728' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:06:05.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:04 smithi104 ceph-mon[32438]: osdmap e329: 8 total, 7 up, 8 in 2024-09-21T19:06:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:05.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[29384]: pgmap v632: 158 pgs: 35 peering, 1 stale+active+clean, 122 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 376 op/s 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3512153728' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[29384]: osdmap e329: 8 total, 7 up, 8 in 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[39192]: pgmap v632: 158 pgs: 35 peering, 1 stale+active+clean, 122 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 376 op/s 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3512153728' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:06:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:04 smithi081 ceph-mon[39192]: osdmap e329: 8 total, 7 up, 8 in 2024-09-21T19:06:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.331 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:05.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:05.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:05.453 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:05.541 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T19:06:05.542 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T19:06:05.542 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T19:06:05.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T19:06:05.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:05.605 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:05.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:05.729 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:05.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:05.854 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:05.956 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:05.978 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:05.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:06.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:06:06.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:06.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:06.207 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:06:06.210 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:06.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:06.237 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/9344435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:06.238 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:06.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/9344435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/9344435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:06.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:06.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:06.483 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:06.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:06.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:06.606 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:06.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:06.730 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:06.831 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:06.852 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:06.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:06.975 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:06.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:07.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:07.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:07.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:06 smithi104 ceph-mon[32438]: pgmap v634: 142 pgs: 3 active+undersized, 29 peering, 1 stale+active+clean, 1 active+undersized+degraded, 108 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 5/597 objects degraded (0.838%) 2024-09-21T19:06:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:06 smithi104 ceph-mon[32438]: Health check failed: Degraded data redundancy: 5/597 objects degraded (0.838%), 1 pg degraded (PG_DEGRADED) 2024-09-21T19:06:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:07.259 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:07.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[39192]: pgmap v634: 142 pgs: 3 active+undersized, 29 peering, 1 stale+active+clean, 1 active+undersized+degraded, 108 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 5/597 objects degraded (0.838%) 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[39192]: Health check failed: Degraded data redundancy: 5/597 objects degraded (0.838%), 1 pg degraded (PG_DEGRADED) 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[29384]: pgmap v634: 142 pgs: 3 active+undersized, 29 peering, 1 stale+active+clean, 1 active+undersized+degraded, 108 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 716 B/s rd, 0 op/s; 5/597 objects degraded (0.838%) 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[29384]: Health check failed: Degraded data redundancy: 5/597 objects degraded (0.838%), 1 pg degraded (PG_DEGRADED) 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:07.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:06:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:06:07.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:07.346 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:07.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:07.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:07.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:07.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:07.836 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:07.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:07.870 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:07.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:07.989 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_3' created 2024-09-21T19:06:07.995 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:06:07.995 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:07.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:08.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:08.119 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:08.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:08.242 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:08.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2853923898' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:08.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:07 smithi104 ceph-mon[32438]: from='client.36050 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:08.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1408758316' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T19:06:08.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2853923898' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-mon[29384]: from='client.36050 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1408758316' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T19:06:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2853923898' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-mon[39192]: from='client.36050 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1408758316' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T19:06:08.344 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:08.366 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:08.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:08.489 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:08.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:08.614 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:08.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T19:06:08.715 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:08.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:06:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:08.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:08.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:08.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:09.017 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:9.f 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:9.d 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:9.b 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:5.15 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:3.d 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:5.7 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:4.7 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:4.6 2024-09-21T19:06:09.035 INFO:teuthology.orchestra.run.smithi104.stdout:5.4 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:5.6 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:4.3 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:5.b 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:1.a 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:3.1d 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:2.2 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:5.19 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:4.8 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:6.1e 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:6.4 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:9.9 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:3.c 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:3.2 2024-09-21T19:06:09.036 INFO:teuthology.orchestra.run.smithi104.stdout:4.1f 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:3.10 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:5.f 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:3.16 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:1.4 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:4.1 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:6.a 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:6.1d 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:3.1c 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:1.1 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:6.19 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:3.1f 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:5.3 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:4.9 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:4.15 2024-09-21T19:06:09.037 INFO:teuthology.orchestra.run.smithi104.stdout:5.1b 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:5.2 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:1.8 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:5.d 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:5.e 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:3.5 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:4.2 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.10 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.11 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.17 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.1a 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.c 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.1f 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:5.12 2024-09-21T19:06:09.038 INFO:teuthology.orchestra.run.smithi104.stdout:6.1c 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:6.7 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:6.d 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:5.16 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:3.4 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:9.5 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:9.6 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:9.7 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:9.8 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:3.17 2024-09-21T19:06:09.039 INFO:teuthology.orchestra.run.smithi104.stdout:9.a 2024-09-21T19:06:09.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:06:09.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:09.140 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:09.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-mon[32438]: pgmap v635: 142 pgs: 28 active+undersized, 9 peering, 16 active+undersized+degraded, 89 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 59/597 objects degraded (9.883%) 2024-09-21T19:06:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T19:06:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-mon[32438]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:06:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1408758316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T19:06:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:08 smithi104 ceph-mon[32438]: osdmap e330: 8 total, 7 up, 8 in 2024-09-21T19:06:09.263 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[29384]: pgmap v635: 142 pgs: 28 active+undersized, 9 peering, 16 active+undersized+degraded, 89 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 59/597 objects degraded (9.883%) 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[29384]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1408758316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[29384]: osdmap e330: 8 total, 7 up, 8 in 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[39192]: pgmap v635: 142 pgs: 28 active+undersized, 9 peering, 16 active+undersized+degraded, 89 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 59/597 objects degraded (9.883%) 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[39192]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:06:09.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1408758316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T19:06:09.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:08 smithi081 ceph-mon[39192]: osdmap e330: 8 total, 7 up, 8 in 2024-09-21T19:06:09.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:09.386 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:09.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:09.509 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:09.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:09.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:09.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:09.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:09.773 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:06:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:06:09.774 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-09-21T19:06:09.774 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:06:09.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:09.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:09.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:09.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.5 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 9.7 --file /var/log/ceph/exp.9.7.5 2024-09-21T19:06:09.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[29384]: osdmap e331: 8 total, 7 up, 8 in 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[29384]: 4.b scrub starts 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[29384]: 4.b scrub ok 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T19:06:10.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T19:06:10.006 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:10.108 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:10.129 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T19:06:10.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:10.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:10.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:10 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:10.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:10 smithi104 ceph-mon[32438]: osdmap e331: 8 total, 7 up, 8 in 2024-09-21T19:06:10.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:10 smithi104 ceph-mon[32438]: 4.b scrub starts 2024-09-21T19:06:10.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:10 smithi104 ceph-mon[32438]: 4.b scrub ok 2024-09-21T19:06:10.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:10 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T19:06:10.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:10 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T19:06:10.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[39192]: osdmap e331: 8 total, 7 up, 8 in 2024-09-21T19:06:10.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[39192]: 4.b scrub starts 2024-09-21T19:06:10.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[39192]: 4.b scrub ok 2024-09-21T19:06:10.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T19:06:10.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:10 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T19:06:10.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:10.406 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:10.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:06:10.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:10.499 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:10.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:10.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:10.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:10.745 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:10.846 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:10.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:10.971 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:10.971 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:10.990 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:10.994 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:10.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:11.012 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:11.012 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:11.029 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:11.029 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:11.032 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:11.032 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:11.058 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:11.058 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:11.059 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:11.075 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:11.076 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:11.092 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:11.092 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:11.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:11.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:11.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:11.241 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:11 smithi104 ceph-mon[32438]: pgmap v638: 158 pgs: 6 creating+peering, 10 unknown, 36 active+undersized, 17 active+undersized+degraded, 89 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:11 smithi104 ceph-mon[32438]: 3.b scrub starts 2024-09-21T19:06:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:11 smithi104 ceph-mon[32438]: 3.b scrub ok 2024-09-21T19:06:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:11 smithi104 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:06:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:11 smithi104 ceph-mon[32438]: osdmap e332: 8 total, 7 up, 8 in 2024-09-21T19:06:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1716700850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[29384]: pgmap v638: 158 pgs: 6 creating+peering, 10 unknown, 36 active+undersized, 17 active+undersized+degraded, 89 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[29384]: 3.b scrub starts 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[29384]: 3.b scrub ok 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[29384]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[29384]: osdmap e332: 8 total, 7 up, 8 in 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1716700850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[39192]: pgmap v638: 158 pgs: 6 creating+peering, 10 unknown, 36 active+undersized, 17 active+undersized+degraded, 89 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:11.307 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[39192]: 3.b scrub starts 2024-09-21T19:06:11.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[39192]: 3.b scrub ok 2024-09-21T19:06:11.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[39192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:06:11.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[39192]: osdmap e332: 8 total, 7 up, 8 in 2024-09-21T19:06:11.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1716700850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:11.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:11.365 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:11.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:06:11.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:11.487 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:11.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:11.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:11.827 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:11.827 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:11.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:11.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:11.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:12.016 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:12.037 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:12.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:12.161 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:12.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.285 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:12.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:12.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:12.407 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:12.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:12.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:12.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:12.654 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:12.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:12.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:12.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:12.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:12.901 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:12.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:13.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:13.005 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T19:06:13.029 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:13.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.153 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:13.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[29384]: pgmap v640: 158 pgs: 6 creating+peering, 8 unknown, 36 active+undersized, 17 active+undersized+degraded, 91 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[29384]: 3.7 scrub starts 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[29384]: 3.7 scrub ok 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2862413999' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[39192]: pgmap v640: 158 pgs: 6 creating+peering, 8 unknown, 36 active+undersized, 17 active+undersized+degraded, 91 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[39192]: 3.7 scrub starts 2024-09-21T19:06:13.239 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[39192]: 3.7 scrub ok 2024-09-21T19:06:13.240 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2862413999' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:06:13.254 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:13.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:06:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:13.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:13.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:13.397 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:13.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:13.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:13 smithi104 ceph-mon[32438]: pgmap v640: 158 pgs: 6 creating+peering, 8 unknown, 36 active+undersized, 17 active+undersized+degraded, 91 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:13 smithi104 ceph-mon[32438]: 3.7 scrub starts 2024-09-21T19:06:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:13 smithi104 ceph-mon[32438]: 3.7 scrub ok 2024-09-21T19:06:13.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2862413999' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:06:13.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:13.555 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:13.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.646 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:13.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:13.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:13.768 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:13.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:13.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:13.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:13.993 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:14.014 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:14.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:14.137 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:14.227 INFO:teuthology.orchestra.run.smithi104.stderr:Exporting 9.7 info 9.7( v 325'16042 (303'11412,325'16042] local-lis/les=274/275 n=0 ec=254/254 lis/c=274/274 les/c/f=275/275/0 sis=274) 2024-09-21T19:06:14.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:Export successful 2024-09-21T19:06:14.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:14.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[29384]: 4.1b scrub starts 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[29384]: 4.1b scrub ok 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2862413999' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[29384]: osdmap e333: 8 total, 7 up, 8 in 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[39192]: 4.1b scrub starts 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[39192]: 4.1b scrub ok 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2862413999' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:06:14.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:14 smithi081 ceph-mon[39192]: osdmap e333: 8 total, 7 up, 8 in 2024-09-21T19:06:14.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:14.384 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:14.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:14.506 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:14 smithi104 ceph-mon[32438]: 4.1b scrub starts 2024-09-21T19:06:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:14 smithi104 ceph-mon[32438]: 4.1b scrub ok 2024-09-21T19:06:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2862413999' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:06:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:14 smithi104 ceph-mon[32438]: osdmap e333: 8 total, 7 up, 8 in 2024-09-21T19:06:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:14.512 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:14.512 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:14.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:14.629 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:14.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:06:14.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:14.752 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:14.753 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:14.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:14.876 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:14.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:14.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:14.996 DEBUG:teuthology.orchestra.run.smithi081:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T19:06:14.998 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T19:06:14.999 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T19:06:14.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:15.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:15.122 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:15.195 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:hints = 1 2024-09-21T19:06:15.195 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T19:06:15.195 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object prefix: benchmark_data_smithi081_155596 2024-09-21T19:06:15.195 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:06:15.195 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 0 0 0 0 0 0 - 0 2024-09-21T19:06:15.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:15.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[29384]: pgmap v642: 158 pgs: 6 creating+peering, 39 active+undersized, 17 active+undersized+degraded, 96 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 899 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[29384]: 5.1c scrub starts 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[29384]: 5.1c scrub ok 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[29384]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 60/597 objects degraded (10.050%), 17 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[39192]: pgmap v642: 158 pgs: 6 creating+peering, 39 active+undersized, 17 active+undersized+degraded, 96 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 899 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[39192]: 5.1c scrub starts 2024-09-21T19:06:15.287 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[39192]: 5.1c scrub ok 2024-09-21T19:06:15.288 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[39192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:06:15.288 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:15 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 60/597 objects degraded (10.050%), 17 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:15.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:06:15.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:15.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:15 smithi104 ceph-mon[32438]: pgmap v642: 158 pgs: 6 creating+peering, 39 active+undersized, 17 active+undersized+degraded, 96 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 899 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:15.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:15 smithi104 ceph-mon[32438]: 5.1c scrub starts 2024-09-21T19:06:15.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:15 smithi104 ceph-mon[32438]: 5.1c scrub ok 2024-09-21T19:06:15.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:15 smithi104 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:06:15.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:15 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 60/597 objects degraded (10.050%), 17 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:15.426 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:15.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.5 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 9.7 2024-09-21T19:06:15.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:15.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:15.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:15.708 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:15.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:15.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:15.846 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:15.914 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:15.948 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:15.976 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:15.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:16.069 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T19:06:16.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:16.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 1 15 3497 3482 217.612 217.625 0.00380551 0.00458137 2024-09-21T19:06:16.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:16.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:16.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:16.370 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:16.370 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:16.387 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:16.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:16.604 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:16.604 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:16.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:16.643 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:16.658 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:16.676 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:16.676 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:16.714 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:16.714 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:16.748 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:16.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:16.787 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:16.787 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:16.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:16.821 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:16.821 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:16.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:16.929 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:16.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:17.028 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:06:17.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:17.073 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:17.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:06:17.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:17.195 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 2 16 6936 6920 216.226 214.875 0.00266668 0.00461575 2024-09-21T19:06:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:17.312 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:17.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: pgmap v643: 158 pgs: 41 active+undersized, 17 active+undersized+degraded, 100 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 765 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: 5.13 scrub starts 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: 5.13 scrub ok 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: 3.12 scrub starts 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: 3.12 scrub ok 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4187939997' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/227481127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: pgmap v643: 158 pgs: 41 active+undersized, 17 active+undersized+degraded, 100 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 765 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: 5.13 scrub starts 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: 5.13 scrub ok 2024-09-21T19:06:17.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: 3.12 scrub starts 2024-09-21T19:06:17.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: 3.12 scrub ok 2024-09-21T19:06:17.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4187939997' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:17.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/227481127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:17.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:06:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:06:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:06:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:17.394 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: pgmap v643: 158 pgs: 41 active+undersized, 17 active+undersized+degraded, 100 active+clean; 583 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 765 B/s rd, 0 op/s; 60/597 objects degraded (10.050%) 2024-09-21T19:06:17.394 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: 5.13 scrub starts 2024-09-21T19:06:17.394 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: 5.13 scrub ok 2024-09-21T19:06:17.394 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: 3.12 scrub starts 2024-09-21T19:06:17.394 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: 3.12 scrub ok 2024-09-21T19:06:17.395 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4187939997' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:17.395 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/227481127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:17.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:17.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:17.586 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:17.618 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:17.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:17.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:17.854 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:17.889 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:17.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:17.894 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:17.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:18.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:18.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:18.163 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:18.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 3 16 10511 10495 218.624 223.438 0.00713646 0.00456635 2024-09-21T19:06:18.265 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:18.298 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:18.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:18 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:06:18.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4187939997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:18.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:18 smithi104 ceph-mon[32438]: osdmap e334: 8 total, 7 up, 8 in 2024-09-21T19:06:18.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:18 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:06:18.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4187939997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:18.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:18 smithi081 ceph-mon[39192]: osdmap e334: 8 total, 7 up, 8 in 2024-09-21T19:06:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:18 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:06:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4187939997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:18 smithi081 ceph-mon[29384]: osdmap e334: 8 total, 7 up, 8 in 2024-09-21T19:06:18.375 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:06:18.387 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:18.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:18.436 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:18.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:18.574 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:18.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:06:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:06:18.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:18.712 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:18.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:18.851 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:18.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:18.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:19.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:19.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:19.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 4 16 13801 13785 215.37 205.625 0.0113761 0.004636 2024-09-21T19:06:19.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:19.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:19.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:19 smithi104 ceph-mon[32438]: pgmap v645: 158 pgs: 38 active+undersized, 20 active+undersized+degraded, 100 active+clean; 247 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 32 MiB/s wr, 514 op/s; 1272/12414 objects degraded (10.246%) 2024-09-21T19:06:19.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:19 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T19:06:19.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:19 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T19:06:19.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:19 smithi081 ceph-mon[29384]: pgmap v645: 158 pgs: 38 active+undersized, 20 active+undersized+degraded, 100 active+clean; 247 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 32 MiB/s wr, 514 op/s; 1272/12414 objects degraded (10.246%) 2024-09-21T19:06:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:19 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T19:06:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:19 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T19:06:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:19 smithi081 ceph-mon[39192]: pgmap v645: 158 pgs: 38 active+undersized, 20 active+undersized+degraded, 100 active+clean; 247 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 32 MiB/s wr, 514 op/s; 1272/12414 objects degraded (10.246%) 2024-09-21T19:06:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:19 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T19:06:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:19 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T19:06:19.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:19.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:19.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:19.542 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:19.589 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:06:19.589 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:06:19.589 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:06:19.589 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:06:19.589 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:06:19.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.685 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:19.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:19.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:19.822 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:19.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:06:19.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:19.945 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:19.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:20.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.092 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:20.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:20.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:20.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 5 16 16588 16572 207.13 174.188 0.00558917 0.00482129 2024-09-21T19:06:20.228 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:20.305 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 1272/12414 objects degraded (10.246%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:06:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:20 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 1272/12414 objects degraded (10.246%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:20.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:20.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:20.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:06:20.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:20 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:20 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:06:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:20 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:06:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:20 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 1272/12414 objects degraded (10.246%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:20.525 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:20.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.663 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:20.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:20.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:20.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:20.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:06:20.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:20.981 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:20.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:21.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:21.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:21.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 6 16 19252 19236 200.353 166.5 0.00514272 0.00498515 2024-09-21T19:06:21.215 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:21 smithi081 ceph-mon[39192]: pgmap v646: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 552 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 2932/27063 objects degraded (10.834%) 2024-09-21T19:06:21.215 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:21 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:06:21.215 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:21 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:06:21.216 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:21 smithi081 ceph-mon[29384]: pgmap v646: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 552 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 2932/27063 objects degraded (10.834%) 2024-09-21T19:06:21.216 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:21 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:06:21.216 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:21 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:06:21.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:21 smithi104 ceph-mon[32438]: pgmap v646: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 552 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 2932/27063 objects degraded (10.834%) 2024-09-21T19:06:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:21 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:06:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:21 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:06:21.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:21.299 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:21.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:21.396 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:21.396 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:21.419 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:21.429 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:21.429 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:21.462 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:21.462 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:21.497 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:21.497 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:21.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:21.534 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:21.535 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:21.535 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:21.546 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:21.568 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:21.568 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:21.600 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:21.601 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:21.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:21.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:21.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.821 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:21.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:21.922 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:21.943 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:21.944 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:21.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:21.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:22.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:22.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:22.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 7 15 21707 21692 193.658 153.5 0.00434577 0.00515776 2024-09-21T19:06:22.197 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:22.235 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:22.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:22.372 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:22.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:22.475 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: 3.1e scrub starts 2024-09-21T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: 3.1e scrub ok 2024-09-21T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3291962592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:22.511 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:22.519 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T19:06:22.519 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T19:06:22.519 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: 3.1e scrub starts 2024-09-21T19:06:22.519 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: 3.1e scrub ok 2024-09-21T19:06:22.519 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3291962592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: 3.1e scrub starts 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: 3.1e scrub ok 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3291962592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:22.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:22.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:22.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:22.652 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:22.710 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:06:22.754 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:22.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:22.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:23.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.074 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:23.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:23.145 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:23.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:23.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 8 16 24186 24170 188.809 154.875 0.00485669 0.00528961 2024-09-21T19:06:23.215 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[39192]: pgmap v647: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 673 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 589 B/s rd, 77 MiB/s wr, 1.24k op/s; 2932/32874 objects degraded (8.919%) 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[39192]: 4.10 scrub starts 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[39192]: 4.10 scrub ok 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:06:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[29384]: pgmap v647: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 673 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 589 B/s rd, 77 MiB/s wr, 1.24k op/s; 2932/32874 objects degraded (8.919%) 2024-09-21T19:06:23.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[29384]: 4.10 scrub starts 2024-09-21T19:06:23.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[29384]: 4.10 scrub ok 2024-09-21T19:06:23.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:23 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:06:23.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:23.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:23.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:23.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:23 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:06:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:23 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:06:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:23 smithi104 ceph-mon[32438]: pgmap v647: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 673 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 589 B/s rd, 77 MiB/s wr, 1.24k op/s; 2932/32874 objects degraded (8.919%) 2024-09-21T19:06:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:23 smithi104 ceph-mon[32438]: 4.10 scrub starts 2024-09-21T19:06:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:23 smithi104 ceph-mon[32438]: 4.10 scrub ok 2024-09-21T19:06:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:23 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:06:23.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:23.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:23.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:23.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:23.898 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:23.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:23.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:24.027 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:24.027 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:24.035 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:24.087 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:06:24.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:24.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 9 16 26502 26486 183.912 144.75 0.0118834 0.00543118 2024-09-21T19:06:24.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:24 smithi104 ceph-mon[32438]: 4.c scrub starts 2024-09-21T19:06:24.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:24 smithi104 ceph-mon[32438]: 4.c scrub ok 2024-09-21T19:06:24.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3001029038' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:06:24.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:24 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:06:24.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:24 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:06:24.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:24 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:24.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:24.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:24.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:24.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[29384]: 4.c scrub starts 2024-09-21T19:06:24.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[29384]: 4.c scrub ok 2024-09-21T19:06:24.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3001029038' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:06:24.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:06:24.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:06:24.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:24.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[39192]: 4.c scrub starts 2024-09-21T19:06:24.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[39192]: 4.c scrub ok 2024-09-21T19:06:24.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3001029038' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:06:24.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:06:24.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:06:24.488 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:24 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:24.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:24.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:06:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.581 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:24.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:24.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.723 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:24.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:24.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:24.869 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:24.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:24.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:25.010 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:25.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:25.153 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:25.154 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: pgmap v648: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.82k op/s; 5765/55323 objects degraded (10.421%) 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: 3.3 scrub starts 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: 3.3 scrub ok 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3001029038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: osdmap e335: 8 total, 7 up, 8 in 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 5765/55323 objects degraded (10.421%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: pgmap v648: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.82k op/s; 5765/55323 objects degraded (10.421%) 2024-09-21T19:06:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: 3.3 scrub starts 2024-09-21T19:06:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: 3.3 scrub ok 2024-09-21T19:06:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3001029038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:06:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: osdmap e335: 8 total, 7 up, 8 in 2024-09-21T19:06:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:25 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 5765/55323 objects degraded (10.421%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:25.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 10 16 28749 28733 179.564 140.438 0.0104423 0.00556432 2024-09-21T19:06:25.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:25.285 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:25.387 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: pgmap v648: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.82k op/s; 5765/55323 objects degraded (10.421%) 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: 3.3 scrub starts 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: 3.3 scrub ok 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3001029038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: osdmap e335: 8 total, 7 up, 8 in 2024-09-21T19:06:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:25 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 5765/55323 objects degraded (10.421%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:25.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:25.613 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:25.613 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:25.633 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:25.634 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:25.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:25.660 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:25.660 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:25.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:25.694 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:25.695 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:25.727 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:25.728 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:25.729 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:25.763 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:25.763 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:25.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:25.796 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:25.828 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:25.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:25.962 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:25.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:26.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:26.196 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 11 16 30526 30510 173.335 111.062 0.00616324 0.00576326 2024-09-21T19:06:26.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:26.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:26.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.362 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:26.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:26.445 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:26 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:06:26.445 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:26 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:06:26.445 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1690863110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:26.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:26.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:06:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:26.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:26 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:06:26.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:26 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:06:26.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1690863110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:26 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:06:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:26 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:06:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1690863110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:26.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:26.628 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:26.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:26.750 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:26.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:26.887 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:26.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:26.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:06:26.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:27.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:27.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:27.144 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:27.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 12 16 31807 31791 165.562 80.0625 0.00223965 0.00601363 2024-09-21T19:06:27.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:27 smithi104 ceph-mon[32438]: pgmap v650: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 591 B/s rd, 152 MiB/s wr, 2.43k op/s; 7378/69825 objects degraded (10.566%) 2024-09-21T19:06:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:27 smithi104 ceph-mon[32438]: osdmap e336: 8 total, 7 up, 8 in 2024-09-21T19:06:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.280 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:27.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:27.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:27 smithi081 ceph-mon[39192]: pgmap v650: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 591 B/s rd, 152 MiB/s wr, 2.43k op/s; 7378/69825 objects degraded (10.566%) 2024-09-21T19:06:27.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:27 smithi081 ceph-mon[39192]: osdmap e336: 8 total, 7 up, 8 in 2024-09-21T19:06:27.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:27 smithi081 ceph-mon[29384]: pgmap v650: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 591 B/s rd, 152 MiB/s wr, 2.43k op/s; 7378/69825 objects degraded (10.566%) 2024-09-21T19:06:27.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:27 smithi081 ceph-mon[29384]: osdmap e336: 8 total, 7 up, 8 in 2024-09-21T19:06:27.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:06:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:06:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:06:27.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:27.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.540 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:27.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:27.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.678 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:27.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:27.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:27.816 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:27.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:27.939 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:27.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:27.986 INFO:teuthology.orchestra.run.smithi104.stdout: marking collection for removal 2024-09-21T19:06:27.986 INFO:teuthology.orchestra.run.smithi104.stdout:setting '_remove' omap key 2024-09-21T19:06:27.986 INFO:teuthology.orchestra.run.smithi104.stdout:finish_remove_pgs 9.7_head removing 9.7 2024-09-21T19:06:28.012 INFO:teuthology.orchestra.run.smithi104.stdout:Remove successful 2024-09-21T19:06:28.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:28.071 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:28.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:28.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:28.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 13 16 33344 33328 160.215 96.0625 0.00295229 0.00618508 2024-09-21T19:06:28.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:28.317 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:28.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:28.450 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:28.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:28.588 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:28.619 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:06:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:06:28.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:28.727 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:28.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:28.864 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:28.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:28.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:28.999 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:29.062 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:06:29.062 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:06:29.062 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:06:29.062 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:06:29.062 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:06:29.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:29.124 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:29.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 14 16 34812 34796 155.324 91.75 0.00351794 0.00643314 2024-09-21T19:06:29.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:29.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:29 smithi104 ceph-mon[32438]: pgmap v652: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.60k op/s; 9307/89394 objects degraded (10.411%) 2024-09-21T19:06:29.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:29.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:29.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:29.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:29.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:29 smithi081 ceph-mon[29384]: pgmap v652: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.60k op/s; 9307/89394 objects degraded (10.411%) 2024-09-21T19:06:29.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:29 smithi081 ceph-mon[39192]: pgmap v652: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.60k op/s; 9307/89394 objects degraded (10.411%) 2024-09-21T19:06:29.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.648 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:29.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:29.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:29.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:29.908 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:29.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:30.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.049 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:30.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:30.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell -n osd.5 --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.9.7.5 2024-09-21T19:06:30.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:30.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:30.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 15 16 36652 36636 152.635 115 0.00617196 0.00654581 2024-09-21T19:06:30.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:30.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:30.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:30.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:30.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:30 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:30.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:30 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:06:30.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:30 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:06:30.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:30 smithi104 ceph-mon[32438]: osdmap e337: 8 total, 7 up, 8 in 2024-09-21T19:06:30.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:30 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 9307/89394 objects degraded (10.411%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:30.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:30 smithi104 ceph-mon[32438]: osdmap e338: 8 total, 7 up, 8 in 2024-09-21T19:06:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:30.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[29384]: osdmap e337: 8 total, 7 up, 8 in 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 9307/89394 objects degraded (10.411%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[29384]: osdmap e338: 8 total, 7 up, 8 in 2024-09-21T19:06:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[39192]: osdmap e337: 8 total, 7 up, 8 in 2024-09-21T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 9307/89394 objects degraded (10.411%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:30 smithi081 ceph-mon[39192]: osdmap e338: 8 total, 7 up, 8 in 2024-09-21T19:06:30.589 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:30.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:30.726 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:30.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:30.866 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:30.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:30.903 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T19:06:30.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:06:30.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:30.999 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:31.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:31.137 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:31.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 16 15 38680 38665 151.021 126.812 0.00600765 0.00661714 2024-09-21T19:06:31.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:31 smithi104 ceph-mon[32438]: pgmap v655: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 913 B/s rd, 108 MiB/s wr, 1.72k op/s; 10326/98778 objects degraded (10.454%) 2024-09-21T19:06:31.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:31.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:31.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.398 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:31.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:31.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:31.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:31.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:31 smithi081 ceph-mon[29384]: pgmap v655: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 913 B/s rd, 108 MiB/s wr, 1.72k op/s; 10326/98778 objects degraded (10.454%) 2024-09-21T19:06:31.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:31 smithi081 ceph-mon[39192]: pgmap v655: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.0 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 913 B/s rd, 108 MiB/s wr, 1.72k op/s; 10326/98778 objects degraded (10.454%) 2024-09-21T19:06:31.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.677 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:31.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:31.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:31.799 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:31.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:31.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:31.932 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:31.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:32.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:32.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:32.193 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:32.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 17 15 40415 40400 148.515 108.438 0.0053062 0.00672833 2024-09-21T19:06:32.210 INFO:teuthology.orchestra.run.smithi104.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T19:06:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:32.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:32.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:32.493 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:32.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:32.633 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:32.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:32.766 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:32.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:32.897 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:32.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:32.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:33.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:33.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:33.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:33.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 18 15 42258 42243 146.663 115.188 0.00657401 0.00681359 2024-09-21T19:06:33.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:33.283 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:33 smithi104 ceph-mon[32438]: pgmap v656: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s; 10326/101799 objects degraded (10.144%) 2024-09-21T19:06:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.309 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:33.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:33.352 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237756 8.22 8.9e11cee2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:06:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 277.60931849500002, 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94932720900000001, 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237756 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:06:33.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738064+0000", 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738066+0000", 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6950000000000001e-06 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738071+0000", 2024-09-21T19:06:33.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1390000000000003e-06 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738083+0000", 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1609e-05 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738088+0000", 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3000000000000001e-06 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738133+0000", 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5189999999999999e-05 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.738139+0000", 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5860000000000004e-06 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.687381+0000", 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.94924230799999998 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.687391+0000", 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0382e-05 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2028 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94695648323712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 276.07499276200002, 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43766471099999998, 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.358 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2028 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5080000000000001e-06 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8600000000000005e-06 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272420+0000", 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6631999999999999e-05 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1506e-05 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272509+0000", 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7401e-05 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272519+0000", 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0308e-05 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.701593+0000", 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.429073594 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.701753+0000", 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000160717 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.710055+0000", 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083011850000000009 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2038 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94695648323712 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 275.79368811500001, 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14878480499999999, 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:06:33.363 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2038 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553695+0000", 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553698+0000", 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0649999999999999e-06 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553706+0000", 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3359999999999995e-06 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553727+0000", 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0899000000000001e-05 2024-09-21T19:06:33.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553737+0000", 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0056e-05 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553819+0000", 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2121999999999997e-05 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:06:33.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553828+0000", 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6789999999999994e-06 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702052+0000", 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.148224774 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702124+0000", 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1186000000000006e-05 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702479+0000", 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000355688 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:239931 8.17 8:ec94a649:::benchmark_data_smithi081_105538_object107699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T19:06:33.368 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 273.80114363400003, 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13502588600000001, 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 239931 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:06:33.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546239+0000", 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546241+0000", 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9300000000000002e-06 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546246+0000", 2024-09-21T19:06:33.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3970000000000002e-06 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546258+0000", 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546265+0000", 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.99e-06 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546441+0000", 2024-09-21T19:06:33.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017672099999999999 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546556+0000", 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114626 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.546806+0000", 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025021199999999998 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.549068+0000", 2024-09-21T19:06:33.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002261833 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681108+0000", 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13204001600000001 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681166+0000", 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8137999999999998e-05 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:59.681265+0000", 2024-09-21T19:06:33.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8869000000000005e-05 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95331 9.7 9:e0eb634d:::benchmark_data_smithi081_131274_object95330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.920524+0000", 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 174.426858753, 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15383042099999999, 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.374 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95331 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920524+0000", 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920524+0000", 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920525+0000", 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3400000000000001e-06 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920589+0000", 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3471000000000003e-05 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920597+0000", 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8339999999999999e-06 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920601+0000", 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6809999999999997e-06 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920638+0000", 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6457e-05 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.920730+0000", 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2324999999999994e-05 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.921080+0000", 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034981600000000003 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926003+0000", 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049228919999999999 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926023+0000", 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9816000000000001e-05 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074262+0000", 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14823952100000001 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074313+0000", 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1267999999999998e-05 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074354+0000", 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1e-05 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.379 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95332 9.7 9:e12c8675:::benchmark_data_smithi081_131274_object95331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.922873+0000", 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 174.42450952900001, 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15164102199999999, 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95332 2024-09-21T19:06:33.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922873+0000", 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922873+0000", 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922875+0000", 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.418e-06 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922926+0000", 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1678999999999998e-05 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922938+0000", 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1871999999999999e-05 2024-09-21T19:06:33.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.922946+0000", 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4399999999999999e-06 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.923421+0000", 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047575800000000001 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.923549+0000", 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012808399999999999 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.923689+0000", 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013961999999999999 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926108+0000", 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00241873 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.926417+0000", 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000308843 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074412+0000", 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.147995024 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074471+0000", 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9035e-05 2024-09-21T19:06:33.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074514+0000", 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3519000000000001e-05 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95346 9.7 9:e6facc78:::benchmark_data_smithi081_131274_object95345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.932556+0000", 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 174.414826678, 2024-09-21T19:06:33.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.142115833, 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95346 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932556+0000", 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932556+0000", 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932558+0000", 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.705e-06 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932617+0000", 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9277000000000002e-05 2024-09-21T19:06:33.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932629+0000", 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.238e-05 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932637+0000", 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1980000000000004e-06 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932693+0000", 2024-09-21T19:06:33.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6020999999999998e-05 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932830+0000", 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013747999999999999 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.932997+0000", 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016660700000000001 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.935340+0000", 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023430180000000001 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.936494+0000", 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011541240000000001 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074569+0000", 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.138075219 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074627+0000", 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7645999999999997e-05 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.074672+0000", 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5157999999999997e-05 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:95350 9.d 9:b86b8700:::benchmark_data_smithi081_131274_object95349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:38.937661+0000", 2024-09-21T19:06:33.392 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 174.40972150799999, 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.136252596, 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95350 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937661+0000", 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937661+0000", 2024-09-21T19:06:33.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937663+0000", 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7120000000000001e-06 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937721+0000", 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7759e-05 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937732+0000", 2024-09-21T19:06:33.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1389000000000001e-05 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937738+0000", 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.02e-06 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937795+0000", 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7093999999999999e-05 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.937930+0000", 2024-09-21T19:06:33.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013523499999999999 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.938083+0000", 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000152292 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.941028+0000", 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029454379999999999 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:38.941051+0000", 2024-09-21T19:06:33.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3169000000000001e-05 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.073785+0000", 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13273373499999999 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.073863+0000", 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7625000000000003e-05 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:39.073914+0000", 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1128000000000002e-05 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:120206 9.e 9:79176c90:::benchmark_data_smithi081_131274_object120205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:00.883836+0000", 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 152.463547108, 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15420758500000001, 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.399 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 120206 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883836+0000", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883836+0000", 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883837+0000", 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2079999999999999e-06 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883868+0000", 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1656000000000003e-05 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883879+0000", 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0866999999999999e-05 2024-09-21T19:06:33.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883887+0000", 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7389999999999999e-06 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.883943+0000", 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6378000000000001e-05 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.884055+0000", 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011174 2024-09-21T19:06:33.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.884194+0000", 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013911800000000001 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.888029+0000", 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038351430000000001 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.888089+0000", 2024-09-21T19:06:33.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9212999999999997e-05 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.037963+0000", 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14987436300000001 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.038005+0000", 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2434000000000001e-05 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.038043+0000", 2024-09-21T19:06:33.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7725999999999998e-05 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:120212 9.d 9:b677527e:::benchmark_data_smithi081_131274_object120211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:00.889552+0000", 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 152.45783052100001, 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15145771499999999, 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.405 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 120212 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889552+0000", 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889552+0000", 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889554+0000", 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5510000000000001e-06 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889582+0000", 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7877000000000001e-05 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889592+0000", 2024-09-21T19:06:33.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.059e-05 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889599+0000", 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2010000000000002e-06 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889706+0000", 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010678700000000001 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.889854+0000", 2024-09-21T19:06:33.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014771200000000001 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.890031+0000", 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017752499999999999 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.893604+0000", 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035727060000000001 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.893635+0000", 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1044000000000003e-05 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040917+0000", 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14728138399999999 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040971+0000", 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4277000000000002e-05 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.041010+0000", 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9060999999999998e-05 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:120227 9.3 9:ce04ec7c:::benchmark_data_smithi081_131274_object120226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:04:00.902473+0000", 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 152.44490917499999, 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13844619799999999, 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.411 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 120227 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902473+0000", 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902473+0000", 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902475+0000", 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.699e-06 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902526+0000", 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0787e-05 2024-09-21T19:06:33.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902544+0000", 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7873999999999999e-05 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902551+0000", 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9859999999999997e-06 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902590+0000", 2024-09-21T19:06:33.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9415000000000001e-05 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902697+0000", 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106425 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,3", 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.902822+0000", 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012568799999999999 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.904654+0000", 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0018317310000000001 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:00.905187+0000", 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053294099999999999 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040812+0000", 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13562534100000001 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040878+0000", 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5909999999999997e-05 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:04:01.040920+0000", 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1400999999999999e-05 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:197913 9.e 9:7202835b:::benchmark_data_smithi081_131274_object70058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:06.808210+0000", 2024-09-21T19:06:33.417 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.539173117000004, 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13873943799999999, 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 197913 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.808210+0000", 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.808210+0000", 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.808210+0000", 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0900000000000001e-07 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.808214+0000", 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9009999999999996e-06 2024-09-21T19:06:33.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.808226+0000", 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1535000000000001e-05 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.808235+0000", 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9840000000000007e-06 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.810663+0000", 2024-09-21T19:06:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00242859 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.810798+0000", 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134299 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.810943+0000", 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014553999999999999 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.813372+0000", 2024-09-21T19:06:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024288930000000001 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.815855+0000", 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024832080000000002 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.946868+0000", 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.131012356 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.946912+0000", 2024-09-21T19:06:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4044e-05 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:06.946949+0000", 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7379000000000002e-05 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198227 9.e 9:7230a87b:::benchmark_data_smithi081_131274_object70372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.172077+0000", 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.175305882000004, 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150248088, 2024-09-21T19:06:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198227 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172077+0000", 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172077+0000", 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172078+0000", 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.689e-06 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172083+0000", 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7949999999999998e-06 2024-09-21T19:06:33.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172094+0000", 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0696999999999999e-05 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172101+0000", 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6730000000000003e-06 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172186+0000", 2024-09-21T19:06:33.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5236999999999997e-05 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172311+0000", 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012496299999999999 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172424+0000", 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113214 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180473+0000", 2024-09-21T19:06:33.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080486370000000008 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.182867+0000", 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023945630000000002 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.322262+0000", 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.139394979 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.322295+0000", 2024-09-21T19:06:33.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2787000000000003e-05 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.322325+0000", 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9853999999999999e-05 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198231 9.e 9:706c781d:::benchmark_data_smithi081_131274_object70376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.172255+0000", 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.175127360000005, 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.17373862700000001, 2024-09-21T19:06:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198231 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172255+0000", 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172255+0000", 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172256+0000", 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4900000000000005e-07 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172259+0000", 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9610000000000001e-06 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172266+0000", 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5660000000000003e-06 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.172271+0000", 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0799999999999996e-06 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.175424+0000", 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031531810000000001 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.175548+0000", 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012370799999999999 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.175657+0000", 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108873 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:33.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.182850+0000", 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0071937659999999999 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.182872+0000", 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1691e-05 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.345897+0000", 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16302512799999999 2024-09-21T19:06:33.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.345949+0000", 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2141000000000001e-05 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.345994+0000", 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4682999999999999e-05 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.436 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198235 9.e 9:7d927309:::benchmark_data_smithi081_131274_object70380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.179054+0000", 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.168328270000003, 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.167299745, 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198235 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.179054+0000", 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.179054+0000", 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.179057+0000", 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2460000000000002e-06 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.179063+0000", 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9359999999999996e-06 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.179079+0000", 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6600999999999999e-05 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.179087+0000", 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1820000000000006e-06 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180513+0000", 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001425796 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180666+0000", 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015243900000000001 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180793+0000", 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012759499999999999 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.182971+0000", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021776439999999998 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.183155+0000", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000184618 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.346277+0000", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.16312188399999999 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.346321+0000", 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4131000000000001e-05 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.346354+0000", 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2672999999999997e-05 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198239 9.d 9:b1e4eb4d:::benchmark_data_smithi081_131274_object70384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.180588+0000", 2024-09-21T19:06:33.442 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.166794734000007, 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150357251, 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198239 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180588+0000", 2024-09-21T19:06:33.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180588+0000", 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180589+0000", 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4339999999999999e-06 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180594+0000", 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5750000000000002e-06 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180605+0000", 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1447000000000001e-05 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180612+0000", 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8070000000000002e-06 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180836+0000", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000223351 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.180934+0000", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8194000000000002e-05 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181032+0000", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8540000000000002e-05 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.185153+0000", 2024-09-21T19:06:33.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041207739999999998 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.187572+0000", 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024189559999999999 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.330825+0000", 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14325256 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.330902+0000", 2024-09-21T19:06:33.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6981e-05 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.330945+0000", 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3631999999999998e-05 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198241 9.d 9:bc7a0fa1:::benchmark_data_smithi081_131274_object70386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.181178+0000", 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.166204765000003, 2024-09-21T19:06:33.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14989962800000001, 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198241 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181178+0000", 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181178+0000", 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181179+0000", 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.099e-06 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181186+0000", 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6560000000000003e-06 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181195+0000", 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9360000000000005e-06 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181202+0000", 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3180000000000001e-06 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.183294+0000", 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020919720000000001 2024-09-21T19:06:33.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.183410+0000", 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011593400000000001 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.183544+0000", 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134609 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.185173+0000", 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001628357 2024-09-21T19:06:33.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.187578+0000", 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024049140000000002 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.330978+0000", 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.143400008 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331044+0000", 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6422999999999997e-05 2024-09-21T19:06:33.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331078+0000", 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3402000000000003e-05 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198242 9.d 9:b3961396:::benchmark_data_smithi081_131274_object70387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.181415+0000", 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.165967569000003, 2024-09-21T19:06:33.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14982523, 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198242 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181415+0000", 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181415+0000", 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181416+0000", 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1060000000000001e-06 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181421+0000", 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4630000000000003e-06 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.456 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181429+0000", 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1519999999999992e-06 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181435+0000", 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5819999999999998e-06 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.185195+0000", 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037598570000000001 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.185266+0000", 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0765000000000004e-05 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.185330+0000", 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4348000000000001e-05 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.187611+0000", 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022804399999999999 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.187628+0000", 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7504e-05 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331111+0000", 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.143482729 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331187+0000", 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6075000000000006e-05 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331240+0000", 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3208999999999998e-05 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.15816.0:198243 9.d 9:b6f6b660:::benchmark_data_smithi081_131274_object70388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:05:07.181592+0000", 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 86.165790391000002, 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.14986010499999999, 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.460 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.15816", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198243 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181592+0000", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181592+0000", 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181593+0000", 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9800000000000002e-07 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181597+0000", 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.36e-06 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181604+0000", 2024-09-21T19:06:33.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2830000000000004e-06 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.181610+0000", 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2419999999999997e-06 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.186260+0000", 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046502490000000004 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.186369+0000", 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108799 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.186485+0000", 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000115574 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.187651+0000", 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011660780000000001 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.187699+0000", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.85e-05 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331354+0000", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.143654426 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331406+0000", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1895999999999999e-05 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:05:07.331452+0000", 2024-09-21T19:06:33.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6799999999999999e-05 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31269 10.c 10:39bdf063:::benchmark_data_smithi081_155596_object31268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.608521+0000", 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.7388613189999997, 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.184089792, 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31269 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608521+0000", 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608521+0000", 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608523+0000", 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5090000000000001e-06 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608559+0000", 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5704999999999998e-05 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608567+0000", 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7960000000000001e-06 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608574+0000", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3790000000000003e-06 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608749+0000", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000174778 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.608889+0000", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014011199999999999 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,4", 2024-09-21T19:06:33.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.609020+0000", 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013114800000000001 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611339+0000", 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023192019999999998 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611437+0000", 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8426999999999998e-05 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:33.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.792526+0000", 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.181088638 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.792577+0000", 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1038000000000002e-05 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.792611+0000", 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4060000000000003e-05 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:33.472 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:33.472 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:06:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.473 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:33.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:33.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:33 smithi081 ceph-mon[29384]: pgmap v656: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s; 10326/101799 objects degraded (10.144%) 2024-09-21T19:06:33.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:33 smithi081 ceph-mon[39192]: pgmap v656: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s; 10326/101799 objects degraded (10.144%) 2024-09-21T19:06:33.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:33.611 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:33.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:33.750 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:33.808 INFO:teuthology.orchestra.run.smithi104.stdout:get_pg_num_history pg_num_history pg_num_history(e326 pg_nums {1={168=11},2={56=3},3={57=32},4={59=32},5={61=32},6={63=32},9={254=16}} deleted_pools ) 2024-09-21T19:06:33.808 INFO:teuthology.orchestra.run.smithi104.stdout:Importing pgid 9.7 2024-09-21T19:06:33.822 INFO:teuthology.orchestra.run.smithi104.stdout:write_pg epoch 325 info 9.7( v 325'16042 (303'11412,325'16042] local-lis/les=274/275 n=0 ec=254/254 lis/c=274/274 les/c/f=275/275/0 sis=274) 2024-09-21T19:06:33.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:33.869 INFO:teuthology.orchestra.run.smithi104.stdout:Import successful 2024-09-21T19:06:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:33.888 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:33.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:33.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:34.029 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:34.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:34.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:34.179 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:34.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 19 16 44129 44113 145.094 116.875 0.0078397 0.00688748 2024-09-21T19:06:34.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:34.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:34.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:34.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:34.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:34.704 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:34.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:34.844 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:34.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:34.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:35.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:35.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:35.197 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:06:35.196425+0000 min lat: 0.00110519 max lat: 0.2501 avg lat: 0.00700608 2024-09-21T19:06:35.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:06:35.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 20 16 45663 45647 142.633 95.875 0.00560653 0.00700608 2024-09-21T19:06:35.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:35.253 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:35.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:35.395 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:35.436 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:35 smithi104 ceph-mon[32438]: pgmap v657: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 123 MiB/s wr, 1.97k op/s; 11978/114627 objects degraded (10.450%) 2024-09-21T19:06:35.436 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:35 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 11978/114627 objects degraded (10.450%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:35.436 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:35 smithi104 ceph-mon[32438]: osdmap e339: 8 total, 7 up, 8 in 2024-09-21T19:06:35.436 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:35 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4183270043' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:35.436 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:35 smithi104 ceph-mon[32438]: from='client.36113 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:35.484 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:06:35.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:35.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[29384]: pgmap v657: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 123 MiB/s wr, 1.97k op/s; 11978/114627 objects degraded (10.450%) 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 11978/114627 objects degraded (10.450%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[29384]: osdmap e339: 8 total, 7 up, 8 in 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4183270043' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[29384]: from='client.36113 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[39192]: pgmap v657: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 123 MiB/s wr, 1.97k op/s; 11978/114627 objects degraded (10.450%) 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 11978/114627 objects degraded (10.450%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[39192]: osdmap e339: 8 total, 7 up, 8 in 2024-09-21T19:06:35.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4183270043' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:35.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:35 smithi081 ceph-mon[39192]: from='client.36113 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:35.636 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:35.676 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:35.687 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:06:35.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:35.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:35.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:35.946 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:35.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:36.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:36.082 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:36.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:06:36.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:36.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 21 16 47556 47540 141.474 118.312 0.0138562 0.00706389 2024-09-21T19:06:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:36.316 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:36.345 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:36.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:36.485 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:36.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:36 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:36 smithi104 ceph-mon[32438]: from='client.36113 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:36 smithi104 ceph-mon[32438]: osdmap e340: 8 total, 7 up, 8 in 2024-09-21T19:06:36.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:36 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:36.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:36 smithi081 ceph-mon[39192]: from='client.36113 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:36.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:36 smithi081 ceph-mon[39192]: osdmap e340: 8 total, 7 up, 8 in 2024-09-21T19:06:36.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:36 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:36.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:36 smithi081 ceph-mon[29384]: from='client.36113 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:36.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:36 smithi081 ceph-mon[29384]: osdmap e340: 8 total, 7 up, 8 in 2024-09-21T19:06:36.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:36.624 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/exp.9.7.5 2024-09-21T19:06:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:36.682 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:36.689 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:36.754 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:36.784 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:36.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:36.916 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:36.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:37.019 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.059 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:37.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:37.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:37.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 22 16 49293 49277 139.977 108.562 0.00549938 0.00713832 2024-09-21T19:06:37.201 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49427 10.b 10:d0a6943b:::benchmark_data_smithi081_155596_object49426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.275053+0000", 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00039363499999999998, 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042899700000000003, 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.279 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49427 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275053+0000", 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275053+0000", 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275055+0000", 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1850000000000001e-06 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275118+0000", 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3106000000000002e-05 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275128+0000", 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9510000000000001e-06 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275134+0000", 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7190000000000004e-06 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275186+0000", 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2191999999999998e-05 2024-09-21T19:06:37.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275325+0000", 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013879499999999999 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275454+0000", 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128868 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.283 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49407 10.b 10:d56a5e53:::benchmark_data_smithi081_155596_object49406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.263496+0000", 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.011950499, 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012072972, 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49407 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263496+0000", 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263496+0000", 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263498+0000", 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5400000000000001e-06 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263565+0000", 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6543000000000003e-05 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263577+0000", 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2233e-05 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263586+0000", 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1079999999999999e-06 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263640+0000", 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4157999999999999e-05 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263746+0000", 2024-09-21T19:06:37.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106023 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.263899+0000", 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015338600000000001 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.265439+0000", 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015399039999999999 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268440+0000", 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003000398 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49412 10.f 10:f0707c9f:::benchmark_data_smithi081_155596_object49411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.266761+0000", 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0086860140000000006, 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088892840000000008, 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49412 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266761+0000", 2024-09-21T19:06:37.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266761+0000", 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266763+0000", 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.629e-06 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266815+0000", 2024-09-21T19:06:37.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2902999999999998e-05 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266830+0000", 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4717e-05 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266838+0000", 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2879999999999993e-06 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.267262+0000", 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00042374999999999997 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.267343+0000", 2024-09-21T19:06:37.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0556000000000006e-05 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,3", 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.267440+0000", 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6792000000000004e-05 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.269593+0000", 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021532449999999998 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:37.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274843+0000", 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052503560000000003 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49413 10.b 10:d3c03969:::benchmark_data_smithi081_155596_object49412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.266854+0000", 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0085927020000000007, 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088651619999999993, 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.295 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49413 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266854+0000", 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266854+0000", 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266855+0000", 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2999999999999999e-07 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266894+0000", 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9384000000000001e-05 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266904+0000", 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1719999999999996e-06 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266912+0000", 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7220000000000005e-06 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.266964+0000", 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1177000000000001e-05 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.267071+0000", 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107113 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.267153+0000", 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2447999999999994e-05 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268567+0000", 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001414324 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.272169+0000", 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003601539 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.299 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49414 10.f 10:f9f30d7b:::benchmark_data_smithi081_155596_object49413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.268054+0000", 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0073925190000000002, 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0077309299999999996, 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49414 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268054+0000", 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268054+0000", 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268056+0000", 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2359999999999999e-06 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268131+0000", 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5607999999999998e-05 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268145+0000", 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3900000000000001e-05 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268153+0000", 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1219999999999995e-06 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268194+0000", 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0998000000000003e-05 2024-09-21T19:06:37.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268298+0000", 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104108 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,3", 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.268459+0000", 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000160679 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.269735+0000", 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012754450000000001 2024-09-21T19:06:37.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274968+0000", 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052333029999999999 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49420 10.7 10:eec03175:::benchmark_data_smithi081_155596_object49419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.273660+0000", 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0017867099999999999, 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021898690000000001, 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.305 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49420 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273660+0000", 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273660+0000", 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273662+0000", 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.483e-06 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273718+0000", 2024-09-21T19:06:37.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5869000000000003e-05 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273727+0000", 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1600000000000004e-06 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273732+0000", 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1970000000000004e-06 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273781+0000", 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9449e-05 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273872+0000", 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0969000000000003e-05 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3", 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.273957+0000", 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4847999999999998e-05 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.275812+0000", 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001854316 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:49424 10.a 10:55c29f68:::benchmark_data_smithi081_155596_object49423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:37.274211+0000", 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0012355529999999999, 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001701412, 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:06:37.311 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 49424 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274211+0000", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274211+0000", 2024-09-21T19:06:37.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274212+0000", 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2200000000000003e-07 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274263+0000", 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0754999999999998e-05 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274271+0000", 2024-09-21T19:06:37.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7640000000000005e-06 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274275+0000", 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3560000000000003e-06 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274319+0000", 2024-09-21T19:06:37.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3519000000000001e-05 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274424+0000", 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010492000000000001 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0", 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:37.274550+0000", 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012616 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.315 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:06:37.316 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:06:37.316 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 7 2024-09-21T19:06:37.316 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:06:37.325 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:37.325 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:37.341 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:37.358 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:37.359 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:37.397 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:37.397 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-mon[39192]: pgmap v660: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 849 B/s rd, 91 MiB/s wr, 1.46k op/s; 13194/125178 objects degraded (10.540%) 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-mon[29384]: pgmap v660: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 849 B/s rd, 91 MiB/s wr, 1.46k op/s; 13194/125178 objects degraded (10.540%) 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:37.398 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:06:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:06:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:06:37.432 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:37.432 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:37.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:37.465 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:37.466 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:37.470 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:37.473 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:37.504 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:37.504 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:37.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:37 smithi104 ceph-mon[32438]: pgmap v660: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 849 B/s rd, 91 MiB/s wr, 1.46k op/s; 13194/125178 objects degraded (10.540%) 2024-09-21T19:06:37.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:37.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:37.535 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:37.535 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:37.576 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:37.601 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:37.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:37.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:37.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:37.872 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:37.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:37.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:38.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:38.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:38.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:38.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:06:38.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 23 16 51072 51056 138.725 111.188 0.01393 0.00720341 2024-09-21T19:06:38.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:38.292 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:38.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:38.433 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/73482328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:38.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:06:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:06:38.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:38.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/73482328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:38.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/73482328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:38.566 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:38.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:38.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:06:38.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:38.701 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:38.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:38.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:38.830 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:38.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:38.962 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:38.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:39.058 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:39.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:39.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:39.138 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:39.138 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:39.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 24 16 52837 52821 137.541 110.312 0.00592475 0.00726639 2024-09-21T19:06:39.214 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:39.244 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:39 smithi104 ceph-mon[32438]: pgmap v661: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 14903/141765 objects degraded (10.512%) 2024-09-21T19:06:39.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:39.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:39.502 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:39 smithi081 ceph-mon[39192]: pgmap v661: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 14903/141765 objects degraded (10.512%) 2024-09-21T19:06:39.502 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:39 smithi081 ceph-mon[29384]: pgmap v661: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 14903/141765 objects degraded (10.512%) 2024-09-21T19:06:39.527 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:39.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:39.668 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:39.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:39.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:39.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:06:39.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:39.947 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:39.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:40.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:40.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:40.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 25 16 54464 54448 136.106 101.688 0.0136487 0.00734223 2024-09-21T19:06:40.222 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:40.312 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:40.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:40.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:40 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:40.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:40 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 14903/141765 objects degraded (10.512%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:40.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:40 smithi104 ceph-mon[32438]: osdmap e341: 8 total, 7 up, 8 in 2024-09-21T19:06:40.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3186791547' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:40.448 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:40 smithi104 ceph-mon[32438]: from='client.36122 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:40.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:40.493 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 14903/141765 objects degraded (10.512%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[39192]: osdmap e341: 8 total, 7 up, 8 in 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3186791547' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[39192]: from='client.36122 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 14903/141765 objects degraded (10.512%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[29384]: osdmap e341: 8 total, 7 up, 8 in 2024-09-21T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3186791547' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:40 smithi081 ceph-mon[29384]: from='client.36122 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:06:40.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:40.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:40.697 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:06:40.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.764 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:40.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:40.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:06:40.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:40.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:40.991 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:40.991 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:40.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:41.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:41.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:41.170 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:41.198 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 26 16 55992 55976 134.542 95.5 0.0132694 0.00742847 2024-09-21T19:06:41.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:41.313 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:41.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:41.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:41 smithi104 ceph-mon[32438]: pgmap v663: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.04k op/s; 15962/151440 objects degraded (10.540%) 2024-09-21T19:06:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:41 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:41 smithi104 ceph-mon[32438]: from='client.36122 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:06:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:41 smithi104 ceph-mon[32438]: osdmap e342: 8 total, 7 up, 8 in 2024-09-21T19:06:41.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[29384]: pgmap v663: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.04k op/s; 15962/151440 objects degraded (10.540%) 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[29384]: from='client.36122 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[29384]: osdmap e342: 8 total, 7 up, 8 in 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[39192]: pgmap v663: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.04k op/s; 15962/151440 objects degraded (10.540%) 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[39192]: from='client.36122 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:06:41.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:41 smithi081 ceph-mon[39192]: osdmap e342: 8 total, 7 up, 8 in 2024-09-21T19:06:41.581 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:41.655 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 7, 6] 2024-09-21T19:06:41.655 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-21T19:06:41.655 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-21T19:06:41.655 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T19:06:41.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl start ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5 2024-09-21T19:06:41.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:41.721 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:41.823 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.859 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:41.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:41.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:41.994 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:41.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:42.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:41 smithi104 systemd[1]: Starting Ceph osd.5 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:06:42.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:42.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:42.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 27 16 57502 57486 133.046 94.375 0.00584456 0.00751119 2024-09-21T19:06:42.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:42.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:42.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:42.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:42.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:42.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:42.665 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:42.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:42.797 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:42.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:42.919 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:42.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:42.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:43.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:43.059 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:43.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:43.200 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 28 15 59170 59155 132.02 104.312 0.0216492 0.00756971 2024-09-21T19:06:43.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:06:43.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:43.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:43 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:06:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:43 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:06:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:43 smithi081 ceph-mon[29384]: pgmap v665: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 102 MiB/s wr, 1.63k op/s; 15962/154539 objects degraded (10.329%) 2024-09-21T19:06:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:43 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:06:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:43 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:06:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:43 smithi081 ceph-mon[39192]: pgmap v665: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 102 MiB/s wr, 1.63k op/s; 15962/154539 objects degraded (10.329%) 2024-09-21T19:06:43.324 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:43.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:43.434 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:43 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:06:43.434 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:43 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:06:43.434 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:43 smithi104 ceph-mon[32438]: pgmap v665: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 102 MiB/s wr, 1.63k op/s; 15962/154539 objects degraded (10.329%) 2024-09-21T19:06:43.454 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:43.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:43.605 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:43.606 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:43.606 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:43.635 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:43.636 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:43.665 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:43.697 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:43.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:43.735 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:43.735 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:43.735 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:43.736 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 podman[122495]: 2024-09-21 19:06:43.336381529 +0000 UTC m=+0.021317374 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:06:43.736 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 podman[122495]: 2024-09-21 19:06:43.464446083 +0000 UTC m=+0.149381889 container create 249c9846cdad75a0932eff2e483c1c72b3649219dcb813932ba4083e73dd3542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T19:06:43.749 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:43.768 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:43.768 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:43.803 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:43.803 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:43.851 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:43.883 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:43.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:43.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 podman[122495]: 2024-09-21 19:06:43.734312378 +0000 UTC m=+0.419248203 container init 249c9846cdad75a0932eff2e483c1c72b3649219dcb813932ba4083e73dd3542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 podman[122495]: 2024-09-21 19:06:43.741912981 +0000 UTC m=+0.426848793 container start 249c9846cdad75a0932eff2e483c1c72b3649219dcb813932ba4083e73dd3542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 podman[122495]: 2024-09-21 19:06:43.850796563 +0000 UTC m=+0.535732390 container attach 249c9846cdad75a0932eff2e483c1c72b3649219dcb813932ba4083e73dd3542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 bash[122495]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:06:44.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:43 smithi104 bash[122495]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T19:06:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:44.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:44.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:44.158 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:44.200 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 29 16 60668 60652 130.693 93.5625 0.0118812 0.00764635 2024-09-21T19:06:44.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.307 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:44.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:44.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:44.435 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:44.506 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/986176798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:44.506 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/986176798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:44.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/986176798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:44.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:44.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61236 10.4 10:2b24650e:::benchmark_data_smithi081_155596_object61235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.580333+0000", 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001573348, 2024-09-21T19:06:44.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016301829999999999, 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61236 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580333+0000", 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580333+0000", 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580335+0000", 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0329999999999998e-06 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580361+0000", 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5485000000000001e-05 2024-09-21T19:06:44.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580497+0000", 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013609399999999999 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580506+0000", 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7839999999999998e-06 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580770+0000", 2024-09-21T19:06:44.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026403199999999998 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580922+0000", 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015144400000000001 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581156+0000", 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023443800000000001 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36077.0:61235 10.1 e342/330 10:8a2b9da0:::benchmark_data_smithi081_155596_object61234:head v 342'3914, mlcod=342'3910)", 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.580559+0000", 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0013474310000000001, 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014791190000000001, 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580559+0000", 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580559+0000", 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580560+0000", 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9800000000000003e-07 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580695+0000", 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013539300000000001 2024-09-21T19:06:44.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580742+0000", 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6508999999999998e-05 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580750+0000", 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1089999999999998e-06 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581132+0000", 2024-09-21T19:06:44.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038229300000000002 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581157+0000", 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5355999999999999e-05 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61237 10.6 10.85520c46 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.581921+0000", 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4294967295.9999852, 2024-09-21T19:06:44.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016308699999999999, 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61237 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581921+0000", 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581921+0000", 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581923+0000", 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7579999999999999e-06 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581950+0000", 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7131000000000002e-05 2024-09-21T19:06:44.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581961+0000", 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0890000000000001e-05 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581969+0000", 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.678e-06 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.596 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61242 10.6 10.62200556 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.581984+0000", 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4294967295.9999223, 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014232900000000001, 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61242 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581984+0000", 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581984+0000", 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581985+0000", 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2500000000000004e-07 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582004+0000", 2024-09-21T19:06:44.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9191999999999999e-05 2024-09-21T19:06:44.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582010+0000", 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3509999999999998e-06 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582016+0000", 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6239999999999999e-06 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.36077.0:61232 10.8 e342/330)", 2024-09-21T19:06:44.600 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.582031+0000", 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4294967295.9998751, 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013452599999999999, 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582031+0000", 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582031+0000", 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582032+0000", 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8999999999999995e-07 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582035+0000", 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1709999999999998e-06 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582041+0000", 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5430000000000001e-06 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582045+0000", 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.651e-06 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61218 10.8 10:1139a11b:::benchmark_data_smithi081_155596_object61217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.603 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.571560+0000", 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.010346213999999999, 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010646588, 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61218 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571560+0000", 2024-09-21T19:06:44.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571560+0000", 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571562+0000", 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.759e-06 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571675+0000", 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113299 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571691+0000", 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5359e-05 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571698+0000", 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.418e-06 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571879+0000", 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018128600000000001 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.572013+0000", 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013400900000000001 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,7", 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.572157+0000", 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000144084 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577880+0000", 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057230379999999997 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580389+0000", 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025083340000000001 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61223 10.4 10:25c87be4:::benchmark_data_smithi081_155596_object61222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.571715+0000", 2024-09-21T19:06:44.608 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.010191799, 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010565801999999999, 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61223 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571715+0000", 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571715+0000", 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.571715+0000", 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3600000000000003e-07 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.573955+0000", 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002239389 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.573966+0000", 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1749000000000001e-05 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.573974+0000", 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.627e-06 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.574266+0000", 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029228499999999998 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.574417+0000", 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150147 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.575649+0000", 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001232499 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580602+0000", 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.004952967 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61232 10.8 10:12856548:::benchmark_data_smithi081_155596_object61231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.577670+0000", 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0042361919999999997, 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046794319999999999, 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61232 2024-09-21T19:06:44.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577670+0000", 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577670+0000", 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577672+0000", 2024-09-21T19:06:44.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.539e-06 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577694+0000", 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2138e-05 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577702+0000", 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5599999999999994e-06 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577708+0000", 2024-09-21T19:06:44.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8629999999999996e-06 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577907+0000", 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000198505 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.578075+0000", 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000167999 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 2,7", 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.578206+0000", 2024-09-21T19:06:44.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013132000000000001 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.580624+0000", 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002417579 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.582316+0000", 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001692128 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:61233 10.6 10:688c6fc7:::benchmark_data_smithi081_155596_object61232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.577716+0000", 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0041903440000000004, 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047012440000000003, 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61233 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.618 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577716+0000", 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577716+0000", 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577717+0000", 2024-09-21T19:06:44.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5099999999999999e-07 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577738+0000", 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.126e-05 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577745+0000", 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6089999999999997e-06 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.577750+0000", 2024-09-21T19:06:44.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2089999999999996e-06 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579324+0000", 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015744139999999999 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579466+0000", 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014185599999999999 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4", 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579605+0000", 2024-09-21T19:06:44.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138555 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.581941+0000", 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023360360000000001 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36077.0:61234 10.a e342/330 10:5c06c175:::benchmark_data_smithi081_155596_object61233:head v 342'3818, mlcod=342'3816)", 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:44.579301+0000", 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0026055599999999998, 2024-09-21T19:06:44.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031818789999999999, 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579301+0000", 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579301+0000", 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579302+0000", 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1340000000000001e-06 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579388+0000", 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5688999999999999e-05 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579409+0000", 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1183e-05 2024-09-21T19:06:44.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579415+0000", 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4110000000000001e-06 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579501+0000", 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5446999999999997e-05 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:44.579520+0000", 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9074000000000001e-05 2024-09-21T19:06:44.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:06:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:06:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:06:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 10 2024-09-21T19:06:44.626 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:06:44.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:44.719 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:44.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:06:44.816 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T19:06:44.817 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T19:06:44.817 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T19:06:44.817 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T19:06:44.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:44.886 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:44.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:44.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.034 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:45.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:45.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:45.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:45.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:45.191 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T19:06:45.191 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T19:06:45.191 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T19:06:45.191 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T19:06:45.191 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-21T19:06:45.192 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-21T19:06:45.192 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T19:06:45.192 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T19:06:45.192 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate[122536]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-21T19:06:45.192 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 bash[122495]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-21T19:06:45.192 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:44 smithi104 podman[122495]: 2024-09-21 19:06:44.857415099 +0000 UTC m=+1.542350923 container died 249c9846cdad75a0932eff2e483c1c72b3649219dcb813932ba4083e73dd3542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T19:06:45.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 30 16 62145 62129 129.411 92.3125 0.00812097 0.00772252 2024-09-21T19:06:45.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:45.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:45.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:45 smithi104 ceph-mon[32438]: pgmap v666: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 17484/166941 objects degraded (10.473%) 2024-09-21T19:06:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:45 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T19:06:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:45 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T19:06:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:45 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 17484/166941 objects degraded (10.473%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:45 smithi104 ceph-mon[32438]: osdmap e343: 8 total, 7 up, 8 in 2024-09-21T19:06:45.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[29384]: pgmap v666: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 17484/166941 objects degraded (10.473%) 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 17484/166941 objects degraded (10.473%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[29384]: osdmap e343: 8 total, 7 up, 8 in 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[39192]: pgmap v666: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 17484/166941 objects degraded (10.473%) 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T19:06:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T19:06:45.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 17484/166941 objects degraded (10.473%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:45.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:45 smithi081 ceph-mon[39192]: osdmap e343: 8 total, 7 up, 8 in 2024-09-21T19:06:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.582 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:45.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:45.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:45.725 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:45.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:45.857 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:45.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:45.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:46.002 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:46.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:46.146 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:46.201 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 31 16 63783 63767 128.539 102.375 0.00950575 0.00777496 2024-09-21T19:06:46.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:46.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:46 smithi081 ceph-mon[39192]: 6.b scrub starts 2024-09-21T19:06:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:46 smithi081 ceph-mon[39192]: 6.b scrub ok 2024-09-21T19:06:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:46 smithi081 ceph-mon[29384]: 6.b scrub starts 2024-09-21T19:06:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:46 smithi081 ceph-mon[29384]: 6.b scrub ok 2024-09-21T19:06:46.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:46 smithi104 ceph-mon[32438]: 6.b scrub starts 2024-09-21T19:06:46.346 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:46 smithi104 ceph-mon[32438]: 6.b scrub ok 2024-09-21T19:06:46.347 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 podman[122712]: 2024-09-21 19:06:46.06530182 +0000 UTC m=+1.199924346 container remove 249c9846cdad75a0932eff2e483c1c72b3649219dcb813932ba4083e73dd3542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T19:06:46.383 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:46.418 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:46.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:06:46.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:46.558 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:46.589 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:46.655 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 podman[122848]: 2024-09-21 19:06:46.24843849 +0000 UTC m=+0.027591780 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:06:46.655 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 podman[122848]: 2024-09-21 19:06:46.383115678 +0000 UTC m=+0.162268958 container create 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T19:06:46.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:46.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.718 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:46.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:46.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:06:46.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:46.880 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:46.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:46.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:47.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 podman[122848]: 2024-09-21 19:06:46.653014943 +0000 UTC m=+0.432168232 container init 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731) 2024-09-21T19:06:47.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 podman[122848]: 2024-09-21 19:06:46.658185129 +0000 UTC m=+0.437338409 container start 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T19:06:47.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 bash[122848]: 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 2024-09-21T19:06:47.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:46 smithi104 systemd[1]: Started Ceph osd.5 for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:06:47.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:47.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:06:47.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.161 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:47.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:47.163 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:47.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 32 16 65172 65156 127.235 86.8125 0.00518188 0.00785481 2024-09-21T19:06:47.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:47.264 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:47.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:47.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:47.409 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-mon[39192]: 6.3 scrub starts 2024-09-21T19:06:47.410 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-mon[39192]: 6.3 scrub ok 2024-09-21T19:06:47.410 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-mon[39192]: pgmap v668: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 850 B/s rd, 84 MiB/s wr, 1.34k op/s; 18396/175728 objects degraded (10.468%) 2024-09-21T19:06:47.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-mon[29384]: 6.3 scrub starts 2024-09-21T19:06:47.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-mon[29384]: 6.3 scrub ok 2024-09-21T19:06:47.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-mon[29384]: pgmap v668: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 850 B/s rd, 84 MiB/s wr, 1.34k op/s; 18396/175728 objects degraded (10.468%) 2024-09-21T19:06:47.410 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:06:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:06:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:06:47.441 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:47.462 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:47 smithi104 ceph-mon[32438]: 6.3 scrub starts 2024-09-21T19:06:47.462 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:47 smithi104 ceph-mon[32438]: 6.3 scrub ok 2024-09-21T19:06:47.462 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:47 smithi104 ceph-mon[32438]: pgmap v668: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 850 B/s rd, 84 MiB/s wr, 1.34k op/s; 18396/175728 objects degraded (10.468%) 2024-09-21T19:06:47.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:47.575 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:47.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:47.705 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:47.731 INFO:teuthology.orchestra.run.smithi104.stderr:no valid command found; 10 closest matches: 2024-09-21T19:06:47.731 INFO:teuthology.orchestra.run.smithi104.stderr:0 2024-09-21T19:06:47.731 INFO:teuthology.orchestra.run.smithi104.stderr:1 2024-09-21T19:06:47.731 INFO:teuthology.orchestra.run.smithi104.stderr:2 2024-09-21T19:06:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:abort 2024-09-21T19:06:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:assert 2024-09-21T19:06:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:config diff 2024-09-21T19:06:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:config diff get 2024-09-21T19:06:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:config get 2024-09-21T19:06:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:config help [] 2024-09-21T19:06:47.733 INFO:teuthology.orchestra.run.smithi104.stderr:config set ... 2024-09-21T19:06:47.733 INFO:teuthology.orchestra.run.smithi104.stderr:admin_socket: invalid command 2024-09-21T19:06:47.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:47.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:47.920 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:06:47.920 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:06:47.920 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:06:47.921 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:06:47.921 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:06:47.945 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:47.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:48.004 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:48.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.153 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:48.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:48.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 33 16 66405 66389 125.714 77.0625 0.0107807 0.00794966 2024-09-21T19:06:48.229 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:06:48.254 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:48.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:48.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:48.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:48.443 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4213963124' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[29384]: from='client.36131 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4213963124' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:48.444 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:48 smithi081 ceph-mon[39192]: from='client.36131 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:48.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:06:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:06:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:48 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T19:06:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:48 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T19:06:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4213963124' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:48 smithi104 ceph-mon[32438]: from='client.36131 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:06:48.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:48.561 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:48.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:48.698 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:48.800 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:48.835 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:48.856 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:06:48.571+0000 7f179d677640 -1 Falling back to public interface 2024-09-21T19:06:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T19:06:48.903 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-21T19:06:48.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:48.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:06:48.965 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:48.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:49.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:49.098 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:49.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:49.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 34 15 67815 67800 124.61 88.1875 0.0065814 0.00801949 2024-09-21T19:06:49.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:49.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.370 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:49.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:49.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: pgmap v669: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 19828/190335 objects degraded (10.417%) 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: 6.e scrub starts 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: 6.e scrub ok 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: from='client.36131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: osdmap e344: 8 total, 7 up, 8 in 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T19:06:49.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: pgmap v669: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 19828/190335 objects degraded (10.417%) 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: 6.e scrub starts 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: 6.e scrub ok 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: from='client.36131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: osdmap e344: 8 total, 7 up, 8 in 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T19:06:49.497 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:49.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: pgmap v669: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 19828/190335 objects degraded (10.417%) 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: 6.e scrub starts 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: 6.e scrub ok 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: from='client.36131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: osdmap e344: 8 total, 7 up, 8 in 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T19:06:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:06:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.509 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:49.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:49.580 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:49.580 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:49.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:49.617 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:49.618 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:49.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:49.650 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:49.689 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:49.689 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:49.720 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:49.720 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:49.720 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T19:06:49.751 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:49.751 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:49.755 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:49.785 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:49.785 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:49.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:49.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:49.923 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:49.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:50.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:50.057 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:50.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:50.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:50.189 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:50.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 35 15 69257 69242 123.625 90.125 0.0121663 0.00808438 2024-09-21T19:06:50.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2569228942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 19828/190335 objects degraded (10.417%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: osdmap e345: 8 total, 7 up, 8 in 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: pgmap v672: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 108 MiB/s wr, 1.74k op/s; 20732/198192 objects degraded (10.461%) 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T19:06:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2569228942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 19828/190335 objects degraded (10.417%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: osdmap e345: 8 total, 7 up, 8 in 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: pgmap v672: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 108 MiB/s wr, 1.74k op/s; 20732/198192 objects degraded (10.461%) 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T19:06:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:50 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T19:06:50.320 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:50.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:50.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2569228942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 19828/190335 objects degraded (10.417%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: osdmap e345: 8 total, 7 up, 8 in 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: pgmap v672: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 108 MiB/s wr, 1.74k op/s; 20732/198192 objects degraded (10.461%) 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T19:06:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:50 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T19:06:50.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:50.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:50.720 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:50.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:50.858 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T19:06:50.890 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T19:06:50.934 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:50.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:50.991 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:50.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:51.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:51.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:51.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 36 16 70765 70749 122.807 94.1875 0.012488 0.00813871 2024-09-21T19:06:51.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:51.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.417 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:51.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:51.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.553 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:51.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:51.602 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:51.602 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:51.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:51.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:51.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:51.969 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:51.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:51 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:06:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:51 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:06:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:51 smithi104 ceph-mon[32438]: 6.2 scrub starts 2024-09-21T19:06:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:51 smithi104 ceph-mon[32438]: 6.2 scrub ok 2024-09-21T19:06:52.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:51 smithi104 ceph-mon[32438]: osdmap e346: 8 total, 7 up, 8 in 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[29384]: 6.2 scrub starts 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[29384]: 6.2 scrub ok 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[29384]: osdmap e346: 8 total, 7 up, 8 in 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[39192]: 6.2 scrub starts 2024-09-21T19:06:52.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[39192]: 6.2 scrub ok 2024-09-21T19:06:52.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:51 smithi081 ceph-mon[39192]: osdmap e346: 8 total, 7 up, 8 in 2024-09-21T19:06:52.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:52.108 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:52.112 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:06:52.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 37 16 72146 72130 121.82 86.3125 0.00817885 0.00820309 2024-09-21T19:06:52.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:52.245 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:52.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:52.394 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:52.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:06:52.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:52.524 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:52.577 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:52.626 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:52.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:52.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:06:52.382+0000 7f179d677640 -1 osd.5 326 log_to_monitors true 2024-09-21T19:06:52.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.848 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:52.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:52.891 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:06:52.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:52.974 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:52.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: pgmap v674: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s; 20732/200880 objects degraded (10.321%) 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: 3.18 scrub starts 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: 3.18 scrub ok 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: from='osd.5 v1:172.21.15.104:6804/3649492645' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T19:06:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: pgmap v674: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s; 20732/200880 objects degraded (10.321%) 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: 3.18 scrub starts 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: 3.18 scrub ok 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: from='osd.5 v1:172.21.15.104:6804/3649492645' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T19:06:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:52 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T19:06:53.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:53.099 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:53.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:53.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 38 16 73079 73063 120.149 58.3125 0.0471653 0.00831054 2024-09-21T19:06:53.221 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:53.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: pgmap v674: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s; 20732/200880 objects degraded (10.321%) 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: 3.18 scrub starts 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: 3.18 scrub ok 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: from='osd.5 v1:172.21.15.104:6804/3649492645' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T19:06:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:52 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T19:06:53.292 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:53.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:53.375 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.394 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:53.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:53.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:53.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:53.675 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:53.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:06:53.377+0000 7f1791419640 -1 osd.5 347 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T19:06:53.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:53.821 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:53.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:06:53.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:53.971 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:53.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:54.033 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T19:06:54.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[39192]: from='osd.5 v1:172.21.15.104:6804/3649492645' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:06:54.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[39192]: osdmap e347: 8 total, 7 up, 8 in 2024-09-21T19:06:54.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:06:54.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[39192]: from='osd.5 ' entity='osd.5' 2024-09-21T19:06:54.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4277693810' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:06:54.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T19:06:54.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[29384]: from='osd.5 v1:172.21.15.104:6804/3649492645' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:06:54.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[29384]: osdmap e347: 8 total, 7 up, 8 in 2024-09-21T19:06:54.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:06:54.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[29384]: from='osd.5 ' entity='osd.5' 2024-09-21T19:06:54.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4277693810' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:06:54.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:54.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:54.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 39 16 74288 74272 119.006 75.5625 0.0173348 0.00839795 2024-09-21T19:06:54.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:06:54.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T19:06:54.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-mon[32438]: from='osd.5 v1:172.21.15.104:6804/3649492645' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:06:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-mon[32438]: osdmap e347: 8 total, 7 up, 8 in 2024-09-21T19:06:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]} : dispatch 2024-09-21T19:06:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-mon[32438]: from='osd.5 ' entity='osd.5' 2024-09-21T19:06:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4277693810' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:06:54.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:54.359 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:06:54.359 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:06:54.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:06:54.369 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:06:54.369 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:06:54.374 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:06:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:54.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:06:54.404 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:06:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:06:54.427 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:06:54.428 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:06:54.445 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:06:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:06:54.461 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:06:54.461 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:06:54.478 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:06:54.479 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:06:54.495 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:06:54.495 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:06:54.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:54.512 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:06:54.513 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:06:54.519 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:54.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:54.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:54.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:54.751 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:06:54.751 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:06:54.751 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T19:06:54.751 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:06:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.767 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:54.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:54.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:54.889 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:54.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:54.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: pgmap v676: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 933 B/s rd, 80 MiB/s wr, 1.28k op/s; 21997/211353 objects degraded (10.408%) 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: 4.19 scrub starts 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: 4.19 scrub ok 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/54966240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:55.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4277693810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: osd.5 v1:172.21.15.104:6804/3649492645 boot 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 21997/211353 objects degraded (10.408%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:55.008 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:54 smithi104 ceph-mon[32438]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T19:06:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: pgmap v676: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 933 B/s rd, 80 MiB/s wr, 1.28k op/s; 21997/211353 objects degraded (10.408%) 2024-09-21T19:06:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T19:06:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T19:06:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: 4.19 scrub starts 2024-09-21T19:06:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: 4.19 scrub ok 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/54966240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4277693810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: osd.5 v1:172.21.15.104:6804/3649492645 boot 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 21997/211353 objects degraded (10.408%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[29384]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: pgmap v676: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 933 B/s rd, 80 MiB/s wr, 1.28k op/s; 21997/211353 objects degraded (10.408%) 2024-09-21T19:06:55.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: 4.19 scrub starts 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: 4.19 scrub ok 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/54966240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4277693810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: osd.5 v1:172.21.15.104:6804/3649492645 boot 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 21997/211353 objects degraded (10.408%), 22 pgs degraded (PG_DEGRADED) 2024-09-21T19:06:55.068 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:54 smithi081 ceph-mon[39192]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T19:06:55.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:55.136 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:55.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:06:55.201150+0000 min lat: 0.00110519 max lat: 0.2501 avg lat: 0.00841316 2024-09-21T19:06:55.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:06:55.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 40 16 74535 74519 116.417 15.4375 0.0037345 0.00841316 2024-09-21T19:06:55.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:55.259 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:55.361 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:55.382 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:55.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T19:06:55.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:55.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:55.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:55.676 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:55.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.801 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:55.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:55.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:55.933 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:55.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:56.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:56.073 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:56.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:56.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 41 16 74847 74831 114.053 19.5 0.0480436 0.00876191 2024-09-21T19:06:56.212 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:55 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:06:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:55 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:06:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:55 smithi104 ceph-mon[32438]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T19:06:56.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:56.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:55 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:06:56.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:55 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:06:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:55 smithi081 ceph-mon[29384]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T19:06:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:55 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:06:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:55 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:06:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:55 smithi081 ceph-mon[39192]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T19:06:56.346 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:56.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:06:56.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:56.501 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:56.516 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:56.603 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:06:56.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:06:56.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:06:56.793 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:06:56.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:56.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:57.008 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:56 smithi081 ceph-mon[39192]: pgmap v680: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 22820/218730 objects degraded (10.433%) 2024-09-21T19:06:57.008 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:56 smithi081 ceph-mon[29384]: pgmap v680: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 22820/218730 objects degraded (10.433%) 2024-09-21T19:06:57.022 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:57.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:57.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:06:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:57.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:06:57.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 42 16 75772 75756 112.714 57.8125 0.0156258 0.00886474 2024-09-21T19:06:57.213 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:56 smithi104 ceph-mon[32438]: pgmap v680: 158 pgs: 36 active+undersized, 22 active+undersized+degraded, 100 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 22820/218730 objects degraded (10.433%) 2024-09-21T19:06:57.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:57.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:06:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:06:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:06:57.331 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:57.432 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:06:57.469 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:06:57.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:06:57.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:57.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:57.759 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:57.867 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T19:06:57.877 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:57.933 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:57.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:57.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/861324508' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:06:57.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:57 smithi081 ceph-mon[29384]: from='client.36158 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:06:57.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/861324508' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:06:57.959 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:57 smithi081 ceph-mon[39192]: from='client.36158 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:06:57.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:06:58.023 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:06:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:06:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:06:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:06:58.024 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:06:58.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.090 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:58.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:58.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:58.202 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 43 16 76642 76626 111.357 54.375 0.0101906 0.00897482 2024-09-21T19:06:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:58.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:58.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/861324508' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:06:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:57 smithi104 ceph-mon[32438]: from='client.36158 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:06:58.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:58.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:58.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:58.504 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:58.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:06:58.650 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:06:58.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:06:58.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:06:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:06:58.784 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:06:58.786 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T19:06:58.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:58.928 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:58.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:59.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:06:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:58 smithi081 ceph-mon[39192]: pgmap v681: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 4 active+undersized+degraded, 137 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 972 op/s; 23204/225525 objects degraded (10.289%); 737 KiB/s, 11 objects/s recovering 2024-09-21T19:06:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:58 smithi081 ceph-mon[39192]: from='client.36158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:06:59.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:58 smithi081 ceph-mon[39192]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T19:06:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:58 smithi081 ceph-mon[29384]: pgmap v681: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 4 active+undersized+degraded, 137 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 972 op/s; 23204/225525 objects degraded (10.289%); 737 KiB/s, 11 objects/s recovering 2024-09-21T19:06:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:58 smithi081 ceph-mon[29384]: from='client.36158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:06:59.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:58 smithi081 ceph-mon[29384]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T19:06:59.097 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:06:59.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:59.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 44 16 77615 77599 110.208 60.8125 0.00568221 0.00906778 2024-09-21T19:06:59.240 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:06:59.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:58 smithi104 ceph-mon[32438]: pgmap v681: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 4 active+undersized+degraded, 137 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 972 op/s; 23204/225525 objects degraded (10.289%); 737 KiB/s, 11 objects/s recovering 2024-09-21T19:06:59.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:58 smithi104 ceph-mon[32438]: from='client.36158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:06:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:58 smithi104 ceph-mon[32438]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T19:06:59.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:06:59.381 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:06:59.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:06:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.514 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:59.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:06:59.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:06:59.602 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:59.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:06:59.616 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:06:59.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:06:59.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:06:59.778 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:06:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:06:59.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:06:59.927 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:06:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:06:59.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:00.028 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:00.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:00.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:06:59 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 23204/225525 objects degraded (10.289%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:00.035 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:00.035 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:06:59 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 23204/225525 objects degraded (10.289%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:00.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:00.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 45 16 78518 78502 109.012 56.4375 0.0265223 0.00916698 2024-09-21T19:07:00.209 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:00.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:00.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:06:59 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 23204/225525 objects degraded (10.289%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:00.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.354 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:00.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:00.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:00.493 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:00.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:00.628 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:00.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:00.765 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:00.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:00.897 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:00.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:00.937 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:07:00.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:01.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:00 smithi081 ceph-mon[39192]: pgmap v683: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 959 B/s rd, 42 MiB/s wr, 672 op/s; 23153/229479 objects degraded (10.089%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T19:07:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/106139874' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:01.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:00 smithi081 ceph-mon[39192]: from='client.36164 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:00 smithi081 ceph-mon[29384]: pgmap v683: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 959 B/s rd, 42 MiB/s wr, 672 op/s; 23153/229479 objects degraded (10.089%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T19:07:01.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/106139874' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:01.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:00 smithi081 ceph-mon[29384]: from='client.36164 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:01.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:01.176 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:01.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 46 16 79390 79374 107.827 54.5 0.0162205 0.00926745 2024-09-21T19:07:01.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:00 smithi104 ceph-mon[32438]: pgmap v683: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 959 B/s rd, 42 MiB/s wr, 672 op/s; 23153/229479 objects degraded (10.089%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T19:07:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/106139874' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:00 smithi104 ceph-mon[32438]: from='client.36164 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:01.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:01.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:01.451 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:01.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:01.592 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:01.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:01.727 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:01.828 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:01.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:01.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:01.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:01.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[29384]: from='client.36164 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:01.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[29384]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T19:07:01.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/310539700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:01.917 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:01.917 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[39192]: from='client.36164 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:01.918 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[39192]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T19:07:01.918 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/310539700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:01.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:01.999 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:02.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:02.135 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:02.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 47 16 80355 80339 106.816 60.3125 0.0335549 0.00935709 2024-09-21T19:07:02.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:02.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:01 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:01 smithi104 ceph-mon[32438]: from='client.36164 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:01 smithi104 ceph-mon[32438]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T19:07:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/310539700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:02.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:02.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:02.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:07:02.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:07:02.384 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:07:02.417 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:02.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:02.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:02.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:02.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:02.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:02.879 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:02.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:02.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:03.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:03.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:03.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:03.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 48 16 81178 81162 105.663 51.4375 0.00275364 0.00944929 2024-09-21T19:07:03.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:03.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:02 smithi104 ceph-mon[32438]: pgmap v685: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 849 B/s rd, 43 MiB/s wr, 686 op/s; 23153/231123 objects degraded (10.018%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T19:07:03.272 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:03.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:02 smithi081 ceph-mon[39192]: pgmap v685: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 849 B/s rd, 43 MiB/s wr, 686 op/s; 23153/231123 objects degraded (10.018%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T19:07:03.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:02 smithi081 ceph-mon[29384]: pgmap v685: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 849 B/s rd, 43 MiB/s wr, 686 op/s; 23153/231123 objects degraded (10.018%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T19:07:03.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:03.407 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:03.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:03.532 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:03.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:03.669 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:03.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.814 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:03.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:03.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:03.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:03.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:03.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:04.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:04.083 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:04.184 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:04.203 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 49 16 82037 82021 104.602 53.6875 0.0182727 0.00955365 2024-09-21T19:07:04.214 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:04.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:04.349 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:04.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:04.482 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:04.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:04.617 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:04.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:04.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:04.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:04.890 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:04.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:04.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:05.026 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:05.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:05.158 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:05.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 50 16 82963 82947 103.667 57.875 0.0111868 0.00963759 2024-09-21T19:07:05.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:04 smithi104 ceph-mon[32438]: pgmap v686: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 807 op/s; 22903/238095 objects degraded (9.619%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T19:07:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:04 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 22903/238095 objects degraded (9.619%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:04 smithi104 ceph-mon[32438]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T19:07:05.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:05.293 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:04 smithi081 ceph-mon[29384]: pgmap v686: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 807 op/s; 22903/238095 objects degraded (9.619%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T19:07:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:04 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 22903/238095 objects degraded (9.619%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:04 smithi081 ceph-mon[29384]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T19:07:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:04 smithi081 ceph-mon[39192]: pgmap v686: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 807 op/s; 22903/238095 objects degraded (9.619%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T19:07:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:04 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 22903/238095 objects degraded (9.619%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:04 smithi081 ceph-mon[39192]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T19:07:05.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.432 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:05.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:05.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:05.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:05.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.712 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:05.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:05.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:05.853 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:05.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:05.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:06.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:06.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 51 16 83798 83782 102.658 52.1875 0.00533376 0.00973528 2024-09-21T19:07:06.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:06.262 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:06.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:06.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:06.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:06.538 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:06.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:06.676 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:06.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:06.816 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:06.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:06.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:07.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:07.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:07.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:07.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 52 16 84740 84724 101.815 58.875 0.0317254 0.0098166 2024-09-21T19:07:07.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:07.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:07.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[39192]: pgmap v688: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 652 B/s rd, 47 MiB/s wr, 747 op/s; 22722/243108 objects degraded (9.346%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T19:07:07.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[39192]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[29384]: pgmap v688: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 652 B/s rd, 47 MiB/s wr, 747 op/s; 22722/243108 objects degraded (9.346%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[29384]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:06 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:07.218 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:06 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:06 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:06 smithi104 ceph-mon[32438]: pgmap v688: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 652 B/s rd, 47 MiB/s wr, 747 op/s; 22722/243108 objects degraded (9.346%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T19:07:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:06 smithi104 ceph-mon[32438]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T19:07:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:06 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:07.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:07.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:07.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:07.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:07.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:07:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:07:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:07:07.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:07.620 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:07.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:07.757 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:07.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:07.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:07.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:08.026 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:08.128 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:08.172 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:08.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 53 16 85548 85532 100.847 50.5 0.0239405 0.00991014 2024-09-21T19:07:08.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.306 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:08.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:08.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:08.442 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:08.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:08.585 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:08.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:08.719 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:08.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:07:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:07:08.821 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:08.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:08.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:08.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.005 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:09.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:09.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:09.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:09.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 54 16 86459 86443 100.034 56.9375 0.0147043 0.00999196 2024-09-21T19:07:09.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:08 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:08 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:08 smithi104 ceph-mon[32438]: pgmap v690: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 22451/251205 objects degraded (8.937%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T19:07:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:08 smithi104 ceph-mon[32438]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T19:07:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.281 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:09.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[29384]: pgmap v690: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 22451/251205 objects degraded (8.937%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T19:07:09.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[29384]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T19:07:09.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:09.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:09.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[39192]: pgmap v690: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 22451/251205 objects degraded (8.937%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T19:07:09.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:08 smithi081 ceph-mon[39192]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T19:07:09.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:09.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:09.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:09.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:09.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:09.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:09.819 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:09.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:09.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:10.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:10.109 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:10.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 55 16 87353 87337 99.2306 55.875 0.00902753 0.0100727 2024-09-21T19:07:10.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:10.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:10.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:10.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:09 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 22451/251205 objects degraded (8.937%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:10.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:10.259 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:09 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 22451/251205 objects degraded (8.937%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:10.259 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:10.259 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:09 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 22451/251205 objects degraded (8.937%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:10.334 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:10.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:10.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:10.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:10.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:10.664 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:10.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.802 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:10.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:10.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:10.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:11.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:11.070 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:11.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:11.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 56 16 88238 88222 98.4463 55.3125 0.0244899 0.0101521 2024-09-21T19:07:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:10 smithi104 ceph-mon[32438]: pgmap v692: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 22273/256287 objects degraded (8.691%); 6.6 MiB/s, 104 objects/s recovering 2024-09-21T19:07:11.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:11.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:10 smithi081 ceph-mon[29384]: pgmap v692: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 22273/256287 objects degraded (8.691%); 6.6 MiB/s, 104 objects/s recovering 2024-09-21T19:07:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:10 smithi081 ceph-mon[39192]: pgmap v692: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 22273/256287 objects degraded (8.691%); 6.6 MiB/s, 104 objects/s recovering 2024-09-21T19:07:11.328 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:11.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:11.472 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:11.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.618 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:11.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:11.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.767 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:11.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:11.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:11.891 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:11.913 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:11.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:12.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:12.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:12.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:12.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:11 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:12.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:11 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:12.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:11 smithi081 ceph-mon[29384]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T19:07:12.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:11 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:12.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:11 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:12.175 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:11 smithi081 ceph-mon[39192]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T19:07:12.190 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:12.204 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 57 16 88972 88956 97.5241 45.875 0.0396263 0.0102482 2024-09-21T19:07:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:11 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:11 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:11 smithi104 ceph-mon[32438]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T19:07:12.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:12.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:07:12.326 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:12.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.467 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:12.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:12.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:12.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:12.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:12.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:12.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:12.866 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:07:12.866 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:07:12.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:12.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:13.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T19:07:13.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:13.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:13.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 58 16 89649 89633 96.5722 42.3125 0.0060405 0.0103474 2024-09-21T19:07:13.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:13 smithi104 ceph-mon[32438]: pgmap v694: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 824 op/s; 22273/257931 objects degraded (8.635%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T19:07:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3570483360' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:07:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:13 smithi104 ceph-mon[32438]: from='client.36173 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:07:13.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:13.292 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:13.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:13 smithi081 ceph-mon[29384]: pgmap v694: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 824 op/s; 22273/257931 objects degraded (8.635%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T19:07:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3570483360' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:07:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:13 smithi081 ceph-mon[29384]: from='client.36173 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:07:13.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:13 smithi081 ceph-mon[39192]: pgmap v694: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 824 op/s; 22273/257931 objects degraded (8.635%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T19:07:13.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3570483360' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:07:13.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:13 smithi081 ceph-mon[39192]: from='client.36173 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:07:13.391 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:07:13.394 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.426 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:13.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:13.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.564 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:13.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:13.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:13.704 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:13.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:13.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:13.977 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:13.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:14.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:14.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.116 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:14.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:14.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 59 16 90540 90524 95.8792 55.6875 0.013409 0.0104253 2024-09-21T19:07:14.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.239 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:14.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[29384]: from='client.36173 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[29384]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[39192]: from='client.36173 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[39192]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:14 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:14.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:14.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:14.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:14.503 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:14.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:14 smithi104 ceph-mon[32438]: from='client.36173 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:07:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:14 smithi104 ceph-mon[32438]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T19:07:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:14 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:14 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:14.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:14.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:14.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:14.777 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:14.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:14.923 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:14.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:15.025 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.059 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:15.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:15.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:15.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:15.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:07:15.203735+0000 min lat: 0.00110519 max lat: 1.53665 avg lat: 0.010492 2024-09-21T19:07:15.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:07:15.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 60 16 91458 91442 95.2374 57.375 0.00350002 0.010492 2024-09-21T19:07:15.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[39192]: pgmap v696: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 45 MiB/s wr, 726 op/s; 21952/264270 objects degraded (8.307%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[39192]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 21952/264270 objects degraded (8.307%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[39192]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[29384]: pgmap v696: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 45 MiB/s wr, 726 op/s; 21952/264270 objects degraded (8.307%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[29384]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 21952/264270 objects degraded (8.307%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:15 smithi081 ceph-mon[29384]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T19:07:15.330 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:15.431 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:15.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:15 smithi104 ceph-mon[32438]: pgmap v696: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 45 MiB/s wr, 726 op/s; 21952/264270 objects degraded (8.307%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T19:07:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:15 smithi104 ceph-mon[32438]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T19:07:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:15 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 21952/264270 objects degraded (8.307%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:15 smithi104 ceph-mon[32438]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T19:07:15.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.603 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:15.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:15.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:15.740 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:15.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:15.867 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:15.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:15.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.005 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:16.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:16.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:16.205 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 61 16 92341 92325 94.5808 55.1875 0.0207534 0.0105679 2024-09-21T19:07:16.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:16.273 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:16.375 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.415 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:16.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:16.517 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:16.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:16.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:16.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:16.828 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:16.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:07:16.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:16.968 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:16.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:17.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:17.103 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:17.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:17.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:17.206 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 62 16 93317 93301 94.0377 61 0.0163976 0.0106292 2024-09-21T19:07:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:17 smithi081 ceph-mon[29384]: pgmap v699: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 882 op/s; 21758/268812 objects degraded (8.094%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T19:07:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:17 smithi081 ceph-mon[39192]: pgmap v699: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 882 op/s; 21758/268812 objects degraded (8.094%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T19:07:17.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:17.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:17.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:17.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:17.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:17.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T19:07:17.482 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:17.483 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:17.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:17.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:17 smithi104 ceph-mon[32438]: pgmap v699: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 882 op/s; 21758/268812 objects degraded (8.094%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T19:07:17.535 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:07:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:07:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:07:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:17.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:17.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:17.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:17.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:17.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:17.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:17.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:18.006 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:18.108 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:18.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:18.206 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 63 16 94181 94165 93.4022 54 0.00995019 0.0107009 2024-09-21T19:07:18.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:18.285 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:18.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:18.424 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:18.526 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:18.549 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.589 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:18.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:18.641 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:18.647 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:07:18.647 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:07:18.691 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:18.692 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:07:18.692 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:07:18.721 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:07:18.721 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:07:18.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:18.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:07:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:07:18.786 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:07:18.786 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:07:18.815 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:07:18.816 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:07:18.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:07:18.837 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:07:18.855 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:18.860 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:07:18.860 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:07:18.882 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:07:18.883 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:07:18.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:18.979 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:18.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:19.036 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:07:19.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[39192]: pgmap v700: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 21468/276894 objects degraded (7.753%); 8.4 MiB/s, 134 objects/s recovering 2024-09-21T19:07:19.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1130973608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:19.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1965717975' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:07:19.037 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[39192]: from='client.26065 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:07:19.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[29384]: pgmap v700: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 21468/276894 objects degraded (7.753%); 8.4 MiB/s, 134 objects/s recovering 2024-09-21T19:07:19.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1130973608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:19.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1965717975' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:07:19.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:19 smithi081 ceph-mon[29384]: from='client.26065 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:07:19.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.109 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:19.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:19.206 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 64 16 94778 94762 92.5252 37.3125 0.00904211 0.0108029 2024-09-21T19:07:19.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:19.244 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:19.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:19.391 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:19.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:19.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:19 smithi104 ceph-mon[32438]: pgmap v700: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 21468/276894 objects degraded (7.753%); 8.4 MiB/s, 134 objects/s recovering 2024-09-21T19:07:19.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1130973608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:19.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1965717975' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:07:19.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:19 smithi104 ceph-mon[32438]: from='client.26065 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:07:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:19.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:19.652 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:19.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:19.798 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:19.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:19.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:20.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:20.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:20.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:20.206 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 65 16 95677 95661 91.9662 56.1875 0.0114927 0.0108676 2024-09-21T19:07:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:20.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:20.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[39192]: from='client.26065 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[39192]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 21468/276894 objects degraded (7.753%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[39192]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[29384]: from='client.26065 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[29384]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 21468/276894 objects degraded (7.753%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:20 smithi081 ceph-mon[29384]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T19:07:20.330 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:20.432 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:20.469 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:20.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:20 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:20 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:20 smithi104 ceph-mon[32438]: from='client.26065 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:07:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:20 smithi104 ceph-mon[32438]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T19:07:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:20 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 21468/276894 objects degraded (7.753%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:20 smithi104 ceph-mon[32438]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T19:07:20.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:20.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:20.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:20.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:20.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:20.882 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:20.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:20.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:21.021 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:21.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:21.207 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 66 16 96529 96513 91.3796 53.25 0.00688343 0.0109311 2024-09-21T19:07:21.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:21.297 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:21.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:21 smithi081 ceph-mon[29384]: pgmap v703: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 898 B/s rd, 64 MiB/s wr, 1.02k op/s; 21242/281766 objects degraded (7.539%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T19:07:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:21 smithi081 ceph-mon[39192]: pgmap v703: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 898 B/s rd, 64 MiB/s wr, 1.02k op/s; 21242/281766 objects degraded (7.539%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T19:07:21.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:21.436 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:21.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:21 smithi104 ceph-mon[32438]: pgmap v703: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 898 B/s rd, 64 MiB/s wr, 1.02k op/s; 21242/281766 objects degraded (7.539%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T19:07:21.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:21.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:21.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:21.706 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:21.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:21.845 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:21.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:21.977 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:21.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:22.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:22.119 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:22.206 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 67 16 97450 97434 90.8748 57.5625 0.0115434 0.010999 2024-09-21T19:07:22.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:22.257 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:22.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:22.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97588 10.3 10:cba88c72:::benchmark_data_smithi081_155596_object97587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.365455+0000", 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.038164679, 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.03819703, 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.407 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97588 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365455+0000", 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365455+0000", 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365467+0000", 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1454000000000001e-05 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365495+0000", 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8442999999999999e-05 2024-09-21T19:07:22.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365505+0000", 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5370000000000003e-06 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365512+0000", 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1609999999999997e-06 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.366021+0000", 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00050906200000000001 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.366166+0000", 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000145246 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.366368+0000", 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020166699999999999 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.380983+0000", 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014615555000000001 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.390980+0000", 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0099966610000000004 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97591 10.9 10:9360b538:::benchmark_data_smithi081_155596_object97590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.412 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.365527+0000", 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.038093316000000002, 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.038205941, 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97591 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365527+0000", 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365527+0000", 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365527+0000", 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3600000000000003e-07 2024-09-21T19:07:22.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365552+0000", 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5004e-05 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365559+0000", 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6619999999999999e-06 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365564+0000", 2024-09-21T19:07:22.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5289999999999999e-06 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.374407+0000", 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0088422339999999992 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.374567+0000", 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000160357 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T19:07:22.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.374730+0000", 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016261199999999999 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.378697+0000", 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039669179999999998 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.383847+0000", 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051502400000000004 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402213+0000", 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018366177000000001 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402259+0000", 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6316999999999997e-05 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.36077.0:97606 10.3 e361/349)", 2024-09-21T19:07:22.418 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.398719+0000", 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0049009860000000004, 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0050865349999999997, 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.398719+0000", 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.398719+0000", 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.398721+0000", 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6920000000000001e-06 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.398727+0000", 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2430000000000002e-06 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.398738+0000", 2024-09-21T19:07:22.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0862e-05 2024-09-21T19:07:22.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.398746+0000", 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0539999999999995e-06 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97592 10.3 10:c71bfa57:::benchmark_data_smithi081_155596_object97591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.365572+0000", 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.038047790999999997, 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.038277395999999998, 2024-09-21T19:07:22.422 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97592 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365572+0000", 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365572+0000", 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365573+0000", 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4600000000000004e-07 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365598+0000", 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5089999999999999e-05 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365604+0000", 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7740000000000001e-06 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.365609+0000", 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2599999999999996e-06 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.374144+0000", 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0085353800000000004 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.374299+0000", 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015465300000000001 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.374465+0000", 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016568099999999999 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.391411+0000", 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016946902999999999 2024-09-21T19:07:22.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.391587+0000", 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017570199999999999 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.36077.0:97606 10.3 e361/349)", 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.399999+0000", 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0036206160000000001, 2024-09-21T19:07:22.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003911894, 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.399999+0000", 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.399999+0000", 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.400001+0000", 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3239999999999999e-06 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.400005+0000", 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8130000000000003e-06 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.400015+0000", 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9909999999999997e-06 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.400027+0000", 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1761e-05 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97621 10.5 10.82a00b95 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.402099+0000", 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001520551, 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018480669999999999, 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:07:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97621 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402099+0000", 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402099+0000", 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402101+0000", 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9549999999999999e-06 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402123+0000", 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1951999999999999e-05 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402134+0000", 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.078e-05 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.402141+0000", 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4610000000000002e-06 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36077.0:97620 10.f e361/330 10:f0239563:::benchmark_data_smithi081_155596_object97619:head v 361'6098, mlcod=361'6096)", 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.403112+0000", 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00050814499999999999, 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00087955900000000003, 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403112+0000", 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403112+0000", 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403113+0000", 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4950000000000001e-06 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403180+0000", 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6679000000000003e-05 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403191+0000", 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1459e-05 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403199+0000", 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4120000000000004e-06 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403259+0000", 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0421999999999998e-05 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.403278+0000", 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8675e-05 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97596 10.3 10:cc6ea1ef:::benchmark_data_smithi081_155596_object97595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.436 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.370858+0000", 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.032761815, 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.033187174999999999, 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97596 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.370858+0000", 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.370858+0000", 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.370860+0000", 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.55e-06 2024-09-21T19:07:22.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.370887+0000", 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7223e-05 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.370896+0000", 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1479999999999995e-06 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.370902+0000", 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.268e-06 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.377708+0000", 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0068060569999999999 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.377853+0000", 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000144365 2024-09-21T19:07:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.378020+0000", 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016772900000000001 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.391749+0000", 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013728463999999999 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.391911+0000", 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016254900000000001 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97605 10.9 10:91519b4d:::benchmark_data_smithi081_155596_object97604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.382213+0000", 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.021407117999999999, 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021902176999999998, 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.442 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97605 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382213+0000", 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382213+0000", 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382215+0000", 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1619999999999998e-06 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382238+0000", 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3518000000000001e-05 2024-09-21T19:07:22.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382247+0000", 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7579999999999998e-06 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382254+0000", 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9099999999999999e-06 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.383882+0000", 2024-09-21T19:07:22.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001628352 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.384027+0000", 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014485 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,7", 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.384187+0000", 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015983600000000001 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.399195+0000", 2024-09-21T19:07:22.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015008155 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.399232+0000", 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6783999999999999e-05 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:97606 10.3 10:caec9dec:::benchmark_data_smithi081_155596_object97605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:22.382266+0000", 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.021354237000000002, 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.021916844000000001, 2024-09-21T19:07:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 97606 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382266+0000", 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:07:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382266+0000", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382266+0000", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1299999999999999e-07 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382286+0000", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9825000000000001e-05 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382292+0000", 2024-09-21T19:07:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8549999999999998e-06 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.382297+0000", 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9069999999999998e-06 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.392042+0000", 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097448929999999993 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.392365+0000", 2024-09-21T19:07:22.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00032280299999999998 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:22.392835+0000", 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00047083299999999998 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 10 2024-09-21T19:07:22.451 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:07:22.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:22.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:22.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:22.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:22.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:22.803 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:22.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:22.943 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:22.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:23.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:23.072 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:23 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T19:07:23.072 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:23 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T19:07:23.072 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:23 smithi081 ceph-mon[39192]: pgmap v704: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 727 B/s rd, 43 MiB/s wr, 688 op/s; 21242/283332 objects degraded (7.497%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T19:07:23.072 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:23 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T19:07:23.072 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:23 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T19:07:23.072 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:23 smithi081 ceph-mon[29384]: pgmap v704: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 727 B/s rd, 43 MiB/s wr, 688 op/s; 21242/283332 objects degraded (7.497%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T19:07:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:23.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:23.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:23.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 68 16 98351 98335 90.3655 56.3125 0.0298171 0.0110602 2024-09-21T19:07:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.226 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:23.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:23.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:23.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:23.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:23 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T19:07:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:23 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T19:07:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:23 smithi104 ceph-mon[32438]: pgmap v704: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 727 B/s rd, 43 MiB/s wr, 688 op/s; 21242/283332 objects degraded (7.497%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T19:07:23.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:23.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:23.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:23.778 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:23.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:23.913 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:23.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:24.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:24.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:24.059 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T19:07:24.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:24.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:07:24.207 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 69 15 99278 99263 89.8964 58 0.0103566 0.0111188 2024-09-21T19:07:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:24.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:24.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3547354835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:07:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:24 smithi081 ceph-mon[39192]: from='client.36182 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:07:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3547354835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:07:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:24 smithi081 ceph-mon[29384]: from='client.36182 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:07:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.362 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:24.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:24.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3547354835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:07:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:24 smithi104 ceph-mon[32438]: from='client.36182 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:07:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.518 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:24.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.678 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:24.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.815 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:24.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:24.947 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:24.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:25.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:25.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:25.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:25.207 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 70 15 100140 100125 89.3818 53.875 0.0373763 0.0111832 2024-09-21T19:07:25.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[39192]: pgmap v705: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 869 op/s; 20922/289653 objects degraded (7.223%); 6.5 MiB/s, 104 objects/s recovering 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[39192]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[39192]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 20922/289653 objects degraded (7.223%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[39192]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[29384]: pgmap v705: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 869 op/s; 20922/289653 objects degraded (7.223%); 6.5 MiB/s, 104 objects/s recovering 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[29384]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:07:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[29384]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T19:07:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:07:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 20922/289653 objects degraded (7.223%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:25.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:25 smithi081 ceph-mon[29384]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T19:07:25.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:25.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:25.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:25.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:25 smithi104 ceph-mon[32438]: pgmap v705: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 869 op/s; 20922/289653 objects degraded (7.223%); 6.5 MiB/s, 104 objects/s recovering 2024-09-21T19:07:25.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:25 smithi104 ceph-mon[32438]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:07:25.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:25 smithi104 ceph-mon[32438]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T19:07:25.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:25 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:07:25.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:25 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 20922/289653 objects degraded (7.223%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:25.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:25 smithi104 ceph-mon[32438]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T19:07:25.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:25.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:25.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:25.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:25.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:25.908 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:25.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:26.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.092 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:26.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:26.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:26.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 71 16 100998 100982 88.8772 53.5625 0.0101714 0.0112462 2024-09-21T19:07:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.228 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:26.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:26.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:26.363 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:26.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.528 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:26.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:26.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:26.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:26.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:26.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:26.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:26.945 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:26.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:27.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:27.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:27.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:27.209 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 72 16 101879 101863 88.4076 55.0625 0.0167176 0.0113052 2024-09-21T19:07:27.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:27.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.378 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:27.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:27.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:07:27.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:27 smithi081 ceph-mon[29384]: pgmap v708: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 848 B/s rd, 45 MiB/s wr, 717 op/s; 20715/294738 objects degraded (7.028%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T19:07:27.472 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:27 smithi081 ceph-mon[39192]: pgmap v708: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 848 B/s rd, 45 MiB/s wr, 717 op/s; 20715/294738 objects degraded (7.028%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T19:07:27.472 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:07:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:07:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:07:27.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:27.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:27 smithi104 ceph-mon[32438]: pgmap v708: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 848 B/s rd, 45 MiB/s wr, 717 op/s; 20715/294738 objects degraded (7.028%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T19:07:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.517 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:27.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:27.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.663 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:27.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:27.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.802 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:27.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:27.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T19:07:27.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:27.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:27.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:28.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.090 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:28.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:28.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:28.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 73 16 102773 102757 87.9616 55.875 0.00800107 0.0113639 2024-09-21T19:07:28.244 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:28.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:28 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T19:07:28.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:28 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T19:07:28.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:28.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:28 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T19:07:28.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:28 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T19:07:28.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:28 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T19:07:28.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:28 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T19:07:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.408 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:28.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:28.417 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:28.429 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:28.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.554 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:28.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:28.655 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:28.689 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:28.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:07:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:07:28.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:28.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:28.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:28.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:28.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:29.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:29.102 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:29.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:29.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 74 16 103600 103584 87.4714 51.6875 0.00812535 0.0114271 2024-09-21T19:07:29.230 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:29 smithi081 ceph-mon[29384]: pgmap v709: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 888 op/s; 20444/303069 objects degraded (6.746%); 6.2 MiB/s, 99 objects/s recovering 2024-09-21T19:07:29.230 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:29 smithi081 ceph-mon[39192]: pgmap v709: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 888 op/s; 20444/303069 objects degraded (6.746%); 6.2 MiB/s, 99 objects/s recovering 2024-09-21T19:07:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.243 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:29.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:29.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:29.468 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:29.477 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:07:29.477 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:07:29.477 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:07:29.477 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:07:29.477 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:07:29.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:29 smithi104 ceph-mon[32438]: pgmap v709: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 888 op/s; 20444/303069 objects degraded (6.746%); 6.2 MiB/s, 99 objects/s recovering 2024-09-21T19:07:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.511 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:29.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:29.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.646 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:29.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:29.731 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:29.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.778 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:29.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:29.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:29.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:29.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:30.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:30.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:30.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:30.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:30.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:30.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1398380950' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:30.102 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[29384]: from='client.26074 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:30.102 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 20444/303069 objects degraded (6.746%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:30.105 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:07:30.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:30.209 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 75 16 104340 104324 86.9218 46.25 0.0147878 0.0114984 2024-09-21T19:07:30.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:30.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:30.349 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:07:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.352 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:30.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:30.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:07:30.426 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:30.427 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:30.427 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:30.427 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1398380950' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:30.427 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[39192]: from='client.26074 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:30.427 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:30 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 20444/303069 objects degraded (6.746%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:30.453 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:30.487 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:30.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:30 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:30.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:30 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:30.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:30 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:30.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1398380950' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:30.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:30 smithi104 ceph-mon[32438]: from='client.26074 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:30.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:30 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 20444/303069 objects degraded (6.746%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:30.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.644 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:30.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:30.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.793 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:30.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:30.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:30.928 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:30.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:31.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:31.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:31.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:31.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 76 16 105235 105219 86.5141 55.9375 0.0198788 0.0115546 2024-09-21T19:07:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.219 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:31.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: pgmap v710: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 20230/307950 objects degraded (6.569%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: from='client.26074 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:07:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:31 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:07:31.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:31.415 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T19:07:31.415 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T19:07:31.415 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: pgmap v710: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 20230/307950 objects degraded (6.569%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T19:07:31.415 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:07:31.415 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: from='client.26074 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: pgmap v710: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 20230/307950 objects degraded (6.569%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: from='client.26074 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:07:31.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:31 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:07:31.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:31.611 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.660 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:31.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:31.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:31.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:31.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:31.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:32.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:32.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:32.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:32.208 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 77 16 106131 106115 86.1178 56 0.00379551 0.0116056 2024-09-21T19:07:32.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:32 smithi104 ceph-mon[32438]: 6.1 scrub starts 2024-09-21T19:07:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:32 smithi104 ceph-mon[32438]: 6.1 scrub ok 2024-09-21T19:07:32.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:32.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:32 smithi081 ceph-mon[29384]: 6.1 scrub starts 2024-09-21T19:07:32.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:32 smithi081 ceph-mon[29384]: 6.1 scrub ok 2024-09-21T19:07:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:32 smithi081 ceph-mon[39192]: 6.1 scrub starts 2024-09-21T19:07:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:32 smithi081 ceph-mon[39192]: 6.1 scrub ok 2024-09-21T19:07:32.346 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:32.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.509 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:32.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:32.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:32.636 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:32.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:32.771 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:32.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:32.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:32.988 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:32.995 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:33.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.071 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:33.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:33.118 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T19:07:33.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:33.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:07:33.211 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 78 16 106898 106882 85.6252 47.9375 0.0139409 0.0116738 2024-09-21T19:07:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.216 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:33.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:33.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:33.443 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[39192]: pgmap v712: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 667 B/s rd, 54 MiB/s wr, 864 op/s; 20230/309528 objects degraded (6.536%); 5.6 MiB/s, 90 objects/s recovering 2024-09-21T19:07:33.443 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1204862742' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:07:33.444 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:33.444 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:33.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[29384]: pgmap v712: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 667 B/s rd, 54 MiB/s wr, 864 op/s; 20230/309528 objects degraded (6.536%); 5.6 MiB/s, 90 objects/s recovering 2024-09-21T19:07:33.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1204862742' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:07:33.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:33.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:33 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:33.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:33.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:33 smithi104 ceph-mon[32438]: pgmap v712: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 667 B/s rd, 54 MiB/s wr, 864 op/s; 20230/309528 objects degraded (6.536%); 5.6 MiB/s, 90 objects/s recovering 2024-09-21T19:07:33.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:33 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1204862742' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:07:33.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:33 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:33.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:33 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:33.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:33.690 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:33.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:33.826 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:33.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:33.961 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:33.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:34.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:34.103 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:34.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:34.211 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 79 16 107737 107721 85.2051 52.4375 0.023518 0.0117314 2024-09-21T19:07:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:34.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:34.376 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:34.421 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1204862742' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:07:34.421 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[39192]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T19:07:34.422 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:07:34.422 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:07:34.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1204862742' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:07:34.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[29384]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T19:07:34.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:07:34.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:34 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:07:34.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1204862742' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:07:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:34 smithi104 ceph-mon[32438]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T19:07:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:34 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:07:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:34 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:07:34.512 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:34.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:34.647 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:34.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:34.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:34.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T19:07:34.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:34.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:35.042 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:35.077 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:35.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:35.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:35.211 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:07:35.210560+0000 min lat: 0.00110519 max lat: 1.53665 avg lat: 0.0117779 2024-09-21T19:07:35.212 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:07:35.212 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 80 16 108663 108647 84.8634 57.875 0.00454104 0.0117779 2024-09-21T19:07:35.303 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:07:35.303 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:07:35.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:35.339 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:35.343 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:07:35.343 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:07:35.409 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:07:35.409 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:07:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:35 smithi081 ceph-mon[29384]: pgmap v714: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s; 19941/315915 objects degraded (6.312%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T19:07:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:35 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 19941/315915 objects degraded (6.312%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:35.409 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:35 smithi081 ceph-mon[39192]: pgmap v714: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s; 19941/315915 objects degraded (6.312%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T19:07:35.410 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:35 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 19941/315915 objects degraded (6.312%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:35.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:35.450 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:07:35.450 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:07:35.487 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:07:35.487 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:07:35.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:35 smithi104 ceph-mon[32438]: pgmap v714: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s; 19941/315915 objects degraded (6.312%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T19:07:35.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:35 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 19941/315915 objects degraded (6.312%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:35.517 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:07:35.517 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:07:35.519 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:35.535 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:07:35.536 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:07:35.576 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:07:35.576 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:07:35.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:07:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:07:35.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:35.666 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:35.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:35.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:35.933 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:35.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:36.029 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:36.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:36.070 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:36.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.210 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:36.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:36.212 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 81 16 109521 109505 84.4777 53.625 0.0253607 0.0118324 2024-09-21T19:07:36.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:36.343 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:36.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:36.486 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:36.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:36 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/219145040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:36.550 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:36 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/219145040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:36.550 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:36 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/219145040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:36.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:36.608 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:36.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:36.784 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:36.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:36.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:37.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.060 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:37.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:37.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:07:37.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:37.190 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:37.214 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 82 16 110482 110466 84.1777 60.0625 0.0115016 0.0118741 2024-09-21T19:07:37.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:37.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:37.355 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-mon[29384]: pgmap v715: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 19709/320649 objects degraded (6.147%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T19:07:37.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:37.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:37.356 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:07:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:07:37.356 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-mon[39192]: pgmap v715: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 19709/320649 objects degraded (6.147%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T19:07:37.356 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:37.356 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:37.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:37.472 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:37.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:37 smithi104 ceph-mon[32438]: pgmap v715: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 732 op/s; 19709/320649 objects degraded (6.147%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T19:07:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:37.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:37.622 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:37.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.771 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:37.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:37.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:37.905 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:37.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:38.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:38.068 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:38.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:38.204 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T19:07:38.206 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:38.214 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 83 16 111346 111330 83.8137 54 0.00962387 0.0119255 2024-09-21T19:07:38.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/656596596' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:07:38.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:38 smithi081 ceph-mon[29384]: from='client.36200 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:07:38.224 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/656596596' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:07:38.224 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:38 smithi081 ceph-mon[39192]: from='client.36200 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:07:38.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:38.362 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:38.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:38.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/656596596' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:07:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:38 smithi104 ceph-mon[32438]: from='client.36200 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:07:38.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:07:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:07:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:38.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:07:38.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:38.643 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:38.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:38.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:38.925 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:38.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:39.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:39.064 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:39.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:39.191 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:39.214 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 84 16 112239 112223 83.4803 55.8125 0.011319 0.0119732 2024-09-21T19:07:39.228 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:07:39.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:39.431 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:39.462 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:39.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: pgmap v716: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s; 19410/328827 objects degraded (5.903%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T19:07:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:07:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T19:07:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4062029025' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:07:39.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:39 smithi104 ceph-mon[32438]: from='client.36206 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:07:39.564 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: pgmap v716: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s; 19410/328827 objects degraded (5.903%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4062029025' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:07:39.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[39192]: from='client.36206 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: pgmap v716: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s; 19410/328827 objects degraded (5.903%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: from='client.36200 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4062029025' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:07:39.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:39 smithi081 ceph-mon[29384]: from='client.36206 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:07:39.601 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:39.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:39.751 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:39.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:39.885 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:39.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:39.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:40.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:40.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:40.169 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:40.214 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 85 16 113176 113160 83.1871 58.5625 0.00339721 0.0120149 2024-09-21T19:07:40.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:40.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:40.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:40.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:40 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:40 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:40 smithi104 ceph-mon[32438]: from='client.36206 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:07:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:40 smithi104 ceph-mon[32438]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T19:07:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:40 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 19410/328827 objects degraded (5.903%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:40.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:40.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[29384]: from='client.36206 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[29384]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 19410/328827 objects degraded (5.903%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[39192]: from='client.36206 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[39192]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T19:07:40.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:40 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 19410/328827 objects degraded (5.903%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:40.585 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:40.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:40.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:40.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:40.869 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:40.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:40.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:41.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:41.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:41.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:41.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 86 16 113947 113931 82.7802 48.1875 0.0325681 0.0120756 2024-09-21T19:07:41.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:41.262 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:41.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:41.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:41.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:41 smithi104 ceph-mon[32438]: pgmap v719: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 774 B/s rd, 56 MiB/s wr, 903 op/s; 19144/333813 objects degraded (5.735%); 7.5 MiB/s, 120 objects/s recovering 2024-09-21T19:07:41.530 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:41.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:41 smithi081 ceph-mon[39192]: pgmap v719: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 774 B/s rd, 56 MiB/s wr, 903 op/s; 19144/333813 objects degraded (5.735%); 7.5 MiB/s, 120 objects/s recovering 2024-09-21T19:07:41.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:41 smithi081 ceph-mon[29384]: pgmap v719: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 774 B/s rd, 56 MiB/s wr, 903 op/s; 19144/333813 objects degraded (5.735%); 7.5 MiB/s, 120 objects/s recovering 2024-09-21T19:07:41.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:41.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:41.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:41.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:42.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:42.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:42.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:42.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 87 16 114796 114780 82.4386 53.0625 0.039806 0.0121249 2024-09-21T19:07:42.218 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:42.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:42.348 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:42.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:42.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:42.621 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:42.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:42.763 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:42.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:42.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:43.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:43.047 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:43.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:43.180 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:43.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 88 16 115668 115652 82.1211 54.5 0.0424497 0.0121716 2024-09-21T19:07:43.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:43.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:43.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.460 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:43.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:43 smithi104 ceph-mon[32438]: pgmap v720: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 811 op/s; 19144/335391 objects degraded (5.708%); 6.2 MiB/s, 99 objects/s recovering 2024-09-21T19:07:43.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:43.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:43 smithi081 ceph-mon[29384]: pgmap v720: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 811 op/s; 19144/335391 objects degraded (5.708%); 6.2 MiB/s, 99 objects/s recovering 2024-09-21T19:07:43.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:43 smithi081 ceph-mon[39192]: pgmap v720: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 811 op/s; 19144/335391 objects degraded (5.708%); 6.2 MiB/s, 99 objects/s recovering 2024-09-21T19:07:43.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:43.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:43.737 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:43.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:43.874 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:43.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:44.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:44.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:44.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 89 15 116598 116583 81.8521 58.1875 0.00903839 0.0122126 2024-09-21T19:07:44.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:44.284 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:44.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:44.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:44.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:44.560 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:44.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:44.683 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:44.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:44.826 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:44.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:44.959 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:44.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:45.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:45.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:45.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:45.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total time run: 90.0186 2024-09-21T19:07:45.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total writes made: 117470 2024-09-21T19:07:45.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Write size: 65536 2024-09-21T19:07:45.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object size: 65536 2024-09-21T19:07:45.215 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Bandwidth (MB/sec): 81.5595 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Bandwidth: 43.2586 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max bandwidth (MB/sec): 223.438 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min bandwidth (MB/sec): 15.4375 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average IOPS: 1304 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev IOPS: 692.137 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max IOPS: 3575 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min IOPS: 247 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average Latency(s): 0.0122562 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Latency(s): 0.0217169 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max latency(s): 1.53665 2024-09-21T19:07:45.216 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min latency(s): 0.00110519 2024-09-21T19:07:45.222 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T19:07:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.232 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:45.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:45.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:45.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:45.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:45.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:45 smithi104 ceph-mon[32438]: pgmap v721: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 882 op/s; 18850/341820 objects degraded (5.515%); 6.7 MiB/s, 107 objects/s recovering 2024-09-21T19:07:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:45 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 18850/341820 objects degraded (5.515%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:45.507 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:45.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:45 smithi081 ceph-mon[39192]: pgmap v721: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 882 op/s; 18850/341820 objects degraded (5.515%); 6.7 MiB/s, 107 objects/s recovering 2024-09-21T19:07:45.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:45 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 18850/341820 objects degraded (5.515%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:45 smithi081 ceph-mon[29384]: pgmap v721: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 882 op/s; 18850/341820 objects degraded (5.515%); 6.7 MiB/s, 107 objects/s recovering 2024-09-21T19:07:45.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:45 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 18850/341820 objects degraded (5.515%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:45.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:45.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:45.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:45.781 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:45.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:45.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:46.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:46.058 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:46.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:46.201 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:46.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:46.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:46.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T19:07:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:46.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:46.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:46.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:46.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:46.881 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:46.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:46.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:47.018 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:47.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:47.153 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:47.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.288 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:47.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:47.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:47.426 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:47.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:47 smithi104 ceph-mon[32438]: pgmap v722: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 750 op/s; 18621/346848 objects degraded (5.369%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T19:07:47.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:47.557 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:47 smithi081 ceph-mon[39192]: pgmap v722: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 750 op/s; 18621/346848 objects degraded (5.369%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T19:07:47.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:47 smithi081 ceph-mon[29384]: pgmap v722: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 750 op/s; 18621/346848 objects degraded (5.369%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T19:07:47.561 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:07:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:07:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:07:47.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:47.636 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:47.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:47.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.840 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:47.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:47.941 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:47.970 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:47.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:48.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:48.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:48.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:48.238 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:48.340 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:48.377 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:48.479 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:48.574 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:07:48.574 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:07:48.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:48.673 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:48.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:07:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:07:48.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:48.804 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:48.902 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:07:48.951 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:07:48.951 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:07:48.951 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:07:48.951 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:07:48.951 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:07:48.951 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:07:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:07:48.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:48.988 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:48.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:49.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:49.128 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:49.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.276 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:49.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:49.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:49.411 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:49.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:49 smithi104 ceph-mon[32438]: pgmap v723: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 932 op/s; 18300/348315 objects degraded (5.254%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T19:07:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1364397318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:49.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:49.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:49 smithi081 ceph-mon[39192]: pgmap v723: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 932 op/s; 18300/348315 objects degraded (5.254%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T19:07:49.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1364397318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:49 smithi081 ceph-mon[29384]: pgmap v723: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 932 op/s; 18300/348315 objects degraded (5.254%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T19:07:49.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1364397318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:07:49.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:07:49.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:49.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:49.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:49.828 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:49.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:07:49.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:49.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:49.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:50.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:50.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:50.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:50.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:50.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:50.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:50 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:50 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 18300/348315 objects degraded (5.254%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:50.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:50 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:50 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 18300/348315 objects degraded (5.254%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:50.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:50.420 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:50.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:50 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:07:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:50 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 18300/348315 objects degraded (5.254%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:50.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:50.575 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:50.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:50.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:50.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:50.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:50.873 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:50.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:50.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:51.005 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:51.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.143 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:51.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:51.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:51.278 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:51.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:51.413 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:51 smithi104 ceph-mon[32438]: pgmap v724: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 38 MiB/s wr, 860 op/s; 18022/345105 objects degraded (5.222%); 6.7 MiB/s, 106 objects/s recovering 2024-09-21T19:07:51.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:51.521 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:51 smithi081 ceph-mon[29384]: pgmap v724: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 38 MiB/s wr, 860 op/s; 18022/345105 objects degraded (5.222%); 6.7 MiB/s, 106 objects/s recovering 2024-09-21T19:07:51.524 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:51 smithi081 ceph-mon[39192]: pgmap v724: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 38 MiB/s wr, 860 op/s; 18022/345105 objects degraded (5.222%); 6.7 MiB/s, 106 objects/s recovering 2024-09-21T19:07:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.581 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:51.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:51.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:51.783 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:51.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:51.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:51.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:51.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:52.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.058 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:52.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:52.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:52.192 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:52.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.326 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:52.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:52.427 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:52.465 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:52.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:07:52.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: pgmap v725: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 938 B/s rd, 33 MiB/s wr, 803 op/s; 18022/343293 objects degraded (5.250%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1374823404' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: pgmap v725: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 938 B/s rd, 33 MiB/s wr, 803 op/s; 18022/343293 objects degraded (5.250%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:52.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:07:52.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:07:52.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1374823404' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:07:52.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:52.589 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:52.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.733 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:52.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: pgmap v725: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 938 B/s rd, 33 MiB/s wr, 803 op/s; 18022/343293 objects degraded (5.250%); 5.8 MiB/s, 93 objects/s recovering 2024-09-21T19:07:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:07:52.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:07:52.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:07:52.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:07:52.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1374823404' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:07:52.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:52.876 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:52.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:52.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:53.011 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T19:07:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:53.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:53.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:53.292 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:53.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:53.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:53.530 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:53.562 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:53.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:53.705 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:53.807 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.847 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:53.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:53.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:53.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:53.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:54.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.111 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:54.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:54.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:54.247 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:54.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:54 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1374823404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:07:54.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:54 smithi081 ceph-mon[39192]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T19:07:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:54 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1374823404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:07:54.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:54 smithi081 ceph-mon[29384]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T19:07:54.349 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.383 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:54.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:54.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:54 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1374823404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:07:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:54 smithi104 ceph-mon[32438]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T19:07:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:54.618 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:54.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:54.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:54.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:54.929 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:54.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:55.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:55.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:55.206 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:55.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:55.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:55 smithi081 ceph-mon[29384]: pgmap v727: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 941 op/s; 17728/335970 objects degraded (5.277%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T19:07:55.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:55 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 17728/335970 objects degraded (5.277%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:55 smithi081 ceph-mon[29384]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T19:07:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:55 smithi081 ceph-mon[39192]: pgmap v727: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 941 op/s; 17728/335970 objects degraded (5.277%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T19:07:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:55 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 17728/335970 objects degraded (5.277%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:55 smithi081 ceph-mon[39192]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T19:07:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:55.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:55.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:55.470 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:55.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:55 smithi104 ceph-mon[32438]: pgmap v727: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 941 op/s; 17728/335970 objects degraded (5.277%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T19:07:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:55 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 17728/335970 objects degraded (5.277%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T19:07:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:55 smithi104 ceph-mon[32438]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T19:07:55.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:55.605 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:55.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:55.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:55.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:55.900 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:55.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:56.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.042 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:56.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:56.143 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:56.179 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:56.280 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:56.314 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:56 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:07:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:56 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:07:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:56 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:07:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:56 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:07:56.416 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:56.455 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:56.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:56 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:07:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:56 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:07:56.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.595 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:56.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:56.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:56.727 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:56.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:56.863 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:56.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:56.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:07:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:57.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:07:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:07:57.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.156 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:57.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:57.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T19:07:57.257 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:57.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-mon[29384]: pgmap v729: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 2.9 MiB/s wr, 824 op/s; 17416/330729 objects degraded (5.266%); 6.9 MiB/s, 110 objects/s recovering 2024-09-21T19:07:57.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3066340085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:57.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-mon[29384]: from='client.36224 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:57.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:07:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:07:57.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-mon[39192]: pgmap v729: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 2.9 MiB/s wr, 824 op/s; 17416/330729 objects degraded (5.266%); 6.9 MiB/s, 110 objects/s recovering 2024-09-21T19:07:57.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3066340085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:57.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:57 smithi081 ceph-mon[39192]: from='client.36224 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.318 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:57.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:57.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:57 smithi104 ceph-mon[32438]: pgmap v729: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 2.9 MiB/s wr, 824 op/s; 17416/330729 objects degraded (5.266%); 6.9 MiB/s, 110 objects/s recovering 2024-09-21T19:07:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3066340085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:57 smithi104 ceph-mon[32438]: from='client.36224 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:07:57.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:57.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:57.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:07:57.728 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:07:57.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:57.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:57.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:57.973 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:58.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:58.130 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:07:58.131 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:07:58.131 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:07:58.131 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:07:58.131 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:07:58.136 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:07:58.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:58.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:58.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:58 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:58 smithi081 ceph-mon[39192]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:58 smithi081 ceph-mon[39192]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T19:07:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:58 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:58 smithi081 ceph-mon[29384]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:58 smithi081 ceph-mon[29384]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T19:07:58.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:07:58.421 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:07:58.457 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:58 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:07:58.457 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:58 smithi104 ceph-mon[32438]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:07:58.457 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:58 smithi104 ceph-mon[32438]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T19:07:58.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:07:58.552 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:07:58.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:58.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:58.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:07:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:07:58.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:58.823 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:58.925 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:58.957 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:58.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:59.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:07:59.098 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:59.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:07:59.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:59.314 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:59.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:59.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:07:59.437 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:59.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:07:59.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:07:59 smithi104 ceph-mon[32438]: pgmap v731: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s; 17099/321531 objects degraded (5.318%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:07:59.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:59.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:07:59 smithi081 ceph-mon[29384]: pgmap v731: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s; 17099/321531 objects degraded (5.318%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:07:59.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:07:59 smithi081 ceph-mon[39192]: pgmap v731: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s; 17099/321531 objects degraded (5.318%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:07:59.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:59.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:59.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:07:59.722 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:07:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:59.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:07:59.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:07:59.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:07:59.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:07:59.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:07:59.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:07:59.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:00.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.146 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:00.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:00.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:00.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:00.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:00 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:00.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:00 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 17099/321531 objects degraded (5.318%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:00.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:00 smithi081 ceph-mon[29384]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T19:08:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:00 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:00 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 17099/321531 objects degraded (5.318%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:00 smithi081 ceph-mon[39192]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T19:08:00.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:00.419 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:00.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:00 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:00 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 17099/321531 objects degraded (5.318%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:00 smithi104 ceph-mon[32438]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T19:08:00.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:00.553 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:00.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:00.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:00.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:00.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:00.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:00.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:01.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.115 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:01.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:01.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:01.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:01.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.392 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:01.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:01.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:01.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:01 smithi104 ceph-mon[32438]: pgmap v733: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 16783/316107 objects degraded (5.309%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T19:08:01.534 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:01.545 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:01 smithi081 ceph-mon[39192]: pgmap v733: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 16783/316107 objects degraded (5.309%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T19:08:01.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:01 smithi081 ceph-mon[29384]: pgmap v733: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s; 16783/316107 objects degraded (5.309%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T19:08:01.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:01.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:01.823 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:01.924 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:01.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:02.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:02.096 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:02.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:02.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:02.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:08:02.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:02.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:02.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:02.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:02.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:02.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:02.552 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:02.554 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:02.580 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:02.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:02.582 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:08:02.582 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:08:02.615 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:08:02.615 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:08:02.661 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:08:02.662 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:08:02.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:02.691 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:08:02.691 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:08:02.721 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:08:02.721 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:08:02.724 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:02.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:02.759 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:08:02.759 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:08:02.790 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:08:02.790 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:08:02.819 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:08:02.819 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:08:02.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:02.868 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:02.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:02.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:03.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:03.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.147 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:03.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:03.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:03.286 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:03.330 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:03 smithi081 ceph-mon[39192]: pgmap v734: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 772 op/s; 16783/314445 objects degraded (5.337%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T19:08:03.330 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3464573259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:03.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:03 smithi081 ceph-mon[29384]: pgmap v734: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 772 op/s; 16783/314445 objects degraded (5.337%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T19:08:03.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3464573259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:03.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:03.424 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:03.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:03 smithi104 ceph-mon[32438]: pgmap v734: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 772 op/s; 16783/314445 objects degraded (5.337%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T19:08:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3464573259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:03.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:03.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:03.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:03.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:03.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:03.999 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:04.085 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T19:08:04.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:04.127 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/142485615' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:08:04.127 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:04 smithi081 ceph-mon[29384]: from='client.36236 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:08:04.127 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/142485615' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:08:04.127 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:04 smithi081 ceph-mon[39192]: from='client.36236 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:08:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.144 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:04.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:04.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.288 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:04.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:04.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/142485615' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:08:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:04 smithi104 ceph-mon[32438]: from='client.36236 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:08:04.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:04.557 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:04.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:04.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:04.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:04.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:04.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:05.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.112 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:05.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[39192]: pgmap v735: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 16399/307293 objects degraded (5.337%); 7.9 MiB/s, 127 objects/s recovering 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[39192]: from='client.36236 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[39192]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 16399/307293 objects degraded (5.337%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[39192]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[29384]: pgmap v735: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 16399/307293 objects degraded (5.337%); 7.9 MiB/s, 127 objects/s recovering 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[29384]: from='client.36236 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:08:05.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[29384]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T19:08:05.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 16399/307293 objects degraded (5.337%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:05.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:05 smithi081 ceph-mon[29384]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T19:08:05.213 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.255 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:05.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:05.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:05.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:05 smithi104 ceph-mon[32438]: pgmap v735: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 16399/307293 objects degraded (5.337%); 7.9 MiB/s, 127 objects/s recovering 2024-09-21T19:08:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:05 smithi104 ceph-mon[32438]: from='client.36236 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:08:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:05 smithi104 ceph-mon[32438]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T19:08:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:05 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 16399/307293 objects degraded (5.337%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:05 smithi104 ceph-mon[32438]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T19:08:05.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:05.545 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:05.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:05.678 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:05.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:05.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T19:08:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.825 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:05.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:05.926 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:05.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:05.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:06.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:06.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:08:06.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:06.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:06.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:06.387 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:06.428 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1880785890' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:06.428 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:06 smithi081 ceph-mon[29384]: from='client.36242 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:06.429 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1880785890' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:06.429 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:06 smithi081 ceph-mon[39192]: from='client.36242 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:06.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:06.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1880785890' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:06.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:06 smithi104 ceph-mon[32438]: from='client.36242 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.527 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:06.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:06.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:06.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:06.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:06.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:06.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:06.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:06.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:06.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:08:07.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:07.141 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:07.141 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:07.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[39192]: pgmap v738: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 775 op/s; 16000/302076 objects degraded (5.297%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T19:08:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[39192]: from='client.36242 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[39192]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[29384]: pgmap v738: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 775 op/s; 16000/302076 objects degraded (5.297%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[29384]: from='client.36242 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[29384]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T19:08:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:07.208 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:08:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:08:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:08:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.228 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:07.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:07.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:07.368 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:07.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:07 smithi104 ceph-mon[32438]: pgmap v738: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 775 op/s; 16000/302076 objects degraded (5.297%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T19:08:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:07 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:07 smithi104 ceph-mon[32438]: from='client.36242 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:08:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:07 smithi104 ceph-mon[32438]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T19:08:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:07 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:07.511 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:07.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.647 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:07.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:07.748 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:07.783 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:07.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:08:07.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:07.934 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:07.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:07.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:07.953 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:08.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:08.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:08.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:08.231 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:08.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:08.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:08.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:08.496 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:08.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:08.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:08.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:08.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:08:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:08:08.765 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:08.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:08.900 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:08.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:08.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:08:09.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.066 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:09.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:09.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:09.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:09.291 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:09 smithi081 ceph-mon[29384]: pgmap v740: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s; 15636/292656 objects degraded (5.343%); 12 MiB/s, 191 objects/s recovering 2024-09-21T19:08:09.292 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:09 smithi081 ceph-mon[39192]: pgmap v740: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s; 15636/292656 objects degraded (5.343%); 12 MiB/s, 191 objects/s recovering 2024-09-21T19:08:09.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.343 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:09.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:09.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:09.479 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:09.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:09 smithi104 ceph-mon[32438]: pgmap v740: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s; 15636/292656 objects degraded (5.343%); 12 MiB/s, 191 objects/s recovering 2024-09-21T19:08:09.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:09.611 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:09.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.759 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:09.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:09.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:09.889 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:09.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:09.933 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:09.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:10.022 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:10.124 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:10.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:10.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:10.311 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:10.405 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:10 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:10.405 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:10 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:10.405 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:10 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:10.405 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:10 smithi104 ceph-mon[32438]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T19:08:10.405 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:10 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 15636/292656 objects degraded (5.343%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:10.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:10.438 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[29384]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 15636/292656 objects degraded (5.343%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[39192]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T19:08:10.538 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:10 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 15636/292656 objects degraded (5.343%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:10.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:10.568 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:10.568 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:10.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:10.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:10.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:10.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:10.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:10.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:10.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:10.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:11.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.100 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:11.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:11.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:11.247 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:11.349 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:11.383 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:11.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[29384]: pgmap v742: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 1.19k op/s; 15204/287154 objects degraded (5.295%); 13 MiB/s, 211 objects/s recovering 2024-09-21T19:08:11.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:11.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:11.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[29384]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T19:08:11.421 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[39192]: pgmap v742: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 1.19k op/s; 15204/287154 objects degraded (5.295%); 13 MiB/s, 211 objects/s recovering 2024-09-21T19:08:11.421 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:11.421 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:11.421 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:11 smithi081 ceph-mon[39192]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T19:08:11.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:11.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:11 smithi104 ceph-mon[32438]: pgmap v742: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 1.19k op/s; 15204/287154 objects degraded (5.295%); 13 MiB/s, 211 objects/s recovering 2024-09-21T19:08:11.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:11 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:11 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:11.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:11 smithi104 ceph-mon[32438]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T19:08:11.508 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:11.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:11.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:11.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:11.764 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:11.866 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:11.891 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:11.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:11.993 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:12.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:12.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:12.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:12.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:12.420 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:12.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:12.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:12.607 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:08:12.607 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:08:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:12.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:12.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:08:12.651 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:08:12.684 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:08:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:08:12.711 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:12.728 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:08:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:08:12.742 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:12.770 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:08:12.771 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:08:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:12.810 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:08:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:08:12.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:12.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:08:12.849 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:08:12.883 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:08:12.883 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:08:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:12.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:12.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:12.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.042 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:13.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:13.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:08:13.127 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T19:08:13.143 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:13.210 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:13 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T19:08:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:13 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T19:08:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:13 smithi104 ceph-mon[32438]: pgmap v744: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 924 op/s; 15204/285447 objects degraded (5.326%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T19:08:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1743386210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1491163786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:08:13.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:13 smithi104 ceph-mon[32438]: from='client.26122 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:08:13.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:13.374 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[29384]: pgmap v744: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 924 op/s; 15204/285447 objects degraded (5.326%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1743386210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1491163786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[29384]: from='client.26122 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[39192]: pgmap v744: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 924 op/s; 15204/285447 objects degraded (5.326%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T19:08:13.436 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1743386210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:13.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1491163786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:08:13.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:13 smithi081 ceph-mon[39192]: from='client.26122 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:08:13.476 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:13.529 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:13.592 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:13.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:13.672 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:13.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:13.796 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:13.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:13.942 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:13.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:14.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:14.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:14.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:14.259 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:08:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:08:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:08:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:08:14.260 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:08:14.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:14.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:14.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.503 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:14.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:14.512 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:14 smithi104 ceph-mon[32438]: from='client.26122 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:08:14.512 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:14 smithi104 ceph-mon[32438]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T19:08:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:14 smithi081 ceph-mon[39192]: from='client.26122 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:08:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:14 smithi081 ceph-mon[39192]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T19:08:14.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:14 smithi081 ceph-mon[29384]: from='client.26122 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:08:14.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:14 smithi081 ceph-mon[29384]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T19:08:14.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:14.628 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:14.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:14.765 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:14.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:14.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:15.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:15.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:15.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:08:15.130 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:15.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:15.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:15 smithi104 ceph-mon[32438]: 5.15 scrub starts 2024-09-21T19:08:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:15 smithi104 ceph-mon[32438]: 5.15 scrub ok 2024-09-21T19:08:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:15 smithi104 ceph-mon[32438]: pgmap v746: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 742 op/s; 14784/279297 objects degraded (5.293%); 8.9 MiB/s, 141 objects/s recovering 2024-09-21T19:08:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:15 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 14784/279297 objects degraded (5.293%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:15.342 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:15.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[39192]: 5.15 scrub starts 2024-09-21T19:08:15.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[39192]: 5.15 scrub ok 2024-09-21T19:08:15.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[39192]: pgmap v746: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 742 op/s; 14784/279297 objects degraded (5.293%); 8.9 MiB/s, 141 objects/s recovering 2024-09-21T19:08:15.437 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 14784/279297 objects degraded (5.293%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:15.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[29384]: 5.15 scrub starts 2024-09-21T19:08:15.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[29384]: 5.15 scrub ok 2024-09-21T19:08:15.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[29384]: pgmap v746: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 742 op/s; 14784/279297 objects degraded (5.293%); 8.9 MiB/s, 141 objects/s recovering 2024-09-21T19:08:15.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:15 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 14784/279297 objects degraded (5.293%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:15.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:15.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:15.566 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:15.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:15.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:15.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:15.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:15.917 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:15.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:16.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:16.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:16.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:16.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:16.197 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:16.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:16.198 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T19:08:16.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:08:16.269 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:08:16.269 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235896 8.22 8:45adf4b4:::benchmark_data_smithi081_105538_object103664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 382.53157957100001, 2024-09-21T19:08:16.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039943074, 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.271 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235896 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:08:16.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675143+0000", 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.729e-06 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675150+0000", 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6820000000000003e-06 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675168+0000", 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7875000000000001e-05 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675178+0000", 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5610000000000004e-06 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731748+0000", 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056570691999999999 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731754+0000", 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.976e-06 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.700981+0000", 2024-09-21T19:08:16.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96922663099999995 2024-09-21T19:08:16.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701110+0000", 2024-09-21T19:08:16.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864500000000001 2024-09-21T19:08:16.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:08:16.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701320+0000", 2024-09-21T19:08:16.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021059200000000001 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702216+0000", 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089577400000000001 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702245+0000", 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8608000000000001e-05 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.714979+0000", 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012734354999999999 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715038+0000", 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8588999999999998e-05 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715083+0000", 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5364999999999997e-05 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235898 8.2 8:4337194e:::benchmark_data_smithi081_105538_object103666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 382.53152619299999, 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0181600850000001, 2024-09-21T19:08:16.282 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235898 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675194+0000", 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1699999999999997e-07 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675197+0000", 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675203+0000", 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2369999999999997e-06 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675208+0000", 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9989999999999999e-06 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731592+0000", 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056383961000000003 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731604+0000", 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2278e-05 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685684+0000", 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95407945800000005 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685861+0000", 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017756400000000001 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.686082+0000", 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022110200000000001 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.687697+0000", 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016145230000000001 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.690382+0000", 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026848850000000001 2024-09-21T19:08:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693261+0000", 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879291 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693314+0000", 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2845e-05 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693353+0000", 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9369000000000002e-05 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237679 8.1b 8:deadaa1e:::benchmark_data_smithi081_105538_object105447:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 380.52207391299999, 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0964987129999999, 2024-09-21T19:08:16.289 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237679 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684647+0000", 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.153e-06 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684651+0000", 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3370000000000001e-06 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684660+0000", 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6470000000000004e-06 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684667+0000", 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684783+0000", 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011565600000000001 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684791+0000", 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7809999999999994e-06 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691384+0000", 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0065922899999999 2024-09-21T19:08:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691392+0000", 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.636e-06 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781063+0000", 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0896703569999999 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781144+0000", 2024-09-21T19:08:16.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1546999999999997e-05 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237686 8.1b 8:df3678b5:::benchmark_data_smithi081_105538_object105454:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 380.522034856, 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.173105171, 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:08:16.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237686 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1299999999999999e-07 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684688+0000", 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5830000000000001e-06 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684693+0000", 2024-09-21T19:08:16.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5910000000000003e-06 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684699+0000", 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684822+0000", 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012324499999999999 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684830+0000", 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6750000000000002e-06 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691420+0000", 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006590262 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691426+0000", 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4559999999999997e-06 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781412+0000", 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.089985687 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for missing object", 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781478+0000", 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6272000000000005e-05 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857711+0000", 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076232366999999995 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857790+0000", 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9029999999999994e-05 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.302 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237687 8.1b 8:dc29a777:::benchmark_data_smithi081_105538_object105455:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 380.52201451799999, 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1710659049999999, 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237687 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4899999999999995e-07 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684708+0000", 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1339999999999998e-06 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684713+0000", 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9749999999999998e-06 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684734+0000", 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1206e-05 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684861+0000", 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012699799999999999 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684870+0000", 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9770000000000006e-06 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691453+0000", 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006583418 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691459+0000", 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2580000000000001e-06 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781524+0000", 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0900641010000001 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781643+0000", 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119735 2024-09-21T19:08:16.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781902+0000", 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025915999999999998 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784476+0000", 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025738229999999998 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784488+0000", 2024-09-21T19:08:16.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1401e-05 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784513+0000", 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5462000000000001e-05 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855646+0000", 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071132476 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855686+0000", 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0157999999999997e-05 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855771+0000", 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5074000000000006e-05 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237708 8.1b 8:dbdbeb85:::benchmark_data_smithi081_105538_object105476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:08:16.310 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 380.50232692200001, 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.151669386, 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237708 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704394+0000", 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.855e-06 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704399+0000", 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9699999999999998e-06 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704413+0000", 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3560000000000001e-05 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704421+0000", 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6399999999999997e-06 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704473+0000", 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.27e-05 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704483+0000", 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4520000000000003e-06 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691484+0000", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98700145800000005 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691491+0000", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782825+0000", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0913340810000001 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782935+0000", 2024-09-21T19:08:16.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110647 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.783135+0000", 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020000399999999999 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784491+0000", 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013554319999999999 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784543+0000", 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2609999999999999e-05 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784566+0000", 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2945000000000001e-05 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855871+0000", 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071304949000000006 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855929+0000", 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7924999999999999e-05 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.856062+0000", 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013279700000000001 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237711 8.1b 8:dd96cf97:::benchmark_data_smithi081_105538_object105479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 380.500127335, 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1506975869999998, 2024-09-21T19:08:16.318 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237711 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706594+0000", 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9099999999999999e-06 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706599+0000", 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0259999999999998e-06 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706618+0000", 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8499e-05 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706633+0000", 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5931999999999999e-05 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706751+0000", 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117395 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706762+0000", 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1316e-05 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691770+0000", 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.985007619 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691776+0000", 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5490000000000003e-06 2024-09-21T19:08:16.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784627+0000", 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0928504649999999 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784699+0000", 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2539000000000005e-05 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784817+0000", 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118029 2024-09-21T19:08:16.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785805+0000", 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098732000000000008 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785853+0000", 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8554000000000003e-05 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785945+0000", 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1835000000000004e-05 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857195+0000", 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071249589000000002 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857226+0000", 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1705999999999998e-05 2024-09-21T19:08:16.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857290+0000", 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3304000000000002e-05 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2026 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94695648315648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 378.93432952900002, 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46135494399999999, 2024-09-21T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2026 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9019999999999994e-06 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272421+0000", 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7499999999999998e-05 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0607e-05 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272504+0000", 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2151999999999996e-05 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272515+0000", 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1241e-05 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702573+0000", 2024-09-21T19:08:16.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.430058259 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702717+0000", 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143931 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.733745+0000", 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031027852000000002 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2027 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 378.93426183000003, 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.464594917, 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2027 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4000000000000001e-07 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272462+0000", 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4199999999999999e-06 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272469+0000", 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272474+0000", 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4249999999999997e-06 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272539+0000", 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5066000000000003e-05 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272549+0000", 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8709999999999992e-06 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704278+0000", 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43172896500000002 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704401+0000", 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012265999999999999 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.737053+0000", 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.032651612000000003 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.336 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2036 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 378.65290189000001, 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15105808700000001, 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2036 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0399999999999996e-07 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553821+0000", 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6769999999999999e-06 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553827+0000", 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553831+0000", 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6550000000000003e-06 2024-09-21T19:08:16.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553873+0000", 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1241e-05 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553879+0000", 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5429999999999999e-06 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704657+0000", 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150777833 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704730+0000", 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3398000000000004e-05 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704876+0000", 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014533600000000001 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76067 9.c e280/254 9:3b8d2f8b:::benchmark_data_smithi081_131274_object76066:head v 280'4754, mlcod=280'4748)", 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 292.86133388600001, 2024-09-21T19:08:16.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23228328300000001, 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345387+0000", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5460000000000001e-06 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570553+0000", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.22516550099999999 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570563+0000", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0163000000000001e-05 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570569+0000", 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7660000000000004e-06 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570644+0000", 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5710999999999994e-05 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570678+0000", 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3608000000000001e-05 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577637+0000", 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069593909999999997 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577669+0000", 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1597000000000003e-05 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29874 10.1 10:8233bd87:::benchmark_data_smithi081_155596_object29873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.816351+0000", 2024-09-21T19:08:16.345 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 110.390368107, 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.117039637, 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29874 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816351+0000", 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816351+0000", 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816352+0000", 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2999999999999999e-07 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816387+0000", 2024-09-21T19:08:16.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5271999999999997e-05 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816395+0000", 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1890000000000001e-06 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816400+0000", 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5899999999999998e-06 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816490+0000", 2024-09-21T19:08:16.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9506000000000007e-05 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816621+0000", 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000131712 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.817621+0000", 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00099927000000000002 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.820057+0000", 2024-09-21T19:08:16.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024366959999999999 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823504+0000", 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034463850000000002 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823623+0000", 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118751 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823676+0000", 2024-09-21T19:08:16.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3559e-05 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933391+0000", 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109714877 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29876 10.1 10:8bb58828:::benchmark_data_smithi081_155596_object29875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.816825+0000", 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 110.38989488199999, 2024-09-21T19:08:16.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.116730545, 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29876 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816825+0000", 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816825+0000", 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816826+0000", 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.42e-06 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816858+0000", 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1702999999999998e-05 2024-09-21T19:08:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816868+0000", 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0596e-05 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816875+0000", 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9920000000000001e-06 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.818840+0000", 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019643870000000002 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.818946+0000", 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106789 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.819064+0000", 2024-09-21T19:08:16.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117364 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.821350+0000", 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022863219999999999 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823572+0000", 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022222549999999998 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933459+0000", 2024-09-21T19:08:16.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109886758 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933506+0000", 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7225999999999998e-05 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933555+0000", 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8733000000000002e-05 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.357 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29887 10.1 10:85f2786f:::benchmark_data_smithi081_155596_object29886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 110.38261396599999, 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12143838799999999, 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29887 2024-09-21T19:08:16.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824107+0000", 2024-09-21T19:08:16.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.767e-06 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824151+0000", 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4229e-05 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933322+0000", 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10917025 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933345+0000", 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3669999999999999e-05 2024-09-21T19:08:16.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934070+0000", 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072502099999999996 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934269+0000", 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867600000000001 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934418+0000", 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014940099999999999 2024-09-21T19:08:16.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.940623+0000", 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062042729999999997 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.940648+0000", 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5347999999999999e-05 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945384+0000", 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047363209999999999 2024-09-21T19:08:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945492+0000", 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107899 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945544+0000", 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1533000000000003e-05 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31276 10.e 10:78850574:::benchmark_data_smithi081_155596_object31275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:08:16.363 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.595397644, 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20212575599999999, 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31276 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611324+0000", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9740000000000001e-06 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611355+0000", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1634999999999998e-05 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611367+0000", 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1613999999999999e-05 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611374+0000", 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8029999999999999e-06 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611557+0000", 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018305700000000001 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611683+0000", 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012569999999999999 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611833+0000", 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015060799999999999 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.613723+0000", 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00188988 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614412+0000", 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068917700000000002 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813312+0000", 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198899348 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813403+0000", 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1162000000000006e-05 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813448+0000", 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4798e-05 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31282 10.e 10:7508a8e6:::benchmark_data_smithi081_155596_object31281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:08:16.369 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:08:16.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:16.370 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.592363284, 2024-09-21T19:08:16.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19924375599999999, 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31282 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614359+0000", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4099999999999998e-06 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614395+0000", 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6782999999999997e-05 2024-09-21T19:08:16.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614409+0000", 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3772e-05 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614417+0000", 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9030000000000004e-06 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614476+0000", 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8980000000000001e-05 2024-09-21T19:08:16.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614640+0000", 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016440500000000001 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614800+0000", 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015944200000000001 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.615994+0000", 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001193784 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.616143+0000", 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000149013 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813485+0000", 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.197342778 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813565+0000", 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9160999999999996e-05 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813600+0000", 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5324999999999998e-05 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31755 10.1 10:87c30842:::benchmark_data_smithi081_155596_object31754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.196436331, 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.163083229, 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.376 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31755 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010285+0000", 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8959999999999999e-06 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010383+0000", 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7940000000000001e-05 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010393+0000", 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0414999999999999e-05 2024-09-21T19:08:16.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010409+0000", 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5746000000000001e-05 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010487+0000", 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8318000000000005e-05 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010631+0000", 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143502 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010769+0000", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013836 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.015045+0000", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042753640000000002 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.015059+0000", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4182000000000001e-05 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173282+0000", 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15822359799999999 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173328+0000", 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5524e-05 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173366+0000", 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8383999999999997e-05 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.381 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:124468 10.1 10:815d4fed:::benchmark_data_smithi081_155596_object6995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:52.507511+0000", 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 23.699208564999999, 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103395511, 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 124468 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507511+0000", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507511+0000", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507512+0000", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.567e-06 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507517+0000", 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1930000000000003e-06 2024-09-21T19:08:16.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507525+0000", 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0970000000000006e-06 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507541+0000", 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6113e-05 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507897+0000", 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035643900000000001 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507987+0000", 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9956000000000004e-05 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.508080+0000", 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2881999999999996e-05 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.511714+0000", 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036335650000000001 2024-09-21T19:08:16.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.515388+0000", 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036739289999999998 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.610755+0000", 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.095367371000000006 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.610856+0000", 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010148499999999999 2024-09-21T19:08:16.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.610906+0000", 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9913999999999997e-05 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:124470 10.1 10:8d7365a5:::benchmark_data_smithi081_155596_object6997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:52.507652+0000", 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 23.699066984000002, 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.104117863, 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.387 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 124470 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507652+0000", 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507652+0000", 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507654+0000", 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0559999999999999e-06 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507657+0000", 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.681e-06 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507664+0000", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5570000000000001e-06 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.507670+0000", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0229999999999998e-06 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.511796+0000", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041257780000000001 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.511902+0000", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106562 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.512007+0000", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104967 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.515427+0000", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034196360000000002 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.515440+0000", 2024-09-21T19:08:16.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3138e-05 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.611687+0000", 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.096247104999999999 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.611733+0000", 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5595e-05 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:52.611770+0000", 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7765000000000002e-05 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.392 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:145379 10.1 10:8264e958:::benchmark_data_smithi081_155596_object27906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:14.961448+0000", 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.245271273, 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.063373692999999995, 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 145379 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:16.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961448+0000", 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961448+0000", 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961450+0000", 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.244e-06 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961456+0000", 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.032e-06 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961473+0000", 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6385999999999999e-05 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961481+0000", 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1030000000000002e-06 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961564+0000", 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2714999999999997e-05 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961665+0000", 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000101482 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.961764+0000", 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8684999999999995e-05 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.964465+0000", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002700788 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.965973+0000", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001508332 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.024691+0000", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058718481000000003 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.024778+0000", 2024-09-21T19:08:16.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6798999999999993e-05 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.024822+0000", 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3646000000000002e-05 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:16.398 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:08:16.398 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:16 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:16.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:16 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:16.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:16 smithi104 ceph-mon[32438]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T19:08:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:16.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:16.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:16 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:16.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:16 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:16.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:16 smithi081 ceph-mon[39192]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T19:08:16.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:16 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:16.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:16 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:16.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:16 smithi081 ceph-mon[29384]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T19:08:16.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:16.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:16.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:16.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:16.741 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:16.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:16.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:16.890 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:16.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:16.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:17.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:17.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:17.166 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:17.177 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:08:17.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:17.304 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:17.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:08:17.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:17.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:08:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:17 smithi104 ceph-mon[32438]: pgmap v748: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 697 op/s; 14349/274602 objects degraded (5.225%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T19:08:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3855364893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:17.509 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:08:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:08:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:08:17.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:17 smithi081 ceph-mon[39192]: pgmap v748: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 697 op/s; 14349/274602 objects degraded (5.225%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T19:08:17.510 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3855364893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:17.510 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:17 smithi081 ceph-mon[29384]: pgmap v748: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 697 op/s; 14349/274602 objects degraded (5.225%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T19:08:17.510 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3855364893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:17.541 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:17.589 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:17.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:17.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:17.786 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:17.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:17.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:17.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:17.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:18.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:18.138 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:08:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:18.143 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:08:18.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:18 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:08:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3855364893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:08:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:18 smithi104 ceph-mon[32438]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T19:08:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.280 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:18.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:18.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:18.411 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:18.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:18.545 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:18.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:18 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:08:18.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3855364893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:08:18.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:18 smithi081 ceph-mon[39192]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T19:08:18.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:18 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:08:18.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3855364893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:08:18.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:18 smithi081 ceph-mon[29384]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T19:08:18.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:18.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:18.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:08:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:08:18.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:18.883 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop_reply(client.36077.0:149047 10.c e379/330)", 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:18.880364+0000", 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00016929900000000001, 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019019100000000001, 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "queued for pg", 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.880364+0000", 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.880364+0000", 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.880365+0000", 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3370000000000001e-06 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.880371+0000", 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7950000000000004e-06 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.880381+0000", 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0098999999999999e-05 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.880388+0000", 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.596e-06 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149047 10.c 10:3a789ca8:::benchmark_data_smithi081_155596_object31574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:18.868289+0000", 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.012244623, 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012316004, 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:18.887 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149047 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868289+0000", 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868289+0000", 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868291+0000", 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.844e-06 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868296+0000", 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0039999999999999e-06 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868311+0000", 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4936000000000001e-05 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868317+0000", 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.844e-06 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868386+0000", 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8224000000000003e-05 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868546+0000", 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016068699999999999 2024-09-21T19:08:18.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,4", 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.868689+0000", 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014221000000000001 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876426+0000", 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0077378300000000002 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149055 10.c 10:3e83702d:::benchmark_data_smithi081_155596_object31582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T19:08:18.891 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:18.872474+0000", 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0080593609999999993, 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081976670000000005, 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149055 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872474+0000", 2024-09-21T19:08:18.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872474+0000", 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872476+0000", 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2469999999999998e-06 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:18.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872483+0000", 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8220000000000001e-06 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872500+0000", 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6660999999999999e-05 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872508+0000", 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5269999999999998e-06 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:18.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.872678+0000", 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017002599999999999 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.873045+0000", 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036625299999999998 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,4", 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.873500+0000", 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000455513 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:18.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:149054 10.1 e379/330 10:82cfb288:::benchmark_data_smithi081_155596_object31581:head v 379'9454, mlcod=379'9452)", 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:18.875220+0000", 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0053129570000000001, 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055096210000000001, 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875220+0000", 2024-09-21T19:08:18.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875220+0000", 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875228+0000", 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4070000000000004e-06 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:18.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875235+0000", 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.187e-06 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875248+0000", 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3111e-05 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875256+0000", 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.464e-06 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:18.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875425+0000", 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000169557 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.875477+0000", 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.147e-05 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:149056 10.8 e379/330 10:149cc45f:::benchmark_data_smithi081_155596_object31583:head v 379'9312, mlcod=379'9309)", 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:18.876333+0000", 2024-09-21T19:08:18.899 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0042006309999999998, 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044437679999999999, 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876333+0000", 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876333+0000", 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876334+0000", 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.482e-06 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876341+0000", 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.793e-06 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876357+0000", 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6379e-05 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876365+0000", 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6869999999999994e-06 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876436+0000", 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0877000000000006e-05 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.876462+0000", 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6404e-05 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.902 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:149057 10.8 e379/330 10:178803bf:::benchmark_data_smithi081_155596_object31584:head v 379'9313, mlcod=379'9309)", 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:18.879797+0000", 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00073605299999999997, 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001023468, 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879797+0000", 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879797+0000", 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879799+0000", 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5820000000000001e-06 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879807+0000", 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1380000000000007e-06 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879815+0000", 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1869999999999997e-06 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879822+0000", 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1250000000000004e-06 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879890+0000", 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8042999999999999e-05 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:18.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:18.879908+0000", 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7782999999999999e-05 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 6 2024-09-21T19:08:18.906 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:08:18.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:18.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:18.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:19.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.120 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:19.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:19.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:19.252 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:19 smithi081 ceph-mon[29384]: pgmap v750: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 13941/265995 objects degraded (5.241%); 13 MiB/s, 210 objects/s recovering 2024-09-21T19:08:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:19 smithi081 ceph-mon[29384]: from='client.36260 v1:172.21.15.81:0/1539850137' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:19 smithi081 ceph-mon[39192]: pgmap v750: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 13941/265995 objects degraded (5.241%); 13 MiB/s, 210 objects/s recovering 2024-09-21T19:08:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:19 smithi081 ceph-mon[39192]: from='client.36260 v1:172.21.15.81:0/1539850137' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:19.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:19.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:19.416 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:19.421 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2024-09-21T19:08:17.740401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5798238771,"num_objects":88665,"num_object_clones":0,"num_object_copies":265995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88665,"num_whiteouts":0,"num_read":2038,"num_read_kb":1790,"num_write":147198,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9613,"num_bytes_recovered":614869240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87435,"ondisk_log_size":87435,"up":474,"acting":469,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20574204,"kb_used_data":16404472,"kb_used_omap":18,"kb_used_meta":4169261,"kb_avail":729387012,"statfs":{"total":767960285184,"available":746892300288,"internally_reserved":0,"allocated":16798179328,"data_stored":16793043559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4269324272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,9,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-424935424,"num_objects":-6484,"num_object_clones":0,"num_object_copies":-19452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6484,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6484,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002311"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":596,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194755+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:08:17.194755+0000","last_peered":"2024-09-21T19:08:17.194755+0000","last_clean":"2024-09-21T19:08:17.194755+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:08:17.194755+0000","last_undegraded":"2024-09-21T19:08:17.194755+0000","last_fullsized":"2024-09-21T19:08:17.194755+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":763,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.159854+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:08:15.159854+0000","last_peered":"2024-09-21T19:08:15.159854+0000","last_clean":"2024-09-21T19:08:15.159854+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:08:15.159854+0000","last_undegraded":"2024-09-21T19:08:15.159854+0000","last_fullsized":"2024-09-21T19:08:15.159854+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":694,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181498+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:08:17.181498+0000","last_peered":"2024-09-21T19:08:17.181498+0000","last_clean":"2024-09-21T19:08:17.181498+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:08:17.181498+0000","last_undegraded":"2024-09-21T19:08:17.181498+0000","last_fullsized":"2024-09-21T19:08:17.181498+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":680,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.166261+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:08:15.166261+0000","last_peered":"2024-09-21T19:08:15.166261+0000","last_clean":"2024-09-21T19:08:15.166261+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:08:15.166261+0000","last_undegraded":"2024-09-21T19:08:15.166261+0000","last_fullsized":"2024-09-21T19:08:15.166261+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":684,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143497+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:08:10.143497+0000","last_peered":"2024-09-21T19:08:10.143497+0000","last_clean":"2024-09-21T19:08:10.143497+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:08:10.143497+0000","last_undegraded":"2024-09-21T19:08:10.143497+0000","last_fullsized":"2024-09-21T19:08:10.143497+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":712,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144034+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:08:10.144034+0000","last_peered":"2024-09-21T19:08:10.144034+0000","last_clean":"2024-09-21T19:08:10.144034+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:08:10.144034+0000","last_undegraded":"2024-09-21T19:08:10.144034+0000","last_fullsized":"2024-09-21T19:08:10.144034+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":809,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143191+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:08:10.143191+0000","last_peered":"2024-09-21T19:08:10.143191+0000","last_clean":"2024-09-21T19:08:10.143191+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:08:10.143191+0000","last_undegraded":"2024-09-21T19:08:10.143191+0000","last_fullsized":"2024-09-21T19:08:10.143191+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:13.608987+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":700,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.159777+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:08:15.159777+0000","last_peered":"2024-09-21T19:08:15.159777+0000","last_clean":"2024-09-21T19:08:15.159777+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:08:15.159777+0000","last_undegraded":"2024-09-21T19:08:15.159777+0000","last_fullsized":"2024-09-21T19:08:15.159777+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":687,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.157170+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:08:15.157170+0000","last_peered":"2024-09-21T19:08:15.157170+0000","last_clean":"2024-09-21T19:08:15.157170+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:08:15.157170+0000","last_undegraded":"2024-09-21T19:08:15.157170+0000","last_fullsized":"2024-09-21T19:08:15.157170+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":744,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.166558+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:08:15.166558+0000","last_peered":"2024-09-21T19:08:15.166558+0000","last_clean":"2024-09-21T19:08:15.166558+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:08:15.166558+0000","last_undegraded":"2024-09-21T19:08:15.166558+0000","last_fullsized":"2024-09-21T19:08:15.166558+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":636,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.155045+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:08:10.155045+0000","last_peered":"2024-09-21T19:08:10.155045+0000","last_clean":"2024-09-21T19:08:10.155045+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:08:10.155045+0000","last_undegraded":"2024-09-21T19:08:10.155045+0000","last_fullsized":"2024-09-21T19:08:10.155045+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":563,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.195350+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:08:17.195350+0000","last_peered":"2024-09-21T19:08:17.195350+0000","last_clean":"2024-09-21T19:08:17.195350+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:08:17.195350+0000","last_undegraded":"2024-09-21T19:08:17.195350+0000","last_fullsized":"2024-09-21T19:08:17.195350+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":240,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.157004+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:08:15.157004+0000","last_peered":"2024-09-21T19:08:15.157004+0000","last_clean":"2024-09-21T19:08:15.157004+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:08:15.157004+0000","last_undegraded":"2024-09-21T19:08:15.157004+0000","last_fullsized":"2024-09-21T19:08:15.157004+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":690,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156161+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:08:15.156161+0000","last_peered":"2024-09-21T19:08:15.156161+0000","last_clean":"2024-09-21T19:08:15.156161+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:08:15.156161+0000","last_undegraded":"2024-09-21T19:08:15.156161+0000","last_fullsized":"2024-09-21T19:08:15.156161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":672,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167131+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:08:15.167131+0000","last_peered":"2024-09-21T19:08:15.167131+0000","last_clean":"2024-09-21T19:08:15.167131+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:08:15.167131+0000","last_undegraded":"2024-09-21T19:08:15.167131+0000","last_fullsized":"2024-09-21T19:08:15.167131+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":242,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181300+0000","last_change":"2024-09-21T19:06:54.696659+0000","last_active":"2024-09-21T19:08:17.181300+0000","last_peered":"2024-09-21T19:08:17.181300+0000","last_clean":"2024-09-21T19:08:17.181300+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:08:17.181300+0000","last_undegraded":"2024-09-21T19:08:17.181300+0000","last_fullsized":"2024-09-21T19:08:17.181300+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:16.368451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:16.368451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":708,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.158928+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:08:15.158928+0000","last_peered":"2024-09-21T19:08:15.158928+0000","last_clean":"2024-09-21T19:08:15.158928+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:08:15.158928+0000","last_undegraded":"2024-09-21T19:08:15.158928+0000","last_fullsized":"2024-09-21T19:08:15.158928+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":578,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194856+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:08:17.194856+0000","last_peered":"2024-09-21T19:08:17.194856+0000","last_clean":"2024-09-21T19:08:17.194856+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:08:17.194856+0000","last_undegraded":"2024-09-21T19:08:17.194856+0000","last_fullsized":"2024-09-21T19:08:17.194856+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":153,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.199397+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:08:17.199397+0000","last_peered":"2024-09-21T19:08:17.199397+0000","last_clean":"2024-09-21T19:08:17.199397+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:08:17.199397+0000","last_undegraded":"2024-09-21T19:08:17.199397+0000","last_fullsized":"2024-09-21T19:08:17.199397+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":697,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181952+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:08:17.181952+0000","last_peered":"2024-09-21T19:08:17.181952+0000","last_clean":"2024-09-21T19:08:17.181952+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:08:17.181952+0000","last_undegraded":"2024-09-21T19:08:17.181952+0000","last_fullsized":"2024-09-21T19:08:17.181952+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:02.338670+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":681,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144634+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:08:10.144634+0000","last_peered":"2024-09-21T19:08:10.144634+0000","last_clean":"2024-09-21T19:08:10.144634+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:08:10.144634+0000","last_undegraded":"2024-09-21T19:08:10.144634+0000","last_fullsized":"2024-09-21T19:08:10.144634+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":711,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.166683+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:08:15.166683+0000","last_peered":"2024-09-21T19:08:15.166683+0000","last_clean":"2024-09-21T19:08:15.166683+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:08:15.166683+0000","last_undegraded":"2024-09-21T19:08:15.166683+0000","last_fullsized":"2024-09-21T19:08:15.166683+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":752,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144985+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:08:10.144985+0000","last_peered":"2024-09-21T19:08:10.144985+0000","last_clean":"2024-09-21T19:08:10.144985+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:08:10.144985+0000","last_undegraded":"2024-09-21T19:08:10.144985+0000","last_fullsized":"2024-09-21T19:08:10.144985+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":695,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156839+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:08:15.156839+0000","last_peered":"2024-09-21T19:08:15.156839+0000","last_clean":"2024-09-21T19:08:15.156839+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:08:15.156839+0000","last_undegraded":"2024-09-21T19:08:15.156839+0000","last_fullsized":"2024-09-21T19:08:15.156839+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":718,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167736+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:08:15.167736+0000","last_peered":"2024-09-21T19:08:15.167736+0000","last_clean":"2024-09-21T19:08:15.167736+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:08:15.167736+0000","last_undegraded":"2024-09-21T19:08:15.167736+0000","last_fullsized":"2024-09-21T19:08:15.167736+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":711,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181582+0000","last_change":"2024-09-21T19:07:33.264047+0000","last_active":"2024-09-21T19:08:17.181582+0000","last_peered":"2024-09-21T19:08:17.181582+0000","last_clean":"2024-09-21T19:08:17.181582+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:08:17.181582+0000","last_undegraded":"2024-09-21T19:08:17.181582+0000","last_fullsized":"2024-09-21T19:08:17.181582+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:33.263975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:33.263975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.063323+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":607,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.202751+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:08:17.202751+0000","last_peered":"2024-09-21T19:08:17.202751+0000","last_clean":"2024-09-21T19:08:17.202751+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:08:17.202751+0000","last_undegraded":"2024-09-21T19:08:17.202751+0000","last_fullsized":"2024-09-21T19:08:17.202751+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":759,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135634+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:08:13.135634+0000","last_peered":"2024-09-21T19:08:13.135634+0000","last_clean":"2024-09-21T19:08:13.135634+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:08:13.135634+0000","last_undegraded":"2024-09-21T19:08:13.135634+0000","last_fullsized":"2024-09-21T19:08:13.135634+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"378'9319","reported_seq":9427,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:16.700046+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:08:16.700046+0000","last_peered":"2024-09-21T19:08:16.700046+0000","last_clean":"2024-09-21T19:08:16.700046+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:08:16.700046+0000","last_undegraded":"2024-09-21T19:08:16.700046+0000","last_fullsized":"2024-09-21T19:08:16.700046+0000","mapping_epoch":330,"log_start":"350'4800","ondisk_log_start":"350'4800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364445696,"num_objects":5561,"num_object_clones":0,"num_object_copies":16683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":148,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.199125+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:08:17.199125+0000","last_peered":"2024-09-21T19:08:17.199125+0000","last_clean":"2024-09-21T19:08:17.199125+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:08:17.199125+0000","last_undegraded":"2024-09-21T19:08:17.199125+0000","last_fullsized":"2024-09-21T19:08:17.199125+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":746,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.182195+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:08:17.182195+0000","last_peered":"2024-09-21T19:08:17.182195+0000","last_clean":"2024-09-21T19:08:17.182195+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:08:17.182195+0000","last_undegraded":"2024-09-21T19:08:17.182195+0000","last_fullsized":"2024-09-21T19:08:17.182195+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":678,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.139179+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:08:10.139179+0000","last_peered":"2024-09-21T19:08:10.139179+0000","last_clean":"2024-09-21T19:08:10.139179+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:08:10.139179+0000","last_undegraded":"2024-09-21T19:08:10.139179+0000","last_fullsized":"2024-09-21T19:08:10.139179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":914,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.159015+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:08:15.159015+0000","last_peered":"2024-09-21T19:08:15.159015+0000","last_clean":"2024-09-21T19:08:15.159015+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:08:15.159015+0000","last_undegraded":"2024-09-21T19:08:15.159015+0000","last_fullsized":"2024-09-21T19:08:15.159015+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":374,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181857+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:08:17.181857+0000","last_peered":"2024-09-21T19:08:17.181857+0000","last_clean":"2024-09-21T19:08:17.181857+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:08:17.181857+0000","last_undegraded":"2024-09-21T19:08:17.181857+0000","last_fullsized":"2024-09-21T19:08:17.181857+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"378'9070","reported_seq":9177,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:16.711839+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:08:16.711839+0000","last_peered":"2024-09-21T19:08:16.711839+0000","last_clean":"2024-09-21T19:08:16.711839+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:08:16.711839+0000","last_undegraded":"2024-09-21T19:08:16.711839+0000","last_fullsized":"2024-09-21T19:08:16.711839+0000","mapping_epoch":330,"log_start":"346'4500","ondisk_log_start":"346'4500","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351797248,"num_objects":5368,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":471,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.157078+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:08:15.157078+0000","last_peered":"2024-09-21T19:08:15.157078+0000","last_clean":"2024-09-21T19:08:15.157078+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:08:15.157078+0000","last_undegraded":"2024-09-21T19:08:15.157078+0000","last_fullsized":"2024-09-21T19:08:15.157078+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":775,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.138945+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:08:10.138945+0000","last_peered":"2024-09-21T19:08:10.138945+0000","last_clean":"2024-09-21T19:08:10.138945+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:08:10.138945+0000","last_undegraded":"2024-09-21T19:08:10.138945+0000","last_fullsized":"2024-09-21T19:08:10.138945+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":700,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144934+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:08:10.144934+0000","last_peered":"2024-09-21T19:08:10.144934+0000","last_clean":"2024-09-21T19:08:10.144934+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:08:10.144934+0000","last_undegraded":"2024-09-21T19:08:10.144934+0000","last_fullsized":"2024-09-21T19:08:10.144934+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":682,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.176617+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:08:15.176617+0000","last_peered":"2024-09-21T19:08:15.176617+0000","last_clean":"2024-09-21T19:08:15.176617+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:08:15.176617+0000","last_undegraded":"2024-09-21T19:08:15.176617+0000","last_fullsized":"2024-09-21T19:08:15.176617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":699,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.166475+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:08:15.166475+0000","last_peered":"2024-09-21T19:08:15.166475+0000","last_clean":"2024-09-21T19:08:15.166475+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:08:15.166475+0000","last_undegraded":"2024-09-21T19:08:15.166475+0000","last_fullsized":"2024-09-21T19:08:15.166475+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"377'9054","reported_seq":9159,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:14.206223+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:08:14.206223+0000","last_peered":"2024-09-21T19:08:14.206223+0000","last_clean":"2024-09-21T19:08:14.206223+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:08:14.206223+0000","last_undegraded":"2024-09-21T19:08:14.206223+0000","last_fullsized":"2024-09-21T19:08:14.206223+0000","mapping_epoch":330,"log_start":"342'3700","ondisk_log_start":"342'3700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5354,"log_dups_size":0,"ondisk_log_size":5354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370016256,"num_objects":5646,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":458,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143246+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:08:10.143246+0000","last_peered":"2024-09-21T19:08:10.143246+0000","last_clean":"2024-09-21T19:08:10.143246+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:08:10.143246+0000","last_undegraded":"2024-09-21T19:08:10.143246+0000","last_fullsized":"2024-09-21T19:08:10.143246+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":750,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.176767+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:08:15.176767+0000","last_peered":"2024-09-21T19:08:15.176767+0000","last_clean":"2024-09-21T19:08:15.176767+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:08:15.176767+0000","last_undegraded":"2024-09-21T19:08:15.176767+0000","last_fullsized":"2024-09-21T19:08:15.176767+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":575,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.203248+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:08:17.203248+0000","last_peered":"2024-09-21T19:08:17.203248+0000","last_clean":"2024-09-21T19:08:17.203248+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:08:17.203248+0000","last_undegraded":"2024-09-21T19:08:17.203248+0000","last_fullsized":"2024-09-21T19:08:17.203248+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":853,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144369+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:08:10.144369+0000","last_peered":"2024-09-21T19:08:10.144369+0000","last_clean":"2024-09-21T19:08:10.144369+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:08:10.144369+0000","last_undegraded":"2024-09-21T19:08:10.144369+0000","last_fullsized":"2024-09-21T19:08:10.144369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":702,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.166976+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:08:15.166976+0000","last_peered":"2024-09-21T19:08:15.166976+0000","last_clean":"2024-09-21T19:08:15.166976+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:08:15.166976+0000","last_undegraded":"2024-09-21T19:08:15.166976+0000","last_fullsized":"2024-09-21T19:08:15.166976+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"378'9361","reported_seq":9468,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:16.826026+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:08:16.826026+0000","last_peered":"2024-09-21T19:08:16.826026+0000","last_clean":"2024-09-21T19:08:16.826026+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:08:16.826026+0000","last_undegraded":"2024-09-21T19:08:16.826026+0000","last_fullsized":"2024-09-21T19:08:16.826026+0000","mapping_epoch":330,"log_start":"344'4300","ondisk_log_start":"344'4300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365494272,"num_objects":5577,"num_object_clones":0,"num_object_copies":16731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9361,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":484,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135796+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:08:13.135796+0000","last_peered":"2024-09-21T19:08:13.135796+0000","last_clean":"2024-09-21T19:08:13.135796+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:08:13.135796+0000","last_undegraded":"2024-09-21T19:08:13.135796+0000","last_fullsized":"2024-09-21T19:08:13.135796+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":729,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.183419+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:08:17.183419+0000","last_peered":"2024-09-21T19:08:17.183419+0000","last_clean":"2024-09-21T19:08:17.183419+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:08:17.183419+0000","last_undegraded":"2024-09-21T19:08:17.183419+0000","last_fullsized":"2024-09-21T19:08:17.183419+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":242,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.145029+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:08:10.145029+0000","last_peered":"2024-09-21T19:08:10.145029+0000","last_clean":"2024-09-21T19:08:10.145029+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:08:10.145029+0000","last_undegraded":"2024-09-21T19:08:10.145029+0000","last_fullsized":"2024-09-21T19:08:10.145029+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":742,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194774+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:08:17.194774+0000","last_peered":"2024-09-21T19:08:17.194774+0000","last_clean":"2024-09-21T19:08:17.194774+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:08:17.194774+0000","last_undegraded":"2024-09-21T19:08:17.194774+0000","last_fullsized":"2024-09-21T19:08:17.194774+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":578,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.195474+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:08:17.195474+0000","last_peered":"2024-09-21T19:08:17.195474+0000","last_clean":"2024-09-21T19:08:17.195474+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:08:17.195474+0000","last_undegraded":"2024-09-21T19:08:17.195474+0000","last_fullsized":"2024-09-21T19:08:17.195474+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"377'8998","reported_seq":9104,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:14.138526+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:08:14.138526+0000","last_peered":"2024-09-21T19:08:14.138526+0000","last_clean":"2024-09-21T19:08:14.138526+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:08:14.138526+0000","last_undegraded":"2024-09-21T19:08:14.138526+0000","last_fullsized":"2024-09-21T19:08:14.138526+0000","mapping_epoch":330,"log_start":"342'3700","ondisk_log_start":"342'3700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5298,"log_dups_size":0,"ondisk_log_size":5298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372047896,"num_objects":5678,"num_object_clones":0,"num_object_copies":17034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8998,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":462,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135086+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:08:13.135086+0000","last_peered":"2024-09-21T19:08:13.135086+0000","last_clean":"2024-09-21T19:08:13.135086+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:08:13.135086+0000","last_undegraded":"2024-09-21T19:08:13.135086+0000","last_fullsized":"2024-09-21T19:08:13.135086+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":623,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194928+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:08:17.194928+0000","last_peered":"2024-09-21T19:08:17.194928+0000","last_clean":"2024-09-21T19:08:17.194928+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:08:17.194928+0000","last_undegraded":"2024-09-21T19:08:17.194928+0000","last_fullsized":"2024-09-21T19:08:17.194928+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":717,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.182093+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:08:17.182093+0000","last_peered":"2024-09-21T19:08:17.182093+0000","last_clean":"2024-09-21T19:08:17.182093+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:08:17.182093+0000","last_undegraded":"2024-09-21T19:08:17.182093+0000","last_fullsized":"2024-09-21T19:08:17.182093+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:18.453776+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":597,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135444+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:08:13.135444+0000","last_peered":"2024-09-21T19:08:13.135444+0000","last_clean":"2024-09-21T19:08:13.135444+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:08:13.135444+0000","last_undegraded":"2024-09-21T19:08:13.135444+0000","last_fullsized":"2024-09-21T19:08:13.135444+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":705,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181405+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:08:17.181405+0000","last_peered":"2024-09-21T19:08:17.181405+0000","last_clean":"2024-09-21T19:08:17.181405+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:08:17.181405+0000","last_undegraded":"2024-09-21T19:08:17.181405+0000","last_fullsized":"2024-09-21T19:08:17.181405+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:39.298144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"377'9141","reported_seq":17659,"reported_epoch":377,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:14.203600+0000","last_change":"2024-09-21T19:06:55.884722+0000","last_active":"2024-09-21T19:08:14.203600+0000","last_peered":"2024-09-21T19:08:14.203600+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.859865+0000","last_became_peered":"2024-09-21T19:06:55.859865+0000","last_unstale":"2024-09-21T19:08:14.203600+0000","last_undegraded":"2024-09-21T19:06:55.708016+0000","last_fullsized":"2024-09-21T19:06:55.683203+0000","mapping_epoch":349,"log_start":"342'3800","ondisk_log_start":"342'3800","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374145024,"num_objects":5709,"num_object_clones":0,"num_object_copies":17127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4206,"num_bytes_recovered":275644416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":5236}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":442,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194272+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:08:17.194272+0000","last_peered":"2024-09-21T19:08:17.194272+0000","last_clean":"2024-09-21T19:08:17.194272+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:08:17.194272+0000","last_undegraded":"2024-09-21T19:08:17.194272+0000","last_fullsized":"2024-09-21T19:08:17.194272+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":987,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181957+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:08:17.181957+0000","last_peered":"2024-09-21T19:08:17.181957+0000","last_clean":"2024-09-21T19:08:17.181957+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:08:17.181957+0000","last_undegraded":"2024-09-21T19:08:17.181957+0000","last_fullsized":"2024-09-21T19:08:17.181957+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:41.293807+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":723,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156477+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:08:15.156477+0000","last_peered":"2024-09-21T19:08:15.156477+0000","last_clean":"2024-09-21T19:08:15.156477+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:08:15.156477+0000","last_undegraded":"2024-09-21T19:08:15.156477+0000","last_fullsized":"2024-09-21T19:08:15.156477+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":264,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.183581+0000","last_change":"2024-09-21T19:06:54.707194+0000","last_active":"2024-09-21T19:08:17.183581+0000","last_peered":"2024-09-21T19:08:17.183581+0000","last_clean":"2024-09-21T19:08:17.183581+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:08:17.183581+0000","last_undegraded":"2024-09-21T19:08:17.183581+0000","last_fullsized":"2024-09-21T19:08:17.183581+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:03:15.425359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:03:15.425359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":563,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.198402+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:08:17.198402+0000","last_peered":"2024-09-21T19:08:17.198402+0000","last_clean":"2024-09-21T19:08:17.198402+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:08:17.198402+0000","last_undegraded":"2024-09-21T19:08:17.198402+0000","last_fullsized":"2024-09-21T19:08:17.198402+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":666,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143776+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:08:10.143776+0000","last_peered":"2024-09-21T19:08:10.143776+0000","last_clean":"2024-09-21T19:08:10.143776+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:08:10.143776+0000","last_undegraded":"2024-09-21T19:08:10.143776+0000","last_fullsized":"2024-09-21T19:08:10.143776+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"379'9159","reported_seq":14467,"reported_epoch":379,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:17.603594+0000","last_change":"2024-09-21T19:06:55.907554+0000","last_active":"2024-09-21T19:08:17.603594+0000","last_peered":"2024-09-21T19:08:17.603594+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.870419+0000","last_became_peered":"2024-09-21T19:06:55.870419+0000","last_unstale":"2024-09-21T19:08:17.603594+0000","last_undegraded":"2024-09-21T19:06:55.799686+0000","last_fullsized":"2024-09-21T19:06:55.681444+0000","mapping_epoch":349,"log_start":"338'2700","ondisk_log_start":"338'2700","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356188160,"num_objects":5435,"num_object_clones":0,"num_object_copies":16305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2599,"num_bytes_recovered":170328064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":3397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":767,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.166934+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:08:15.166934+0000","last_peered":"2024-09-21T19:08:15.166934+0000","last_clean":"2024-09-21T19:08:15.166934+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:08:15.166934+0000","last_undegraded":"2024-09-21T19:08:15.166934+0000","last_fullsized":"2024-09-21T19:08:15.166934+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":182,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181928+0000","last_change":"2024-09-21T19:06:54.705027+0000","last_active":"2024-09-21T19:08:17.181928+0000","last_peered":"2024-09-21T19:08:17.181928+0000","last_clean":"2024-09-21T19:08:17.181928+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:08:17.181928+0000","last_undegraded":"2024-09-21T19:08:17.181928+0000","last_fullsized":"2024-09-21T19:08:17.181928+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:03:40.251822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:40.251822+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":739,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.182828+0000","last_change":"2024-09-21T19:07:29.296652+0000","last_active":"2024-09-21T19:08:17.182828+0000","last_peered":"2024-09-21T19:08:17.182828+0000","last_clean":"2024-09-21T19:08:17.182828+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:08:17.182828+0000","last_undegraded":"2024-09-21T19:08:17.182828+0000","last_fullsized":"2024-09-21T19:08:17.182828+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:29.296562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:29.296562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:51.422184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":141,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.212205+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:08:17.212205+0000","last_peered":"2024-09-21T19:08:17.212205+0000","last_clean":"2024-09-21T19:08:17.212205+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:08:17.212205+0000","last_undegraded":"2024-09-21T19:08:17.212205+0000","last_fullsized":"2024-09-21T19:08:17.212205+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":140,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.202078+0000","last_change":"2024-09-21T19:06:54.694760+0000","last_active":"2024-09-21T19:08:17.202078+0000","last_peered":"2024-09-21T19:08:17.202078+0000","last_clean":"2024-09-21T19:08:17.202078+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:08:17.202078+0000","last_undegraded":"2024-09-21T19:08:17.202078+0000","last_fullsized":"2024-09-21T19:08:17.202078+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":692,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167642+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:08:15.167642+0000","last_peered":"2024-09-21T19:08:15.167642+0000","last_clean":"2024-09-21T19:08:15.167642+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:08:15.167642+0000","last_undegraded":"2024-09-21T19:08:15.167642+0000","last_fullsized":"2024-09-21T19:08:15.167642+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"379'9229","reported_seq":9338,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.502717+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:08:17.502717+0000","last_peered":"2024-09-21T19:08:17.502717+0000","last_clean":"2024-09-21T19:08:17.502717+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:08:17.502717+0000","last_undegraded":"2024-09-21T19:08:17.502717+0000","last_fullsized":"2024-09-21T19:08:17.502717+0000","mapping_epoch":330,"log_start":"344'4200","ondisk_log_start":"344'4200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360906752,"num_objects":5507,"num_object_clones":0,"num_object_copies":16521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9229,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":475,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.184187+0000","last_change":"2024-09-21T19:07:30.274632+0000","last_active":"2024-09-21T19:08:17.184187+0000","last_peered":"2024-09-21T19:08:17.184187+0000","last_clean":"2024-09-21T19:08:17.184187+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:08:17.184187+0000","last_undegraded":"2024-09-21T19:08:17.184187+0000","last_fullsized":"2024-09-21T19:08:17.184187+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:07:30.274412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:07:30.274412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:58.929542+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":685,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.202840+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:08:17.202840+0000","last_peered":"2024-09-21T19:08:17.202840+0000","last_clean":"2024-09-21T19:08:17.202840+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:08:17.202840+0000","last_undegraded":"2024-09-21T19:08:17.202840+0000","last_fullsized":"2024-09-21T19:08:17.202840+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":571,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194641+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:08:17.194641+0000","last_peered":"2024-09-21T19:08:17.194641+0000","last_clean":"2024-09-21T19:08:17.194641+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:08:17.194641+0000","last_undegraded":"2024-09-21T19:08:17.194641+0000","last_fullsized":"2024-09-21T19:08:17.194641+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1617,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.671709+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:08:15.671709+0000","last_peered":"2024-09-21T19:08:15.671709+0000","last_clean":"2024-09-21T19:08:15.671709+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:08:15.671709+0000","last_undegraded":"2024-09-21T19:08:15.671709+0000","last_fullsized":"2024-09-21T19:08:15.671709+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":898,"num_read_kb":893,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":637,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.155835+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:08:10.155835+0000","last_peered":"2024-09-21T19:08:10.155835+0000","last_clean":"2024-09-21T19:08:10.155835+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:08:10.155835+0000","last_undegraded":"2024-09-21T19:08:10.155835+0000","last_fullsized":"2024-09-21T19:08:10.155835+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":691,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181717+0000","last_change":"2024-09-21T19:06:54.695510+0000","last_active":"2024-09-21T19:08:17.181717+0000","last_peered":"2024-09-21T19:08:17.181717+0000","last_clean":"2024-09-21T19:08:17.181717+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:08:17.181717+0000","last_undegraded":"2024-09-21T19:08:17.181717+0000","last_fullsized":"2024-09-21T19:08:17.181717+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:54.568845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:54.568845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"377'9004","reported_seq":9109,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:14.201777+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:08:14.201777+0000","last_peered":"2024-09-21T19:08:14.201777+0000","last_clean":"2024-09-21T19:08:14.201777+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:08:14.201777+0000","last_undegraded":"2024-09-21T19:08:14.201777+0000","last_fullsized":"2024-09-21T19:08:14.201777+0000","mapping_epoch":330,"log_start":"342'3700","ondisk_log_start":"342'3700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369491968,"num_objects":5638,"num_object_clones":0,"num_object_copies":16914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":850,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.134012+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:08:13.134012+0000","last_peered":"2024-09-21T19:08:13.134012+0000","last_clean":"2024-09-21T19:08:13.134012+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:08:13.134012+0000","last_undegraded":"2024-09-21T19:08:13.134012+0000","last_fullsized":"2024-09-21T19:08:13.134012+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":701,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143431+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:08:10.143431+0000","last_peered":"2024-09-21T19:08:10.143431+0000","last_clean":"2024-09-21T19:08:10.143431+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:08:10.143431+0000","last_undegraded":"2024-09-21T19:08:10.143431+0000","last_fullsized":"2024-09-21T19:08:10.143431+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":781,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.171211+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:08:15.171211+0000","last_peered":"2024-09-21T19:08:15.171211+0000","last_clean":"2024-09-21T19:08:15.171211+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:08:15.171211+0000","last_undegraded":"2024-09-21T19:08:15.171211+0000","last_fullsized":"2024-09-21T19:08:15.171211+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":706,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167833+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:08:15.167833+0000","last_peered":"2024-09-21T19:08:15.167833+0000","last_clean":"2024-09-21T19:08:15.167833+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:08:15.167833+0000","last_undegraded":"2024-09-21T19:08:15.167833+0000","last_fullsized":"2024-09-21T19:08:15.167833+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":228,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144591+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:08:10.144591+0000","last_peered":"2024-09-21T19:08:10.144591+0000","last_clean":"2024-09-21T19:08:10.144591+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:08:10.144591+0000","last_undegraded":"2024-09-21T19:08:10.144591+0000","last_fullsized":"2024-09-21T19:08:10.144591+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"377'8900","reported_seq":9006,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:14.198650+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:08:14.198650+0000","last_peered":"2024-09-21T19:08:14.198650+0000","last_clean":"2024-09-21T19:08:14.198650+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:08:14.198650+0000","last_undegraded":"2024-09-21T19:08:14.198650+0000","last_fullsized":"2024-09-21T19:08:14.198650+0000","mapping_epoch":330,"log_start":"342'3600","ondisk_log_start":"342'3600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363331584,"num_objects":5544,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8900,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":743,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.159101+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:08:15.159101+0000","last_peered":"2024-09-21T19:08:15.159101+0000","last_clean":"2024-09-21T19:08:15.159101+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:08:15.159101+0000","last_undegraded":"2024-09-21T19:08:15.159101+0000","last_fullsized":"2024-09-21T19:08:15.159101+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":711,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.177013+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:08:15.177013+0000","last_peered":"2024-09-21T19:08:15.177013+0000","last_clean":"2024-09-21T19:08:15.177013+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:08:15.177013+0000","last_undegraded":"2024-09-21T19:08:15.177013+0000","last_fullsized":"2024-09-21T19:08:15.177013+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":696,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156665+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:08:15.156665+0000","last_peered":"2024-09-21T19:08:15.156665+0000","last_clean":"2024-09-21T19:08:15.156665+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:08:15.156665+0000","last_undegraded":"2024-09-21T19:08:15.156665+0000","last_fullsized":"2024-09-21T19:08:15.156665+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":672,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144072+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:08:10.144072+0000","last_peered":"2024-09-21T19:08:10.144072+0000","last_clean":"2024-09-21T19:08:10.144072+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:08:10.144072+0000","last_undegraded":"2024-09-21T19:08:10.144072+0000","last_fullsized":"2024-09-21T19:08:10.144072+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"379'9250","reported_seq":14499,"reported_epoch":379,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:17.605969+0000","last_change":"2024-09-21T19:06:55.916390+0000","last_active":"2024-09-21T19:08:17.605969+0000","last_peered":"2024-09-21T19:08:17.605969+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.879505+0000","last_became_peered":"2024-09-21T19:06:55.879505+0000","last_unstale":"2024-09-21T19:08:17.605969+0000","last_undegraded":"2024-09-21T19:06:55.764320+0000","last_fullsized":"2024-09-21T19:06:55.681165+0000","mapping_epoch":349,"log_start":"338'2800","ondisk_log_start":"338'2800","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356777984,"num_objects":5444,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2570,"num_bytes_recovered":168427520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3352}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":144,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.199600+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:08:17.199600+0000","last_peered":"2024-09-21T19:08:17.199600+0000","last_clean":"2024-09-21T19:08:17.199600+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:08:17.199600+0000","last_undegraded":"2024-09-21T19:08:17.199600+0000","last_fullsized":"2024-09-21T19:08:17.199600+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":792,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156945+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:08:15.156945+0000","last_peered":"2024-09-21T19:08:15.156945+0000","last_clean":"2024-09-21T19:08:15.156945+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:08:15.156945+0000","last_undegraded":"2024-09-21T19:08:15.156945+0000","last_fullsized":"2024-09-21T19:08:15.156945+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":608,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.138676+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:08:13.138676+0000","last_peered":"2024-09-21T19:08:13.138676+0000","last_clean":"2024-09-21T19:08:13.138676+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:08:13.138676+0000","last_undegraded":"2024-09-21T19:08:13.138676+0000","last_fullsized":"2024-09-21T19:08:13.138676+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":133,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.133122+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:08:13.133122+0000","last_peered":"2024-09-21T19:08:13.133122+0000","last_clean":"2024-09-21T19:08:13.133122+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:08:13.133122+0000","last_undegraded":"2024-09-21T19:08:13.133122+0000","last_fullsized":"2024-09-21T19:08:13.133122+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":137,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.199749+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:08:17.199749+0000","last_peered":"2024-09-21T19:08:17.199749+0000","last_clean":"2024-09-21T19:08:17.199749+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:08:17.199749+0000","last_undegraded":"2024-09-21T19:08:17.199749+0000","last_fullsized":"2024-09-21T19:08:17.199749+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"379'9287","reported_seq":9394,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.582028+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:08:17.582028+0000","last_peered":"2024-09-21T19:08:17.582028+0000","last_clean":"2024-09-21T19:08:17.582028+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:08:17.582028+0000","last_undegraded":"2024-09-21T19:08:17.582028+0000","last_fullsized":"2024-09-21T19:08:17.582028+0000","mapping_epoch":330,"log_start":"343'4100","ondisk_log_start":"343'4100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:45.180663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357105664,"num_objects":5449,"num_object_clones":0,"num_object_copies":16347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":789,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167171+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:08:15.167171+0000","last_peered":"2024-09-21T19:08:15.167171+0000","last_clean":"2024-09-21T19:08:15.167171+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:08:15.167171+0000","last_undegraded":"2024-09-21T19:08:15.167171+0000","last_fullsized":"2024-09-21T19:08:15.167171+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":718,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156882+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:08:15.156882+0000","last_peered":"2024-09-21T19:08:15.156882+0000","last_clean":"2024-09-21T19:08:15.156882+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:08:15.156882+0000","last_undegraded":"2024-09-21T19:08:15.156882+0000","last_fullsized":"2024-09-21T19:08:15.156882+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":714,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143967+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:08:10.143967+0000","last_peered":"2024-09-21T19:08:10.143967+0000","last_clean":"2024-09-21T19:08:10.143967+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:08:10.143967+0000","last_undegraded":"2024-09-21T19:08:10.143967+0000","last_fullsized":"2024-09-21T19:08:10.143967+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":177,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.200860+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:08:17.200860+0000","last_peered":"2024-09-21T19:08:17.200860+0000","last_clean":"2024-09-21T19:08:17.200860+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:08:17.200860+0000","last_undegraded":"2024-09-21T19:08:17.200860+0000","last_fullsized":"2024-09-21T19:08:17.200860+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":691,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.158837+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:08:15.158837+0000","last_peered":"2024-09-21T19:08:15.158837+0000","last_clean":"2024-09-21T19:08:15.158837+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:08:15.158837+0000","last_undegraded":"2024-09-21T19:08:15.158837+0000","last_fullsized":"2024-09-21T19:08:15.158837+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"379'9147","reported_seq":9257,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.616939+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:08:17.616939+0000","last_peered":"2024-09-21T19:08:17.616939+0000","last_clean":"2024-09-21T19:08:17.616939+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:08:17.616939+0000","last_undegraded":"2024-09-21T19:08:17.616939+0000","last_fullsized":"2024-09-21T19:08:17.616939+0000","mapping_epoch":330,"log_start":"338'2700","ondisk_log_start":"338'2700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347275264,"num_objects":5299,"num_object_clones":0,"num_object_copies":15897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":582,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.195006+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:08:17.195006+0000","last_peered":"2024-09-21T19:08:17.195006+0000","last_clean":"2024-09-21T19:08:17.195006+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:08:17.195006+0000","last_undegraded":"2024-09-21T19:08:17.195006+0000","last_fullsized":"2024-09-21T19:08:17.195006+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":676,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144637+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:08:10.144637+0000","last_peered":"2024-09-21T19:08:10.144637+0000","last_clean":"2024-09-21T19:08:10.144637+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:08:10.144637+0000","last_undegraded":"2024-09-21T19:08:10.144637+0000","last_fullsized":"2024-09-21T19:08:10.144637+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":129,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.199033+0000","last_change":"2024-09-21T19:05:42.909550+0000","last_active":"2024-09-21T19:08:17.199033+0000","last_peered":"2024-09-21T19:08:17.199033+0000","last_clean":"2024-09-21T19:08:17.199033+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:08:17.199033+0000","last_undegraded":"2024-09-21T19:08:17.199033+0000","last_fullsized":"2024-09-21T19:08:17.199033+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":577,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194559+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:08:17.194559+0000","last_peered":"2024-09-21T19:08:17.194559+0000","last_clean":"2024-09-21T19:08:17.194559+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:08:17.194559+0000","last_undegraded":"2024-09-21T19:08:17.194559+0000","last_fullsized":"2024-09-21T19:08:17.194559+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"379'9113","reported_seq":9223,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.536406+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:08:17.536406+0000","last_peered":"2024-09-21T19:08:17.536406+0000","last_clean":"2024-09-21T19:08:17.536406+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:08:17.536406+0000","last_undegraded":"2024-09-21T19:08:17.536406+0000","last_fullsized":"2024-09-21T19:08:17.536406+0000","mapping_epoch":330,"log_start":"343'4100","ondisk_log_start":"343'4100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355008512,"num_objects":5417,"num_object_clones":0,"num_object_copies":16251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9113,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":627,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.138507+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:08:13.138507+0000","last_peered":"2024-09-21T19:08:13.138507+0000","last_clean":"2024-09-21T19:08:13.138507+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:08:13.138507+0000","last_undegraded":"2024-09-21T19:08:13.138507+0000","last_fullsized":"2024-09-21T19:08:13.138507+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":710,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167999+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:08:15.167999+0000","last_peered":"2024-09-21T19:08:15.167999+0000","last_clean":"2024-09-21T19:08:15.167999+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:08:15.167999+0000","last_undegraded":"2024-09-21T19:08:15.167999+0000","last_fullsized":"2024-09-21T19:08:15.167999+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":140,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.205432+0000","last_change":"2024-09-21T19:06:54.697181+0000","last_active":"2024-09-21T19:08:17.205432+0000","last_peered":"2024-09-21T19:08:17.205432+0000","last_clean":"2024-09-21T19:08:17.205432+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:08:17.205432+0000","last_undegraded":"2024-09-21T19:08:17.205432+0000","last_fullsized":"2024-09-21T19:08:17.205432+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":682,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.171118+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:08:15.171118+0000","last_peered":"2024-09-21T19:08:15.171118+0000","last_clean":"2024-09-21T19:08:15.171118+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:08:15.171118+0000","last_undegraded":"2024-09-21T19:08:15.171118+0000","last_fullsized":"2024-09-21T19:08:15.171118+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"377'9053","reported_seq":9166,"reported_epoch":377,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:14.158443+0000","last_change":"2024-09-21T19:06:55.921138+0000","last_active":"2024-09-21T19:08:14.158443+0000","last_peered":"2024-09-21T19:08:14.158443+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:08:14.158443+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"342'3700","ondisk_log_start":"342'3700","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371916800,"num_objects":5675,"num_object_clones":0,"num_object_copies":17025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":1086}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":747,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.176889+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:08:15.176889+0000","last_peered":"2024-09-21T19:08:15.176889+0000","last_clean":"2024-09-21T19:08:15.176889+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:08:15.176889+0000","last_undegraded":"2024-09-21T19:08:15.176889+0000","last_fullsized":"2024-09-21T19:08:15.176889+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":704,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.168414+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:08:15.168414+0000","last_peered":"2024-09-21T19:08:15.168414+0000","last_clean":"2024-09-21T19:08:15.168414+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:08:15.168414+0000","last_undegraded":"2024-09-21T19:08:15.168414+0000","last_fullsized":"2024-09-21T19:08:15.168414+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":129,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.197731+0000","last_change":"2024-09-21T19:05:42.902451+0000","last_active":"2024-09-21T19:08:17.197731+0000","last_peered":"2024-09-21T19:08:17.197731+0000","last_clean":"2024-09-21T19:08:17.197731+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:08:17.197731+0000","last_undegraded":"2024-09-21T19:08:17.197731+0000","last_fullsized":"2024-09-21T19:08:17.197731+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":666,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143582+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:08:10.143582+0000","last_peered":"2024-09-21T19:08:10.143582+0000","last_clean":"2024-09-21T19:08:10.143582+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:08:10.143582+0000","last_undegraded":"2024-09-21T19:08:10.143582+0000","last_fullsized":"2024-09-21T19:08:10.143582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"379'9391","reported_seq":9499,"reported_epoch":379,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:17.531195+0000","last_change":"2024-09-21T19:06:55.875703+0000","last_active":"2024-09-21T19:08:17.531195+0000","last_peered":"2024-09-21T19:08:17.531195+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:08:17.531195+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"344'4300","ondisk_log_start":"344'4300","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361693184,"num_objects":5519,"num_object_clones":0,"num_object_copies":16557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":770}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":303,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.169330+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:08:15.169330+0000","last_peered":"2024-09-21T19:08:15.169330+0000","last_clean":"2024-09-21T19:08:15.169330+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:08:15.169330+0000","last_undegraded":"2024-09-21T19:08:15.169330+0000","last_fullsized":"2024-09-21T19:08:15.169330+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":718,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.168737+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:08:15.168737+0000","last_peered":"2024-09-21T19:08:15.168737+0000","last_clean":"2024-09-21T19:08:15.168737+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:08:15.168737+0000","last_undegraded":"2024-09-21T19:08:15.168737+0000","last_fullsized":"2024-09-21T19:08:15.168737+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":167,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.139023+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:08:10.139023+0000","last_peered":"2024-09-21T19:08:10.139023+0000","last_clean":"2024-09-21T19:08:10.139023+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:08:10.139023+0000","last_undegraded":"2024-09-21T19:08:10.139023+0000","last_fullsized":"2024-09-21T19:08:10.139023+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":704,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135518+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:08:13.135518+0000","last_peered":"2024-09-21T19:08:13.135518+0000","last_clean":"2024-09-21T19:08:13.135518+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:08:13.135518+0000","last_undegraded":"2024-09-21T19:08:13.135518+0000","last_fullsized":"2024-09-21T19:08:13.135518+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":639,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135961+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:08:13.135961+0000","last_peered":"2024-09-21T19:08:13.135961+0000","last_clean":"2024-09-21T19:08:13.135961+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:08:13.135961+0000","last_undegraded":"2024-09-21T19:08:13.135961+0000","last_fullsized":"2024-09-21T19:08:13.135961+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":700,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144849+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:08:10.144849+0000","last_peered":"2024-09-21T19:08:10.144849+0000","last_clean":"2024-09-21T19:08:10.144849+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:08:10.144849+0000","last_undegraded":"2024-09-21T19:08:10.144849+0000","last_fullsized":"2024-09-21T19:08:10.144849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":236,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143691+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:08:10.143691+0000","last_peered":"2024-09-21T19:08:10.143691+0000","last_clean":"2024-09-21T19:08:10.143691+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:08:10.143691+0000","last_undegraded":"2024-09-21T19:08:10.143691+0000","last_fullsized":"2024-09-21T19:08:10.143691+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":671,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167555+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:08:15.167555+0000","last_peered":"2024-09-21T19:08:15.167555+0000","last_clean":"2024-09-21T19:08:15.167555+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:08:15.167555+0000","last_undegraded":"2024-09-21T19:08:15.167555+0000","last_fullsized":"2024-09-21T19:08:15.167555+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":582,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.202926+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:08:17.202926+0000","last_peered":"2024-09-21T19:08:17.202926+0000","last_clean":"2024-09-21T19:08:17.202926+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:08:17.202926+0000","last_undegraded":"2024-09-21T19:08:17.202926+0000","last_fullsized":"2024-09-21T19:08:17.202926+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":691,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167376+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:08:15.167376+0000","last_peered":"2024-09-21T19:08:15.167376+0000","last_clean":"2024-09-21T19:08:15.167376+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:08:15.167376+0000","last_undegraded":"2024-09-21T19:08:15.167376+0000","last_fullsized":"2024-09-21T19:08:15.167376+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":670,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144760+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:08:10.144760+0000","last_peered":"2024-09-21T19:08:10.144760+0000","last_clean":"2024-09-21T19:08:10.144760+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:08:10.144760+0000","last_undegraded":"2024-09-21T19:08:10.144760+0000","last_fullsized":"2024-09-21T19:08:10.144760+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":129,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.199489+0000","last_change":"2024-09-21T19:05:42.905153+0000","last_active":"2024-09-21T19:08:17.199489+0000","last_peered":"2024-09-21T19:08:17.199489+0000","last_clean":"2024-09-21T19:08:17.199489+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:08:17.199489+0000","last_undegraded":"2024-09-21T19:08:17.199489+0000","last_fullsized":"2024-09-21T19:08:17.199489+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":293,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167782+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:08:15.167782+0000","last_peered":"2024-09-21T19:08:15.167782+0000","last_clean":"2024-09-21T19:08:15.167782+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:08:15.167782+0000","last_undegraded":"2024-09-21T19:08:15.167782+0000","last_fullsized":"2024-09-21T19:08:15.167782+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":579,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.197706+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:08:17.197706+0000","last_peered":"2024-09-21T19:08:17.197706+0000","last_clean":"2024-09-21T19:08:17.197706+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:08:17.197706+0000","last_undegraded":"2024-09-21T19:08:17.197706+0000","last_fullsized":"2024-09-21T19:08:17.197706+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":883,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144207+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:08:10.144207+0000","last_peered":"2024-09-21T19:08:10.144207+0000","last_clean":"2024-09-21T19:08:10.144207+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:08:10.144207+0000","last_undegraded":"2024-09-21T19:08:10.144207+0000","last_fullsized":"2024-09-21T19:08:10.144207+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":694,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156444+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:08:15.156444+0000","last_peered":"2024-09-21T19:08:15.156444+0000","last_clean":"2024-09-21T19:08:15.156444+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:08:15.156444+0000","last_undegraded":"2024-09-21T19:08:15.156444+0000","last_fullsized":"2024-09-21T19:08:15.156444+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":696,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.144425+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:08:10.144425+0000","last_peered":"2024-09-21T19:08:10.144425+0000","last_clean":"2024-09-21T19:08:10.144425+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:08:10.144425+0000","last_undegraded":"2024-09-21T19:08:10.144425+0000","last_fullsized":"2024-09-21T19:08:10.144425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":694,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.169023+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:08:15.169023+0000","last_peered":"2024-09-21T19:08:15.169023+0000","last_clean":"2024-09-21T19:08:15.169023+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:08:15.169023+0000","last_undegraded":"2024-09-21T19:08:15.169023+0000","last_fullsized":"2024-09-21T19:08:15.169023+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":339,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.181537+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:08:17.181537+0000","last_peered":"2024-09-21T19:08:17.181537+0000","last_clean":"2024-09-21T19:08:17.181537+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:08:17.181537+0000","last_undegraded":"2024-09-21T19:08:17.181537+0000","last_fullsized":"2024-09-21T19:08:17.181537+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":571,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.203600+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:08:17.203600+0000","last_peered":"2024-09-21T19:08:17.203600+0000","last_clean":"2024-09-21T19:08:17.203600+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:08:17.203600+0000","last_undegraded":"2024-09-21T19:08:17.203600+0000","last_fullsized":"2024-09-21T19:08:17.203600+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":773,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.156257+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:08:15.156257+0000","last_peered":"2024-09-21T19:08:15.156257+0000","last_clean":"2024-09-21T19:08:15.156257+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:08:15.156257+0000","last_undegraded":"2024-09-21T19:08:15.156257+0000","last_fullsized":"2024-09-21T19:08:15.156257+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":692,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.157077+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:08:15.157077+0000","last_peered":"2024-09-21T19:08:15.157077+0000","last_clean":"2024-09-21T19:08:15.157077+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:08:15.157077+0000","last_undegraded":"2024-09-21T19:08:15.157077+0000","last_fullsized":"2024-09-21T19:08:15.157077+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":694,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.183357+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:08:17.183357+0000","last_peered":"2024-09-21T19:08:17.183357+0000","last_clean":"2024-09-21T19:08:17.183357+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:08:17.183357+0000","last_undegraded":"2024-09-21T19:08:17.183357+0000","last_fullsized":"2024-09-21T19:08:17.183357+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":691,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.139088+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:08:10.139088+0000","last_peered":"2024-09-21T19:08:10.139088+0000","last_clean":"2024-09-21T19:08:10.139088+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:08:10.139088+0000","last_undegraded":"2024-09-21T19:08:10.139088+0000","last_fullsized":"2024-09-21T19:08:10.139088+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":773,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.183678+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:08:17.183678+0000","last_peered":"2024-09-21T19:08:17.183678+0000","last_clean":"2024-09-21T19:08:17.183678+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:08:17.183678+0000","last_undegraded":"2024-09-21T19:08:17.183678+0000","last_fullsized":"2024-09-21T19:08:17.183678+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:44.281618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":688,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167872+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:08:15.167872+0000","last_peered":"2024-09-21T19:08:15.167872+0000","last_clean":"2024-09-21T19:08:15.167872+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:08:15.167872+0000","last_undegraded":"2024-09-21T19:08:15.167872+0000","last_fullsized":"2024-09-21T19:08:15.167872+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":692,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143716+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:08:10.143716+0000","last_peered":"2024-09-21T19:08:10.143716+0000","last_clean":"2024-09-21T19:08:10.143716+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:08:10.143716+0000","last_undegraded":"2024-09-21T19:08:10.143716+0000","last_fullsized":"2024-09-21T19:08:10.143716+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":590,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.195550+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:08:17.195550+0000","last_peered":"2024-09-21T19:08:17.195550+0000","last_clean":"2024-09-21T19:08:17.195550+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:08:17.195550+0000","last_undegraded":"2024-09-21T19:08:17.195550+0000","last_fullsized":"2024-09-21T19:08:17.195550+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":716,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.139298+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:08:10.139298+0000","last_peered":"2024-09-21T19:08:10.139298+0000","last_clean":"2024-09-21T19:08:10.139298+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:08:10.139298+0000","last_undegraded":"2024-09-21T19:08:10.139298+0000","last_fullsized":"2024-09-21T19:08:10.139298+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":629,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.155439+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:08:10.155439+0000","last_peered":"2024-09-21T19:08:10.155439+0000","last_clean":"2024-09-21T19:08:10.155439+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:08:10.155439+0000","last_undegraded":"2024-09-21T19:08:10.155439+0000","last_fullsized":"2024-09-21T19:08:10.155439+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":707,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.165892+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:08:15.165892+0000","last_peered":"2024-09-21T19:08:15.165892+0000","last_clean":"2024-09-21T19:08:15.165892+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:08:15.165892+0000","last_undegraded":"2024-09-21T19:08:15.165892+0000","last_fullsized":"2024-09-21T19:08:15.165892+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":691,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.143856+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:08:10.143856+0000","last_peered":"2024-09-21T19:08:10.143856+0000","last_clean":"2024-09-21T19:08:10.143856+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:08:10.143856+0000","last_undegraded":"2024-09-21T19:08:10.143856+0000","last_fullsized":"2024-09-21T19:08:10.143856+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":715,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.110862+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:08:10.110862+0000","last_peered":"2024-09-21T19:08:10.110862+0000","last_clean":"2024-09-21T19:08:10.110862+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:08:10.110862+0000","last_undegraded":"2024-09-21T19:08:10.110862+0000","last_fullsized":"2024-09-21T19:08:10.110862+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":574,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.194627+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:08:17.194627+0000","last_peered":"2024-09-21T19:08:17.194627+0000","last_clean":"2024-09-21T19:08:17.194627+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:08:17.194627+0000","last_undegraded":"2024-09-21T19:08:17.194627+0000","last_fullsized":"2024-09-21T19:08:17.194627+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":606,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.133292+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:08:13.133292+0000","last_peered":"2024-09-21T19:08:13.133292+0000","last_clean":"2024-09-21T19:08:13.133292+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:08:13.133292+0000","last_undegraded":"2024-09-21T19:08:13.133292+0000","last_fullsized":"2024-09-21T19:08:13.133292+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":682,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T19:08:15.167040+0000","last_change":"2024-09-21T19:05:42.904326+0000","last_active":"2024-09-21T19:08:15.167040+0000","last_peered":"2024-09-21T19:08:15.167040+0000","last_clean":"2024-09-21T19:08:15.167040+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:08:15.167040+0000","last_undegraded":"2024-09-21T19:08:15.167040+0000","last_fullsized":"2024-09-21T19:08:15.167040+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":690,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.182235+0000","last_change":"2024-09-21T19:07:21.428355+0000","last_active":"2024-09-21T19:08:17.182235+0000","last_peered":"2024-09-21T19:08:17.182235+0000","last_clean":"2024-09-21T19:08:17.182235+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:08:17.182235+0000","last_undegraded":"2024-09-21T19:08:17.182235+0000","last_fullsized":"2024-09-21T19:08:17.182235+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:21.428212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:21.428212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.151249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":708,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T19:08:10.139095+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:08:10.139095+0000","last_peered":"2024-09-21T19:08:10.139095+0000","last_clean":"2024-09-21T19:08:10.139095+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:08:10.139095+0000","last_undegraded":"2024-09-21T19:08:10.139095+0000","last_fullsized":"2024-09-21T19:08:10.139095+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":671,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-21T19:08:13.135717+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:08:13.135717+0000","last_peered":"2024-09-21T19:08:13.135717+0000","last_clean":"2024-09-21T19:08:13.135717+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:08:13.135717+0000","last_undegraded":"2024-09-21T19:08:13.135717+0000","last_fullsized":"2024-09-21T19:08:13.135717+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":594,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T19:08:17.203434+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:08:17.203434+0000","last_peered":"2024-09-21T19:08:17.203434+0000","last_clean":"2024-09-21T19:08:17.203434+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:08:17.203434+0000","last_undegraded":"2024-09-21T19:08:17.203434+0000","last_fullsized":"2024-09-21T19:08:17.203434+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5797642264,"num_objects":88466,"num_object_clones":0,"num_object_copies":265398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88466,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146476,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9378,"num_bytes_recovered":614596608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16700223488,"data_stored":16700211272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85776,"ondisk_log_size":85776,"up":48,"acting":43,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":898,"num_read_kb":893,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266874,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2033704,"kb_used_data":1434712,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91711448,"statfs":{"total":95995035648,"available":93912522752,"internally_reserved":0,"allocated":1469145088,"data_stored":1468505308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464324,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2775000,"kb_used_data":2536072,"kb_used_omap":4,"kb_used_meta":238843,"kb_avail":90970152,"statfs":{"total":95995035648,"available":93153435648,"internally_reserved":0,"allocated":2596937728,"data_stored":2596306760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":244575908},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619025,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1297032,"kb_used_data":1125688,"kb_used_omap":4,"kb_used_meta":171323,"kb_avail":92448120,"statfs":{"total":95995035648,"available":94666874880,"internally_reserved":0,"allocated":1152704512,"data_stored":1152083233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":175435427},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019080,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3306284,"kb_used_data":2823900,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":90438868,"statfs":{"total":95995035648,"available":92609400832,"internally_reserved":0,"allocated":2891673600,"data_stored":2891030028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182604,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2544616,"kb_used_data":1823384,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91200536,"statfs":{"total":95995035648,"available":93389348864,"internally_reserved":0,"allocated":1867145216,"data_stored":1866492487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411539,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2633108,"kb_used_data":2096452,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91112044,"statfs":{"total":95995035648,"available":93298733056,"internally_reserved":0,"allocated":2146766848,"data_stored":2146147578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1703132,"kb_used_data":1044424,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":92042020,"statfs":{"total":95995035648,"available":94251028480,"internally_reserved":0,"allocated":1069490176,"data_stored":1068809264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4281328,"kb_used_data":3519840,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":89463824,"statfs":{"total":95995035648,"available":91610955776,"internally_reserved":0,"allocated":3604316160,"data_stored":3603668901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3581349888,"data_stored":3581345816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1062338560,"data_stored":1062338560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2126053376,"data_stored":2126053376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1861615616,"data_stored":1861615616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2882473984,"data_stored":2882469912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1151664128,"data_stored":1151664128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2582253568,"data_stored":2582249496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1452474368,"data_stored":1452474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:08:19.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:08:19.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:19.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:19.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:19 smithi104 ceph-mon[32438]: pgmap v750: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 13941/265995 objects degraded (5.241%); 13 MiB/s, 210 objects/s recovering 2024-09-21T19:08:19.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:19 smithi104 ceph-mon[32438]: from='client.36260 v1:172.21.15.81:0/1539850137' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:19.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:19.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:19.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:19.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:08:19.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:19.878 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:19.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:19.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:20.002 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:20.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:20.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:20.146 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:20.227 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:20.259 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:20.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:20.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:20.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:20 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:20.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:20 smithi104 ceph-mon[32438]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T19:08:20.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:20 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 13941/265995 objects degraded (5.241%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:20.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:20 smithi104 ceph-mon[32438]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T19:08:20.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:20.530 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[29384]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 13941/265995 objects degraded (5.241%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[29384]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[39192]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 13941/265995 objects degraded (5.241%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:20.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:20 smithi081 ceph-mon[39192]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T19:08:20.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:20.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:08:20.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.660 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:20.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:20.718 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:08:20.718 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:08:20.718 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:08:20.718 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:08:20.718 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:08:20.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:20.902 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:20.924 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:20.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:21.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:21.063 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:21.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.205 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:21.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:21.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:21.333 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:21.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:21.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:21.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:21 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T19:08:21.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:21 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T19:08:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:21 smithi104 ceph-mon[32438]: pgmap v753: 158 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s; 13468/260703 objects degraded (5.166%); 12 MiB/s, 191 objects/s recovering 2024-09-21T19:08:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:21 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T19:08:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:21 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T19:08:21.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:21 smithi081 ceph-mon[29384]: pgmap v753: 158 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s; 13468/260703 objects degraded (5.166%); 12 MiB/s, 191 objects/s recovering 2024-09-21T19:08:21.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:21 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T19:08:21.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:21 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T19:08:21.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:21 smithi081 ceph-mon[39192]: pgmap v753: 158 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s; 13468/260703 objects degraded (5.166%); 12 MiB/s, 191 objects/s recovering 2024-09-21T19:08:21.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:21.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:21.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:21.730 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:21.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:21.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:21.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:22.010 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:22.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:22.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:22.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.290 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:22.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:22.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:22.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:22 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T19:08:22.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:22 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T19:08:22.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1391725100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:22.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:22.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:22.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:22.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1391725100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1391725100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:22.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:22.597 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:08:22.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:22.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:22.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:22.808 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:22.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:22.960 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:22.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:22.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:08:22.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:08:22.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:08:22.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:08:22.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:08:22.976 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:08:22.977 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:08:22.977 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:08:22.977 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:08:22.977 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:08:22.992 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:23.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:23.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:23.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:23.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:23.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:23.441 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:23 smithi104 ceph-mon[32438]: pgmap v754: 158 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 857 op/s; 13468/259173 objects degraded (5.197%); 9.2 MiB/s, 146 objects/s recovering 2024-09-21T19:08:23.441 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:23 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:08:23.441 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:23 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:08:23.441 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:23 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:08:23.441 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:23 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:08:23.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[39192]: pgmap v754: 158 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 857 op/s; 13468/259173 objects degraded (5.197%); 9.2 MiB/s, 146 objects/s recovering 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[29384]: pgmap v754: 158 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 857 op/s; 13468/259173 objects degraded (5.197%); 9.2 MiB/s, 146 objects/s recovering 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:08:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:23 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:08:23.567 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:23.567 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:23.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:23.637 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:23.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:23.760 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:23.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:23.893 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:23.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:23.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:24.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:24.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:08:24.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:24.395 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:24.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:24.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.587 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:24.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:24.658 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:24.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:24.723 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:24.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:24.856 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:24.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:24.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:24.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:25.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:25.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:25.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:25.254 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:25.254 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.287 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:25.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:25.384 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:25.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:25.445 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:25 smithi104 ceph-mon[32438]: pgmap v755: 158 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 962 op/s; 12793/253023 objects degraded (5.056%); 15 MiB/s, 237 objects/s recovering 2024-09-21T19:08:25.445 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:25 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 12793/253023 objects degraded (5.056%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:25.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:25.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:25 smithi081 ceph-mon[29384]: pgmap v755: 158 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 962 op/s; 12793/253023 objects degraded (5.056%); 15 MiB/s, 237 objects/s recovering 2024-09-21T19:08:25.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:25 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 12793/253023 objects degraded (5.056%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:25.537 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:25 smithi081 ceph-mon[39192]: pgmap v755: 158 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 780 B/s rd, 0 B/s wr, 962 op/s; 12793/253023 objects degraded (5.056%); 15 MiB/s, 237 objects/s recovering 2024-09-21T19:08:25.537 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:25 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 12793/253023 objects degraded (5.056%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:25.566 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:25.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:25.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:25.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:25.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:25.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:25.980 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:25.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:26.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:26.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:08:26.122 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:26.225 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:26.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:26.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:26.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:26.524 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:26.556 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:26.645 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:08:26.648 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:08:26.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:26.692 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:26.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:26.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:26.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:26.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:26.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:27.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:27.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:27.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:27.236 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:08:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:08:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:08:27.252 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:08:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:27.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:08:27.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:27.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:27.446 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2024-09-21T19:08:25.742878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5418850867,"num_objects":82876,"num_object_clones":0,"num_object_copies":248628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82876,"num_whiteouts":0,"num_read":2043,"num_read_kb":1795,"num_write":152987,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10761,"num_bytes_recovered":690104568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87224,"ondisk_log_size":87224,"up":474,"acting":470,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19683484,"kb_used_data":15473544,"kb_used_omap":18,"kb_used_meta":4209517,"kb_avail":730277732,"statfs":{"total":767960285184,"available":747804397568,"internally_reserved":0,"allocated":15844909056,"data_stored":15839628497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4310546416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-379387904,"num_objects":-5789,"num_object_clones":0,"num_object_copies":-17367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5789,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5789,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1148,"num_bytes_recovered":75235328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002477"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":601,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.756091+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:08:19.756091+0000","last_peered":"2024-09-21T19:08:19.756091+0000","last_clean":"2024-09-21T19:08:19.756091+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:08:19.756091+0000","last_undegraded":"2024-09-21T19:08:19.756091+0000","last_fullsized":"2024-09-21T19:08:19.756091+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":769,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705675+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:08:19.705675+0000","last_peered":"2024-09-21T19:08:19.705675+0000","last_clean":"2024-09-21T19:08:19.705675+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:08:19.705675+0000","last_undegraded":"2024-09-21T19:08:19.705675+0000","last_fullsized":"2024-09-21T19:08:19.705675+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":698,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707479+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:08:19.707479+0000","last_peered":"2024-09-21T19:08:19.707479+0000","last_clean":"2024-09-21T19:08:19.707479+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:08:19.707479+0000","last_undegraded":"2024-09-21T19:08:19.707479+0000","last_fullsized":"2024-09-21T19:08:19.707479+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":686,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705809+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:08:19.705809+0000","last_peered":"2024-09-21T19:08:19.705809+0000","last_clean":"2024-09-21T19:08:19.705809+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:08:19.705809+0000","last_undegraded":"2024-09-21T19:08:19.705809+0000","last_fullsized":"2024-09-21T19:08:19.705809+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":695,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705959+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:08:19.705959+0000","last_peered":"2024-09-21T19:08:19.705959+0000","last_clean":"2024-09-21T19:08:19.705959+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:08:19.705959+0000","last_undegraded":"2024-09-21T19:08:19.705959+0000","last_fullsized":"2024-09-21T19:08:19.705959+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752147+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:08:19.752147+0000","last_peered":"2024-09-21T19:08:19.752147+0000","last_clean":"2024-09-21T19:08:19.752147+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:08:19.752147+0000","last_undegraded":"2024-09-21T19:08:19.752147+0000","last_fullsized":"2024-09-21T19:08:19.752147+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":819,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752061+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:08:19.752061+0000","last_peered":"2024-09-21T19:08:19.752061+0000","last_clean":"2024-09-21T19:08:19.752061+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:08:19.752061+0000","last_undegraded":"2024-09-21T19:08:19.752061+0000","last_fullsized":"2024-09-21T19:08:19.752061+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":706,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705621+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:08:19.705621+0000","last_peered":"2024-09-21T19:08:19.705621+0000","last_clean":"2024-09-21T19:08:19.705621+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:08:19.705621+0000","last_undegraded":"2024-09-21T19:08:19.705621+0000","last_fullsized":"2024-09-21T19:08:19.705621+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":693,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706622+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:08:19.706622+0000","last_peered":"2024-09-21T19:08:19.706622+0000","last_clean":"2024-09-21T19:08:19.706622+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:08:19.706622+0000","last_undegraded":"2024-09-21T19:08:19.706622+0000","last_fullsized":"2024-09-21T19:08:19.706622+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":750,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705971+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:08:19.705971+0000","last_peered":"2024-09-21T19:08:19.705971+0000","last_clean":"2024-09-21T19:08:19.705971+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:08:19.705971+0000","last_undegraded":"2024-09-21T19:08:19.705971+0000","last_fullsized":"2024-09-21T19:08:19.705971+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":646,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.711819+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:08:19.711819+0000","last_peered":"2024-09-21T19:08:19.711819+0000","last_clean":"2024-09-21T19:08:19.711819+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:08:19.711819+0000","last_undegraded":"2024-09-21T19:08:19.711819+0000","last_fullsized":"2024-09-21T19:08:19.711819+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":567,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712829+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:08:19.712829+0000","last_peered":"2024-09-21T19:08:19.712829+0000","last_clean":"2024-09-21T19:08:19.712829+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:08:19.712829+0000","last_undegraded":"2024-09-21T19:08:19.712829+0000","last_fullsized":"2024-09-21T19:08:19.712829+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":246,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706485+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:08:19.706485+0000","last_peered":"2024-09-21T19:08:19.706485+0000","last_clean":"2024-09-21T19:08:19.706485+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:08:19.706485+0000","last_undegraded":"2024-09-21T19:08:19.706485+0000","last_fullsized":"2024-09-21T19:08:19.706485+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":696,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705974+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:08:19.705974+0000","last_peered":"2024-09-21T19:08:19.705974+0000","last_clean":"2024-09-21T19:08:19.705974+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:08:19.705974+0000","last_undegraded":"2024-09-21T19:08:19.705974+0000","last_fullsized":"2024-09-21T19:08:19.705974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":678,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781457+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:08:19.781457+0000","last_peered":"2024-09-21T19:08:19.781457+0000","last_clean":"2024-09-21T19:08:19.781457+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:08:19.781457+0000","last_undegraded":"2024-09-21T19:08:19.781457+0000","last_fullsized":"2024-09-21T19:08:19.781457+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":254,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:21.206201+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:08:21.206201+0000","last_peered":"2024-09-21T19:08:21.206201+0000","last_clean":"2024-09-21T19:08:21.206201+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:08:21.206201+0000","last_undegraded":"2024-09-21T19:08:21.206201+0000","last_fullsized":"2024-09-21T19:08:21.206201+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.570833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":714,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706560+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:08:19.706560+0000","last_peered":"2024-09-21T19:08:19.706560+0000","last_clean":"2024-09-21T19:08:19.706560+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:08:19.706560+0000","last_undegraded":"2024-09-21T19:08:19.706560+0000","last_fullsized":"2024-09-21T19:08:19.706560+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":583,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755377+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:08:19.755377+0000","last_peered":"2024-09-21T19:08:19.755377+0000","last_clean":"2024-09-21T19:08:19.755377+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:08:19.755377+0000","last_undegraded":"2024-09-21T19:08:19.755377+0000","last_fullsized":"2024-09-21T19:08:19.755377+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":157,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755687+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:08:19.755687+0000","last_peered":"2024-09-21T19:08:19.755687+0000","last_clean":"2024-09-21T19:08:19.755687+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:08:19.755687+0000","last_undegraded":"2024-09-21T19:08:19.755687+0000","last_fullsized":"2024-09-21T19:08:19.755687+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":701,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752105+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:08:19.752105+0000","last_peered":"2024-09-21T19:08:19.752105+0000","last_clean":"2024-09-21T19:08:19.752105+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:08:19.752105+0000","last_undegraded":"2024-09-21T19:08:19.752105+0000","last_fullsized":"2024-09-21T19:08:19.752105+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:02.338670+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":691,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.753336+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:08:19.753336+0000","last_peered":"2024-09-21T19:08:19.753336+0000","last_clean":"2024-09-21T19:08:19.753336+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:08:19.753336+0000","last_undegraded":"2024-09-21T19:08:19.753336+0000","last_fullsized":"2024-09-21T19:08:19.753336+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":717,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706061+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:08:19.706061+0000","last_peered":"2024-09-21T19:08:19.706061+0000","last_clean":"2024-09-21T19:08:19.706061+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:08:19.706061+0000","last_undegraded":"2024-09-21T19:08:19.706061+0000","last_fullsized":"2024-09-21T19:08:19.706061+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":762,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752860+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:08:19.752860+0000","last_peered":"2024-09-21T19:08:19.752860+0000","last_clean":"2024-09-21T19:08:19.752860+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:08:19.752860+0000","last_undegraded":"2024-09-21T19:08:19.752860+0000","last_fullsized":"2024-09-21T19:08:19.752860+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":701,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706284+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:08:19.706284+0000","last_peered":"2024-09-21T19:08:19.706284+0000","last_clean":"2024-09-21T19:08:19.706284+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:08:19.706284+0000","last_undegraded":"2024-09-21T19:08:19.706284+0000","last_fullsized":"2024-09-21T19:08:19.706284+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782268+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:08:19.782268+0000","last_peered":"2024-09-21T19:08:19.782268+0000","last_clean":"2024-09-21T19:08:19.782268+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:08:19.782268+0000","last_undegraded":"2024-09-21T19:08:19.782268+0000","last_fullsized":"2024-09-21T19:08:19.782268+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":715,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707503+0000","last_change":"2024-09-21T19:07:33.264047+0000","last_active":"2024-09-21T19:08:19.707503+0000","last_peered":"2024-09-21T19:08:19.707503+0000","last_clean":"2024-09-21T19:08:19.707503+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:08:19.707503+0000","last_undegraded":"2024-09-21T19:08:19.707503+0000","last_fullsized":"2024-09-21T19:08:19.707503+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:33.263975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:33.263975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.063323+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":611,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755545+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:08:19.755545+0000","last_peered":"2024-09-21T19:08:19.755545+0000","last_clean":"2024-09-21T19:08:19.755545+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:08:19.755545+0000","last_undegraded":"2024-09-21T19:08:19.755545+0000","last_fullsized":"2024-09-21T19:08:19.755545+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":769,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709882+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:08:19.709882+0000","last_peered":"2024-09-21T19:08:19.709882+0000","last_clean":"2024-09-21T19:08:19.709882+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:08:19.709882+0000","last_undegraded":"2024-09-21T19:08:19.709882+0000","last_fullsized":"2024-09-21T19:08:19.709882+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"381'9583","reported_seq":9697,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:21.694522+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:08:21.694522+0000","last_peered":"2024-09-21T19:08:21.694522+0000","last_clean":"2024-09-21T19:08:21.694522+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:08:21.694522+0000","last_undegraded":"2024-09-21T19:08:21.694522+0000","last_fullsized":"2024-09-21T19:08:21.694522+0000","mapping_epoch":330,"log_start":"352'5100","ondisk_log_start":"352'5100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":15891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":152,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755257+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:08:19.755257+0000","last_peered":"2024-09-21T19:08:19.755257+0000","last_clean":"2024-09-21T19:08:19.755257+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:08:19.755257+0000","last_undegraded":"2024-09-21T19:08:19.755257+0000","last_fullsized":"2024-09-21T19:08:19.755257+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":750,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752912+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:08:19.752912+0000","last_peered":"2024-09-21T19:08:19.752912+0000","last_clean":"2024-09-21T19:08:19.752912+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:08:19.752912+0000","last_undegraded":"2024-09-21T19:08:19.752912+0000","last_fullsized":"2024-09-21T19:08:19.752912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":689,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.751946+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:08:19.751946+0000","last_peered":"2024-09-21T19:08:19.751946+0000","last_clean":"2024-09-21T19:08:19.751946+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:08:19.751946+0000","last_undegraded":"2024-09-21T19:08:19.751946+0000","last_fullsized":"2024-09-21T19:08:19.751946+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":921,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706612+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:08:19.706612+0000","last_peered":"2024-09-21T19:08:19.706612+0000","last_clean":"2024-09-21T19:08:19.706612+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:08:19.706612+0000","last_undegraded":"2024-09-21T19:08:19.706612+0000","last_fullsized":"2024-09-21T19:08:19.706612+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":378,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752001+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:08:19.752001+0000","last_peered":"2024-09-21T19:08:19.752001+0000","last_clean":"2024-09-21T19:08:19.752001+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:08:19.752001+0000","last_undegraded":"2024-09-21T19:08:19.752001+0000","last_fullsized":"2024-09-21T19:08:19.752001+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"381'9316","reported_seq":9428,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:21.689918+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:08:21.689918+0000","last_peered":"2024-09-21T19:08:21.689918+0000","last_clean":"2024-09-21T19:08:21.689918+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:08:21.689918+0000","last_undegraded":"2024-09-21T19:08:21.689918+0000","last_fullsized":"2024-09-21T19:08:21.689918+0000","mapping_epoch":330,"log_start":"351'4800","ondisk_log_start":"351'4800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4516,"log_dups_size":0,"ondisk_log_size":4516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335675392,"num_objects":5122,"num_object_clones":0,"num_object_copies":15366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9316,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":477,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706543+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:08:19.706543+0000","last_peered":"2024-09-21T19:08:19.706543+0000","last_clean":"2024-09-21T19:08:19.706543+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:08:19.706543+0000","last_undegraded":"2024-09-21T19:08:19.706543+0000","last_fullsized":"2024-09-21T19:08:19.706543+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":785,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706070+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:08:19.706070+0000","last_peered":"2024-09-21T19:08:19.706070+0000","last_clean":"2024-09-21T19:08:19.706070+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:08:19.706070+0000","last_undegraded":"2024-09-21T19:08:19.706070+0000","last_fullsized":"2024-09-21T19:08:19.706070+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.753639+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:08:19.753639+0000","last_peered":"2024-09-21T19:08:19.753639+0000","last_clean":"2024-09-21T19:08:19.753639+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:08:19.753639+0000","last_undegraded":"2024-09-21T19:08:19.753639+0000","last_fullsized":"2024-09-21T19:08:19.753639+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":688,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705739+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:08:19.705739+0000","last_peered":"2024-09-21T19:08:19.705739+0000","last_clean":"2024-09-21T19:08:19.705739+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:08:19.705739+0000","last_undegraded":"2024-09-21T19:08:19.705739+0000","last_fullsized":"2024-09-21T19:08:19.705739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":705,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705940+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:08:19.705940+0000","last_peered":"2024-09-21T19:08:19.705940+0000","last_clean":"2024-09-21T19:08:19.705940+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:08:19.705940+0000","last_undegraded":"2024-09-21T19:08:19.705940+0000","last_fullsized":"2024-09-21T19:08:19.705940+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"381'9606","reported_seq":9720,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:24.192248+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:08:24.192248+0000","last_peered":"2024-09-21T19:08:24.192248+0000","last_clean":"2024-09-21T19:08:24.192248+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:08:24.192248+0000","last_undegraded":"2024-09-21T19:08:24.192248+0000","last_fullsized":"2024-09-21T19:08:24.192248+0000","mapping_epoch":330,"log_start":"345'4300","ondisk_log_start":"345'4300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333840384,"num_objects":5094,"num_object_clones":0,"num_object_copies":15282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":468,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752354+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:08:19.752354+0000","last_peered":"2024-09-21T19:08:19.752354+0000","last_clean":"2024-09-21T19:08:19.752354+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:08:19.752354+0000","last_undegraded":"2024-09-21T19:08:19.752354+0000","last_fullsized":"2024-09-21T19:08:19.752354+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":756,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705839+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:08:19.705839+0000","last_peered":"2024-09-21T19:08:19.705839+0000","last_clean":"2024-09-21T19:08:19.705839+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:08:19.705839+0000","last_undegraded":"2024-09-21T19:08:19.705839+0000","last_fullsized":"2024-09-21T19:08:19.705839+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":579,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755831+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:08:19.755831+0000","last_peered":"2024-09-21T19:08:19.755831+0000","last_clean":"2024-09-21T19:08:19.755831+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:08:19.755831+0000","last_undegraded":"2024-09-21T19:08:19.755831+0000","last_fullsized":"2024-09-21T19:08:19.755831+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":865,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705916+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:08:19.705916+0000","last_peered":"2024-09-21T19:08:19.705916+0000","last_clean":"2024-09-21T19:08:19.705916+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:08:19.705916+0000","last_undegraded":"2024-09-21T19:08:19.705916+0000","last_fullsized":"2024-09-21T19:08:19.705916+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":708,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781323+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:08:19.781323+0000","last_peered":"2024-09-21T19:08:19.781323+0000","last_clean":"2024-09-21T19:08:19.781323+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:08:19.781323+0000","last_undegraded":"2024-09-21T19:08:19.781323+0000","last_fullsized":"2024-09-21T19:08:19.781323+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"381'9630","reported_seq":9743,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:21.842706+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:08:21.842706+0000","last_peered":"2024-09-21T19:08:21.842706+0000","last_clean":"2024-09-21T19:08:21.842706+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:08:21.842706+0000","last_undegraded":"2024-09-21T19:08:21.842706+0000","last_fullsized":"2024-09-21T19:08:21.842706+0000","mapping_epoch":330,"log_start":"346'4600","ondisk_log_start":"346'4600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347865088,"num_objects":5308,"num_object_clones":0,"num_object_copies":15924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9630,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":492,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.710020+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:08:19.710020+0000","last_peered":"2024-09-21T19:08:19.710020+0000","last_clean":"2024-09-21T19:08:19.710020+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:08:19.710020+0000","last_undegraded":"2024-09-21T19:08:19.710020+0000","last_fullsized":"2024-09-21T19:08:19.710020+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":733,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752845+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:08:19.752845+0000","last_peered":"2024-09-21T19:08:19.752845+0000","last_clean":"2024-09-21T19:08:19.752845+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:08:19.752845+0000","last_undegraded":"2024-09-21T19:08:19.752845+0000","last_fullsized":"2024-09-21T19:08:19.752845+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":252,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752947+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:08:19.752947+0000","last_peered":"2024-09-21T19:08:19.752947+0000","last_clean":"2024-09-21T19:08:19.752947+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:08:19.752947+0000","last_undegraded":"2024-09-21T19:08:19.752947+0000","last_fullsized":"2024-09-21T19:08:19.752947+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":748,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755290+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:08:19.755290+0000","last_peered":"2024-09-21T19:08:19.755290+0000","last_clean":"2024-09-21T19:08:19.755290+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:08:19.755290+0000","last_undegraded":"2024-09-21T19:08:19.755290+0000","last_fullsized":"2024-09-21T19:08:19.755290+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":582,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712961+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:08:19.712961+0000","last_peered":"2024-09-21T19:08:19.712961+0000","last_clean":"2024-09-21T19:08:19.712961+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:08:19.712961+0000","last_undegraded":"2024-09-21T19:08:19.712961+0000","last_fullsized":"2024-09-21T19:08:19.712961+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"381'9526","reported_seq":9641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:24.200532+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:08:24.200532+0000","last_peered":"2024-09-21T19:08:24.200532+0000","last_clean":"2024-09-21T19:08:24.200532+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:08:24.200532+0000","last_undegraded":"2024-09-21T19:08:24.200532+0000","last_fullsized":"2024-09-21T19:08:24.200532+0000","mapping_epoch":330,"log_start":"344'4200","ondisk_log_start":"344'4200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5326,"log_dups_size":0,"ondisk_log_size":5326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337444888,"num_objects":5150,"num_object_clones":0,"num_object_copies":15450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9526,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":470,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709911+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:08:19.709911+0000","last_peered":"2024-09-21T19:08:19.709911+0000","last_clean":"2024-09-21T19:08:19.709911+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:08:19.709911+0000","last_undegraded":"2024-09-21T19:08:19.709911+0000","last_fullsized":"2024-09-21T19:08:19.709911+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":628,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755472+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:08:19.755472+0000","last_peered":"2024-09-21T19:08:19.755472+0000","last_clean":"2024-09-21T19:08:19.755472+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:08:19.755472+0000","last_undegraded":"2024-09-21T19:08:19.755472+0000","last_fullsized":"2024-09-21T19:08:19.755472+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":721,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707808+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:08:19.707808+0000","last_peered":"2024-09-21T19:08:19.707808+0000","last_clean":"2024-09-21T19:08:19.707808+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:08:19.707808+0000","last_undegraded":"2024-09-21T19:08:19.707808+0000","last_fullsized":"2024-09-21T19:08:19.707808+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":605,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709046+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:08:19.709046+0000","last_peered":"2024-09-21T19:08:19.709046+0000","last_clean":"2024-09-21T19:08:19.709046+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:08:19.709046+0000","last_undegraded":"2024-09-21T19:08:19.709046+0000","last_fullsized":"2024-09-21T19:08:19.709046+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":709,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707337+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:08:19.707337+0000","last_peered":"2024-09-21T19:08:19.707337+0000","last_clean":"2024-09-21T19:08:19.707337+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:08:19.707337+0000","last_undegraded":"2024-09-21T19:08:19.707337+0000","last_fullsized":"2024-09-21T19:08:19.707337+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:39.298144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"381'9686","reported_seq":19167,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:24.186860+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:08:24.186860+0000","last_peered":"2024-09-21T19:08:24.186860+0000","last_clean":"2024-09-21T19:08:24.186860+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:08:24.186860+0000","last_undegraded":"2024-09-21T19:08:24.186860+0000","last_fullsized":"2024-09-21T19:08:24.186860+0000","mapping_epoch":380,"log_start":"345'4400","ondisk_log_start":"345'4400","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338427904,"num_objects":5164,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9686,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":446,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712554+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:08:19.712554+0000","last_peered":"2024-09-21T19:08:19.712554+0000","last_clean":"2024-09-21T19:08:19.712554+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:08:19.712554+0000","last_undegraded":"2024-09-21T19:08:19.712554+0000","last_fullsized":"2024-09-21T19:08:19.712554+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":992,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707651+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:08:19.707651+0000","last_peered":"2024-09-21T19:08:19.707651+0000","last_clean":"2024-09-21T19:08:19.707651+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:08:19.707651+0000","last_undegraded":"2024-09-21T19:08:19.707651+0000","last_fullsized":"2024-09-21T19:08:19.707651+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:41.293807+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":729,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706197+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:08:19.706197+0000","last_peered":"2024-09-21T19:08:19.706197+0000","last_clean":"2024-09-21T19:08:19.706197+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:08:19.706197+0000","last_undegraded":"2024-09-21T19:08:19.706197+0000","last_fullsized":"2024-09-21T19:08:19.706197+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":276,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752939+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:08:19.752939+0000","last_peered":"2024-09-21T19:08:19.752939+0000","last_clean":"2024-09-21T19:08:19.752939+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:08:19.752939+0000","last_undegraded":"2024-09-21T19:08:19.752939+0000","last_fullsized":"2024-09-21T19:08:19.752939+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:25.256087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":568,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.754528+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:08:19.754528+0000","last_peered":"2024-09-21T19:08:19.754528+0000","last_clean":"2024-09-21T19:08:19.754528+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:08:19.754528+0000","last_undegraded":"2024-09-21T19:08:19.754528+0000","last_fullsized":"2024-09-21T19:08:19.754528+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":676,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.751838+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:08:19.751838+0000","last_peered":"2024-09-21T19:08:19.751838+0000","last_clean":"2024-09-21T19:08:19.751838+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:08:19.751838+0000","last_undegraded":"2024-09-21T19:08:19.751838+0000","last_fullsized":"2024-09-21T19:08:19.751838+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"381'9412","reported_seq":15269,"reported_epoch":381,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:22.624716+0000","last_change":"2024-09-21T19:06:55.907554+0000","last_active":"2024-09-21T19:08:22.624716+0000","last_peered":"2024-09-21T19:08:22.624716+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.870419+0000","last_became_peered":"2024-09-21T19:06:55.870419+0000","last_unstale":"2024-09-21T19:08:22.624716+0000","last_undegraded":"2024-09-21T19:06:55.799686+0000","last_fullsized":"2024-09-21T19:06:55.681444+0000","mapping_epoch":349,"log_start":"340'3000","ondisk_log_start":"340'3000","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6412,"log_dups_size":0,"ondisk_log_size":6412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339607552,"num_objects":5182,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2872,"num_bytes_recovered":188219392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":3417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":773,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781283+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:08:19.781283+0000","last_peered":"2024-09-21T19:08:19.781283+0000","last_clean":"2024-09-21T19:08:19.781283+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:08:19.781283+0000","last_undegraded":"2024-09-21T19:08:19.781283+0000","last_fullsized":"2024-09-21T19:08:19.781283+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":194,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:22.212329+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:08:22.212329+0000","last_peered":"2024-09-21T19:08:22.212329+0000","last_clean":"2024-09-21T19:08:22.212329+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:08:22.212329+0000","last_undegraded":"2024-09-21T19:08:22.212329+0000","last_fullsized":"2024-09-21T19:08:22.212329+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":743,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707338+0000","last_change":"2024-09-21T19:07:29.296652+0000","last_active":"2024-09-21T19:08:19.707338+0000","last_peered":"2024-09-21T19:08:19.707338+0000","last_clean":"2024-09-21T19:08:19.707338+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:08:19.707338+0000","last_undegraded":"2024-09-21T19:08:19.707338+0000","last_fullsized":"2024-09-21T19:08:19.707338+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:29.296562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:29.296562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:51.422184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":145,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.756226+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:08:19.756226+0000","last_peered":"2024-09-21T19:08:19.756226+0000","last_clean":"2024-09-21T19:08:19.756226+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:08:19.756226+0000","last_undegraded":"2024-09-21T19:08:19.756226+0000","last_fullsized":"2024-09-21T19:08:19.756226+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":144,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.756258+0000","last_change":"2024-09-21T19:06:54.694760+0000","last_active":"2024-09-21T19:08:19.756258+0000","last_peered":"2024-09-21T19:08:19.756258+0000","last_clean":"2024-09-21T19:08:19.756258+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:08:19.756258+0000","last_undegraded":"2024-09-21T19:08:19.756258+0000","last_fullsized":"2024-09-21T19:08:19.756258+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":698,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781957+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:08:19.781957+0000","last_peered":"2024-09-21T19:08:19.781957+0000","last_clean":"2024-09-21T19:08:19.781957+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:08:19.781957+0000","last_undegraded":"2024-09-21T19:08:19.781957+0000","last_fullsized":"2024-09-21T19:08:19.781957+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"381'9486","reported_seq":9600,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:22.584633+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:08:22.584633+0000","last_peered":"2024-09-21T19:08:22.584633+0000","last_clean":"2024-09-21T19:08:22.584633+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:08:22.584633+0000","last_undegraded":"2024-09-21T19:08:22.584633+0000","last_fullsized":"2024-09-21T19:08:22.584633+0000","mapping_epoch":330,"log_start":"345'4400","ondisk_log_start":"345'4400","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344064000,"num_objects":5250,"num_object_clones":0,"num_object_copies":15750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9486,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":479,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752645+0000","last_change":"2024-09-21T19:07:30.274632+0000","last_active":"2024-09-21T19:08:19.752645+0000","last_peered":"2024-09-21T19:08:19.752645+0000","last_clean":"2024-09-21T19:08:19.752645+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:08:19.752645+0000","last_undegraded":"2024-09-21T19:08:19.752645+0000","last_fullsized":"2024-09-21T19:08:19.752645+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:07:30.274412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:07:30.274412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:58.929542+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":689,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755646+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:08:19.755646+0000","last_peered":"2024-09-21T19:08:19.755646+0000","last_clean":"2024-09-21T19:08:19.755646+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:08:19.755646+0000","last_undegraded":"2024-09-21T19:08:19.755646+0000","last_fullsized":"2024-09-21T19:08:19.755646+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":576,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712833+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:08:19.712833+0000","last_peered":"2024-09-21T19:08:19.712833+0000","last_clean":"2024-09-21T19:08:19.712833+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:08:19.712833+0000","last_undegraded":"2024-09-21T19:08:19.712833+0000","last_fullsized":"2024-09-21T19:08:19.712833+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1628,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:20.679554+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:08:20.679554+0000","last_peered":"2024-09-21T19:08:20.679554+0000","last_clean":"2024-09-21T19:08:20.679554+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:08:20.679554+0000","last_undegraded":"2024-09-21T19:08:20.679554+0000","last_fullsized":"2024-09-21T19:08:20.679554+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":903,"num_read_kb":898,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":647,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712217+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:08:19.712217+0000","last_peered":"2024-09-21T19:08:19.712217+0000","last_clean":"2024-09-21T19:08:19.712217+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:08:19.712217+0000","last_undegraded":"2024-09-21T19:08:19.712217+0000","last_fullsized":"2024-09-21T19:08:19.712217+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":695,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707495+0000","last_change":"2024-09-21T19:06:54.695510+0000","last_active":"2024-09-21T19:08:19.707495+0000","last_peered":"2024-09-21T19:08:19.707495+0000","last_clean":"2024-09-21T19:08:19.707495+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:08:19.707495+0000","last_undegraded":"2024-09-21T19:08:19.707495+0000","last_fullsized":"2024-09-21T19:08:19.707495+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:54.568845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:54.568845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"381'9558","reported_seq":9672,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:24.214246+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:08:24.214246+0000","last_peered":"2024-09-21T19:08:24.214246+0000","last_clean":"2024-09-21T19:08:24.214246+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:08:24.214246+0000","last_undegraded":"2024-09-21T19:08:24.214246+0000","last_fullsized":"2024-09-21T19:08:24.214246+0000","mapping_epoch":330,"log_start":"344'4200","ondisk_log_start":"344'4200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5358,"log_dups_size":0,"ondisk_log_size":5358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9558,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":860,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.710019+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:08:19.710019+0000","last_peered":"2024-09-21T19:08:19.710019+0000","last_clean":"2024-09-21T19:08:19.710019+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:08:19.710019+0000","last_undegraded":"2024-09-21T19:08:19.710019+0000","last_fullsized":"2024-09-21T19:08:19.710019+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706156+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:08:19.706156+0000","last_peered":"2024-09-21T19:08:19.706156+0000","last_clean":"2024-09-21T19:08:19.706156+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:08:19.706156+0000","last_undegraded":"2024-09-21T19:08:19.706156+0000","last_fullsized":"2024-09-21T19:08:19.706156+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":787,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781500+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:08:19.781500+0000","last_peered":"2024-09-21T19:08:19.781500+0000","last_clean":"2024-09-21T19:08:19.781500+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:08:19.781500+0000","last_undegraded":"2024-09-21T19:08:19.781500+0000","last_fullsized":"2024-09-21T19:08:19.781500+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":712,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782315+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:08:19.782315+0000","last_peered":"2024-09-21T19:08:19.782315+0000","last_clean":"2024-09-21T19:08:19.782315+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:08:19.782315+0000","last_undegraded":"2024-09-21T19:08:19.782315+0000","last_fullsized":"2024-09-21T19:08:19.782315+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":238,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.753277+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:08:19.753277+0000","last_peered":"2024-09-21T19:08:19.753277+0000","last_clean":"2024-09-21T19:08:19.753277+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:08:19.753277+0000","last_undegraded":"2024-09-21T19:08:19.753277+0000","last_fullsized":"2024-09-21T19:08:19.753277+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"381'9428","reported_seq":9542,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:24.192507+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:08:24.192507+0000","last_peered":"2024-09-21T19:08:24.192507+0000","last_clean":"2024-09-21T19:08:24.192507+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:08:24.192507+0000","last_undegraded":"2024-09-21T19:08:24.192507+0000","last_fullsized":"2024-09-21T19:08:24.192507+0000","mapping_epoch":330,"log_start":"343'4100","ondisk_log_start":"343'4100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328728576,"num_objects":5016,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":749,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706656+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:08:19.706656+0000","last_peered":"2024-09-21T19:08:19.706656+0000","last_clean":"2024-09-21T19:08:19.706656+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:08:19.706656+0000","last_undegraded":"2024-09-21T19:08:19.706656+0000","last_fullsized":"2024-09-21T19:08:19.706656+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":718,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706022+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:08:19.706022+0000","last_peered":"2024-09-21T19:08:19.706022+0000","last_clean":"2024-09-21T19:08:19.706022+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:08:19.706022+0000","last_undegraded":"2024-09-21T19:08:19.706022+0000","last_fullsized":"2024-09-21T19:08:19.706022+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":702,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706574+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:08:19.706574+0000","last_peered":"2024-09-21T19:08:19.706574+0000","last_clean":"2024-09-21T19:08:19.706574+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:08:19.706574+0000","last_undegraded":"2024-09-21T19:08:19.706574+0000","last_fullsized":"2024-09-21T19:08:19.706574+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":682,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752450+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:08:19.752450+0000","last_peered":"2024-09-21T19:08:19.752450+0000","last_clean":"2024-09-21T19:08:19.752450+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:08:19.752450+0000","last_undegraded":"2024-09-21T19:08:19.752450+0000","last_fullsized":"2024-09-21T19:08:19.752450+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"381'9498","reported_seq":15276,"reported_epoch":381,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:22.614045+0000","last_change":"2024-09-21T19:06:55.916390+0000","last_active":"2024-09-21T19:08:22.614045+0000","last_peered":"2024-09-21T19:08:22.614045+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.879505+0000","last_became_peered":"2024-09-21T19:06:55.879505+0000","last_unstale":"2024-09-21T19:08:22.614045+0000","last_undegraded":"2024-09-21T19:06:55.764320+0000","last_fullsized":"2024-09-21T19:06:55.681165+0000","mapping_epoch":349,"log_start":"340'3100","ondisk_log_start":"340'3100","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340525056,"num_objects":5196,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9498,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":185532416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3365}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":148,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.754931+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:08:19.754931+0000","last_peered":"2024-09-21T19:08:19.754931+0000","last_clean":"2024-09-21T19:08:19.754931+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:08:19.754931+0000","last_undegraded":"2024-09-21T19:08:19.754931+0000","last_fullsized":"2024-09-21T19:08:19.754931+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":798,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706420+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:08:19.706420+0000","last_peered":"2024-09-21T19:08:19.706420+0000","last_clean":"2024-09-21T19:08:19.706420+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:08:19.706420+0000","last_undegraded":"2024-09-21T19:08:19.706420+0000","last_fullsized":"2024-09-21T19:08:19.706420+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":616,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709396+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:08:19.709396+0000","last_peered":"2024-09-21T19:08:19.709396+0000","last_clean":"2024-09-21T19:08:19.709396+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:08:19.709396+0000","last_undegraded":"2024-09-21T19:08:19.709396+0000","last_fullsized":"2024-09-21T19:08:19.709396+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":141,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709574+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:08:19.709574+0000","last_peered":"2024-09-21T19:08:19.709574+0000","last_clean":"2024-09-21T19:08:19.709574+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:08:19.709574+0000","last_undegraded":"2024-09-21T19:08:19.709574+0000","last_fullsized":"2024-09-21T19:08:19.709574+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":141,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.758840+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:08:19.758840+0000","last_peered":"2024-09-21T19:08:19.758840+0000","last_clean":"2024-09-21T19:08:19.758840+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:08:19.758840+0000","last_undegraded":"2024-09-21T19:08:19.758840+0000","last_fullsized":"2024-09-21T19:08:19.758840+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"381'9532","reported_seq":9646,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:22.560150+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:08:22.560150+0000","last_peered":"2024-09-21T19:08:22.560150+0000","last_clean":"2024-09-21T19:08:22.560150+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:08:22.560150+0000","last_undegraded":"2024-09-21T19:08:22.560150+0000","last_fullsized":"2024-09-21T19:08:22.560150+0000","mapping_epoch":330,"log_start":"344'4300","ondisk_log_start":"344'4300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5232,"log_dups_size":0,"ondisk_log_size":5232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:45.180663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341049344,"num_objects":5204,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9532,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":795,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.783450+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:08:19.783450+0000","last_peered":"2024-09-21T19:08:19.783450+0000","last_clean":"2024-09-21T19:08:19.783450+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:08:19.783450+0000","last_undegraded":"2024-09-21T19:08:19.783450+0000","last_fullsized":"2024-09-21T19:08:19.783450+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706368+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:08:19.706368+0000","last_peered":"2024-09-21T19:08:19.706368+0000","last_clean":"2024-09-21T19:08:19.706368+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:08:19.706368+0000","last_undegraded":"2024-09-21T19:08:19.706368+0000","last_fullsized":"2024-09-21T19:08:19.706368+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752640+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:08:19.752640+0000","last_peered":"2024-09-21T19:08:19.752640+0000","last_clean":"2024-09-21T19:08:19.752640+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:08:19.752640+0000","last_undegraded":"2024-09-21T19:08:19.752640+0000","last_fullsized":"2024-09-21T19:08:19.752640+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":182,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.756212+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:08:19.756212+0000","last_peered":"2024-09-21T19:08:19.756212+0000","last_clean":"2024-09-21T19:08:19.756212+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:08:19.756212+0000","last_undegraded":"2024-09-21T19:08:19.756212+0000","last_fullsized":"2024-09-21T19:08:19.756212+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":697,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706435+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:08:19.706435+0000","last_peered":"2024-09-21T19:08:19.706435+0000","last_clean":"2024-09-21T19:08:19.706435+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:08:19.706435+0000","last_undegraded":"2024-09-21T19:08:19.706435+0000","last_fullsized":"2024-09-21T19:08:19.706435+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"381'9387","reported_seq":9501,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:22.534210+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:08:22.534210+0000","last_peered":"2024-09-21T19:08:22.534210+0000","last_clean":"2024-09-21T19:08:22.534210+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:08:22.534210+0000","last_undegraded":"2024-09-21T19:08:22.534210+0000","last_fullsized":"2024-09-21T19:08:22.534210+0000","mapping_epoch":330,"log_start":"340'3000","ondisk_log_start":"340'3000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331546624,"num_objects":5059,"num_object_clones":0,"num_object_copies":15177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":586,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755524+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:08:19.755524+0000","last_peered":"2024-09-21T19:08:19.755524+0000","last_clean":"2024-09-21T19:08:19.755524+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:08:19.755524+0000","last_undegraded":"2024-09-21T19:08:19.755524+0000","last_fullsized":"2024-09-21T19:08:19.755524+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":686,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.753258+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:08:19.753258+0000","last_peered":"2024-09-21T19:08:19.753258+0000","last_clean":"2024-09-21T19:08:19.753258+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:08:19.753258+0000","last_undegraded":"2024-09-21T19:08:19.753258+0000","last_fullsized":"2024-09-21T19:08:19.753258+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":133,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755193+0000","last_change":"2024-09-21T19:05:42.909550+0000","last_active":"2024-09-21T19:08:19.755193+0000","last_peered":"2024-09-21T19:08:19.755193+0000","last_clean":"2024-09-21T19:08:19.755193+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:08:19.755193+0000","last_undegraded":"2024-09-21T19:08:19.755193+0000","last_fullsized":"2024-09-21T19:08:19.755193+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":581,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712757+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:08:19.712757+0000","last_peered":"2024-09-21T19:08:19.712757+0000","last_clean":"2024-09-21T19:08:19.712757+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:08:19.712757+0000","last_undegraded":"2024-09-21T19:08:19.712757+0000","last_fullsized":"2024-09-21T19:08:19.712757+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"381'9368","reported_seq":9482,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:22.586605+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:08:22.586605+0000","last_peered":"2024-09-21T19:08:22.586605+0000","last_clean":"2024-09-21T19:08:22.586605+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:08:22.586605+0000","last_undegraded":"2024-09-21T19:08:22.586605+0000","last_fullsized":"2024-09-21T19:08:22.586605+0000","mapping_epoch":330,"log_start":"345'4300","ondisk_log_start":"345'4300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338296832,"num_objects":5162,"num_object_clones":0,"num_object_copies":15486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9368,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":635,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709270+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:08:19.709270+0000","last_peered":"2024-09-21T19:08:19.709270+0000","last_clean":"2024-09-21T19:08:19.709270+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:08:19.709270+0000","last_undegraded":"2024-09-21T19:08:19.709270+0000","last_fullsized":"2024-09-21T19:08:19.709270+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":716,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782342+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:08:19.782342+0000","last_peered":"2024-09-21T19:08:19.782342+0000","last_clean":"2024-09-21T19:08:19.782342+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:08:19.782342+0000","last_undegraded":"2024-09-21T19:08:19.782342+0000","last_fullsized":"2024-09-21T19:08:19.782342+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":144,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.756217+0000","last_change":"2024-09-21T19:06:54.697181+0000","last_active":"2024-09-21T19:08:19.756217+0000","last_peered":"2024-09-21T19:08:19.756217+0000","last_clean":"2024-09-21T19:08:19.756217+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:08:19.756217+0000","last_undegraded":"2024-09-21T19:08:19.756217+0000","last_fullsized":"2024-09-21T19:08:19.756217+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":688,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781490+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:08:19.781490+0000","last_peered":"2024-09-21T19:08:19.781490+0000","last_clean":"2024-09-21T19:08:19.781490+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:08:19.781490+0000","last_undegraded":"2024-09-21T19:08:19.781490+0000","last_fullsized":"2024-09-21T19:08:19.781490+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"381'9575","reported_seq":9697,"reported_epoch":381,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:24.206869+0000","last_change":"2024-09-21T19:06:55.921138+0000","last_active":"2024-09-21T19:08:24.206869+0000","last_peered":"2024-09-21T19:08:24.206869+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:08:24.206869+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"345'4300","ondisk_log_start":"345'4300","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337707008,"num_objects":5153,"num_object_clones":0,"num_object_copies":15459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9575,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":564}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":754,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705943+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:08:19.705943+0000","last_peered":"2024-09-21T19:08:19.705943+0000","last_clean":"2024-09-21T19:08:19.705943+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:08:19.705943+0000","last_undegraded":"2024-09-21T19:08:19.705943+0000","last_fullsized":"2024-09-21T19:08:19.705943+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":710,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782782+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:08:19.782782+0000","last_peered":"2024-09-21T19:08:19.782782+0000","last_clean":"2024-09-21T19:08:19.782782+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:08:19.782782+0000","last_undegraded":"2024-09-21T19:08:19.782782+0000","last_fullsized":"2024-09-21T19:08:19.782782+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":133,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.754840+0000","last_change":"2024-09-21T19:05:42.902451+0000","last_active":"2024-09-21T19:08:19.754840+0000","last_peered":"2024-09-21T19:08:19.754840+0000","last_clean":"2024-09-21T19:08:19.754840+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:08:19.754840+0000","last_undegraded":"2024-09-21T19:08:19.754840+0000","last_fullsized":"2024-09-21T19:08:19.754840+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":676,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706186+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:08:19.706186+0000","last_peered":"2024-09-21T19:08:19.706186+0000","last_clean":"2024-09-21T19:08:19.706186+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:08:19.706186+0000","last_undegraded":"2024-09-21T19:08:19.706186+0000","last_fullsized":"2024-09-21T19:08:19.706186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"381'9674","reported_seq":10070,"reported_epoch":381,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:22.587955+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:08:22.587955+0000","last_peered":"2024-09-21T19:08:22.587955+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:08:22.587955+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"346'4600","ondisk_log_start":"346'4600","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9674,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":9240576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":628}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":309,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781392+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:08:19.781392+0000","last_peered":"2024-09-21T19:08:19.781392+0000","last_clean":"2024-09-21T19:08:19.781392+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:08:19.781392+0000","last_undegraded":"2024-09-21T19:08:19.781392+0000","last_fullsized":"2024-09-21T19:08:19.781392+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782123+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:08:19.782123+0000","last_peered":"2024-09-21T19:08:19.782123+0000","last_clean":"2024-09-21T19:08:19.782123+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:08:19.782123+0000","last_undegraded":"2024-09-21T19:08:19.782123+0000","last_fullsized":"2024-09-21T19:08:19.782123+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":179,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705930+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:08:19.705930+0000","last_peered":"2024-09-21T19:08:19.705930+0000","last_clean":"2024-09-21T19:08:19.705930+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:08:19.705930+0000","last_undegraded":"2024-09-21T19:08:19.705930+0000","last_fullsized":"2024-09-21T19:08:19.705930+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":712,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709106+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:08:19.709106+0000","last_peered":"2024-09-21T19:08:19.709106+0000","last_clean":"2024-09-21T19:08:19.709106+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:08:19.709106+0000","last_undegraded":"2024-09-21T19:08:19.709106+0000","last_fullsized":"2024-09-21T19:08:19.709106+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":647,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709259+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:08:19.709259+0000","last_peered":"2024-09-21T19:08:19.709259+0000","last_clean":"2024-09-21T19:08:19.709259+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:08:19.709259+0000","last_undegraded":"2024-09-21T19:08:19.709259+0000","last_fullsized":"2024-09-21T19:08:19.709259+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":710,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.753538+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:08:19.753538+0000","last_peered":"2024-09-21T19:08:19.753538+0000","last_clean":"2024-09-21T19:08:19.753538+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:08:19.753538+0000","last_undegraded":"2024-09-21T19:08:19.753538+0000","last_fullsized":"2024-09-21T19:08:19.753538+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":246,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706248+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:08:19.706248+0000","last_peered":"2024-09-21T19:08:19.706248+0000","last_clean":"2024-09-21T19:08:19.706248+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:08:19.706248+0000","last_undegraded":"2024-09-21T19:08:19.706248+0000","last_fullsized":"2024-09-21T19:08:19.706248+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":677,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781873+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:08:19.781873+0000","last_peered":"2024-09-21T19:08:19.781873+0000","last_clean":"2024-09-21T19:08:19.781873+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:08:19.781873+0000","last_undegraded":"2024-09-21T19:08:19.781873+0000","last_fullsized":"2024-09-21T19:08:19.781873+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":586,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755740+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:08:19.755740+0000","last_peered":"2024-09-21T19:08:19.755740+0000","last_clean":"2024-09-21T19:08:19.755740+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:08:19.755740+0000","last_undegraded":"2024-09-21T19:08:19.755740+0000","last_fullsized":"2024-09-21T19:08:19.755740+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":697,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.783635+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:08:19.783635+0000","last_peered":"2024-09-21T19:08:19.783635+0000","last_clean":"2024-09-21T19:08:19.783635+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:08:19.783635+0000","last_undegraded":"2024-09-21T19:08:19.783635+0000","last_fullsized":"2024-09-21T19:08:19.783635+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":680,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.753399+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:08:19.753399+0000","last_peered":"2024-09-21T19:08:19.753399+0000","last_clean":"2024-09-21T19:08:19.753399+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:08:19.753399+0000","last_undegraded":"2024-09-21T19:08:19.753399+0000","last_fullsized":"2024-09-21T19:08:19.753399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":133,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755783+0000","last_change":"2024-09-21T19:05:42.905153+0000","last_active":"2024-09-21T19:08:19.755783+0000","last_peered":"2024-09-21T19:08:19.755783+0000","last_clean":"2024-09-21T19:08:19.755783+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:08:19.755783+0000","last_undegraded":"2024-09-21T19:08:19.755783+0000","last_fullsized":"2024-09-21T19:08:19.755783+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":299,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782093+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:08:19.782093+0000","last_peered":"2024-09-21T19:08:19.782093+0000","last_clean":"2024-09-21T19:08:19.782093+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:08:19.782093+0000","last_undegraded":"2024-09-21T19:08:19.782093+0000","last_fullsized":"2024-09-21T19:08:19.782093+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":583,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.754221+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:08:19.754221+0000","last_peered":"2024-09-21T19:08:19.754221+0000","last_clean":"2024-09-21T19:08:19.754221+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:08:19.754221+0000","last_undegraded":"2024-09-21T19:08:19.754221+0000","last_fullsized":"2024-09-21T19:08:19.754221+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":895,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752591+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:08:19.752591+0000","last_peered":"2024-09-21T19:08:19.752591+0000","last_clean":"2024-09-21T19:08:19.752591+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:08:19.752591+0000","last_undegraded":"2024-09-21T19:08:19.752591+0000","last_fullsized":"2024-09-21T19:08:19.752591+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":700,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706618+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:08:19.706618+0000","last_peered":"2024-09-21T19:08:19.706618+0000","last_clean":"2024-09-21T19:08:19.706618+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:08:19.706618+0000","last_undegraded":"2024-09-21T19:08:19.706618+0000","last_fullsized":"2024-09-21T19:08:19.706618+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":706,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752788+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:08:19.752788+0000","last_peered":"2024-09-21T19:08:19.752788+0000","last_clean":"2024-09-21T19:08:19.752788+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:08:19.752788+0000","last_undegraded":"2024-09-21T19:08:19.752788+0000","last_fullsized":"2024-09-21T19:08:19.752788+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":700,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781788+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:08:19.781788+0000","last_peered":"2024-09-21T19:08:19.781788+0000","last_clean":"2024-09-21T19:08:19.781788+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:08:19.781788+0000","last_undegraded":"2024-09-21T19:08:19.781788+0000","last_fullsized":"2024-09-21T19:08:19.781788+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":344,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707768+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:08:19.707768+0000","last_peered":"2024-09-21T19:08:19.707768+0000","last_clean":"2024-09-21T19:08:19.707768+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:08:19.707768+0000","last_undegraded":"2024-09-21T19:08:19.707768+0000","last_fullsized":"2024-09-21T19:08:19.707768+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":575,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.756001+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:08:19.756001+0000","last_peered":"2024-09-21T19:08:19.756001+0000","last_clean":"2024-09-21T19:08:19.756001+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:08:19.756001+0000","last_undegraded":"2024-09-21T19:08:19.756001+0000","last_fullsized":"2024-09-21T19:08:19.756001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":779,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706082+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:08:19.706082+0000","last_peered":"2024-09-21T19:08:19.706082+0000","last_clean":"2024-09-21T19:08:19.706082+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:08:19.706082+0000","last_undegraded":"2024-09-21T19:08:19.706082+0000","last_fullsized":"2024-09-21T19:08:19.706082+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":698,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706872+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:08:19.706872+0000","last_peered":"2024-09-21T19:08:19.706872+0000","last_clean":"2024-09-21T19:08:19.706872+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:08:19.706872+0000","last_undegraded":"2024-09-21T19:08:19.706872+0000","last_fullsized":"2024-09-21T19:08:19.706872+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":698,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752507+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:08:19.752507+0000","last_peered":"2024-09-21T19:08:19.752507+0000","last_clean":"2024-09-21T19:08:19.752507+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:08:19.752507+0000","last_undegraded":"2024-09-21T19:08:19.752507+0000","last_fullsized":"2024-09-21T19:08:19.752507+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":701,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705932+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:08:19.705932+0000","last_peered":"2024-09-21T19:08:19.705932+0000","last_clean":"2024-09-21T19:08:19.705932+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:08:19.705932+0000","last_undegraded":"2024-09-21T19:08:19.705932+0000","last_fullsized":"2024-09-21T19:08:19.705932+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":777,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752655+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:08:19.752655+0000","last_peered":"2024-09-21T19:08:19.752655+0000","last_clean":"2024-09-21T19:08:19.752655+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:08:19.752655+0000","last_undegraded":"2024-09-21T19:08:19.752655+0000","last_fullsized":"2024-09-21T19:08:19.752655+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:44.281618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":694,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.782172+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:08:19.782172+0000","last_peered":"2024-09-21T19:08:19.782172+0000","last_clean":"2024-09-21T19:08:19.782172+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:08:19.782172+0000","last_undegraded":"2024-09-21T19:08:19.782172+0000","last_fullsized":"2024-09-21T19:08:19.782172+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":702,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752377+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:08:19.752377+0000","last_peered":"2024-09-21T19:08:19.752377+0000","last_clean":"2024-09-21T19:08:19.752377+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:08:19.752377+0000","last_undegraded":"2024-09-21T19:08:19.752377+0000","last_fullsized":"2024-09-21T19:08:19.752377+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":594,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.754124+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:08:19.754124+0000","last_peered":"2024-09-21T19:08:19.754124+0000","last_clean":"2024-09-21T19:08:19.754124+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:08:19.754124+0000","last_undegraded":"2024-09-21T19:08:19.754124+0000","last_fullsized":"2024-09-21T19:08:19.754124+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":726,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.706096+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:08:19.706096+0000","last_peered":"2024-09-21T19:08:19.706096+0000","last_clean":"2024-09-21T19:08:19.706096+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:08:19.706096+0000","last_undegraded":"2024-09-21T19:08:19.706096+0000","last_fullsized":"2024-09-21T19:08:19.706096+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":639,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.712275+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:08:19.712275+0000","last_peered":"2024-09-21T19:08:19.712275+0000","last_clean":"2024-09-21T19:08:19.712275+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:08:19.712275+0000","last_undegraded":"2024-09-21T19:08:19.712275+0000","last_fullsized":"2024-09-21T19:08:19.712275+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781139+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:08:19.781139+0000","last_peered":"2024-09-21T19:08:19.781139+0000","last_clean":"2024-09-21T19:08:19.781139+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:08:19.781139+0000","last_undegraded":"2024-09-21T19:08:19.781139+0000","last_fullsized":"2024-09-21T19:08:19.781139+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":701,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752556+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:08:19.752556+0000","last_peered":"2024-09-21T19:08:19.752556+0000","last_clean":"2024-09-21T19:08:19.752556+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:08:19.752556+0000","last_undegraded":"2024-09-21T19:08:19.752556+0000","last_fullsized":"2024-09-21T19:08:19.752556+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":725,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.705864+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:08:19.705864+0000","last_peered":"2024-09-21T19:08:19.705864+0000","last_clean":"2024-09-21T19:08:19.705864+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:08:19.705864+0000","last_undegraded":"2024-09-21T19:08:19.705864+0000","last_fullsized":"2024-09-21T19:08:19.705864+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":578,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755906+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:08:19.755906+0000","last_peered":"2024-09-21T19:08:19.755906+0000","last_clean":"2024-09-21T19:08:19.755906+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:08:19.755906+0000","last_undegraded":"2024-09-21T19:08:19.755906+0000","last_fullsized":"2024-09-21T19:08:19.755906+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":614,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.708949+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:08:19.708949+0000","last_peered":"2024-09-21T19:08:19.708949+0000","last_clean":"2024-09-21T19:08:19.708949+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:08:19.708949+0000","last_undegraded":"2024-09-21T19:08:19.708949+0000","last_fullsized":"2024-09-21T19:08:19.708949+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":688,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.781378+0000","last_change":"2024-09-21T19:05:42.904326+0000","last_active":"2024-09-21T19:08:19.781378+0000","last_peered":"2024-09-21T19:08:19.781378+0000","last_clean":"2024-09-21T19:08:19.781378+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:08:19.781378+0000","last_undegraded":"2024-09-21T19:08:19.781378+0000","last_fullsized":"2024-09-21T19:08:19.781378+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":694,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.707631+0000","last_change":"2024-09-21T19:07:21.428355+0000","last_active":"2024-09-21T19:08:19.707631+0000","last_peered":"2024-09-21T19:08:19.707631+0000","last_clean":"2024-09-21T19:08:19.707631+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:08:19.707631+0000","last_undegraded":"2024-09-21T19:08:19.707631+0000","last_fullsized":"2024-09-21T19:08:19.707631+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:21.428212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:21.428212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.151249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":718,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.752003+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:08:19.752003+0000","last_peered":"2024-09-21T19:08:19.752003+0000","last_clean":"2024-09-21T19:08:19.752003+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:08:19.752003+0000","last_undegraded":"2024-09-21T19:08:19.752003+0000","last_fullsized":"2024-09-21T19:08:19.752003+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":679,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.709929+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:08:19.709929+0000","last_peered":"2024-09-21T19:08:19.709929+0000","last_clean":"2024-09-21T19:08:19.709929+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:08:19.709929+0000","last_undegraded":"2024-09-21T19:08:19.709929+0000","last_fullsized":"2024-09-21T19:08:19.709929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":598,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T19:08:19.755920+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:08:19.755920+0000","last_peered":"2024-09-21T19:08:19.755920+0000","last_clean":"2024-09-21T19:08:19.755920+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:08:19.755920+0000","last_undegraded":"2024-09-21T19:08:19.755920+0000","last_fullsized":"2024-09-21T19:08:19.755920+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5418254360,"num_objects":82677,"num_object_clones":0,"num_object_copies":248031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152265,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10526,"num_bytes_recovered":689831936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15752179712,"data_stored":15752167496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85565,"ondisk_log_size":85565,"up":48,"acting":44,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":903,"num_read_kb":898,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266875,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1952532,"kb_used_data":1366012,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91792620,"statfs":{"total":95995035648,"available":93995642880,"internally_reserved":0,"allocated":1398796288,"data_stored":1398140974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":600570314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464326,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534024,"kb_used_data":2295160,"kb_used_omap":4,"kb_used_meta":238843,"kb_avail":91211128,"statfs":{"total":95995035648,"available":93400195072,"internally_reserved":0,"allocated":2350243840,"data_stored":2349592217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":244575908},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619027,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1311940,"kb_used_data":1123956,"kb_used_omap":4,"kb_used_meta":187963,"kb_avail":92433212,"statfs":{"total":95995035648,"available":94651609088,"internally_reserved":0,"allocated":1150930944,"data_stored":1150283847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":192474787},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019081,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3178264,"kb_used_data":2687552,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":90566888,"statfs":{"total":95995035648,"available":92740493312,"internally_reserved":0,"allocated":2752053248,"data_stored":2751394142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182606,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2366756,"kb_used_data":1653908,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91378396,"statfs":{"total":95995035648,"available":93571477504,"internally_reserved":0,"allocated":1693601792,"data_stored":1692923245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":729938378},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411540,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2536184,"kb_used_data":1995368,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91208968,"statfs":{"total":95995035648,"available":93397983232,"internally_reserved":0,"allocated":2043256832,"data_stored":2042622028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1659312,"kb_used_data":996320,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92085840,"statfs":{"total":95995035648,"available":94295900160,"internally_reserved":0,"allocated":1020231680,"data_stored":1019540405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4144472,"kb_used_data":3355268,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":89600680,"statfs":{"total":95995035648,"available":91751096320,"internally_reserved":0,"allocated":3435794432,"data_stored":3435131639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3414757376,"data_stored":3414753304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1014300672,"data_stored":1014300672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2026700800,"data_stored":2026700800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1687748608,"data_stored":1687748608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2745110528,"data_stored":2745106456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1145241600,"data_stored":1145241600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2335903744,"data_stored":2335899672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1382416384,"data_stored":1382416384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:08:27.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T19:08:27.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:27.502 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:27 smithi081 ceph-mon[29384]: pgmap v756: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 724 op/s; 12793/248628 objects degraded (5.145%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T19:08:27.503 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:27 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2254754131' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:08:27.503 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:27 smithi081 ceph-mon[39192]: pgmap v756: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 724 op/s; 12793/248628 objects degraded (5.145%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T19:08:27.503 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:27 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2254754131' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:08:27.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:27 smithi104 ceph-mon[32438]: pgmap v756: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 724 op/s; 12793/248628 objects degraded (5.145%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T19:08:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:27 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2254754131' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:08:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.526 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:27.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:27.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.665 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:27.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:27.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:27.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:27.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:27.941 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:27.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:28.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:28.078 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:28.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:28.271 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:28.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.345 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:28.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[29384]: from='client.26146 v1:172.21.15.81:0/4003188757' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2254754131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[29384]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[39192]: from='client.26146 v1:172.21.15.81:0/4003188757' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:28.391 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:28.392 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2254754131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:08:28.392 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:28 smithi081 ceph-mon[39192]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T19:08:28.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:28.481 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:28.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:28.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:28.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-mon[32438]: from='client.26146 v1:172.21.15.81:0/4003188757' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:28.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:28.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2254754131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:08:28.491 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-mon[32438]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T19:08:28.583 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:28.620 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:28.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:28.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:08:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:08:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:28.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:28.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:28.897 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:28.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:28.972 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:28.972 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:28.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:29.029 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:29.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:29.172 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:29.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:29.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:29.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:29.440 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:29.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:29 smithi104 ceph-mon[32438]: pgmap v758: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s; 12239/239979 objects degraded (5.100%); 9.6 MiB/s, 152 objects/s recovering 2024-09-21T19:08:29.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:29.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:29 smithi081 ceph-mon[29384]: pgmap v758: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s; 12239/239979 objects degraded (5.100%); 9.6 MiB/s, 152 objects/s recovering 2024-09-21T19:08:29.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:29 smithi081 ceph-mon[39192]: pgmap v758: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s; 12239/239979 objects degraded (5.100%); 9.6 MiB/s, 152 objects/s recovering 2024-09-21T19:08:29.585 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:29.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:29.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:29.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:29.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:08:29.945 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:29.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:30.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:30.084 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:30.108 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:08:30.109 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12239/239979 objects degraded (5.100%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1141,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":382,"num_osds":8,"num_up_osds":8,"osd_up_since":1726945614,"num_in_osds":8,"osd_in_since":1726945541,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":154},{"state_name":"active+recovering+undersized+degraded+remapped","count":3},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":158,"num_pools":7,"num_objects":79993,"data_bytes":5229910579,"bytes_used":19592781824,"bytes_avail":748367503360,"bytes_total":767960285184,"degraded_objects":12239,"degraded_total":239979,"degraded_ratio":0.051000295859221015,"recovering_objects_per_sec":152,"recovering_bytes_per_sec":10019551,"recovering_keys_per_sec":0,"num_objects_recovered":1229,"num_bytes_recovered":80543744,"num_keys_recovered":0,"read_bytes_sec":1273,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":859},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-21T19:07:53.731883+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T19:08:30.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:08:30.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:30.216 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:30.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:30.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:30.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:30.491 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:30.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:30 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:30.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2719838008' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:08:30.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:30 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 12239/239979 objects degraded (5.100%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:30.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:30 smithi104 ceph-mon[32438]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2719838008' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 12239/239979 objects degraded (5.100%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[39192]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2719838008' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 12239/239979 objects degraded (5.100%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:30 smithi081 ceph-mon[29384]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T19:08:30.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:30.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:30.766 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:30.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:30.871 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:30.871 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:30.908 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:30.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:31.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.044 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:31.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:31.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:31.178 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:31.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:31.313 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:31.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:31.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:31.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:31 smithi104 ceph-mon[32438]: pgmap v760: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 12239/234498 objects degraded (5.219%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:08:31.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:08:31.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:31.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:31 smithi081 ceph-mon[29384]: pgmap v760: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 12239/234498 objects degraded (5.219%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:08:31.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:31 smithi081 ceph-mon[39192]: pgmap v760: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 12239/234498 objects degraded (5.219%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:08:31.602 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:31.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:31.735 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:31.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:31.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:31.908 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:31.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:32.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:32.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:32.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:32.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:32.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.403 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:32.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:32.457 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:32.457 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:32.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:32.539 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:32.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:32.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:32.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:32.818 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:32.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:32.960 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:32.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:33.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:33.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:33.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:33.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:08:33.232 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:33.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:33.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:33.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:33 smithi104 ceph-mon[32438]: pgmap v761: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s; 12239/232848 objects degraded (5.256%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T19:08:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.512 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:33.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:33.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:33 smithi081 ceph-mon[29384]: pgmap v761: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s; 12239/232848 objects degraded (5.256%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T19:08:33.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:33 smithi081 ceph-mon[39192]: pgmap v761: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s; 12239/232848 objects degraded (5.256%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T19:08:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:33.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.664 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:33.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:33.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.799 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:33.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:33.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:33.938 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:33.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:34.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:34.072 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:34.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:34.196 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:34.196 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:34.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:34.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/172898783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:34.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/172898783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:34.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:08:34.332 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:08:34.333 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:08:34.344 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:08:34.344 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:08:34.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:34.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:34.380 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:08:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:08:34.413 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:08:34.413 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:08:34.445 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:08:34.445 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:08:34.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:34.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:34.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/172898783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:34.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:34.641 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:34.742 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:34.777 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:34.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:08:34.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:34.914 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:34.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:34.972 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T19:08:34.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:08:35.016 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:35.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:35.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:35.267 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:08:35.267 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:35 smithi081 ceph-mon[29384]: pgmap v762: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 11659/225981 objects degraded (5.159%); 8.9 MiB/s, 141 objects/s recovering 2024-09-21T19:08:35.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:35.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:35.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.501 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:35.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:35.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:35 smithi104 ceph-mon[32438]: pgmap v762: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 11659/225981 objects degraded (5.159%); 8.9 MiB/s, 141 objects/s recovering 2024-09-21T19:08:35.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:35 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 11659/225981 objects degraded (5.159%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:35.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:35 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 11659/225981 objects degraded (5.159%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:35.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:35 smithi081 ceph-mon[39192]: pgmap v762: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 11659/225981 objects degraded (5.159%); 8.9 MiB/s, 141 objects/s recovering 2024-09-21T19:08:35.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:35 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 11659/225981 objects degraded (5.159%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:35.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:35.704 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:35.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.775 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:35.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:35.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:100431 10.9 e363/349 10:938b8e0d:::benchmark_data_smithi081_155596_object100430:head v 363'6268, mlcod=334'828)", 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:25.523167+0000", 2024-09-21T19:08:35.877 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 70.348112809, 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060867209999999998, 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.523167+0000", 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.523167+0000", 2024-09-21T19:08:35.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.523168+0000", 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1800000000000004e-07 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.523274+0000", 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105201 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.523284+0000", 2024-09-21T19:08:35.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0317e-05 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.523290+0000", 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7420000000000003e-06 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.532479+0000", 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091897929999999999 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.532503+0000", 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3575999999999999e-05 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.583976+0000", 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051472677000000001 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:25.584035+0000", 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8986000000000001e-05 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103680 10.7 e363/349 10:e4a084ff:::benchmark_data_smithi081_155596_object103679:head v 363'6476, mlcod=333'3)", 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.300623+0000", 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.570656839999998, 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10987514600000001, 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300623+0000", 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300623+0000", 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300625+0000", 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8139999999999999e-06 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300651+0000", 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5850000000000002e-05 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300662+0000", 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0917999999999999e-05 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300670+0000", 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.525e-06 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300749+0000", 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9420999999999997e-05 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.300770+0000", 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1220999999999999e-05 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.410448+0000", 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10967792 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.410499+0000", 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0476999999999998e-05 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.885 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103685 10.7 e363/349 10:e32169f0:::benchmark_data_smithi081_155596_object103684:head v 363'6477, mlcod=333'3)", 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.303613+0000", 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.567666966000004, 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11359659, 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.303613+0000", 2024-09-21T19:08:35.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.303613+0000", 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.303615+0000", 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4980000000000001e-06 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.303635+0000", 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0665999999999998e-05 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.303644+0000", 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7870000000000007e-06 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.303649+0000", 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2059999999999998e-06 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307087+0000", 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034374470000000002 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307110+0000", 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3552000000000001e-05 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.417158+0000", 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110047782 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.417210+0000", 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1651999999999999e-05 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103676 10.3 e363/349 10:c4037dfe:::benchmark_data_smithi081_155596_object103675:head v 363'6433, mlcod=334'968)", 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.304025+0000", 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.567254872999996, 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103374193, 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.890 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304025+0000", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304025+0000", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304027+0000", 2024-09-21T19:08:35.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.598e-06 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304101+0000", 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4095000000000001e-05 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304111+0000", 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0406e-05 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304120+0000", 2024-09-21T19:08:35.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9199999999999993e-06 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304177+0000", 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6502000000000002e-05 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.304199+0000", 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1826000000000001e-05 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.312815+0000", 2024-09-21T19:08:35.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0086162459999999993 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407400+0000", 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.094584600000000005 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103687 10.7 e363/349 10:e6515bfc:::benchmark_data_smithi081_155596_object103686:head v 363'6478, mlcod=333'3)", 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.307142+0000", 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.564138482999994, 2024-09-21T19:08:35.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.11010115299999999, 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307142+0000", 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307142+0000", 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307143+0000", 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.672e-06 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307166+0000", 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2717999999999998e-05 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307176+0000", 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6180000000000001e-06 2024-09-21T19:08:35.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307183+0000", 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7309999999999996e-06 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.312710+0000", 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055278000000000002 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.312734+0000", 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3876999999999999e-05 2024-09-21T19:08:35.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.417220+0000", 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10448618699999999 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.417243+0000", 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2549999999999999e-05 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103678 10.3 e363/349 10:cf947b76:::benchmark_data_smithi081_155596_object103677:head v 363'6434, mlcod=334'968)", 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.307808+0000", 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.563472121000004, 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.127021143, 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307808+0000", 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307808+0000", 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307810+0000", 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4810000000000001e-06 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307847+0000", 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7373999999999998e-05 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307857+0000", 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0009e-05 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.307864+0000", 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4949999999999996e-06 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407450+0000", 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.099586559000000005 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407473+0000", 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3272999999999999e-05 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.434747+0000", 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027274033 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.434829+0000", 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1919e-05 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.902 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103688 10.7 e363/349 10:ea21833f:::benchmark_data_smithi081_155596_object103687:head v 363'6479, mlcod=333'3)", 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.310294+0000", 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.560986510999996, 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.114124349, 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.310294+0000", 2024-09-21T19:08:35.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.310294+0000", 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.310295+0000", 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.742e-06 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.310319+0000", 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3187e-05 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.310329+0000", 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0577999999999999e-05 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.310337+0000", 2024-09-21T19:08:35.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2599999999999999e-06 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.410546+0000", 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.100209805 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.410571+0000", 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4445999999999999e-05 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.424290+0000", 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013718789 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.424418+0000", 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012854199999999999 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDPGPush(10.a 363/349 [PushOp(10:5d904c98:::benchmark_data_smithi081_155596_object21134:head, version: 334'1320, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:5d904c98:::benchmark_data_smithi081_155596_object21134:head@334'1320, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.317468+0000", 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.553812434999998, 2024-09-21T19:08:35.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.092123699000000003, 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.317468+0000", 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.317468+0000", 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.317470+0000", 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2649999999999999e-06 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.317535+0000", 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4597999999999994e-05 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407351+0000", 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.089816654999999995 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407387+0000", 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5787000000000001e-05 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407511+0000", 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124086 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407524+0000", 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2672e-05 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.409592+0000", 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020676359999999999 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:103700 10.3 e363/349 10:cbe3d919:::benchmark_data_smithi081_155596_object103699:head v 363'6435, mlcod=334'968)", 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:29.318884+0000", 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.552396392000006, 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.120843848, 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.912 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.318884+0000", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.318884+0000", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.318886+0000", 2024-09-21T19:08:35.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9350000000000001e-06 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.318921+0000", 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4823999999999997e-05 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407361+0000", 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.088440011999999998 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.407371+0000", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0027000000000001e-05 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.413110+0000", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005739816 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.413159+0000", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8912999999999997e-05 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.439689+0000", 2024-09-21T19:08:35.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026530043 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:29.439728+0000", 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8278000000000002e-05 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:106260 10.6 e364/349 10:66e56b21:::benchmark_data_smithi081_155596_object106259:head v 364'6772, mlcod=0'0)", 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:32.355313+0000", 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 63.515967498000002, 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.063481582999999994, 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.916 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355313+0000", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355313+0000", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355315+0000", 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.193e-06 2024-09-21T19:08:35.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355364+0000", 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9169999999999998e-05 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355385+0000", 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0611999999999998e-05 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355394+0000", 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5480000000000007e-06 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355620+0000", 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022584999999999999 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.355679+0000", 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8860000000000002e-05 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.418739+0000", 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.063059923000000004 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.418794+0000", 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5427000000000003e-05 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:106264 10.6 e364/349 10:61434769:::benchmark_data_smithi081_155596_object106263:head v 364'6773, mlcod=0'0)", 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:07:32.360809+0000", 2024-09-21T19:08:35.920 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 63.510471123999999, 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058022365999999999, 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.360809+0000", 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.360809+0000", 2024-09-21T19:08:35.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.360810+0000", 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6899999999999996e-07 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.360842+0000", 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1646999999999997e-05 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.360850+0000", 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7619999999999995e-06 2024-09-21T19:08:35.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.360855+0000", 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1880000000000001e-06 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.365083+0000", 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042284719999999996 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.365143+0000", 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9392999999999999e-05 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.418799+0000", 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.053656025000000003 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:07:32.418832+0000", 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2910000000000002e-05 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:144545 10.6 e377/349 10:64efe071:::benchmark_data_smithi081_155596_object27072:head v 377'9161, mlcod=0'0)", 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:14.052268+0000", 2024-09-21T19:08:35.924 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.819011885999998, 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.061152353, 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.052268+0000", 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.052268+0000", 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.052270+0000", 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.556e-06 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.052277+0000", 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.793e-06 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.052291+0000", 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4470999999999999e-05 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.052298+0000", 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0480000000000001e-06 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.062915+0000", 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010616361 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.062937+0000", 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2410999999999999e-05 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.113378+0000", 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.050441468000000003 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.113421+0000", 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2245e-05 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:144547 10.6 e377/349 10:646854ca:::benchmark_data_smithi081_155596_object27074:head v 377'9162, mlcod=0'0)", 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:14.054884+0000", 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.816395792000002, 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058563022999999999, 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.054884+0000", 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.054884+0000", 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.054885+0000", 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6800000000000009e-07 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.054891+0000", 2024-09-21T19:08:35.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5500000000000002e-06 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.054900+0000", 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9420000000000001e-06 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.054905+0000", 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8060000000000002e-06 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.070163+0000", 2024-09-21T19:08:35.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015258012 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.070192+0000", 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8847999999999999e-05 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.113423+0000", 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.043231546000000003 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.113447+0000", 2024-09-21T19:08:35.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4351000000000001e-05 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:144549 10.6 e377/349 10:6128d681:::benchmark_data_smithi081_155596_object27076:head v 377'9163, mlcod=0'0)", 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:14.057743+0000", 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.813537239999999, 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.055729155000000002, 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.057743+0000", 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.057743+0000", 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.057744+0000", 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.184e-06 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.057750+0000", 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6629999999999998e-06 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.057761+0000", 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1432e-05 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.057767+0000", 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.071e-06 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.076872+0000", 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019104920000000001 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.076897+0000", 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4990999999999998e-05 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.113450+0000", 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.036552411 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.113472+0000", 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2483000000000001e-05 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:144633 10.6 e377/349 10:6786691f:::benchmark_data_smithi081_155596_object27160:head v 377'9174, mlcod=0'0)", 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:14.160850+0000", 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.710430161000001, 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.061125051999999999, 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.937 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.160850+0000", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.160850+0000", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.160852+0000", 2024-09-21T19:08:35.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6819999999999999e-06 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.160861+0000", 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0410000000000003e-06 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.160880+0000", 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9020000000000001e-05 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.160889+0000", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3190000000000001e-06 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.161749+0000", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00085961399999999997 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.161797+0000", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7769000000000003e-05 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.221936+0000", 2024-09-21T19:08:35.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060139423999999997 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.221975+0000", 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9183000000000001e-05 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:144638 10.6 e377/349 10:665e71c4:::benchmark_data_smithi081_155596_object27165:head v 377'9175, mlcod=0'0)", 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:14.165193+0000", 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.706087162999999, 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056806085999999999, 2024-09-21T19:08:35.941 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.165193+0000", 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.165193+0000", 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.165195+0000", 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.53e-06 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.165202+0000", 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3830000000000003e-06 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.165212+0000", 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4949999999999997e-06 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.165217+0000", 2024-09-21T19:08:35.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9410000000000004e-06 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.169896+0000", 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046783160000000001 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.169942+0000", 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5751999999999998e-05 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.221978+0000", 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052036934 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:14.221999+0000", 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0735000000000001e-05 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:145391 10.9 e377/349 10:9e718fcd:::benchmark_data_smithi081_155596_object27918:head v 377'9018, mlcod=338'2205)", 2024-09-21T19:08:35.946 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:15.050116+0000", 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.821164267, 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.059655213999999998, 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050116+0000", 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050116+0000", 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050118+0000", 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.809e-06 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050127+0000", 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5980000000000003e-06 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050140+0000", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2686e-05 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050162+0000", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2005e-05 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050846+0000", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068363699999999996 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.050867+0000", 2024-09-21T19:08:35.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1107e-05 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.109721+0000", 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.058854624000000001 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.109771+0000", 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9747999999999999e-05 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:145402 10.9 e377/349 10:9359de22:::benchmark_data_smithi081_155596_object27929:head v 377'9019, mlcod=338'2205)", 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:15.053916+0000", 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.817364277999999, 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.055887376000000002, 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.053916+0000", 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.053916+0000", 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.053917+0000", 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7999999999999995e-07 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.053922+0000", 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9899999999999997e-06 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.053928+0000", 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1109999999999996e-06 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.053933+0000", 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0159999999999999e-06 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.058249+0000", 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043158279999999999 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.058270+0000", 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1146999999999999e-05 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.109778+0000", 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051508058000000002 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.109803+0000", 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5646000000000001e-05 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.954 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:145443 10.6 e377/349 10:6ae099a1:::benchmark_data_smithi081_155596_object27970:head v 377'9235, mlcod=0'0)", 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:15.094900+0000", 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.776380635999999, 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.062431794999999998, 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.094900+0000", 2024-09-21T19:08:35.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.094900+0000", 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.094902+0000", 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.094910+0000", 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7690000000000002e-06 2024-09-21T19:08:35.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.094926+0000", 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5953000000000001e-05 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.094935+0000", 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4179999999999997e-06 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.105014+0000", 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010078912000000001 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.105035+0000", 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1361000000000001e-05 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.157230+0000", 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.052195167000000001 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.157331+0000", 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010125500000000001 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36077.0:145446 10.6 e377/349 10:6d0b237c:::benchmark_data_smithi081_155596_object27973:head v 377'9236, mlcod=0'0)", 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:15.096635+0000", 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.77464509, 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.060746359, 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.096635+0000", 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.096635+0000", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.096637+0000", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.415e-06 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.096644+0000", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8120000000000008e-06 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.096656+0000", 2024-09-21T19:08:35.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1622e-05 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.096663+0000", 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6710000000000002e-06 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.111852+0000", 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015189408 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.111928+0000", 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6089999999999998e-05 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.157334+0000", 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.045405819 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:15.157382+0000", 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7521999999999997e-05 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:08:35.962 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:08:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:35.963 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:35.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:36.064 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:36.098 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:36.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:36.239 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:36.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:36.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:36.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:36.495 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:36.505 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:36.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:36.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:36.737 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:36.758 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:36.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:08:36.860 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:36.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:36.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:36 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:36 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:36 smithi104 ceph-mon[32438]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T19:08:37.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:36 smithi104 ceph-mon[32438]: pgmap v764: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 793 op/s; 11659/220935 objects degraded (5.277%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T19:08:37.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[29384]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[29384]: pgmap v764: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 793 op/s; 11659/220935 objects degraded (5.277%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[39192]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T19:08:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:36 smithi081 ceph-mon[39192]: pgmap v764: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 793 op/s; 11659/220935 objects degraded (5.277%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T19:08:37.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:37.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:37.248 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:37.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:37.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:37.392 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:37.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:37.534 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:37.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:08:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:08:37.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:37.670 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:37.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:08:37.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:08:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.835 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:37.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:37.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:37.972 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:37.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:38.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:37 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1295835574' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:38.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:37 smithi104 ceph-mon[32438]: from='client.36296 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:38.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1295835574' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[29384]: from='client.36296 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1295835574' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[39192]: from='client.36296 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:38.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:38.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.108 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:38.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:38.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:38.246 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:38.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:38.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:38.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:38.504 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:38.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:38.644 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:38.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:38.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:08:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:08:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:38 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:38 smithi104 ceph-mon[32438]: pgmap v765: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 818 op/s; 11518/214800 objects degraded (5.362%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T19:08:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:38 smithi104 ceph-mon[32438]: from='client.36296 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:08:38.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:38 smithi104 ceph-mon[32438]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T19:08:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.778 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:38.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:38.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:38.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:39.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:39.057 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[29384]: pgmap v765: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 818 op/s; 11518/214800 objects degraded (5.362%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[29384]: from='client.36296 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[29384]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[39192]: pgmap v765: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 818 op/s; 11518/214800 objects degraded (5.362%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T19:08:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[39192]: from='client.36296 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:08:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:38 smithi081 ceph-mon[39192]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T19:08:39.159 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:39.194 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:39.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:39.327 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:39.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:39.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:39.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:39.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:39.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:39.717 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:39.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:39.854 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:39.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:39.956 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:40.000 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:40.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:40.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:39 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 11518/214800 objects degraded (5.362%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:40.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:40.041 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:39 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 11518/214800 objects degraded (5.362%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:40.041 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:40.041 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:39 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 11518/214800 objects degraded (5.362%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:40.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:40.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:40.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:40.291 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:40.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:40.425 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:40.446 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:08:40.450 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:08:40.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:40.560 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:40.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:40.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:40.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:40.824 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:40.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:40.966 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:40.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:41.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:40 smithi104 ceph-mon[32438]: pgmap v767: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 11024/206847 objects degraded (5.330%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T19:08:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:40 smithi081 ceph-mon[29384]: pgmap v767: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 11024/206847 objects degraded (5.330%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T19:08:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:40 smithi081 ceph-mon[39192]: pgmap v767: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 11024/206847 objects degraded (5.330%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T19:08:41.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:41.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:41.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:41.336 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:41.375 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:41.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:41.514 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:41.573 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":767,"stamp":"2024-09-21T19:08:39.747746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4506130995,"num_objects":68949,"num_object_clones":0,"num_object_copies":206847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68949,"num_whiteouts":0,"num_read":2058,"num_read_kb":1810,"num_write":166914,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12530,"num_bytes_recovered":806037752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87251,"ondisk_log_size":87251,"up":474,"acting":470,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17344248,"kb_used_data":13040104,"kb_used_omap":18,"kb_used_meta":4303917,"kb_avail":732616968,"statfs":{"total":767960285184,"available":750199775232,"internally_reserved":0,"allocated":13353066496,"data_stored":13347661887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4407212016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,8,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-568000512,"num_objects":-8667,"num_object_clones":0,"num_object_copies":-26001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8667,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8667,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1215,"num_bytes_recovered":79626240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002727"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":607,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.748010+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:08:35.748010+0000","last_peered":"2024-09-21T19:08:35.748010+0000","last_clean":"2024-09-21T19:08:35.748010+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:08:35.748010+0000","last_undegraded":"2024-09-21T19:08:35.748010+0000","last_fullsized":"2024-09-21T19:08:35.748010+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":775,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.759233+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:08:35.759233+0000","last_peered":"2024-09-21T19:08:35.759233+0000","last_clean":"2024-09-21T19:08:35.759233+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:08:35.759233+0000","last_undegraded":"2024-09-21T19:08:35.759233+0000","last_fullsized":"2024-09-21T19:08:35.759233+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.748048+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:08:35.748048+0000","last_peered":"2024-09-21T19:08:35.748048+0000","last_clean":"2024-09-21T19:08:35.748048+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:08:35.748048+0000","last_undegraded":"2024-09-21T19:08:35.748048+0000","last_fullsized":"2024-09-21T19:08:35.748048+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:31.228363+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":692,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756989+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:08:35.756989+0000","last_peered":"2024-09-21T19:08:35.756989+0000","last_clean":"2024-09-21T19:08:35.756989+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:08:35.756989+0000","last_undegraded":"2024-09-21T19:08:35.756989+0000","last_fullsized":"2024-09-21T19:08:35.756989+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":703,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768651+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:08:37.768651+0000","last_peered":"2024-09-21T19:08:37.768651+0000","last_clean":"2024-09-21T19:08:37.768651+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:08:37.768651+0000","last_undegraded":"2024-09-21T19:08:37.768651+0000","last_fullsized":"2024-09-21T19:08:37.768651+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":730,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768353+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:08:37.768353+0000","last_peered":"2024-09-21T19:08:37.768353+0000","last_clean":"2024-09-21T19:08:37.768353+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:08:37.768353+0000","last_undegraded":"2024-09-21T19:08:37.768353+0000","last_fullsized":"2024-09-21T19:08:37.768353+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":827,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.767203+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:08:37.767203+0000","last_peered":"2024-09-21T19:08:37.767203+0000","last_clean":"2024-09-21T19:08:37.767203+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:08:37.767203+0000","last_undegraded":"2024-09-21T19:08:37.767203+0000","last_fullsized":"2024-09-21T19:08:37.767203+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":712,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.759142+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:08:35.759142+0000","last_peered":"2024-09-21T19:08:35.759142+0000","last_clean":"2024-09-21T19:08:35.759142+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:08:35.759142+0000","last_undegraded":"2024-09-21T19:08:35.759142+0000","last_fullsized":"2024-09-21T19:08:35.759142+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763633+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:08:35.763633+0000","last_peered":"2024-09-21T19:08:35.763633+0000","last_clean":"2024-09-21T19:08:35.763633+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:08:35.763633+0000","last_undegraded":"2024-09-21T19:08:35.763633+0000","last_fullsized":"2024-09-21T19:08:35.763633+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":756,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757317+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:08:35.757317+0000","last_peered":"2024-09-21T19:08:35.757317+0000","last_clean":"2024-09-21T19:08:35.757317+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:08:35.757317+0000","last_undegraded":"2024-09-21T19:08:35.757317+0000","last_fullsized":"2024-09-21T19:08:35.757317+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":654,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.782031+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:08:37.782031+0000","last_peered":"2024-09-21T19:08:37.782031+0000","last_clean":"2024-09-21T19:08:37.782031+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:08:37.782031+0000","last_undegraded":"2024-09-21T19:08:37.782031+0000","last_fullsized":"2024-09-21T19:08:37.782031+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":573,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.749351+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:08:35.749351+0000","last_peered":"2024-09-21T19:08:35.749351+0000","last_clean":"2024-09-21T19:08:35.749351+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:08:35.749351+0000","last_undegraded":"2024-09-21T19:08:35.749351+0000","last_fullsized":"2024-09-21T19:08:35.749351+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":252,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.764921+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:08:35.764921+0000","last_peered":"2024-09-21T19:08:35.764921+0000","last_clean":"2024-09-21T19:08:35.764921+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:08:35.764921+0000","last_undegraded":"2024-09-21T19:08:35.764921+0000","last_fullsized":"2024-09-21T19:08:35.764921+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":702,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757065+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:08:35.757065+0000","last_peered":"2024-09-21T19:08:35.757065+0000","last_clean":"2024-09-21T19:08:35.757065+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:08:35.757065+0000","last_undegraded":"2024-09-21T19:08:35.757065+0000","last_fullsized":"2024-09-21T19:08:35.757065+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":684,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.736839+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:08:35.736839+0000","last_peered":"2024-09-21T19:08:35.736839+0000","last_clean":"2024-09-21T19:08:35.736839+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:08:35.736839+0000","last_undegraded":"2024-09-21T19:08:35.736839+0000","last_fullsized":"2024-09-21T19:08:35.736839+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":260,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747873+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:08:35.747873+0000","last_peered":"2024-09-21T19:08:35.747873+0000","last_clean":"2024-09-21T19:08:35.747873+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:08:35.747873+0000","last_undegraded":"2024-09-21T19:08:35.747873+0000","last_fullsized":"2024-09-21T19:08:35.747873+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.570833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763204+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:08:35.763204+0000","last_peered":"2024-09-21T19:08:35.763204+0000","last_clean":"2024-09-21T19:08:35.763204+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:08:35.763204+0000","last_undegraded":"2024-09-21T19:08:35.763204+0000","last_fullsized":"2024-09-21T19:08:35.763204+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":589,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747908+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:08:35.747908+0000","last_peered":"2024-09-21T19:08:35.747908+0000","last_clean":"2024-09-21T19:08:35.747908+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:08:35.747908+0000","last_undegraded":"2024-09-21T19:08:35.747908+0000","last_fullsized":"2024-09-21T19:08:35.747908+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":163,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750809+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:08:35.750809+0000","last_peered":"2024-09-21T19:08:35.750809+0000","last_clean":"2024-09-21T19:08:35.750809+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:08:35.750809+0000","last_undegraded":"2024-09-21T19:08:35.750809+0000","last_fullsized":"2024-09-21T19:08:35.750809+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":707,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.753503+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:08:35.753503+0000","last_peered":"2024-09-21T19:08:35.753503+0000","last_clean":"2024-09-21T19:08:35.753503+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:08:35.753503+0000","last_undegraded":"2024-09-21T19:08:35.753503+0000","last_fullsized":"2024-09-21T19:08:35.753503+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:02.338670+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":699,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.772778+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:08:37.772778+0000","last_peered":"2024-09-21T19:08:37.772778+0000","last_clean":"2024-09-21T19:08:37.772778+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:08:37.772778+0000","last_undegraded":"2024-09-21T19:08:37.772778+0000","last_fullsized":"2024-09-21T19:08:37.772778+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":723,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757526+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:08:35.757526+0000","last_peered":"2024-09-21T19:08:35.757526+0000","last_clean":"2024-09-21T19:08:35.757526+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:08:35.757526+0000","last_undegraded":"2024-09-21T19:08:35.757526+0000","last_fullsized":"2024-09-21T19:08:35.757526+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":770,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.767973+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:08:37.767973+0000","last_peered":"2024-09-21T19:08:37.767973+0000","last_clean":"2024-09-21T19:08:37.767973+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:08:37.767973+0000","last_undegraded":"2024-09-21T19:08:37.767973+0000","last_fullsized":"2024-09-21T19:08:37.767973+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":707,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.764710+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:08:35.764710+0000","last_peered":"2024-09-21T19:08:35.764710+0000","last_clean":"2024-09-21T19:08:35.764710+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:08:35.764710+0000","last_undegraded":"2024-09-21T19:08:35.764710+0000","last_fullsized":"2024-09-21T19:08:35.764710+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":730,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737927+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:08:35.737927+0000","last_peered":"2024-09-21T19:08:35.737927+0000","last_clean":"2024-09-21T19:08:35.737927+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:08:35.737927+0000","last_undegraded":"2024-09-21T19:08:35.737927+0000","last_fullsized":"2024-09-21T19:08:35.737927+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":721,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.748119+0000","last_change":"2024-09-21T19:07:33.264047+0000","last_active":"2024-09-21T19:08:35.748119+0000","last_peered":"2024-09-21T19:08:35.748119+0000","last_clean":"2024-09-21T19:08:35.748119+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:08:35.748119+0000","last_undegraded":"2024-09-21T19:08:35.748119+0000","last_fullsized":"2024-09-21T19:08:35.748119+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:33.263975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:33.263975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.063323+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":617,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756633+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:08:35.756633+0000","last_peered":"2024-09-21T19:08:35.756633+0000","last_clean":"2024-09-21T19:08:35.756633+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:08:35.756633+0000","last_undegraded":"2024-09-21T19:08:35.756633+0000","last_fullsized":"2024-09-21T19:08:35.756633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":780,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.775905+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:08:37.775905+0000","last_peered":"2024-09-21T19:08:37.775905+0000","last_clean":"2024-09-21T19:08:37.775905+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:08:37.775905+0000","last_undegraded":"2024-09-21T19:08:37.775905+0000","last_fullsized":"2024-09-21T19:08:37.775905+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"384'10444","reported_seq":10564,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:36.362942+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:08:36.362942+0000","last_peered":"2024-09-21T19:08:36.362942+0000","last_clean":"2024-09-21T19:08:36.362942+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:08:36.362942+0000","last_undegraded":"2024-09-21T19:08:36.362942+0000","last_fullsized":"2024-09-21T19:08:36.362942+0000","mapping_epoch":330,"log_start":"359'5900","ondisk_log_start":"359'5900","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290717696,"num_objects":4436,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":158,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750402+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:08:35.750402+0000","last_peered":"2024-09-21T19:08:35.750402+0000","last_clean":"2024-09-21T19:08:35.750402+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:08:35.750402+0000","last_undegraded":"2024-09-21T19:08:35.750402+0000","last_fullsized":"2024-09-21T19:08:35.750402+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":756,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752969+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:08:35.752969+0000","last_peered":"2024-09-21T19:08:35.752969+0000","last_clean":"2024-09-21T19:08:35.752969+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:08:35.752969+0000","last_undegraded":"2024-09-21T19:08:35.752969+0000","last_fullsized":"2024-09-21T19:08:35.752969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:35.238469+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":697,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766509+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:08:37.766509+0000","last_peered":"2024-09-21T19:08:37.766509+0000","last_clean":"2024-09-21T19:08:37.766509+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:08:37.766509+0000","last_undegraded":"2024-09-21T19:08:37.766509+0000","last_fullsized":"2024-09-21T19:08:37.766509+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":930,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763289+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:08:35.763289+0000","last_peered":"2024-09-21T19:08:35.763289+0000","last_clean":"2024-09-21T19:08:35.763289+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:08:35.763289+0000","last_undegraded":"2024-09-21T19:08:35.763289+0000","last_fullsized":"2024-09-21T19:08:35.763289+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":384,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.753414+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:08:35.753414+0000","last_peered":"2024-09-21T19:08:35.753414+0000","last_clean":"2024-09-21T19:08:35.753414+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:08:35.753414+0000","last_undegraded":"2024-09-21T19:08:35.753414+0000","last_fullsized":"2024-09-21T19:08:35.753414+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"384'10100","reported_seq":10220,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:36.281811+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:08:36.281811+0000","last_peered":"2024-09-21T19:08:36.281811+0000","last_clean":"2024-09-21T19:08:36.281811+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:08:36.281811+0000","last_undegraded":"2024-09-21T19:08:36.281811+0000","last_fullsized":"2024-09-21T19:08:36.281811+0000","mapping_epoch":330,"log_start":"358'5600","ondisk_log_start":"358'5600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284295168,"num_objects":4338,"num_object_clones":0,"num_object_copies":13014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":483,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763552+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:08:35.763552+0000","last_peered":"2024-09-21T19:08:35.763552+0000","last_clean":"2024-09-21T19:08:35.763552+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:08:35.763552+0000","last_undegraded":"2024-09-21T19:08:35.763552+0000","last_fullsized":"2024-09-21T19:08:35.763552+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":793,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766286+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:08:37.766286+0000","last_peered":"2024-09-21T19:08:37.766286+0000","last_clean":"2024-09-21T19:08:37.766286+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:08:37.766286+0000","last_undegraded":"2024-09-21T19:08:37.766286+0000","last_fullsized":"2024-09-21T19:08:37.766286+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":719,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.775888+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:08:37.775888+0000","last_peered":"2024-09-21T19:08:37.775888+0000","last_clean":"2024-09-21T19:08:37.775888+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:08:37.775888+0000","last_undegraded":"2024-09-21T19:08:37.775888+0000","last_fullsized":"2024-09-21T19:08:37.775888+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":694,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.758485+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:08:35.758485+0000","last_peered":"2024-09-21T19:08:35.758485+0000","last_clean":"2024-09-21T19:08:35.758485+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:08:35.758485+0000","last_undegraded":"2024-09-21T19:08:35.758485+0000","last_fullsized":"2024-09-21T19:08:35.758485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":711,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757234+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:08:35.757234+0000","last_peered":"2024-09-21T19:08:35.757234+0000","last_clean":"2024-09-21T19:08:35.757234+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:08:35.757234+0000","last_undegraded":"2024-09-21T19:08:35.757234+0000","last_fullsized":"2024-09-21T19:08:35.757234+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"385'10443","reported_seq":10565,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:39.219347+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:08:39.219347+0000","last_peered":"2024-09-21T19:08:39.219347+0000","last_clean":"2024-09-21T19:08:39.219347+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:08:39.219347+0000","last_undegraded":"2024-09-21T19:08:39.219347+0000","last_fullsized":"2024-09-21T19:08:39.219347+0000","mapping_epoch":330,"log_start":"352'5100","ondisk_log_start":"352'5100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5343,"log_dups_size":0,"ondisk_log_size":5343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278986752,"num_objects":4257,"num_object_clones":0,"num_object_copies":12771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10443,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":476,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768171+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:08:37.768171+0000","last_peered":"2024-09-21T19:08:37.768171+0000","last_clean":"2024-09-21T19:08:37.768171+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:08:37.768171+0000","last_undegraded":"2024-09-21T19:08:37.768171+0000","last_fullsized":"2024-09-21T19:08:37.768171+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":762,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.758638+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:08:35.758638+0000","last_peered":"2024-09-21T19:08:35.758638+0000","last_clean":"2024-09-21T19:08:35.758638+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:08:35.758638+0000","last_undegraded":"2024-09-21T19:08:35.758638+0000","last_fullsized":"2024-09-21T19:08:35.758638+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":585,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756898+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:08:35.756898+0000","last_peered":"2024-09-21T19:08:35.756898+0000","last_clean":"2024-09-21T19:08:35.756898+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:08:35.756898+0000","last_undegraded":"2024-09-21T19:08:35.756898+0000","last_fullsized":"2024-09-21T19:08:35.756898+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":876,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766091+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:08:37.766091+0000","last_peered":"2024-09-21T19:08:37.766091+0000","last_clean":"2024-09-21T19:08:37.766091+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:08:37.766091+0000","last_undegraded":"2024-09-21T19:08:37.766091+0000","last_fullsized":"2024-09-21T19:08:37.766091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":714,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.738130+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:08:35.738130+0000","last_peered":"2024-09-21T19:08:35.738130+0000","last_clean":"2024-09-21T19:08:35.738130+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:08:35.738130+0000","last_undegraded":"2024-09-21T19:08:35.738130+0000","last_fullsized":"2024-09-21T19:08:35.738130+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"384'10503","reported_seq":10623,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:36.835507+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:08:36.835507+0000","last_peered":"2024-09-21T19:08:36.835507+0000","last_clean":"2024-09-21T19:08:36.835507+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:08:36.835507+0000","last_undegraded":"2024-09-21T19:08:36.835507+0000","last_fullsized":"2024-09-21T19:08:36.835507+0000","mapping_epoch":330,"log_start":"355'5500","ondisk_log_start":"355'5500","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290652160,"num_objects":4435,"num_object_clones":0,"num_object_copies":13305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":500,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.776006+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:08:37.776006+0000","last_peered":"2024-09-21T19:08:37.776006+0000","last_clean":"2024-09-21T19:08:37.776006+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:08:37.776006+0000","last_undegraded":"2024-09-21T19:08:37.776006+0000","last_fullsized":"2024-09-21T19:08:37.776006+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":739,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750782+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:08:35.750782+0000","last_peered":"2024-09-21T19:08:35.750782+0000","last_clean":"2024-09-21T19:08:35.750782+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:08:35.750782+0000","last_undegraded":"2024-09-21T19:08:35.750782+0000","last_fullsized":"2024-09-21T19:08:35.750782+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:33.202656+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":260,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768067+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:08:37.768067+0000","last_peered":"2024-09-21T19:08:37.768067+0000","last_clean":"2024-09-21T19:08:37.768067+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:08:37.768067+0000","last_undegraded":"2024-09-21T19:08:37.768067+0000","last_fullsized":"2024-09-21T19:08:37.768067+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":758,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747859+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:08:35.747859+0000","last_peered":"2024-09-21T19:08:35.747859+0000","last_clean":"2024-09-21T19:08:35.747859+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:08:35.747859+0000","last_undegraded":"2024-09-21T19:08:35.747859+0000","last_fullsized":"2024-09-21T19:08:35.747859+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":588,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.749477+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:08:35.749477+0000","last_peered":"2024-09-21T19:08:35.749477+0000","last_clean":"2024-09-21T19:08:35.749477+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:08:35.749477+0000","last_undegraded":"2024-09-21T19:08:35.749477+0000","last_fullsized":"2024-09-21T19:08:35.749477+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"385'10380","reported_seq":10503,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:39.244963+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:08:39.244963+0000","last_peered":"2024-09-21T19:08:39.244963+0000","last_clean":"2024-09-21T19:08:39.244963+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:08:39.244963+0000","last_undegraded":"2024-09-21T19:08:39.244963+0000","last_fullsized":"2024-09-21T19:08:39.244963+0000","mapping_epoch":330,"log_start":"353'5100","ondisk_log_start":"353'5100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281477144,"num_objects":4296,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10380,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":478,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773665+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:08:37.773665+0000","last_peered":"2024-09-21T19:08:37.773665+0000","last_clean":"2024-09-21T19:08:37.773665+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:08:37.773665+0000","last_undegraded":"2024-09-21T19:08:37.773665+0000","last_fullsized":"2024-09-21T19:08:37.773665+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":634,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747993+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:08:35.747993+0000","last_peered":"2024-09-21T19:08:35.747993+0000","last_clean":"2024-09-21T19:08:35.747993+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:08:35.747993+0000","last_undegraded":"2024-09-21T19:08:35.747993+0000","last_fullsized":"2024-09-21T19:08:35.747993+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":727,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752394+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:08:35.752394+0000","last_peered":"2024-09-21T19:08:35.752394+0000","last_clean":"2024-09-21T19:08:35.752394+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:08:35.752394+0000","last_undegraded":"2024-09-21T19:08:35.752394+0000","last_fullsized":"2024-09-21T19:08:35.752394+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:34.228030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":613,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773049+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:08:37.773049+0000","last_peered":"2024-09-21T19:08:37.773049+0000","last_clean":"2024-09-21T19:08:37.773049+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:08:37.773049+0000","last_undegraded":"2024-09-21T19:08:37.773049+0000","last_fullsized":"2024-09-21T19:08:37.773049+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":715,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747970+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:08:35.747970+0000","last_peered":"2024-09-21T19:08:35.747970+0000","last_clean":"2024-09-21T19:08:35.747970+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:08:35.747970+0000","last_undegraded":"2024-09-21T19:08:35.747970+0000","last_fullsized":"2024-09-21T19:08:35.747970+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:39.298144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"385'10563","reported_seq":20052,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:39.205236+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:08:39.205236+0000","last_peered":"2024-09-21T19:08:39.205236+0000","last_clean":"2024-09-21T19:08:39.205236+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:08:39.205236+0000","last_undegraded":"2024-09-21T19:08:39.205236+0000","last_fullsized":"2024-09-21T19:08:39.205236+0000","mapping_epoch":380,"log_start":"352'5200","ondisk_log_start":"352'5200","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280952832,"num_objects":4287,"num_object_clones":0,"num_object_copies":12861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10563,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":452,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.746988+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:08:35.746988+0000","last_peered":"2024-09-21T19:08:35.746988+0000","last_clean":"2024-09-21T19:08:35.746988+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:08:35.746988+0000","last_undegraded":"2024-09-21T19:08:35.746988+0000","last_fullsized":"2024-09-21T19:08:35.746988+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1002,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752370+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:08:35.752370+0000","last_peered":"2024-09-21T19:08:35.752370+0000","last_clean":"2024-09-21T19:08:35.752370+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:08:35.752370+0000","last_undegraded":"2024-09-21T19:08:35.752370+0000","last_fullsized":"2024-09-21T19:08:35.752370+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:41.293807+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":735,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757363+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:08:35.757363+0000","last_peered":"2024-09-21T19:08:35.757363+0000","last_clean":"2024-09-21T19:08:35.757363+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:08:35.757363+0000","last_undegraded":"2024-09-21T19:08:35.757363+0000","last_fullsized":"2024-09-21T19:08:35.757363+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":282,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.751589+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:08:35.751589+0000","last_peered":"2024-09-21T19:08:35.751589+0000","last_clean":"2024-09-21T19:08:35.751589+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:08:35.751589+0000","last_undegraded":"2024-09-21T19:08:35.751589+0000","last_fullsized":"2024-09-21T19:08:35.751589+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:25.256087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":574,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.748505+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:08:35.748505+0000","last_peered":"2024-09-21T19:08:35.748505+0000","last_clean":"2024-09-21T19:08:35.748505+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:08:35.748505+0000","last_undegraded":"2024-09-21T19:08:35.748505+0000","last_fullsized":"2024-09-21T19:08:35.748505+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":685,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768102+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:08:37.768102+0000","last_peered":"2024-09-21T19:08:37.768102+0000","last_clean":"2024-09-21T19:08:37.768102+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:08:37.768102+0000","last_undegraded":"2024-09-21T19:08:37.768102+0000","last_fullsized":"2024-09-21T19:08:37.768102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"384'10316","reported_seq":17554,"reported_epoch":384,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:37.767464+0000","last_change":"2024-09-21T19:06:55.907554+0000","last_active":"2024-09-21T19:08:37.767464+0000","last_peered":"2024-09-21T19:08:37.767464+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.870419+0000","last_became_peered":"2024-09-21T19:06:55.870419+0000","last_unstale":"2024-09-21T19:08:37.767464+0000","last_undegraded":"2024-09-21T19:06:55.799686+0000","last_fullsized":"2024-09-21T19:06:55.681444+0000","mapping_epoch":349,"log_start":"343'3900","ondisk_log_start":"343'3900","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280363008,"num_objects":4278,"num_object_clones":0,"num_object_copies":12834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3559,"num_bytes_recovered":233242624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":3200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":779,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.736648+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:08:35.736648+0000","last_peered":"2024-09-21T19:08:35.736648+0000","last_clean":"2024-09-21T19:08:35.736648+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:08:35.736648+0000","last_undegraded":"2024-09-21T19:08:35.736648+0000","last_fullsized":"2024-09-21T19:08:35.736648+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":200,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752707+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:08:35.752707+0000","last_peered":"2024-09-21T19:08:35.752707+0000","last_clean":"2024-09-21T19:08:35.752707+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:08:35.752707+0000","last_undegraded":"2024-09-21T19:08:35.752707+0000","last_fullsized":"2024-09-21T19:08:35.752707+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":749,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.758813+0000","last_change":"2024-09-21T19:07:29.296652+0000","last_active":"2024-09-21T19:08:35.758813+0000","last_peered":"2024-09-21T19:08:35.758813+0000","last_clean":"2024-09-21T19:08:35.758813+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:08:35.758813+0000","last_undegraded":"2024-09-21T19:08:35.758813+0000","last_fullsized":"2024-09-21T19:08:35.758813+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:29.296562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:29.296562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:51.422184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":151,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.753319+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:08:35.753319+0000","last_peered":"2024-09-21T19:08:35.753319+0000","last_clean":"2024-09-21T19:08:35.753319+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:08:35.753319+0000","last_undegraded":"2024-09-21T19:08:35.753319+0000","last_fullsized":"2024-09-21T19:08:35.753319+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":150,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.751938+0000","last_change":"2024-09-21T19:06:54.694760+0000","last_active":"2024-09-21T19:08:35.751938+0000","last_peered":"2024-09-21T19:08:35.751938+0000","last_clean":"2024-09-21T19:08:35.751938+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:08:35.751938+0000","last_undegraded":"2024-09-21T19:08:35.751938+0000","last_fullsized":"2024-09-21T19:08:35.751938+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737400+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:08:35.737400+0000","last_peered":"2024-09-21T19:08:35.737400+0000","last_clean":"2024-09-21T19:08:35.737400+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:08:35.737400+0000","last_undegraded":"2024-09-21T19:08:35.737400+0000","last_fullsized":"2024-09-21T19:08:35.737400+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"384'10360","reported_seq":10480,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.618747+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:08:37.618747+0000","last_peered":"2024-09-21T19:08:37.618747+0000","last_clean":"2024-09-21T19:08:37.618747+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:08:37.618747+0000","last_undegraded":"2024-09-21T19:08:37.618747+0000","last_fullsized":"2024-09-21T19:08:37.618747+0000","mapping_epoch":330,"log_start":"354'5300","ondisk_log_start":"354'5300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286785536,"num_objects":4376,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10360,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":485,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752507+0000","last_change":"2024-09-21T19:07:30.274632+0000","last_active":"2024-09-21T19:08:35.752507+0000","last_peered":"2024-09-21T19:08:35.752507+0000","last_clean":"2024-09-21T19:08:35.752507+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:08:35.752507+0000","last_undegraded":"2024-09-21T19:08:35.752507+0000","last_fullsized":"2024-09-21T19:08:35.752507+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:07:30.274412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:07:30.274412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:58.929542+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":695,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756734+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:08:35.756734+0000","last_peered":"2024-09-21T19:08:35.756734+0000","last_clean":"2024-09-21T19:08:35.756734+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:08:35.756734+0000","last_undegraded":"2024-09-21T19:08:35.756734+0000","last_fullsized":"2024-09-21T19:08:35.756734+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":582,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747315+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:08:35.747315+0000","last_peered":"2024-09-21T19:08:35.747315+0000","last_clean":"2024-09-21T19:08:35.747315+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:08:35.747315+0000","last_undegraded":"2024-09-21T19:08:35.747315+0000","last_fullsized":"2024-09-21T19:08:35.747315+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1649,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.738691+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:08:35.738691+0000","last_peered":"2024-09-21T19:08:35.738691+0000","last_clean":"2024-09-21T19:08:35.738691+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:08:35.738691+0000","last_undegraded":"2024-09-21T19:08:35.738691+0000","last_fullsized":"2024-09-21T19:08:35.738691+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":918,"num_read_kb":913,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":655,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.784929+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:08:37.784929+0000","last_peered":"2024-09-21T19:08:37.784929+0000","last_clean":"2024-09-21T19:08:37.784929+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:08:37.784929+0000","last_undegraded":"2024-09-21T19:08:37.784929+0000","last_fullsized":"2024-09-21T19:08:37.784929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":709,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752195+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:08:35.752195+0000","last_peered":"2024-09-21T19:08:35.752195+0000","last_clean":"2024-09-21T19:08:35.752195+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:08:35.752195+0000","last_undegraded":"2024-09-21T19:08:35.752195+0000","last_fullsized":"2024-09-21T19:08:35.752195+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:30.010503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"385'10461","reported_seq":10583,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:39.229886+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:08:39.229886+0000","last_peered":"2024-09-21T19:08:39.229886+0000","last_clean":"2024-09-21T19:08:39.229886+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:08:39.229886+0000","last_undegraded":"2024-09-21T19:08:39.229886+0000","last_fullsized":"2024-09-21T19:08:39.229886+0000","mapping_epoch":330,"log_start":"352'5100","ondisk_log_start":"352'5100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274006016,"num_objects":4181,"num_object_clones":0,"num_object_copies":12543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":871,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773762+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:08:37.773762+0000","last_peered":"2024-09-21T19:08:37.773762+0000","last_clean":"2024-09-21T19:08:37.773762+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:08:37.773762+0000","last_undegraded":"2024-09-21T19:08:37.773762+0000","last_fullsized":"2024-09-21T19:08:37.773762+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":719,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.767562+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:08:37.767562+0000","last_peered":"2024-09-21T19:08:37.767562+0000","last_clean":"2024-09-21T19:08:37.767562+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:08:37.767562+0000","last_undegraded":"2024-09-21T19:08:37.767562+0000","last_fullsized":"2024-09-21T19:08:37.767562+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":793,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.738731+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:08:35.738731+0000","last_peered":"2024-09-21T19:08:35.738731+0000","last_clean":"2024-09-21T19:08:35.738731+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:08:35.738731+0000","last_undegraded":"2024-09-21T19:08:35.738731+0000","last_fullsized":"2024-09-21T19:08:35.738731+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":718,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.738086+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:08:35.738086+0000","last_peered":"2024-09-21T19:08:35.738086+0000","last_clean":"2024-09-21T19:08:35.738086+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:08:35.738086+0000","last_undegraded":"2024-09-21T19:08:35.738086+0000","last_fullsized":"2024-09-21T19:08:35.738086+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":246,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.772686+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:08:37.772686+0000","last_peered":"2024-09-21T19:08:37.772686+0000","last_clean":"2024-09-21T19:08:37.772686+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:08:37.772686+0000","last_undegraded":"2024-09-21T19:08:37.772686+0000","last_fullsized":"2024-09-21T19:08:37.772686+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"385'10328","reported_seq":10450,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:39.234790+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:08:39.234790+0000","last_peered":"2024-09-21T19:08:39.234790+0000","last_clean":"2024-09-21T19:08:39.234790+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:08:39.234790+0000","last_undegraded":"2024-09-21T19:08:39.234790+0000","last_fullsized":"2024-09-21T19:08:39.234790+0000","mapping_epoch":330,"log_start":"352'5000","ondisk_log_start":"352'5000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269746176,"num_objects":4116,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":755,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763377+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:08:35.763377+0000","last_peered":"2024-09-21T19:08:35.763377+0000","last_clean":"2024-09-21T19:08:35.763377+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:08:35.763377+0000","last_undegraded":"2024-09-21T19:08:35.763377+0000","last_fullsized":"2024-09-21T19:08:35.763377+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":724,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.758880+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:08:35.758880+0000","last_peered":"2024-09-21T19:08:35.758880+0000","last_clean":"2024-09-21T19:08:35.758880+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:08:35.758880+0000","last_undegraded":"2024-09-21T19:08:35.758880+0000","last_fullsized":"2024-09-21T19:08:35.758880+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":708,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757401+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:08:35.757401+0000","last_peered":"2024-09-21T19:08:35.757401+0000","last_clean":"2024-09-21T19:08:35.757401+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:08:35.757401+0000","last_undegraded":"2024-09-21T19:08:35.757401+0000","last_fullsized":"2024-09-21T19:08:35.757401+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":690,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.767719+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:08:37.767719+0000","last_peered":"2024-09-21T19:08:37.767719+0000","last_clean":"2024-09-21T19:08:37.767719+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:08:37.767719+0000","last_undegraded":"2024-09-21T19:08:37.767719+0000","last_fullsized":"2024-09-21T19:08:37.767719+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"384'10382","reported_seq":17534,"reported_epoch":384,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:37.771804+0000","last_change":"2024-09-21T19:06:55.916390+0000","last_active":"2024-09-21T19:08:37.771804+0000","last_peered":"2024-09-21T19:08:37.771804+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.879505+0000","last_became_peered":"2024-09-21T19:06:55.879505+0000","last_unstale":"2024-09-21T19:08:37.771804+0000","last_undegraded":"2024-09-21T19:06:55.764320+0000","last_fullsized":"2024-09-21T19:06:55.681165+0000","mapping_epoch":349,"log_start":"343'3900","ondisk_log_start":"343'3900","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6482,"log_dups_size":0,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282591232,"num_objects":4312,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10382,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3516,"num_bytes_recovered":230424576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":154,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.751519+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:08:35.751519+0000","last_peered":"2024-09-21T19:08:35.751519+0000","last_clean":"2024-09-21T19:08:35.751519+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:08:35.751519+0000","last_undegraded":"2024-09-21T19:08:35.751519+0000","last_fullsized":"2024-09-21T19:08:35.751519+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":804,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763380+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:08:35.763380+0000","last_peered":"2024-09-21T19:08:35.763380+0000","last_clean":"2024-09-21T19:08:35.763380+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:08:35.763380+0000","last_undegraded":"2024-09-21T19:08:35.763380+0000","last_fullsized":"2024-09-21T19:08:35.763380+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":624,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773475+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:08:37.773475+0000","last_peered":"2024-09-21T19:08:37.773475+0000","last_clean":"2024-09-21T19:08:37.773475+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:08:37.773475+0000","last_undegraded":"2024-09-21T19:08:37.773475+0000","last_fullsized":"2024-09-21T19:08:37.773475+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":149,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773166+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:08:37.773166+0000","last_peered":"2024-09-21T19:08:37.773166+0000","last_clean":"2024-09-21T19:08:37.773166+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:08:37.773166+0000","last_undegraded":"2024-09-21T19:08:37.773166+0000","last_fullsized":"2024-09-21T19:08:37.773166+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":147,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752249+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:08:35.752249+0000","last_peered":"2024-09-21T19:08:35.752249+0000","last_clean":"2024-09-21T19:08:35.752249+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:08:35.752249+0000","last_undegraded":"2024-09-21T19:08:35.752249+0000","last_fullsized":"2024-09-21T19:08:35.752249+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"384'10385","reported_seq":10505,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.604732+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:08:37.604732+0000","last_peered":"2024-09-21T19:08:37.604732+0000","last_clean":"2024-09-21T19:08:37.604732+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:08:37.604732+0000","last_undegraded":"2024-09-21T19:08:37.604732+0000","last_fullsized":"2024-09-21T19:08:37.604732+0000","mapping_epoch":330,"log_start":"353'5200","ondisk_log_start":"353'5200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:45.180663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285147136,"num_objects":4351,"num_object_clones":0,"num_object_copies":13053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10385,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":801,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737194+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:08:35.737194+0000","last_peered":"2024-09-21T19:08:35.737194+0000","last_clean":"2024-09-21T19:08:35.737194+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:08:35.737194+0000","last_undegraded":"2024-09-21T19:08:35.737194+0000","last_fullsized":"2024-09-21T19:08:35.737194+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":730,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.764791+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:08:35.764791+0000","last_peered":"2024-09-21T19:08:35.764791+0000","last_clean":"2024-09-21T19:08:35.764791+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:08:35.764791+0000","last_undegraded":"2024-09-21T19:08:35.764791+0000","last_fullsized":"2024-09-21T19:08:35.764791+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":732,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.771215+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:08:37.771215+0000","last_peered":"2024-09-21T19:08:37.771215+0000","last_clean":"2024-09-21T19:08:37.771215+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:08:37.771215+0000","last_undegraded":"2024-09-21T19:08:37.771215+0000","last_fullsized":"2024-09-21T19:08:37.771215+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":192,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756103+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:08:35.756103+0000","last_peered":"2024-09-21T19:08:35.756103+0000","last_clean":"2024-09-21T19:08:35.756103+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:08:35.756103+0000","last_undegraded":"2024-09-21T19:08:35.756103+0000","last_fullsized":"2024-09-21T19:08:35.756103+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":703,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.763089+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:08:35.763089+0000","last_peered":"2024-09-21T19:08:35.763089+0000","last_clean":"2024-09-21T19:08:35.763089+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:08:35.763089+0000","last_undegraded":"2024-09-21T19:08:35.763089+0000","last_fullsized":"2024-09-21T19:08:35.763089+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"384'10195","reported_seq":10315,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.761202+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:08:37.761202+0000","last_peered":"2024-09-21T19:08:37.761202+0000","last_clean":"2024-09-21T19:08:37.761202+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:08:37.761202+0000","last_undegraded":"2024-09-21T19:08:37.761202+0000","last_fullsized":"2024-09-21T19:08:37.761202+0000","mapping_epoch":330,"log_start":"343'3800","ondisk_log_start":"343'3800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6395,"log_dups_size":0,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278593536,"num_objects":4251,"num_object_clones":0,"num_object_copies":12753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10195,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":592,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.748083+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:08:35.748083+0000","last_peered":"2024-09-21T19:08:35.748083+0000","last_clean":"2024-09-21T19:08:35.748083+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:08:35.748083+0000","last_undegraded":"2024-09-21T19:08:35.748083+0000","last_fullsized":"2024-09-21T19:08:35.748083+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":694,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768391+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:08:37.768391+0000","last_peered":"2024-09-21T19:08:37.768391+0000","last_clean":"2024-09-21T19:08:37.768391+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:08:37.768391+0000","last_undegraded":"2024-09-21T19:08:37.768391+0000","last_fullsized":"2024-09-21T19:08:37.768391+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":139,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750318+0000","last_change":"2024-09-21T19:05:42.909550+0000","last_active":"2024-09-21T19:08:35.750318+0000","last_peered":"2024-09-21T19:08:35.750318+0000","last_clean":"2024-09-21T19:08:35.750318+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:08:35.750318+0000","last_undegraded":"2024-09-21T19:08:35.750318+0000","last_fullsized":"2024-09-21T19:08:35.750318+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":587,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747230+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:08:35.747230+0000","last_peered":"2024-09-21T19:08:35.747230+0000","last_clean":"2024-09-21T19:08:35.747230+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:08:35.747230+0000","last_undegraded":"2024-09-21T19:08:35.747230+0000","last_fullsized":"2024-09-21T19:08:35.747230+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"384'10246","reported_seq":10366,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.622636+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:08:37.622636+0000","last_peered":"2024-09-21T19:08:37.622636+0000","last_clean":"2024-09-21T19:08:37.622636+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:08:37.622636+0000","last_undegraded":"2024-09-21T19:08:37.622636+0000","last_fullsized":"2024-09-21T19:08:37.622636+0000","mapping_epoch":330,"log_start":"354'5200","ondisk_log_start":"354'5200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280756224,"num_objects":4284,"num_object_clones":0,"num_object_copies":12852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10246,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":643,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773366+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:08:37.773366+0000","last_peered":"2024-09-21T19:08:37.773366+0000","last_clean":"2024-09-21T19:08:37.773366+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:08:37.773366+0000","last_undegraded":"2024-09-21T19:08:37.773366+0000","last_fullsized":"2024-09-21T19:08:37.773366+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":722,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737762+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:08:35.737762+0000","last_peered":"2024-09-21T19:08:35.737762+0000","last_clean":"2024-09-21T19:08:35.737762+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:08:35.737762+0000","last_undegraded":"2024-09-21T19:08:35.737762+0000","last_fullsized":"2024-09-21T19:08:35.737762+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":150,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750816+0000","last_change":"2024-09-21T19:06:54.697181+0000","last_active":"2024-09-21T19:08:35.750816+0000","last_peered":"2024-09-21T19:08:35.750816+0000","last_clean":"2024-09-21T19:08:35.750816+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:08:35.750816+0000","last_undegraded":"2024-09-21T19:08:35.750816+0000","last_fullsized":"2024-09-21T19:08:35.750816+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":695,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.738672+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:08:35.738672+0000","last_peered":"2024-09-21T19:08:35.738672+0000","last_clean":"2024-09-21T19:08:35.738672+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:08:35.738672+0000","last_undegraded":"2024-09-21T19:08:35.738672+0000","last_fullsized":"2024-09-21T19:08:35.738672+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"385'10477","reported_seq":10607,"reported_epoch":385,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:39.251463+0000","last_change":"2024-09-21T19:06:55.921138+0000","last_active":"2024-09-21T19:08:39.251463+0000","last_peered":"2024-09-21T19:08:39.251463+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:08:39.251463+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"354'5200","ondisk_log_start":"354'5200","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278593536,"num_objects":4251,"num_object_clones":0,"num_object_copies":12753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-338}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":760,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.758778+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:08:35.758778+0000","last_peered":"2024-09-21T19:08:35.758778+0000","last_clean":"2024-09-21T19:08:35.758778+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:08:35.758778+0000","last_undegraded":"2024-09-21T19:08:35.758778+0000","last_fullsized":"2024-09-21T19:08:35.758778+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737361+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:08:35.737361+0000","last_peered":"2024-09-21T19:08:35.737361+0000","last_clean":"2024-09-21T19:08:35.737361+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:08:35.737361+0000","last_undegraded":"2024-09-21T19:08:35.737361+0000","last_fullsized":"2024-09-21T19:08:35.737361+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":139,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.751431+0000","last_change":"2024-09-21T19:05:42.902451+0000","last_active":"2024-09-21T19:08:35.751431+0000","last_peered":"2024-09-21T19:08:35.751431+0000","last_clean":"2024-09-21T19:08:35.751431+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:08:35.751431+0000","last_undegraded":"2024-09-21T19:08:35.751431+0000","last_fullsized":"2024-09-21T19:08:35.751431+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":684,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768732+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:08:37.768732+0000","last_peered":"2024-09-21T19:08:37.768732+0000","last_clean":"2024-09-21T19:08:37.768732+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:08:37.768732+0000","last_undegraded":"2024-09-21T19:08:37.768732+0000","last_fullsized":"2024-09-21T19:08:37.768732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"384'10609","reported_seq":11805,"reported_epoch":384,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:37.656783+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:08:37.656783+0000","last_peered":"2024-09-21T19:08:37.656783+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:08:37.656783+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"355'5600","ondisk_log_start":"355'5600","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281870336,"num_objects":4301,"num_object_clones":0,"num_object_copies":12903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":35258368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":90}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":315,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737658+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:08:35.737658+0000","last_peered":"2024-09-21T19:08:35.737658+0000","last_clean":"2024-09-21T19:08:35.737658+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:08:35.737658+0000","last_undegraded":"2024-09-21T19:08:35.737658+0000","last_fullsized":"2024-09-21T19:08:35.737658+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":730,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.736933+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:08:35.736933+0000","last_peered":"2024-09-21T19:08:35.736933+0000","last_clean":"2024-09-21T19:08:35.736933+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:08:35.736933+0000","last_undegraded":"2024-09-21T19:08:35.736933+0000","last_fullsized":"2024-09-21T19:08:35.736933+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":190,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766598+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:08:37.766598+0000","last_peered":"2024-09-21T19:08:37.766598+0000","last_clean":"2024-09-21T19:08:37.766598+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:08:37.766598+0000","last_undegraded":"2024-09-21T19:08:37.766598+0000","last_fullsized":"2024-09-21T19:08:37.766598+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":720,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.773055+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:08:37.773055+0000","last_peered":"2024-09-21T19:08:37.773055+0000","last_clean":"2024-09-21T19:08:37.773055+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:08:37.773055+0000","last_undegraded":"2024-09-21T19:08:37.773055+0000","last_fullsized":"2024-09-21T19:08:37.773055+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":655,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.772481+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:08:37.772481+0000","last_peered":"2024-09-21T19:08:37.772481+0000","last_clean":"2024-09-21T19:08:37.772481+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:08:37.772481+0000","last_undegraded":"2024-09-21T19:08:37.772481+0000","last_fullsized":"2024-09-21T19:08:37.772481+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":718,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.775748+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:08:37.775748+0000","last_peered":"2024-09-21T19:08:37.775748+0000","last_clean":"2024-09-21T19:08:37.775748+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:08:37.775748+0000","last_undegraded":"2024-09-21T19:08:37.775748+0000","last_fullsized":"2024-09-21T19:08:37.775748+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":254,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768009+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:08:37.768009+0000","last_peered":"2024-09-21T19:08:37.768009+0000","last_clean":"2024-09-21T19:08:37.768009+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:08:37.768009+0000","last_undegraded":"2024-09-21T19:08:37.768009+0000","last_fullsized":"2024-09-21T19:08:37.768009+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":683,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737352+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:08:35.737352+0000","last_peered":"2024-09-21T19:08:35.737352+0000","last_clean":"2024-09-21T19:08:35.737352+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:08:35.737352+0000","last_undegraded":"2024-09-21T19:08:35.737352+0000","last_fullsized":"2024-09-21T19:08:35.737352+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":592,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756840+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:08:35.756840+0000","last_peered":"2024-09-21T19:08:35.756840+0000","last_clean":"2024-09-21T19:08:35.756840+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:08:35.756840+0000","last_undegraded":"2024-09-21T19:08:35.756840+0000","last_fullsized":"2024-09-21T19:08:35.756840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":703,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737303+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:08:35.737303+0000","last_peered":"2024-09-21T19:08:35.737303+0000","last_clean":"2024-09-21T19:08:35.737303+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:08:35.737303+0000","last_undegraded":"2024-09-21T19:08:35.737303+0000","last_fullsized":"2024-09-21T19:08:35.737303+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":688,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.775498+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:08:37.775498+0000","last_peered":"2024-09-21T19:08:37.775498+0000","last_clean":"2024-09-21T19:08:37.775498+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:08:37.775498+0000","last_undegraded":"2024-09-21T19:08:37.775498+0000","last_fullsized":"2024-09-21T19:08:37.775498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":139,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750896+0000","last_change":"2024-09-21T19:05:42.905153+0000","last_active":"2024-09-21T19:08:35.750896+0000","last_peered":"2024-09-21T19:08:35.750896+0000","last_clean":"2024-09-21T19:08:35.750896+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:08:35.750896+0000","last_undegraded":"2024-09-21T19:08:35.750896+0000","last_fullsized":"2024-09-21T19:08:35.750896+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":305,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737552+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:08:35.737552+0000","last_peered":"2024-09-21T19:08:35.737552+0000","last_clean":"2024-09-21T19:08:35.737552+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:08:35.737552+0000","last_undegraded":"2024-09-21T19:08:35.737552+0000","last_fullsized":"2024-09-21T19:08:35.737552+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":589,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747490+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:08:35.747490+0000","last_peered":"2024-09-21T19:08:35.747490+0000","last_clean":"2024-09-21T19:08:35.747490+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:08:35.747490+0000","last_undegraded":"2024-09-21T19:08:35.747490+0000","last_fullsized":"2024-09-21T19:08:35.747490+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":906,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.767831+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:08:37.767831+0000","last_peered":"2024-09-21T19:08:37.767831+0000","last_clean":"2024-09-21T19:08:37.767831+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:08:37.767831+0000","last_undegraded":"2024-09-21T19:08:37.767831+0000","last_fullsized":"2024-09-21T19:08:37.767831+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":706,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757317+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:08:35.757317+0000","last_peered":"2024-09-21T19:08:35.757317+0000","last_clean":"2024-09-21T19:08:35.757317+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:08:35.757317+0000","last_undegraded":"2024-09-21T19:08:35.757317+0000","last_fullsized":"2024-09-21T19:08:35.757317+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":714,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.768259+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:08:37.768259+0000","last_peered":"2024-09-21T19:08:37.768259+0000","last_clean":"2024-09-21T19:08:37.768259+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:08:37.768259+0000","last_undegraded":"2024-09-21T19:08:37.768259+0000","last_fullsized":"2024-09-21T19:08:37.768259+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":706,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.738452+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:08:35.738452+0000","last_peered":"2024-09-21T19:08:35.738452+0000","last_clean":"2024-09-21T19:08:35.738452+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:08:35.738452+0000","last_undegraded":"2024-09-21T19:08:35.738452+0000","last_fullsized":"2024-09-21T19:08:35.738452+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":354,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.753364+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:08:35.753364+0000","last_peered":"2024-09-21T19:08:35.753364+0000","last_clean":"2024-09-21T19:08:35.753364+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:08:35.753364+0000","last_undegraded":"2024-09-21T19:08:35.753364+0000","last_fullsized":"2024-09-21T19:08:35.753364+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":581,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757019+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:08:35.757019+0000","last_peered":"2024-09-21T19:08:35.757019+0000","last_clean":"2024-09-21T19:08:35.757019+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:08:35.757019+0000","last_undegraded":"2024-09-21T19:08:35.757019+0000","last_fullsized":"2024-09-21T19:08:35.757019+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":785,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.757161+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:08:35.757161+0000","last_peered":"2024-09-21T19:08:35.757161+0000","last_clean":"2024-09-21T19:08:35.757161+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:08:35.757161+0000","last_undegraded":"2024-09-21T19:08:35.757161+0000","last_fullsized":"2024-09-21T19:08:35.757161+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.764993+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:08:35.764993+0000","last_peered":"2024-09-21T19:08:35.764993+0000","last_clean":"2024-09-21T19:08:35.764993+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:08:35.764993+0000","last_undegraded":"2024-09-21T19:08:35.764993+0000","last_fullsized":"2024-09-21T19:08:35.764993+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.750018+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:08:35.750018+0000","last_peered":"2024-09-21T19:08:35.750018+0000","last_clean":"2024-09-21T19:08:35.750018+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:08:35.750018+0000","last_undegraded":"2024-09-21T19:08:35.750018+0000","last_fullsized":"2024-09-21T19:08:35.750018+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.268143+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":709,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766751+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:08:37.766751+0000","last_peered":"2024-09-21T19:08:37.766751+0000","last_clean":"2024-09-21T19:08:37.766751+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:08:37.766751+0000","last_undegraded":"2024-09-21T19:08:37.766751+0000","last_fullsized":"2024-09-21T19:08:37.766751+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":783,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.751451+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:08:35.751451+0000","last_peered":"2024-09-21T19:08:35.751451+0000","last_clean":"2024-09-21T19:08:35.751451+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:08:35.751451+0000","last_undegraded":"2024-09-21T19:08:35.751451+0000","last_fullsized":"2024-09-21T19:08:35.751451+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:44.281618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":700,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.737633+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:08:35.737633+0000","last_peered":"2024-09-21T19:08:35.737633+0000","last_clean":"2024-09-21T19:08:35.737633+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:08:35.737633+0000","last_undegraded":"2024-09-21T19:08:35.737633+0000","last_fullsized":"2024-09-21T19:08:35.737633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":710,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.771062+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:08:37.771062+0000","last_peered":"2024-09-21T19:08:37.771062+0000","last_clean":"2024-09-21T19:08:37.771062+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:08:37.771062+0000","last_undegraded":"2024-09-21T19:08:37.771062+0000","last_fullsized":"2024-09-21T19:08:37.771062+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":600,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.749552+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:08:35.749552+0000","last_peered":"2024-09-21T19:08:35.749552+0000","last_clean":"2024-09-21T19:08:35.749552+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:08:35.749552+0000","last_undegraded":"2024-09-21T19:08:35.749552+0000","last_fullsized":"2024-09-21T19:08:35.749552+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":734,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766957+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:08:37.766957+0000","last_peered":"2024-09-21T19:08:37.766957+0000","last_clean":"2024-09-21T19:08:37.766957+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:08:37.766957+0000","last_undegraded":"2024-09-21T19:08:37.766957+0000","last_fullsized":"2024-09-21T19:08:37.766957+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":647,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.780359+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:08:37.780359+0000","last_peered":"2024-09-21T19:08:37.780359+0000","last_clean":"2024-09-21T19:08:37.780359+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:08:37.780359+0000","last_undegraded":"2024-09-21T19:08:37.780359+0000","last_fullsized":"2024-09-21T19:08:37.780359+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":719,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.736587+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:08:35.736587+0000","last_peered":"2024-09-21T19:08:35.736587+0000","last_clean":"2024-09-21T19:08:35.736587+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:08:35.736587+0000","last_undegraded":"2024-09-21T19:08:35.736587+0000","last_fullsized":"2024-09-21T19:08:35.736587+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":709,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.771158+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:08:37.771158+0000","last_peered":"2024-09-21T19:08:37.771158+0000","last_clean":"2024-09-21T19:08:37.771158+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:08:37.771158+0000","last_undegraded":"2024-09-21T19:08:37.771158+0000","last_fullsized":"2024-09-21T19:08:37.771158+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":733,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766199+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:08:37.766199+0000","last_peered":"2024-09-21T19:08:37.766199+0000","last_clean":"2024-09-21T19:08:37.766199+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:08:37.766199+0000","last_undegraded":"2024-09-21T19:08:37.766199+0000","last_fullsized":"2024-09-21T19:08:37.766199+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":584,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.747875+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:08:35.747875+0000","last_peered":"2024-09-21T19:08:35.747875+0000","last_clean":"2024-09-21T19:08:35.747875+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:08:35.747875+0000","last_undegraded":"2024-09-21T19:08:35.747875+0000","last_fullsized":"2024-09-21T19:08:35.747875+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":622,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.772142+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:08:37.772142+0000","last_peered":"2024-09-21T19:08:37.772142+0000","last_clean":"2024-09-21T19:08:37.772142+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:08:37.772142+0000","last_undegraded":"2024-09-21T19:08:37.772142+0000","last_fullsized":"2024-09-21T19:08:37.772142+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":694,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.736746+0000","last_change":"2024-09-21T19:05:42.904326+0000","last_active":"2024-09-21T19:08:35.736746+0000","last_peered":"2024-09-21T19:08:35.736746+0000","last_clean":"2024-09-21T19:08:35.736746+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:08:35.736746+0000","last_undegraded":"2024-09-21T19:08:35.736746+0000","last_fullsized":"2024-09-21T19:08:35.736746+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":700,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.752500+0000","last_change":"2024-09-21T19:07:21.428355+0000","last_active":"2024-09-21T19:08:35.752500+0000","last_peered":"2024-09-21T19:08:35.752500+0000","last_clean":"2024-09-21T19:08:35.752500+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:08:35.752500+0000","last_undegraded":"2024-09-21T19:08:35.752500+0000","last_fullsized":"2024-09-21T19:08:35.752500+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:21.428212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:21.428212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.151249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":726,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.766422+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:08:37.766422+0000","last_peered":"2024-09-21T19:08:37.766422+0000","last_clean":"2024-09-21T19:08:37.766422+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:08:37.766422+0000","last_undegraded":"2024-09-21T19:08:37.766422+0000","last_fullsized":"2024-09-21T19:08:37.766422+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":687,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-21T19:08:37.775956+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:08:37.775956+0000","last_peered":"2024-09-21T19:08:37.775956+0000","last_clean":"2024-09-21T19:08:37.775956+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:08:37.775956+0000","last_undegraded":"2024-09-21T19:08:37.775956+0000","last_fullsized":"2024-09-21T19:08:37.775956+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":604,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T19:08:35.756956+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:08:35.756956+0000","last_peered":"2024-09-21T19:08:35.756956+0000","last_clean":"2024-09-21T19:08:35.756956+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:08:35.756956+0000","last_undegraded":"2024-09-21T19:08:35.756956+0000","last_fullsized":"2024-09-21T19:08:35.756956+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4505534488,"num_objects":68750,"num_object_clones":0,"num_object_copies":206250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68750,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166192,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12295,"num_bytes_recovered":805765120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13235728384,"data_stored":13235716168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85592,"ondisk_log_size":85592,"up":48,"acting":44,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":918,"num_read_kb":913,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266878,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728688,"kb_used_data":1129696,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92016464,"statfs":{"total":95995035648,"available":94224859136,"internally_reserved":0,"allocated":1156808704,"data_stored":1156137890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464329,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2191292,"kb_used_data":1912168,"kb_used_omap":4,"kb_used_meta":279099,"kb_avail":91553860,"statfs":{"total":95995035648,"available":93751152640,"internally_reserved":0,"allocated":1958060032,"data_stored":1957387709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":285798052},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619030,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234600,"kb_used_data":1041048,"kb_used_omap":4,"kb_used_meta":193531,"kb_avail":92510552,"statfs":{"total":95995035648,"available":94730805248,"internally_reserved":0,"allocated":1066033152,"data_stored":1065370555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":198176419},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019084,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2794984,"kb_used_data":2276504,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90950168,"statfs":{"total":95995035648,"available":93132972032,"internally_reserved":0,"allocated":2331140096,"data_stored":2330470640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182609,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2096332,"kb_used_data":1383416,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91648820,"statfs":{"total":95995035648,"available":93848391680,"internally_reserved":0,"allocated":1416617984,"data_stored":1415923937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":729938378},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411543,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2218460,"kb_used_data":1673612,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91526692,"statfs":{"total":95995035648,"available":93723332608,"internally_reserved":0,"allocated":1713778688,"data_stored":1713128384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575067,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497752,"kb_used_data":826564,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92247400,"statfs":{"total":95995035648,"available":94461337600,"internally_reserved":0,"allocated":846401536,"data_stored":845694761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738593,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3582140,"kb_used_data":2797096,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":90163012,"statfs":{"total":95995035648,"available":92326924288,"internally_reserved":0,"allocated":2864226304,"data_stored":2863548011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2840727552,"data_stored":2840723480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":841023488,"data_stored":841023488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1691484160,"data_stored":1691484160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1407582208,"data_stored":1407582208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2302808064,"data_stored":2302803992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1061355520,"data_stored":1061355520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1937117184,"data_stored":1937113112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1153630208,"data_stored":1153630208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:08:41.611 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:41.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:41.655 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:41.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:41.773 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:41 smithi081 ceph-mon[29384]: from='client.16419 v1:172.21.15.81:0/3273259948' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:41.809 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:41.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:41.957 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:41.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:42.042 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:41 smithi081 ceph-mon[39192]: from='client.16419 v1:172.21.15.81:0/3273259948' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:42.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.093 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:42.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:42.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:42.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:42.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:41 smithi104 ceph-mon[32438]: from='client.16419 v1:172.21.15.81:0/3273259948' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:42.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:42.363 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:42.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:42.505 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:42.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:42.629 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:42.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:42.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:08:42.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:42.893 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:42.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:42.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:43.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:43.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:43.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[29384]: pgmap v768: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 871 op/s; 11024/205071 objects degraded (5.376%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T19:08:43.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[29384]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T19:08:43.010 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1770529919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:43.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:43.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:43.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[39192]: pgmap v768: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 871 op/s; 11024/205071 objects degraded (5.376%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T19:08:43.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[39192]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T19:08:43.010 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1770529919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:43.023 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:43.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:43.168 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:43.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:42 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:42 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:42 smithi104 ceph-mon[32438]: pgmap v768: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 871 op/s; 11024/205071 objects degraded (5.376%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T19:08:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:42 smithi104 ceph-mon[32438]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T19:08:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1770529919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:43.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:43.301 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:43.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:43.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:43.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:43.566 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:43.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:43.704 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:43.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:43.840 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:43.941 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:43.977 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:43.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:44.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:44.106 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:44.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:44.240 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:44.314 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:44.314 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:44.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.380 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:44.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:44.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:44.515 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:44.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:08:44.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.651 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:44.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:44.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:44.788 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:44.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:44.916 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:44.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:45.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.049 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:45.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:44 smithi081 ceph-mon[29384]: pgmap v770: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 10305/197709 objects degraded (5.212%); 11 MiB/s, 169 objects/s recovering 2024-09-21T19:08:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:44 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 10305/197709 objects degraded (5.212%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:45.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:44 smithi081 ceph-mon[29384]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T19:08:45.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:44 smithi081 ceph-mon[39192]: pgmap v770: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 10305/197709 objects degraded (5.212%); 11 MiB/s, 169 objects/s recovering 2024-09-21T19:08:45.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:44 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 10305/197709 objects degraded (5.212%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:45.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:44 smithi081 ceph-mon[39192]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T19:08:45.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:45.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:44 smithi104 ceph-mon[32438]: pgmap v770: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 10305/197709 objects degraded (5.212%); 11 MiB/s, 169 objects/s recovering 2024-09-21T19:08:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:44 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 10305/197709 objects degraded (5.212%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:44 smithi104 ceph-mon[32438]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T19:08:45.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:45.327 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:45.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:45.460 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:45.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:45.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:45.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:45.707 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:45.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:45.848 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:45.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:45.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:46.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:46.117 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:46.195 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:08:46.196 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:08:46.197 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:08:46.197 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:08:46.197 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:08:46.197 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:08:46.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:46.248 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:46.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:46.383 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:46.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:46.522 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:46.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:46.654 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:46.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:46.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:46.933 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:46.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:47.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:47.044 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:46 smithi081 ceph-mon[39192]: pgmap v772: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 939 op/s; 10305/192297 objects degraded (5.359%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T19:08:47.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:46 smithi081 ceph-mon[29384]: pgmap v772: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 939 op/s; 10305/192297 objects degraded (5.359%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T19:08:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:47.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:08:47.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.205 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:47.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:46 smithi104 ceph-mon[32438]: pgmap v772: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 939 op/s; 10305/192297 objects degraded (5.359%); 9.5 MiB/s, 151 objects/s recovering 2024-09-21T19:08:47.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:47.319 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:08:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:08:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:08:47.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:47.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:47.503 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:47.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:47.620 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:47.692 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.693 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:47.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:47.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.827 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:47.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:47.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:47.962 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:47.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:48.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:48.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:48.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:48.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:48.247 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:48.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:48.376 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:48.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:48.498 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:48.555 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:08:48.557 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:08:48.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:48.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:48.776 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:48.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:08:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:08:48.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:48.918 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:48.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:49.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:49.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:49.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:48 smithi081 ceph-mon[39192]: pgmap v773: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 10148/185637 objects degraded (5.467%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T19:08:49.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:48 smithi081 ceph-mon[29384]: pgmap v773: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 10148/185637 objects degraded (5.467%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T19:08:49.153 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.189 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:49.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:49.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:48 smithi104 ceph-mon[32438]: pgmap v773: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 884 op/s; 10148/185637 objects degraded (5.467%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T19:08:49.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:49.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:49.427 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:49.461 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:49.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:49.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:49.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:08:49.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:49.753 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:49.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:49.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:50.012 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:50.053 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:50.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:49 smithi081 ceph-mon[39192]: from='client.36314 v1:172.21.15.81:0/2979486501' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:50.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:49 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 10148/185637 objects degraded (5.467%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:49 smithi081 ceph-mon[29384]: from='client.36314 v1:172.21.15.81:0/2979486501' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:50.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:49 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 10148/185637 objects degraded (5.467%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:50.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:50.193 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:50.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:08:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:49 smithi104 ceph-mon[32438]: from='client.36314 v1:172.21.15.81:0/2979486501' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:08:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:49 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 10148/185637 objects degraded (5.467%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:50.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:50.328 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:50.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:50.464 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:50.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:50.595 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:50.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:50.738 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:50.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:50.870 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:50.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:50.993 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:50.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:51.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:50 smithi081 ceph-mon[39192]: pgmap v774: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 9463/177192 objects degraded (5.341%); 12 MiB/s, 195 objects/s recovering 2024-09-21T19:08:51.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:50 smithi081 ceph-mon[29384]: pgmap v774: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 9463/177192 objects degraded (5.341%); 12 MiB/s, 195 objects/s recovering 2024-09-21T19:08:51.094 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.131 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:51.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:51.232 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:50 smithi104 ceph-mon[32438]: pgmap v774: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 9463/177192 objects degraded (5.341%); 12 MiB/s, 195 objects/s recovering 2024-09-21T19:08:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.267 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:51.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:51.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.402 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:51.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:51.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:51.535 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:51.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:51.677 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:51.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:51.815 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2024-09-21T19:08:47.750776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4042791475,"num_objects":61879,"num_object_clones":0,"num_object_copies":185637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61879,"num_whiteouts":0,"num_read":2068,"num_read_kb":1820,"num_write":173984,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13406,"num_bytes_recovered":863447288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87321,"ondisk_log_size":87321,"up":474,"acting":470,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15985132,"kb_used_data":11667160,"kb_used_omap":18,"kb_used_meta":4317805,"kb_avail":733976084,"statfs":{"total":767960285184,"available":751591510016,"internally_reserved":0,"allocated":11947171840,"data_stored":11941663681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4421433328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,3,5,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-463339520,"num_objects":-7070,"num_object_clones":0,"num_object_copies":-21210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7070,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":57409536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003029"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":613,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.729753+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:08:44.729753+0000","last_peered":"2024-09-21T19:08:44.729753+0000","last_clean":"2024-09-21T19:08:44.729753+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:08:44.729753+0000","last_undegraded":"2024-09-21T19:08:44.729753+0000","last_fullsized":"2024-09-21T19:08:44.729753+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":781,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713280+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:08:44.713280+0000","last_peered":"2024-09-21T19:08:44.713280+0000","last_clean":"2024-09-21T19:08:44.713280+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:08:44.713280+0000","last_undegraded":"2024-09-21T19:08:44.713280+0000","last_fullsized":"2024-09-21T19:08:44.713280+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":710,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719320+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:08:44.719320+0000","last_peered":"2024-09-21T19:08:44.719320+0000","last_clean":"2024-09-21T19:08:44.719320+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:08:44.719320+0000","last_undegraded":"2024-09-21T19:08:44.719320+0000","last_fullsized":"2024-09-21T19:08:44.719320+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:31.228363+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":698,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713794+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:08:44.713794+0000","last_peered":"2024-09-21T19:08:44.713794+0000","last_clean":"2024-09-21T19:08:44.713794+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:08:44.713794+0000","last_undegraded":"2024-09-21T19:08:44.713794+0000","last_fullsized":"2024-09-21T19:08:44.713794+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":705,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.780935+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:08:41.780935+0000","last_peered":"2024-09-21T19:08:41.780935+0000","last_clean":"2024-09-21T19:08:41.780935+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:08:41.780935+0000","last_undegraded":"2024-09-21T19:08:41.780935+0000","last_fullsized":"2024-09-21T19:08:41.780935+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":732,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.777970+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:08:41.777970+0000","last_peered":"2024-09-21T19:08:41.777970+0000","last_clean":"2024-09-21T19:08:41.777970+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:08:41.777970+0000","last_undegraded":"2024-09-21T19:08:41.777970+0000","last_fullsized":"2024-09-21T19:08:41.777970+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":829,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775633+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:08:41.775633+0000","last_peered":"2024-09-21T19:08:41.775633+0000","last_clean":"2024-09-21T19:08:41.775633+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:08:41.775633+0000","last_undegraded":"2024-09-21T19:08:41.775633+0000","last_fullsized":"2024-09-21T19:08:41.775633+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":718,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713138+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:08:44.713138+0000","last_peered":"2024-09-21T19:08:44.713138+0000","last_clean":"2024-09-21T19:08:44.713138+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:08:44.713138+0000","last_undegraded":"2024-09-21T19:08:44.713138+0000","last_fullsized":"2024-09-21T19:08:44.713138+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":705,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714767+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:08:44.714767+0000","last_peered":"2024-09-21T19:08:44.714767+0000","last_clean":"2024-09-21T19:08:44.714767+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:08:44.714767+0000","last_undegraded":"2024-09-21T19:08:44.714767+0000","last_fullsized":"2024-09-21T19:08:44.714767+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":762,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713935+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:08:44.713935+0000","last_peered":"2024-09-21T19:08:44.713935+0000","last_clean":"2024-09-21T19:08:44.713935+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:08:44.713935+0000","last_undegraded":"2024-09-21T19:08:44.713935+0000","last_fullsized":"2024-09-21T19:08:44.713935+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":656,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.791986+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:08:41.791986+0000","last_peered":"2024-09-21T19:08:41.791986+0000","last_clean":"2024-09-21T19:08:41.791986+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:08:41.791986+0000","last_undegraded":"2024-09-21T19:08:41.791986+0000","last_fullsized":"2024-09-21T19:08:41.791986+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":579,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.726285+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:08:44.726285+0000","last_peered":"2024-09-21T19:08:44.726285+0000","last_clean":"2024-09-21T19:08:44.726285+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:08:44.726285+0000","last_undegraded":"2024-09-21T19:08:44.726285+0000","last_fullsized":"2024-09-21T19:08:44.726285+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":258,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714278+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:08:44.714278+0000","last_peered":"2024-09-21T19:08:44.714278+0000","last_clean":"2024-09-21T19:08:44.714278+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:08:44.714278+0000","last_undegraded":"2024-09-21T19:08:44.714278+0000","last_fullsized":"2024-09-21T19:08:44.714278+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":708,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713200+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:08:44.713200+0000","last_peered":"2024-09-21T19:08:44.713200+0000","last_clean":"2024-09-21T19:08:44.713200+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:08:44.713200+0000","last_undegraded":"2024-09-21T19:08:44.713200+0000","last_fullsized":"2024-09-21T19:08:44.713200+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":690,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721200+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:08:44.721200+0000","last_peered":"2024-09-21T19:08:44.721200+0000","last_clean":"2024-09-21T19:08:44.721200+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:08:44.721200+0000","last_undegraded":"2024-09-21T19:08:44.721200+0000","last_fullsized":"2024-09-21T19:08:44.721200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":266,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.717881+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:08:44.717881+0000","last_peered":"2024-09-21T19:08:44.717881+0000","last_clean":"2024-09-21T19:08:44.717881+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:08:44.717881+0000","last_undegraded":"2024-09-21T19:08:44.717881+0000","last_fullsized":"2024-09-21T19:08:44.717881+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.570833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":726,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714309+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:08:44.714309+0000","last_peered":"2024-09-21T19:08:44.714309+0000","last_clean":"2024-09-21T19:08:44.714309+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:08:44.714309+0000","last_undegraded":"2024-09-21T19:08:44.714309+0000","last_fullsized":"2024-09-21T19:08:44.714309+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":595,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.729981+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:08:44.729981+0000","last_peered":"2024-09-21T19:08:44.729981+0000","last_clean":"2024-09-21T19:08:44.729981+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:08:44.729981+0000","last_undegraded":"2024-09-21T19:08:44.729981+0000","last_fullsized":"2024-09-21T19:08:44.729981+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":167,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.778251+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:08:41.778251+0000","last_peered":"2024-09-21T19:08:41.778251+0000","last_clean":"2024-09-21T19:08:41.778251+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:08:41.778251+0000","last_undegraded":"2024-09-21T19:08:41.778251+0000","last_fullsized":"2024-09-21T19:08:41.778251+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.720658+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:08:44.720658+0000","last_peered":"2024-09-21T19:08:44.720658+0000","last_clean":"2024-09-21T19:08:44.720658+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:08:44.720658+0000","last_undegraded":"2024-09-21T19:08:44.720658+0000","last_fullsized":"2024-09-21T19:08:44.720658+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:02.338670+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":701,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.778342+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:08:41.778342+0000","last_peered":"2024-09-21T19:08:41.778342+0000","last_clean":"2024-09-21T19:08:41.778342+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:08:41.778342+0000","last_undegraded":"2024-09-21T19:08:41.778342+0000","last_fullsized":"2024-09-21T19:08:41.778342+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":729,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713963+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:08:44.713963+0000","last_peered":"2024-09-21T19:08:44.713963+0000","last_clean":"2024-09-21T19:08:44.713963+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:08:44.713963+0000","last_undegraded":"2024-09-21T19:08:44.713963+0000","last_fullsized":"2024-09-21T19:08:44.713963+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":772,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.779594+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:08:41.779594+0000","last_peered":"2024-09-21T19:08:41.779594+0000","last_clean":"2024-09-21T19:08:41.779594+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:08:41.779594+0000","last_undegraded":"2024-09-21T19:08:41.779594+0000","last_fullsized":"2024-09-21T19:08:41.779594+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714073+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:08:44.714073+0000","last_peered":"2024-09-21T19:08:44.714073+0000","last_clean":"2024-09-21T19:08:44.714073+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:08:44.714073+0000","last_undegraded":"2024-09-21T19:08:44.714073+0000","last_fullsized":"2024-09-21T19:08:44.714073+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":736,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721478+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:08:44.721478+0000","last_peered":"2024-09-21T19:08:44.721478+0000","last_clean":"2024-09-21T19:08:44.721478+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:08:44.721478+0000","last_undegraded":"2024-09-21T19:08:44.721478+0000","last_fullsized":"2024-09-21T19:08:44.721478+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":727,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719343+0000","last_change":"2024-09-21T19:07:33.264047+0000","last_active":"2024-09-21T19:08:44.719343+0000","last_peered":"2024-09-21T19:08:44.719343+0000","last_clean":"2024-09-21T19:08:44.719343+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:08:44.719343+0000","last_undegraded":"2024-09-21T19:08:44.719343+0000","last_fullsized":"2024-09-21T19:08:44.719343+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:33.263975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:33.263975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.063323+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":623,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.737518+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:08:44.737518+0000","last_peered":"2024-09-21T19:08:44.737518+0000","last_clean":"2024-09-21T19:08:44.737518+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:08:44.737518+0000","last_undegraded":"2024-09-21T19:08:44.737518+0000","last_fullsized":"2024-09-21T19:08:44.737518+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":783,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.774664+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:08:41.774664+0000","last_peered":"2024-09-21T19:08:41.774664+0000","last_clean":"2024-09-21T19:08:41.774664+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:08:41.774664+0000","last_undegraded":"2024-09-21T19:08:41.774664+0000","last_fullsized":"2024-09-21T19:08:41.774664+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"387'11046","reported_seq":11172,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:46.375875+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:08:46.375875+0000","last_peered":"2024-09-21T19:08:46.375875+0000","last_clean":"2024-09-21T19:08:46.375875+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:08:46.375875+0000","last_undegraded":"2024-09-21T19:08:46.375875+0000","last_fullsized":"2024-09-21T19:08:46.375875+0000","mapping_epoch":330,"log_start":"363'6500","ondisk_log_start":"363'6500","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251265024,"num_objects":3834,"num_object_clones":0,"num_object_copies":11502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":162,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.777881+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:08:41.777881+0000","last_peered":"2024-09-21T19:08:41.777881+0000","last_clean":"2024-09-21T19:08:41.777881+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:08:41.777881+0000","last_undegraded":"2024-09-21T19:08:41.777881+0000","last_fullsized":"2024-09-21T19:08:41.777881+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":762,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.720729+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:08:44.720729+0000","last_peered":"2024-09-21T19:08:44.720729+0000","last_clean":"2024-09-21T19:08:44.720729+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:08:44.720729+0000","last_undegraded":"2024-09-21T19:08:44.720729+0000","last_fullsized":"2024-09-21T19:08:44.720729+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:35.238469+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":699,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775367+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:08:41.775367+0000","last_peered":"2024-09-21T19:08:41.775367+0000","last_clean":"2024-09-21T19:08:41.775367+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:08:41.775367+0000","last_undegraded":"2024-09-21T19:08:41.775367+0000","last_fullsized":"2024-09-21T19:08:41.775367+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":938,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714509+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:08:44.714509+0000","last_peered":"2024-09-21T19:08:44.714509+0000","last_clean":"2024-09-21T19:08:44.714509+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:08:44.714509+0000","last_undegraded":"2024-09-21T19:08:44.714509+0000","last_fullsized":"2024-09-21T19:08:44.714509+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":390,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.720568+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:08:44.720568+0000","last_peered":"2024-09-21T19:08:44.720568+0000","last_clean":"2024-09-21T19:08:44.720568+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:08:44.720568+0000","last_undegraded":"2024-09-21T19:08:44.720568+0000","last_fullsized":"2024-09-21T19:08:44.720568+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"387'10704","reported_seq":10828,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:46.361427+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:08:46.361427+0000","last_peered":"2024-09-21T19:08:46.361427+0000","last_clean":"2024-09-21T19:08:46.361427+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:08:46.361427+0000","last_undegraded":"2024-09-21T19:08:46.361427+0000","last_fullsized":"2024-09-21T19:08:46.361427+0000","mapping_epoch":330,"log_start":"363'6200","ondisk_log_start":"363'6200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":489,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714766+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:08:44.714766+0000","last_peered":"2024-09-21T19:08:44.714766+0000","last_clean":"2024-09-21T19:08:44.714766+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:08:44.714766+0000","last_undegraded":"2024-09-21T19:08:44.714766+0000","last_fullsized":"2024-09-21T19:08:44.714766+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":795,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.774737+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:08:41.774737+0000","last_peered":"2024-09-21T19:08:41.774737+0000","last_clean":"2024-09-21T19:08:41.774737+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:08:41.774737+0000","last_undegraded":"2024-09-21T19:08:41.774737+0000","last_fullsized":"2024-09-21T19:08:41.774737+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":721,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.779423+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:08:41.779423+0000","last_peered":"2024-09-21T19:08:41.779423+0000","last_clean":"2024-09-21T19:08:41.779423+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:08:41.779423+0000","last_undegraded":"2024-09-21T19:08:41.779423+0000","last_fullsized":"2024-09-21T19:08:41.779423+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":700,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713341+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:08:44.713341+0000","last_peered":"2024-09-21T19:08:44.713341+0000","last_clean":"2024-09-21T19:08:44.713341+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:08:44.713341+0000","last_undegraded":"2024-09-21T19:08:44.713341+0000","last_fullsized":"2024-09-21T19:08:44.713341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":717,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713884+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:08:44.713884+0000","last_peered":"2024-09-21T19:08:44.713884+0000","last_clean":"2024-09-21T19:08:44.713884+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:08:44.713884+0000","last_undegraded":"2024-09-21T19:08:44.713884+0000","last_fullsized":"2024-09-21T19:08:44.713884+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"386'10761","reported_seq":10885,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.158362+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:08:44.158362+0000","last_peered":"2024-09-21T19:08:44.158362+0000","last_clean":"2024-09-21T19:08:44.158362+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:08:44.158362+0000","last_undegraded":"2024-09-21T19:08:44.158362+0000","last_fullsized":"2024-09-21T19:08:44.158362+0000","mapping_epoch":330,"log_start":"355'5400","ondisk_log_start":"355'5400","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258146304,"num_objects":3939,"num_object_clones":0,"num_object_copies":11817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10761,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":478,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775564+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:08:41.775564+0000","last_peered":"2024-09-21T19:08:41.775564+0000","last_clean":"2024-09-21T19:08:41.775564+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:08:41.775564+0000","last_undegraded":"2024-09-21T19:08:41.775564+0000","last_fullsized":"2024-09-21T19:08:41.775564+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":768,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713459+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:08:44.713459+0000","last_peered":"2024-09-21T19:08:44.713459+0000","last_clean":"2024-09-21T19:08:44.713459+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:08:44.713459+0000","last_undegraded":"2024-09-21T19:08:44.713459+0000","last_fullsized":"2024-09-21T19:08:44.713459+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":591,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.737767+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:08:44.737767+0000","last_peered":"2024-09-21T19:08:44.737767+0000","last_clean":"2024-09-21T19:08:44.737767+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:08:44.737767+0000","last_undegraded":"2024-09-21T19:08:44.737767+0000","last_fullsized":"2024-09-21T19:08:44.737767+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":879,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.774583+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:08:41.774583+0000","last_peered":"2024-09-21T19:08:41.774583+0000","last_clean":"2024-09-21T19:08:41.774583+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:08:41.774583+0000","last_undegraded":"2024-09-21T19:08:41.774583+0000","last_fullsized":"2024-09-21T19:08:41.774583+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":720,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721667+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:08:44.721667+0000","last_peered":"2024-09-21T19:08:44.721667+0000","last_clean":"2024-09-21T19:08:44.721667+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:08:44.721667+0000","last_undegraded":"2024-09-21T19:08:44.721667+0000","last_fullsized":"2024-09-21T19:08:44.721667+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"387'11121","reported_seq":11247,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:46.831542+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:08:46.831542+0000","last_peered":"2024-09-21T19:08:46.831542+0000","last_clean":"2024-09-21T19:08:46.831542+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:08:46.831542+0000","last_undegraded":"2024-09-21T19:08:46.831542+0000","last_fullsized":"2024-09-21T19:08:46.831542+0000","mapping_epoch":330,"log_start":"361'6100","ondisk_log_start":"361'6100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250150912,"num_objects":3817,"num_object_clones":0,"num_object_copies":11451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11121,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":502,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.774817+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:08:41.774817+0000","last_peered":"2024-09-21T19:08:41.774817+0000","last_clean":"2024-09-21T19:08:41.774817+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:08:41.774817+0000","last_undegraded":"2024-09-21T19:08:41.774817+0000","last_fullsized":"2024-09-21T19:08:41.774817+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":745,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722158+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:08:44.722158+0000","last_peered":"2024-09-21T19:08:44.722158+0000","last_clean":"2024-09-21T19:08:44.722158+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:08:44.722158+0000","last_undegraded":"2024-09-21T19:08:44.722158+0000","last_fullsized":"2024-09-21T19:08:44.722158+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:33.202656+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":262,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.779679+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:08:41.779679+0000","last_peered":"2024-09-21T19:08:41.779679+0000","last_clean":"2024-09-21T19:08:41.779679+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:08:41.779679+0000","last_undegraded":"2024-09-21T19:08:41.779679+0000","last_fullsized":"2024-09-21T19:08:41.779679+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":766,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.729971+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:08:44.729971+0000","last_peered":"2024-09-21T19:08:44.729971+0000","last_clean":"2024-09-21T19:08:44.729971+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:08:44.729971+0000","last_undegraded":"2024-09-21T19:08:44.729971+0000","last_fullsized":"2024-09-21T19:08:44.729971+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":594,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.726407+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:08:44.726407+0000","last_peered":"2024-09-21T19:08:44.726407+0000","last_clean":"2024-09-21T19:08:44.726407+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:08:44.726407+0000","last_undegraded":"2024-09-21T19:08:44.726407+0000","last_fullsized":"2024-09-21T19:08:44.726407+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"386'10674","reported_seq":10799,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.230971+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:08:44.230971+0000","last_peered":"2024-09-21T19:08:44.230971+0000","last_clean":"2024-09-21T19:08:44.230971+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:08:44.230971+0000","last_undegraded":"2024-09-21T19:08:44.230971+0000","last_fullsized":"2024-09-21T19:08:44.230971+0000","mapping_epoch":330,"log_start":"355'5400","ondisk_log_start":"355'5400","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262209560,"num_objects":4002,"num_object_clones":0,"num_object_copies":12006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10674,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":480,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.772603+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:08:41.772603+0000","last_peered":"2024-09-21T19:08:41.772603+0000","last_clean":"2024-09-21T19:08:41.772603+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:08:41.772603+0000","last_undegraded":"2024-09-21T19:08:41.772603+0000","last_fullsized":"2024-09-21T19:08:41.772603+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":640,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.730015+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:08:44.730015+0000","last_peered":"2024-09-21T19:08:44.730015+0000","last_clean":"2024-09-21T19:08:44.730015+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:08:44.730015+0000","last_undegraded":"2024-09-21T19:08:44.730015+0000","last_fullsized":"2024-09-21T19:08:44.730015+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":733,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719820+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:08:44.719820+0000","last_peered":"2024-09-21T19:08:44.719820+0000","last_clean":"2024-09-21T19:08:44.719820+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:08:44.719820+0000","last_undegraded":"2024-09-21T19:08:44.719820+0000","last_fullsized":"2024-09-21T19:08:44.719820+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:34.228030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":615,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.772910+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:08:41.772910+0000","last_peered":"2024-09-21T19:08:41.772910+0000","last_clean":"2024-09-21T19:08:41.772910+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:08:41.772910+0000","last_undegraded":"2024-09-21T19:08:41.772910+0000","last_fullsized":"2024-09-21T19:08:41.772910+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":721,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719246+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:08:44.719246+0000","last_peered":"2024-09-21T19:08:44.719246+0000","last_clean":"2024-09-21T19:08:44.719246+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:08:44.719246+0000","last_undegraded":"2024-09-21T19:08:44.719246+0000","last_fullsized":"2024-09-21T19:08:44.719246+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.253514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"386'10874","reported_seq":20365,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.221511+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:08:44.221511+0000","last_peered":"2024-09-21T19:08:44.221511+0000","last_clean":"2024-09-21T19:08:44.221511+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:08:44.221511+0000","last_undegraded":"2024-09-21T19:08:44.221511+0000","last_fullsized":"2024-09-21T19:08:44.221511+0000","mapping_epoch":380,"log_start":"355'5600","ondisk_log_start":"355'5600","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260571136,"num_objects":3976,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":458,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.725861+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:08:44.725861+0000","last_peered":"2024-09-21T19:08:44.725861+0000","last_clean":"2024-09-21T19:08:44.725861+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:08:44.725861+0000","last_undegraded":"2024-09-21T19:08:44.725861+0000","last_fullsized":"2024-09-21T19:08:44.725861+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1010,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719732+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:08:44.719732+0000","last_peered":"2024-09-21T19:08:44.719732+0000","last_clean":"2024-09-21T19:08:44.719732+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:08:44.719732+0000","last_undegraded":"2024-09-21T19:08:44.719732+0000","last_fullsized":"2024-09-21T19:08:44.719732+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:45.305870+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":741,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713507+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:08:44.713507+0000","last_peered":"2024-09-21T19:08:44.713507+0000","last_clean":"2024-09-21T19:08:44.713507+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:08:44.713507+0000","last_undegraded":"2024-09-21T19:08:44.713507+0000","last_fullsized":"2024-09-21T19:08:44.713507+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":288,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722423+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:08:44.722423+0000","last_peered":"2024-09-21T19:08:44.722423+0000","last_clean":"2024-09-21T19:08:44.722423+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:08:44.722423+0000","last_undegraded":"2024-09-21T19:08:44.722423+0000","last_fullsized":"2024-09-21T19:08:44.722423+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:25.256087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":580,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.730798+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:08:44.730798+0000","last_peered":"2024-09-21T19:08:44.730798+0000","last_clean":"2024-09-21T19:08:44.730798+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:08:44.730798+0000","last_undegraded":"2024-09-21T19:08:44.730798+0000","last_fullsized":"2024-09-21T19:08:44.730798+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":687,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.777699+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:08:41.777699+0000","last_peered":"2024-09-21T19:08:41.777699+0000","last_clean":"2024-09-21T19:08:41.777699+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:08:41.777699+0000","last_undegraded":"2024-09-21T19:08:41.777699+0000","last_fullsized":"2024-09-21T19:08:41.777699+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"386'10636","reported_seq":18455,"reported_epoch":386,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:42.789340+0000","last_change":"2024-09-21T19:06:55.907554+0000","last_active":"2024-09-21T19:08:42.789340+0000","last_peered":"2024-09-21T19:08:42.789340+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.870419+0000","last_became_peered":"2024-09-21T19:06:55.870419+0000","last_unstale":"2024-09-21T19:08:42.789340+0000","last_undegraded":"2024-09-21T19:06:55.799686+0000","last_fullsized":"2024-09-21T19:06:55.681444+0000","mapping_epoch":349,"log_start":"344'4200","ondisk_log_start":"344'4200","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10636,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3847,"num_bytes_recovered":252116992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":3168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":785,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721029+0000","last_change":"2024-09-21T19:02:16.837235+0000","last_active":"2024-09-21T19:08:44.721029+0000","last_peered":"2024-09-21T19:08:44.721029+0000","last_clean":"2024-09-21T19:08:44.721029+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:08:44.721029+0000","last_undegraded":"2024-09-21T19:08:44.721029+0000","last_fullsized":"2024-09-21T19:08:44.721029+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:16.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:16.837162+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":206,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.720807+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:08:44.720807+0000","last_peered":"2024-09-21T19:08:44.720807+0000","last_clean":"2024-09-21T19:08:44.720807+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:08:44.720807+0000","last_undegraded":"2024-09-21T19:08:44.720807+0000","last_fullsized":"2024-09-21T19:08:44.720807+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":755,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.718020+0000","last_change":"2024-09-21T19:07:29.296652+0000","last_active":"2024-09-21T19:08:44.718020+0000","last_peered":"2024-09-21T19:08:44.718020+0000","last_clean":"2024-09-21T19:08:44.718020+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:08:44.718020+0000","last_undegraded":"2024-09-21T19:08:44.718020+0000","last_fullsized":"2024-09-21T19:08:44.718020+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:07:29.296562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:07:29.296562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:51.422184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":155,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.784760+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:08:41.784760+0000","last_peered":"2024-09-21T19:08:41.784760+0000","last_clean":"2024-09-21T19:08:41.784760+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:08:41.784760+0000","last_undegraded":"2024-09-21T19:08:41.784760+0000","last_fullsized":"2024-09-21T19:08:41.784760+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":154,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.781108+0000","last_change":"2024-09-21T19:06:54.694760+0000","last_active":"2024-09-21T19:08:41.781108+0000","last_peered":"2024-09-21T19:08:41.781108+0000","last_clean":"2024-09-21T19:08:41.781108+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:08:41.781108+0000","last_undegraded":"2024-09-21T19:08:41.781108+0000","last_fullsized":"2024-09-21T19:08:41.781108+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":710,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721476+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:08:44.721476+0000","last_peered":"2024-09-21T19:08:44.721476+0000","last_clean":"2024-09-21T19:08:44.721476+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:08:44.721476+0000","last_undegraded":"2024-09-21T19:08:44.721476+0000","last_fullsized":"2024-09-21T19:08:44.721476+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"387'11022","reported_seq":11148,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:47.650222+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:08:47.650222+0000","last_peered":"2024-09-21T19:08:47.650222+0000","last_clean":"2024-09-21T19:08:47.650222+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:08:47.650222+0000","last_undegraded":"2024-09-21T19:08:47.650222+0000","last_fullsized":"2024-09-21T19:08:47.650222+0000","mapping_epoch":330,"log_start":"361'6000","ondisk_log_start":"361'6000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243400704,"num_objects":3714,"num_object_clones":0,"num_object_copies":11142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":491,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719919+0000","last_change":"2024-09-21T19:07:30.274632+0000","last_active":"2024-09-21T19:08:44.719919+0000","last_peered":"2024-09-21T19:08:44.719919+0000","last_clean":"2024-09-21T19:08:44.719919+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:08:44.719919+0000","last_undegraded":"2024-09-21T19:08:44.719919+0000","last_fullsized":"2024-09-21T19:08:44.719919+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:07:30.274412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:07:30.274412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:58.929542+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":701,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.737606+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:08:44.737606+0000","last_peered":"2024-09-21T19:08:44.737606+0000","last_clean":"2024-09-21T19:08:44.737606+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:08:44.737606+0000","last_undegraded":"2024-09-21T19:08:44.737606+0000","last_fullsized":"2024-09-21T19:08:44.737606+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":588,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.726179+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:08:44.726179+0000","last_peered":"2024-09-21T19:08:44.726179+0000","last_clean":"2024-09-21T19:08:44.726179+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:08:44.726179+0000","last_undegraded":"2024-09-21T19:08:44.726179+0000","last_fullsized":"2024-09-21T19:08:44.726179+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1665,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:45.713881+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:08:45.713881+0000","last_peered":"2024-09-21T19:08:45.713881+0000","last_clean":"2024-09-21T19:08:45.713881+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:08:45.713881+0000","last_undegraded":"2024-09-21T19:08:45.713881+0000","last_fullsized":"2024-09-21T19:08:45.713881+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":928,"num_read_kb":923,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":657,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.790642+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:08:41.790642+0000","last_peered":"2024-09-21T19:08:41.790642+0000","last_clean":"2024-09-21T19:08:41.790642+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:08:41.790642+0000","last_undegraded":"2024-09-21T19:08:41.790642+0000","last_fullsized":"2024-09-21T19:08:41.790642+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":715,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719315+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:08:44.719315+0000","last_peered":"2024-09-21T19:08:44.719315+0000","last_clean":"2024-09-21T19:08:44.719315+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:08:44.719315+0000","last_undegraded":"2024-09-21T19:08:44.719315+0000","last_fullsized":"2024-09-21T19:08:44.719315+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:30.010503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"386'10763","reported_seq":10887,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.245456+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:08:44.245456+0000","last_peered":"2024-09-21T19:08:44.245456+0000","last_clean":"2024-09-21T19:08:44.245456+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:08:44.245456+0000","last_undegraded":"2024-09-21T19:08:44.245456+0000","last_fullsized":"2024-09-21T19:08:44.245456+0000","mapping_epoch":330,"log_start":"355'5400","ondisk_log_start":"355'5400","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254214144,"num_objects":3879,"num_object_clones":0,"num_object_copies":11637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":874,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.772069+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:08:41.772069+0000","last_peered":"2024-09-21T19:08:41.772069+0000","last_clean":"2024-09-21T19:08:41.772069+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:08:41.772069+0000","last_undegraded":"2024-09-21T19:08:41.772069+0000","last_fullsized":"2024-09-21T19:08:41.772069+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":721,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775379+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:08:41.775379+0000","last_peered":"2024-09-21T19:08:41.775379+0000","last_clean":"2024-09-21T19:08:41.775379+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:08:41.775379+0000","last_undegraded":"2024-09-21T19:08:41.775379+0000","last_fullsized":"2024-09-21T19:08:41.775379+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":799,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721676+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:08:44.721676+0000","last_peered":"2024-09-21T19:08:44.721676+0000","last_clean":"2024-09-21T19:08:44.721676+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:08:44.721676+0000","last_undegraded":"2024-09-21T19:08:44.721676+0000","last_fullsized":"2024-09-21T19:08:44.721676+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":724,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721553+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:08:44.721553+0000","last_peered":"2024-09-21T19:08:44.721553+0000","last_clean":"2024-09-21T19:08:44.721553+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:08:44.721553+0000","last_undegraded":"2024-09-21T19:08:44.721553+0000","last_fullsized":"2024-09-21T19:08:44.721553+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":248,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.778270+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:08:41.778270+0000","last_peered":"2024-09-21T19:08:41.778270+0000","last_clean":"2024-09-21T19:08:41.778270+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:08:41.778270+0000","last_undegraded":"2024-09-21T19:08:41.778270+0000","last_fullsized":"2024-09-21T19:08:41.778270+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"386'10641","reported_seq":10765,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.216709+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:08:44.216709+0000","last_peered":"2024-09-21T19:08:44.216709+0000","last_clean":"2024-09-21T19:08:44.216709+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:08:44.216709+0000","last_undegraded":"2024-09-21T19:08:44.216709+0000","last_fullsized":"2024-09-21T19:08:44.216709+0000","mapping_epoch":330,"log_start":"355'5300","ondisk_log_start":"355'5300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249233408,"num_objects":3803,"num_object_clones":0,"num_object_copies":11409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10641,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":761,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714632+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:08:44.714632+0000","last_peered":"2024-09-21T19:08:44.714632+0000","last_clean":"2024-09-21T19:08:44.714632+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:08:44.714632+0000","last_undegraded":"2024-09-21T19:08:44.714632+0000","last_fullsized":"2024-09-21T19:08:44.714632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":730,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713640+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:08:44.713640+0000","last_peered":"2024-09-21T19:08:44.713640+0000","last_clean":"2024-09-21T19:08:44.713640+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:08:44.713640+0000","last_undegraded":"2024-09-21T19:08:44.713640+0000","last_fullsized":"2024-09-21T19:08:44.713640+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":714,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714383+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:08:44.714383+0000","last_peered":"2024-09-21T19:08:44.714383+0000","last_clean":"2024-09-21T19:08:44.714383+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:08:44.714383+0000","last_undegraded":"2024-09-21T19:08:44.714383+0000","last_fullsized":"2024-09-21T19:08:44.714383+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":692,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775826+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:08:41.775826+0000","last_peered":"2024-09-21T19:08:41.775826+0000","last_clean":"2024-09-21T19:08:41.775826+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:08:41.775826+0000","last_undegraded":"2024-09-21T19:08:41.775826+0000","last_fullsized":"2024-09-21T19:08:41.775826+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"386'10707","reported_seq":18423,"reported_epoch":386,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:42.766253+0000","last_change":"2024-09-21T19:06:55.916390+0000","last_active":"2024-09-21T19:08:42.766253+0000","last_peered":"2024-09-21T19:08:42.766253+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.879505+0000","last_became_peered":"2024-09-21T19:06:55.879505+0000","last_unstale":"2024-09-21T19:08:42.766253+0000","last_undegraded":"2024-09-21T19:06:55.764320+0000","last_fullsized":"2024-09-21T19:06:55.681165+0000","mapping_epoch":349,"log_start":"345'4300","ondisk_log_start":"345'4300","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261292032,"num_objects":3987,"num_object_clones":0,"num_object_copies":11961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10707,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3796,"num_bytes_recovered":248774656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3121}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":158,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.780557+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:08:41.780557+0000","last_peered":"2024-09-21T19:08:41.780557+0000","last_clean":"2024-09-21T19:08:41.780557+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:08:41.780557+0000","last_undegraded":"2024-09-21T19:08:41.780557+0000","last_fullsized":"2024-09-21T19:08:41.780557+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":810,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714628+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:08:44.714628+0000","last_peered":"2024-09-21T19:08:44.714628+0000","last_clean":"2024-09-21T19:08:44.714628+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:08:44.714628+0000","last_undegraded":"2024-09-21T19:08:44.714628+0000","last_fullsized":"2024-09-21T19:08:44.714628+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":626,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.773144+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:08:41.773144+0000","last_peered":"2024-09-21T19:08:41.773144+0000","last_clean":"2024-09-21T19:08:41.773144+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:08:41.773144+0000","last_undegraded":"2024-09-21T19:08:41.773144+0000","last_fullsized":"2024-09-21T19:08:41.773144+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":151,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.771695+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:08:41.771695+0000","last_peered":"2024-09-21T19:08:41.771695+0000","last_clean":"2024-09-21T19:08:41.771695+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:08:41.771695+0000","last_undegraded":"2024-09-21T19:08:41.771695+0000","last_fullsized":"2024-09-21T19:08:41.771695+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":151,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.782142+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:08:41.782142+0000","last_peered":"2024-09-21T19:08:41.782142+0000","last_clean":"2024-09-21T19:08:41.782142+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:08:41.782142+0000","last_undegraded":"2024-09-21T19:08:41.782142+0000","last_fullsized":"2024-09-21T19:08:41.782142+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"387'11040","reported_seq":11166,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:47.581708+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:08:47.581708+0000","last_peered":"2024-09-21T19:08:47.581708+0000","last_clean":"2024-09-21T19:08:47.581708+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:08:47.581708+0000","last_undegraded":"2024-09-21T19:08:47.581708+0000","last_fullsized":"2024-09-21T19:08:47.581708+0000","mapping_epoch":330,"log_start":"359'5800","ondisk_log_start":"359'5800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:46.323337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242221056,"num_objects":3696,"num_object_clones":0,"num_object_copies":11088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":807,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.723274+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:08:44.723274+0000","last_peered":"2024-09-21T19:08:44.723274+0000","last_clean":"2024-09-21T19:08:44.723274+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:08:44.723274+0000","last_undegraded":"2024-09-21T19:08:44.723274+0000","last_fullsized":"2024-09-21T19:08:44.723274+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":736,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714136+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:08:44.714136+0000","last_peered":"2024-09-21T19:08:44.714136+0000","last_clean":"2024-09-21T19:08:44.714136+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:08:44.714136+0000","last_undegraded":"2024-09-21T19:08:44.714136+0000","last_fullsized":"2024-09-21T19:08:44.714136+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":734,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.782876+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:08:41.782876+0000","last_peered":"2024-09-21T19:08:41.782876+0000","last_clean":"2024-09-21T19:08:41.782876+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:08:41.782876+0000","last_undegraded":"2024-09-21T19:08:41.782876+0000","last_fullsized":"2024-09-21T19:08:41.782876+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":197,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.778073+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:08:41.778073+0000","last_peered":"2024-09-21T19:08:41.778073+0000","last_clean":"2024-09-21T19:08:41.778073+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:08:41.778073+0000","last_undegraded":"2024-09-21T19:08:41.778073+0000","last_fullsized":"2024-09-21T19:08:41.778073+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":709,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714231+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:08:44.714231+0000","last_peered":"2024-09-21T19:08:44.714231+0000","last_clean":"2024-09-21T19:08:44.714231+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:08:44.714231+0000","last_undegraded":"2024-09-21T19:08:44.714231+0000","last_fullsized":"2024-09-21T19:08:44.714231+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"386'10467","reported_seq":10591,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:42.776792+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:08:42.776792+0000","last_peered":"2024-09-21T19:08:42.776792+0000","last_clean":"2024-09-21T19:08:42.776792+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:08:42.776792+0000","last_undegraded":"2024-09-21T19:08:42.776792+0000","last_fullsized":"2024-09-21T19:08:42.776792+0000","mapping_epoch":330,"log_start":"343'4000","ondisk_log_start":"343'4000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6467,"log_dups_size":0,"ondisk_log_size":6467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":11937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10467,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":598,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.730066+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:08:44.730066+0000","last_peered":"2024-09-21T19:08:44.730066+0000","last_clean":"2024-09-21T19:08:44.730066+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:08:44.730066+0000","last_undegraded":"2024-09-21T19:08:44.730066+0000","last_fullsized":"2024-09-21T19:08:44.730066+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":696,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.776296+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:08:41.776296+0000","last_peered":"2024-09-21T19:08:41.776296+0000","last_clean":"2024-09-21T19:08:41.776296+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:08:41.776296+0000","last_undegraded":"2024-09-21T19:08:41.776296+0000","last_fullsized":"2024-09-21T19:08:41.776296+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":143,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.777801+0000","last_change":"2024-09-21T19:05:42.909550+0000","last_active":"2024-09-21T19:08:41.777801+0000","last_peered":"2024-09-21T19:08:41.777801+0000","last_clean":"2024-09-21T19:08:41.777801+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:08:41.777801+0000","last_undegraded":"2024-09-21T19:08:41.777801+0000","last_fullsized":"2024-09-21T19:08:41.777801+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":593,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.726102+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:08:44.726102+0000","last_peered":"2024-09-21T19:08:44.726102+0000","last_clean":"2024-09-21T19:08:44.726102+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:08:44.726102+0000","last_undegraded":"2024-09-21T19:08:44.726102+0000","last_fullsized":"2024-09-21T19:08:44.726102+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"387'10847","reported_seq":10973,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:47.681012+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:08:47.681012+0000","last_peered":"2024-09-21T19:08:47.681012+0000","last_clean":"2024-09-21T19:08:47.681012+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:08:47.681012+0000","last_undegraded":"2024-09-21T19:08:47.681012+0000","last_fullsized":"2024-09-21T19:08:47.681012+0000","mapping_epoch":330,"log_start":"359'5800","ondisk_log_start":"359'5800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5047,"log_dups_size":0,"ondisk_log_size":5047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241369088,"num_objects":3683,"num_object_clones":0,"num_object_copies":11049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":645,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.773068+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:08:41.773068+0000","last_peered":"2024-09-21T19:08:41.773068+0000","last_clean":"2024-09-21T19:08:41.773068+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:08:41.773068+0000","last_undegraded":"2024-09-21T19:08:41.773068+0000","last_fullsized":"2024-09-21T19:08:41.773068+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":728,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722127+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:08:44.722127+0000","last_peered":"2024-09-21T19:08:44.722127+0000","last_clean":"2024-09-21T19:08:44.722127+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:08:44.722127+0000","last_undegraded":"2024-09-21T19:08:44.722127+0000","last_fullsized":"2024-09-21T19:08:44.722127+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":154,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.779983+0000","last_change":"2024-09-21T19:06:54.697181+0000","last_active":"2024-09-21T19:08:41.779983+0000","last_peered":"2024-09-21T19:08:41.779983+0000","last_clean":"2024-09-21T19:08:41.779983+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:08:41.779983+0000","last_undegraded":"2024-09-21T19:08:41.779983+0000","last_fullsized":"2024-09-21T19:08:41.779983+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.357087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.357087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":701,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721668+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:08:44.721668+0000","last_peered":"2024-09-21T19:08:44.721668+0000","last_clean":"2024-09-21T19:08:44.721668+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:08:44.721668+0000","last_undegraded":"2024-09-21T19:08:44.721668+0000","last_fullsized":"2024-09-21T19:08:44.721668+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"386'10743","reported_seq":10875,"reported_epoch":386,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:44.229887+0000","last_change":"2024-09-21T19:06:55.921138+0000","last_active":"2024-09-21T19:08:44.229887+0000","last_peered":"2024-09-21T19:08:44.229887+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:08:44.229887+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"355'5400","ondisk_log_start":"355'5400","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5343,"log_dups_size":0,"ondisk_log_size":5343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261160960,"num_objects":3985,"num_object_clones":0,"num_object_copies":11955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-604}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":766,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713594+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:08:44.713594+0000","last_peered":"2024-09-21T19:08:44.713594+0000","last_clean":"2024-09-21T19:08:44.713594+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:08:44.713594+0000","last_undegraded":"2024-09-21T19:08:44.713594+0000","last_fullsized":"2024-09-21T19:08:44.713594+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":722,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722074+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:08:44.722074+0000","last_peered":"2024-09-21T19:08:44.722074+0000","last_clean":"2024-09-21T19:08:44.722074+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:08:44.722074+0000","last_undegraded":"2024-09-21T19:08:44.722074+0000","last_fullsized":"2024-09-21T19:08:44.722074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":143,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.780478+0000","last_change":"2024-09-21T19:05:42.902451+0000","last_active":"2024-09-21T19:08:41.780478+0000","last_peered":"2024-09-21T19:08:41.780478+0000","last_clean":"2024-09-21T19:08:41.780478+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:08:41.780478+0000","last_undegraded":"2024-09-21T19:08:41.780478+0000","last_fullsized":"2024-09-21T19:08:41.780478+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:48.464805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:01:48.464805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":686,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.781187+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:08:41.781187+0000","last_peered":"2024-09-21T19:08:41.781187+0000","last_clean":"2024-09-21T19:08:41.781187+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:08:41.781187+0000","last_undegraded":"2024-09-21T19:08:41.781187+0000","last_fullsized":"2024-09-21T19:08:41.781187+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"387'11216","reported_seq":13033,"reported_epoch":387,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:08:47.668396+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:08:47.668396+0000","last_peered":"2024-09-21T19:08:47.668396+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:08:47.668396+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"361'6200","ondisk_log_start":"361'6200","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242089984,"num_objects":3694,"num_object_clones":0,"num_object_copies":11082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":846,"num_bytes_recovered":55443456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":-209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":321,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721218+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:08:44.721218+0000","last_peered":"2024-09-21T19:08:44.721218+0000","last_clean":"2024-09-21T19:08:44.721218+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:08:44.721218+0000","last_undegraded":"2024-09-21T19:08:44.721218+0000","last_fullsized":"2024-09-21T19:08:44.721218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":736,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722289+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:08:44.722289+0000","last_peered":"2024-09-21T19:08:44.722289+0000","last_clean":"2024-09-21T19:08:44.722289+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:08:44.722289+0000","last_undegraded":"2024-09-21T19:08:44.722289+0000","last_fullsized":"2024-09-21T19:08:44.722289+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":193,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775069+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:08:41.775069+0000","last_peered":"2024-09-21T19:08:41.775069+0000","last_clean":"2024-09-21T19:08:41.775069+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:08:41.775069+0000","last_undegraded":"2024-09-21T19:08:41.775069+0000","last_fullsized":"2024-09-21T19:08:41.775069+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":722,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.772927+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:08:41.772927+0000","last_peered":"2024-09-21T19:08:41.772927+0000","last_clean":"2024-09-21T19:08:41.772927+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:08:41.772927+0000","last_undegraded":"2024-09-21T19:08:41.772927+0000","last_fullsized":"2024-09-21T19:08:41.772927+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":657,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.772574+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:08:41.772574+0000","last_peered":"2024-09-21T19:08:41.772574+0000","last_clean":"2024-09-21T19:08:41.772574+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:08:41.772574+0000","last_undegraded":"2024-09-21T19:08:41.772574+0000","last_fullsized":"2024-09-21T19:08:41.772574+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":720,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.779260+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:08:41.779260+0000","last_peered":"2024-09-21T19:08:41.779260+0000","last_clean":"2024-09-21T19:08:41.779260+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:08:41.779260+0000","last_undegraded":"2024-09-21T19:08:41.779260+0000","last_fullsized":"2024-09-21T19:08:41.779260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":256,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.777602+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:08:41.777602+0000","last_peered":"2024-09-21T19:08:41.777602+0000","last_clean":"2024-09-21T19:08:41.777602+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:08:41.777602+0000","last_undegraded":"2024-09-21T19:08:41.777602+0000","last_fullsized":"2024-09-21T19:08:41.777602+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":689,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721392+0000","last_change":"2024-09-21T19:02:41.040381+0000","last_active":"2024-09-21T19:08:44.721392+0000","last_peered":"2024-09-21T19:08:44.721392+0000","last_clean":"2024-09-21T19:08:44.721392+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:08:44.721392+0000","last_undegraded":"2024-09-21T19:08:44.721392+0000","last_fullsized":"2024-09-21T19:08:44.721392+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:41.040334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:41.040334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":598,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.737687+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:08:44.737687+0000","last_peered":"2024-09-21T19:08:44.737687+0000","last_clean":"2024-09-21T19:08:44.737687+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:08:44.737687+0000","last_undegraded":"2024-09-21T19:08:44.737687+0000","last_fullsized":"2024-09-21T19:08:44.737687+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":709,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.723627+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:08:44.723627+0000","last_peered":"2024-09-21T19:08:44.723627+0000","last_clean":"2024-09-21T19:08:44.723627+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:08:44.723627+0000","last_undegraded":"2024-09-21T19:08:44.723627+0000","last_fullsized":"2024-09-21T19:08:44.723627+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":690,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.778984+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:08:41.778984+0000","last_peered":"2024-09-21T19:08:41.778984+0000","last_clean":"2024-09-21T19:08:41.778984+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:08:41.778984+0000","last_undegraded":"2024-09-21T19:08:41.778984+0000","last_fullsized":"2024-09-21T19:08:41.778984+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":143,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.778317+0000","last_change":"2024-09-21T19:05:42.905153+0000","last_active":"2024-09-21T19:08:41.778317+0000","last_peered":"2024-09-21T19:08:41.778317+0000","last_clean":"2024-09-21T19:08:41.778317+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:08:41.778317+0000","last_undegraded":"2024-09-21T19:08:41.778317+0000","last_fullsized":"2024-09-21T19:08:41.778317+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:44.187862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:44.187862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":311,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721899+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:08:44.721899+0000","last_peered":"2024-09-21T19:08:44.721899+0000","last_clean":"2024-09-21T19:08:44.721899+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:08:44.721899+0000","last_undegraded":"2024-09-21T19:08:44.721899+0000","last_fullsized":"2024-09-21T19:08:44.721899+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":595,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.727950+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:08:44.727950+0000","last_peered":"2024-09-21T19:08:44.727950+0000","last_clean":"2024-09-21T19:08:44.727950+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:08:44.727950+0000","last_undegraded":"2024-09-21T19:08:44.727950+0000","last_fullsized":"2024-09-21T19:08:44.727950+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":909,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775944+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:08:41.775944+0000","last_peered":"2024-09-21T19:08:41.775944+0000","last_clean":"2024-09-21T19:08:41.775944+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:08:41.775944+0000","last_undegraded":"2024-09-21T19:08:41.775944+0000","last_fullsized":"2024-09-21T19:08:41.775944+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":712,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714359+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:08:44.714359+0000","last_peered":"2024-09-21T19:08:44.714359+0000","last_clean":"2024-09-21T19:08:44.714359+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:08:44.714359+0000","last_undegraded":"2024-09-21T19:08:44.714359+0000","last_fullsized":"2024-09-21T19:08:44.714359+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":716,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.776124+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:08:41.776124+0000","last_peered":"2024-09-21T19:08:41.776124+0000","last_clean":"2024-09-21T19:08:41.776124+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:08:41.776124+0000","last_undegraded":"2024-09-21T19:08:41.776124+0000","last_fullsized":"2024-09-21T19:08:41.776124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":712,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721827+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:08:44.721827+0000","last_peered":"2024-09-21T19:08:44.721827+0000","last_clean":"2024-09-21T19:08:44.721827+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:08:44.721827+0000","last_undegraded":"2024-09-21T19:08:44.721827+0000","last_fullsized":"2024-09-21T19:08:44.721827+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":362,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.720516+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:08:44.720516+0000","last_peered":"2024-09-21T19:08:44.720516+0000","last_clean":"2024-09-21T19:08:44.720516+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:08:44.720516+0000","last_undegraded":"2024-09-21T19:08:44.720516+0000","last_fullsized":"2024-09-21T19:08:44.720516+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":587,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.737912+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:08:44.737912+0000","last_peered":"2024-09-21T19:08:44.737912+0000","last_clean":"2024-09-21T19:08:44.737912+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:08:44.737912+0000","last_undegraded":"2024-09-21T19:08:44.737912+0000","last_fullsized":"2024-09-21T19:08:44.737912+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":791,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.713286+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:08:44.713286+0000","last_peered":"2024-09-21T19:08:44.713286+0000","last_clean":"2024-09-21T19:08:44.713286+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:08:44.713286+0000","last_undegraded":"2024-09-21T19:08:44.713286+0000","last_fullsized":"2024-09-21T19:08:44.713286+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":710,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.714375+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:08:44.714375+0000","last_peered":"2024-09-21T19:08:44.714375+0000","last_clean":"2024-09-21T19:08:44.714375+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:08:44.714375+0000","last_undegraded":"2024-09-21T19:08:44.714375+0000","last_fullsized":"2024-09-21T19:08:44.714375+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":710,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722059+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:08:44.722059+0000","last_peered":"2024-09-21T19:08:44.722059+0000","last_clean":"2024-09-21T19:08:44.722059+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:08:44.722059+0000","last_undegraded":"2024-09-21T19:08:44.722059+0000","last_fullsized":"2024-09-21T19:08:44.722059+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.268143+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":711,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775197+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:08:41.775197+0000","last_peered":"2024-09-21T19:08:41.775197+0000","last_clean":"2024-09-21T19:08:41.775197+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:08:41.775197+0000","last_undegraded":"2024-09-21T19:08:41.775197+0000","last_fullsized":"2024-09-21T19:08:41.775197+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":789,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.722300+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:08:44.722300+0000","last_peered":"2024-09-21T19:08:44.722300+0000","last_clean":"2024-09-21T19:08:44.722300+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:08:44.722300+0000","last_undegraded":"2024-09-21T19:08:44.722300+0000","last_fullsized":"2024-09-21T19:08:44.722300+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":706,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721982+0000","last_change":"2024-09-21T19:02:35.088491+0000","last_active":"2024-09-21T19:08:44.721982+0000","last_peered":"2024-09-21T19:08:44.721982+0000","last_clean":"2024-09-21T19:08:44.721982+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:08:44.721982+0000","last_undegraded":"2024-09-21T19:08:44.721982+0000","last_fullsized":"2024-09-21T19:08:44.721982+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.088444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.088444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":712,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.782653+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:08:41.782653+0000","last_peered":"2024-09-21T19:08:41.782653+0000","last_clean":"2024-09-21T19:08:41.782653+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:08:41.782653+0000","last_undegraded":"2024-09-21T19:08:41.782653+0000","last_fullsized":"2024-09-21T19:08:41.782653+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":606,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.726488+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:08:44.726488+0000","last_peered":"2024-09-21T19:08:44.726488+0000","last_clean":"2024-09-21T19:08:44.726488+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:08:44.726488+0000","last_undegraded":"2024-09-21T19:08:44.726488+0000","last_fullsized":"2024-09-21T19:08:44.726488+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":736,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775365+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:08:41.775365+0000","last_peered":"2024-09-21T19:08:41.775365+0000","last_clean":"2024-09-21T19:08:41.775365+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:08:41.775365+0000","last_undegraded":"2024-09-21T19:08:41.775365+0000","last_fullsized":"2024-09-21T19:08:41.775365+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":649,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.789281+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:08:41.789281+0000","last_peered":"2024-09-21T19:08:41.789281+0000","last_clean":"2024-09-21T19:08:41.789281+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:08:41.789281+0000","last_undegraded":"2024-09-21T19:08:41.789281+0000","last_fullsized":"2024-09-21T19:08:41.789281+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":725,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721021+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:08:44.721021+0000","last_peered":"2024-09-21T19:08:44.721021+0000","last_clean":"2024-09-21T19:08:44.721021+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:08:44.721021+0000","last_undegraded":"2024-09-21T19:08:44.721021+0000","last_fullsized":"2024-09-21T19:08:44.721021+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":711,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.782968+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:08:41.782968+0000","last_peered":"2024-09-21T19:08:41.782968+0000","last_clean":"2024-09-21T19:08:41.782968+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:08:41.782968+0000","last_undegraded":"2024-09-21T19:08:41.782968+0000","last_fullsized":"2024-09-21T19:08:41.782968+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":735,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.774638+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:08:41.774638+0000","last_peered":"2024-09-21T19:08:41.774638+0000","last_clean":"2024-09-21T19:08:41.774638+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:08:41.774638+0000","last_undegraded":"2024-09-21T19:08:41.774638+0000","last_fullsized":"2024-09-21T19:08:41.774638+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":590,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.729633+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:08:44.729633+0000","last_peered":"2024-09-21T19:08:44.729633+0000","last_clean":"2024-09-21T19:08:44.729633+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:08:44.729633+0000","last_undegraded":"2024-09-21T19:08:44.729633+0000","last_fullsized":"2024-09-21T19:08:44.729633+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":624,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.771375+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:08:41.771375+0000","last_peered":"2024-09-21T19:08:41.771375+0000","last_clean":"2024-09-21T19:08:41.771375+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:08:41.771375+0000","last_undegraded":"2024-09-21T19:08:41.771375+0000","last_fullsized":"2024-09-21T19:08:41.771375+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":700,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.721153+0000","last_change":"2024-09-21T19:05:42.904326+0000","last_active":"2024-09-21T19:08:44.721153+0000","last_peered":"2024-09-21T19:08:44.721153+0000","last_clean":"2024-09-21T19:08:44.721153+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:08:44.721153+0000","last_undegraded":"2024-09-21T19:08:44.721153+0000","last_fullsized":"2024-09-21T19:08:44.721153+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.128848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.128848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":706,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.719485+0000","last_change":"2024-09-21T19:07:21.428355+0000","last_active":"2024-09-21T19:08:44.719485+0000","last_peered":"2024-09-21T19:08:44.719485+0000","last_clean":"2024-09-21T19:08:44.719485+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:08:44.719485+0000","last_undegraded":"2024-09-21T19:08:44.719485+0000","last_fullsized":"2024-09-21T19:08:44.719485+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:21.428212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:21.428212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:08:47.151249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":728,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.775283+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:08:41.775283+0000","last_peered":"2024-09-21T19:08:41.775283+0000","last_clean":"2024-09-21T19:08:41.775283+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:08:41.775283+0000","last_undegraded":"2024-09-21T19:08:41.775283+0000","last_fullsized":"2024-09-21T19:08:41.775283+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":689,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-21T19:08:41.774720+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:08:41.774720+0000","last_peered":"2024-09-21T19:08:41.774720+0000","last_clean":"2024-09-21T19:08:41.774720+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:08:41.774720+0000","last_undegraded":"2024-09-21T19:08:41.774720+0000","last_fullsized":"2024-09-21T19:08:41.774720+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":610,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T19:08:44.737842+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:08:44.737842+0000","last_peered":"2024-09-21T19:08:44.737842+0000","last_clean":"2024-09-21T19:08:44.737842+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:08:44.737842+0000","last_undegraded":"2024-09-21T19:08:44.737842+0000","last_fullsized":"2024-09-21T19:08:44.737842+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4042194968,"num_objects":61680,"num_object_clones":0,"num_object_copies":185040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173262,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13171,"num_bytes_recovered":863174656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11841974272,"data_stored":11841962056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85662,"ondisk_log_size":85662,"up":48,"acting":44,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":928,"num_read_kb":923,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266880,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1565524,"kb_used_data":970692,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92179628,"statfs":{"total":95995035648,"available":94391939072,"internally_reserved":0,"allocated":993988608,"data_stored":993302244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464330,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2049732,"kb_used_data":1778932,"kb_used_omap":4,"kb_used_meta":270779,"kb_avail":91695420,"statfs":{"total":95995035648,"available":93896110080,"internally_reserved":0,"allocated":1821626368,"data_stored":1820948895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":277278372},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619031,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1236932,"kb_used_data":1017008,"kb_used_omap":4,"kb_used_meta":219899,"kb_avail":92508220,"statfs":{"total":95995035648,"available":94728417280,"internally_reserved":0,"allocated":1041416192,"data_stored":1040743245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":225177251},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019086,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2486044,"kb_used_data":1967560,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91259108,"statfs":{"total":95995035648,"available":93449326592,"internally_reserved":0,"allocated":2014781440,"data_stored":2014091284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182610,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2008868,"kb_used_data":1287696,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91736284,"statfs":{"total":95995035648,"available":93937954816,"internally_reserved":0,"allocated":1318600704,"data_stored":1317896307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411544,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2071672,"kb_used_data":1535140,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91673480,"statfs":{"total":95995035648,"available":93873643520,"internally_reserved":0,"allocated":1571983360,"data_stored":1571322706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1369212,"kb_used_data":706344,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92375940,"statfs":{"total":95995035648,"available":94592962560,"internally_reserved":0,"allocated":723296256,"data_stored":722573931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3197148,"kb_used_data":2403788,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":90548004,"statfs":{"total":95995035648,"available":92721156096,"internally_reserved":0,"allocated":2461478912,"data_stored":2460785069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2435977216,"data_stored":2435973144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":717619200,"data_stored":717619200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1569980416,"data_stored":1569980416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1309605888,"data_stored":1309605888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1985417216,"data_stored":1985413144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1036648448,"data_stored":1036648448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1798049792,"data_stored":1798045720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":988676096,"data_stored":988676096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:08:51.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T19:08:51.834 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:51.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:51.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:52.072 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:52.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:52.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:52.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:52.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:52.527 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:52.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:52.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:52.659 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:52.662 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:08:52.662 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:08:52.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.804 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:52.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:52.815 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:08:52.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:52.945 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:52.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:53.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[29384]: pgmap v775: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 9463/175113 objects degraded (5.404%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/180811921' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[29384]: from='client.36320 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[39192]: pgmap v775: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 9463/175113 objects degraded (5.404%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:08:53.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/180811921' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:53.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:52 smithi081 ceph-mon[39192]: from='client.36320 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:53.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:53.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:53.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:52 smithi104 ceph-mon[32438]: pgmap v775: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 9463/175113 objects degraded (5.404%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T19:08:53.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:08:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:08:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/180811921' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:52 smithi104 ceph-mon[32438]: from='client.36320 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:08:53.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:53.364 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:53.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:53.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:53.625 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:53.646 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:53.727 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.770 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:53.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:53.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:53.910 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:53.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:54.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:54.050 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[29384]: from='client.36320 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[29384]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[39192]: from='client.36320 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[39192]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:08:54.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:53 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:54.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:08:54.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:54.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:54.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:53 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:08:54.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:53 smithi104 ceph-mon[32438]: from='client.36320 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:08:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:53 smithi104 ceph-mon[32438]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T19:08:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:08:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:08:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:53 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:08:54.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:54.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:54.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:54.602 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:54.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:54.729 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:54.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:54.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:54.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:54.994 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:54.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:55.095 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:55.122 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:55.173 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:08:55.173 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:08:55.195 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:08:55.195 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:08:55.206 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:55.216 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:08:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:08:55.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:55.225 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:08:55.225 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8580/167655 objects degraded (5.118%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1166,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":390,"num_osds":8,"num_up_osds":8,"osd_up_since":1726945614,"num_in_osds":8,"osd_in_since":1726945541,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":154},{"state_name":"active+recovering+undersized+degraded+remapped","count":3},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":158,"num_pools":7,"num_objects":55885,"data_bytes":3649968691,"bytes_used":15221727232,"bytes_avail":752738557952,"bytes_total":767960285184,"degraded_objects":8580,"degraded_total":167655,"degraded_ratio":0.051176523217321288,"recovering_objects_per_sec":190,"recovering_bytes_per_sec":12502483,"recovering_keys_per_sec":0,"num_objects_recovered":1725,"num_bytes_recovered":113049600,"num_keys_recovered":0,"read_bytes_sec":1132,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":908},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-21T19:07:53.731883+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T19:08:55.235 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:08:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:08:55.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:55.256 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:08:55.256 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:08:55.273 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:08:55.273 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:08:55.290 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:08:55.291 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:08:55.308 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:08:55.308 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:08:55.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:08:55.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: 1.2 scrub starts 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: 1.2 scrub ok 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: pgmap v777: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 909 op/s; 8580/167655 objects degraded (5.118%); 12 MiB/s, 190 objects/s recovering 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 8580/167655 objects degraded (5.118%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[29384]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: 1.2 scrub starts 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: 1.2 scrub ok 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: pgmap v777: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 909 op/s; 8580/167655 objects degraded (5.118%); 12 MiB/s, 190 objects/s recovering 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T19:08:55.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 8580/167655 objects degraded (5.118%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:55.310 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:55 smithi081 ceph-mon[39192]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T19:08:55.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:55.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:55.494 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:55.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:08:55.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:08:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: 1.2 scrub starts 2024-09-21T19:08:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: 1.2 scrub ok 2024-09-21T19:08:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: pgmap v777: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 909 op/s; 8580/167655 objects degraded (5.118%); 12 MiB/s, 190 objects/s recovering 2024-09-21T19:08:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T19:08:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 8580/167655 objects degraded (5.118%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:08:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:55 smithi104 ceph-mon[32438]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T19:08:55.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:55.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:55.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:08:55.720 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:55.845 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:55.883 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:55.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:55.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:55.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:56.016 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:56.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:56.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:56.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:56.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:56.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:56.412 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:56.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:56 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T19:08:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:56 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T19:08:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/877141880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1646165647' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:08:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:56 smithi104 ceph-mon[32438]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T19:08:56.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:56.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:56.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/877141880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1646165647' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[29384]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/877141880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1646165647' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:08:56.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:56 smithi081 ceph-mon[39192]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T19:08:56.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:56.671 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:56.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:08:56.803 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:08:56.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:56.943 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:56.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:56.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:57.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:57.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:57.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:57.215 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:57.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:57.344 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:57.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:57.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:57 smithi104 ceph-mon[32438]: pgmap v781: 158 pgs: 4 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 8554/162003 objects degraded (5.280%); 14 MiB/s, 227 objects/s recovering 2024-09-21T19:08:57.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:57 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:57 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:57 smithi104 ceph-mon[32438]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T19:08:57.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[39192]: pgmap v781: 158 pgs: 4 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 8554/162003 objects degraded (5.280%); 14 MiB/s, 227 objects/s recovering 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[39192]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[29384]: pgmap v781: 158 pgs: 4 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 8554/162003 objects degraded (5.280%); 14 MiB/s, 227 objects/s recovering 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-mon[29384]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T19:08:57.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:08:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:08:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:08:57.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:57.604 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:57.706 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:57.741 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:08:57.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:08:57.880 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:57.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:08:57.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:58.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:58.157 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:58.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:58.300 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T19:08:58.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:08:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:58.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:58.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:58.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:58 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:08:58.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:58 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:08:58.522 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:58 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:08:58.522 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:58 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:08:58.522 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:58 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:08:58.523 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:58 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:08:58.538 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:58.570 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:58.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:58.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:58.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:08:58.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:08:58.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:58.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:58.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:59.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:08:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:08:59.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:59.121 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:59.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[29384]: 6.12 scrub starts 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[29384]: 6.12 scrub ok 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[29384]: pgmap v783: 158 pgs: 4 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 8285/156801 objects degraded (5.284%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[29384]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[39192]: 6.12 scrub starts 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[39192]: 6.12 scrub ok 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[39192]: pgmap v783: 158 pgs: 4 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 8285/156801 objects degraded (5.284%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T19:08:59.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:08:59 smithi081 ceph-mon[39192]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T19:08:59.333 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:08:59.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235880 8.1e 8:7f5b8aa9:::benchmark_data_smithi081_105538_object103648:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 425.70450979899999, 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.088566871, 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:08:59.384 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235880 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:08:59.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674018+0000", 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7300000000000001e-07 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674021+0000", 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1380000000000001e-06 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674025+0000", 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3499999999999999e-06 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.674029+0000", 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1690000000000002e-06 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842647+0000", 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16861802400000001 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.842654+0000", 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2509999999999997e-06 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758247+0000", 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91559294899999999 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.758376+0000", 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012917699999999999 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762523+0000", 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041469219999999999 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.762585+0000", 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1418e-05 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237693 8.21 8:86cd0139:::benchmark_data_smithi081_105538_object105461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 423.679085606, 2024-09-21T19:08:59.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.19476636, 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237693 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699442+0000", 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699444+0000", 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999999999999999e-06 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699448+0000", 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9110000000000003e-06 2024-09-21T19:08:59.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699456+0000", 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3259999999999996e-06 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.699471+0000", 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4681000000000001e-05 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700689+0000", 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012186180000000001 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.700697+0000", 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4739999999999999e-06 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698487+0000", 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.99779048800000003 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698497+0000", 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1120000000000002e-06 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889014+0000", 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1905176980000001 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889174+0000", 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000159545 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.889403+0000", 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022944299999999999 2024-09-21T19:08:59.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891432+0000", 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002028448 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891637+0000", 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020514200000000001 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891727+0000", 2024-09-21T19:08:59.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0705000000000007e-05 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.893997+0000", 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002269802 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894105+0000", 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107367 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894208+0000", 2024-09-21T19:08:59.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0001038 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237741 8.21 8:86a63524:::benchmark_data_smithi081_105538_object105509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:08:59.397 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 423.65584497700002, 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1717911829999998, 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237741 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722683+0000", 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722684+0000", 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0270000000000001e-06 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722688+0000", 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7929999999999999e-06 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722697+0000", 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1290000000000002e-06 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722703+0000", 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6200000000000001e-06 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722891+0000", 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018788600000000001 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:55.722902+0000", 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0541e-05 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702948+0000", 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98004672199999998 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.702955+0000", 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0549999999999996e-06 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890451+0000", 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1874960109999999 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890572+0000", 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012139800000000001 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,5", 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.890731+0000", 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015913599999999999 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.891711+0000", 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00097963900000000003 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892154+0000", 2024-09-21T19:08:59.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044374699999999998 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.892258+0000", 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010389400000000001 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894257+0000", 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019987550000000001 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894360+0000", 2024-09-21T19:08:59.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010312200000000001 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.894474+0000", 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113708 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237675 8.1c 8:3e931efd:::benchmark_data_smithi081_105538_object105443:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 422.68517685400002, 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.207533752, 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.405 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237675 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693351+0000", 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693353+0000", 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9609999999999999e-06 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693358+0000", 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6030000000000004e-06 2024-09-21T19:08:59.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694353+0000", 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00099412400000000009 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694362+0000", 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3500000000000003e-06 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698188+0000", 2024-09-21T19:08:59.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038259919999999998 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698196+0000", 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7100000000000007e-06 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900800+0000", 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2026040019999999 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.900885+0000", 2024-09-21T19:08:59.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5010000000000001e-05 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237706 8.1c 8:39f9b21e:::benchmark_data_smithi081_105538_object105474:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 422.684155831, 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2136628949999999, 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.410 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237706 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694372+0000", 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694373+0000", 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7300000000000005e-07 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694377+0000", 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5800000000000002e-06 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694385+0000", 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3030000000000002e-06 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694390+0000", 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4140000000000002e-06 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698219+0000", 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038285289999999998 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698225+0000", 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4269999999999996e-06 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901135+0000", 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2029100589999999 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901255+0000", 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120221 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.901527+0000", 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000272045 2024-09-21T19:08:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907437+0000", 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059092579999999997 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907688+0000", 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025114199999999999 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907731+0000", 2024-09-21T19:08:59.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3180000000000003e-05 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907774+0000", 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3256000000000001e-05 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907856+0000", 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1582000000000006e-05 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907917+0000", 2024-09-21T19:08:59.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0930000000000001e-05 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908035+0000", 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118096 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237735 8.1c 8:3aa9ee71:::benchmark_data_smithi081_105538_object105503:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 422.68413124800003, 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2197030149999999, 2024-09-21T19:08:59.417 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237735 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694396+0000", 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694397+0000", 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7000000000000005e-07 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694400+0000", 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7470000000000001e-06 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694405+0000", 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6230000000000002e-06 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694410+0000", 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.972e-06 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698247+0000", 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038369789999999999 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.698253+0000", 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9390000000000002e-06 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.904941+0000", 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.206687686 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905096+0000", 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000155236 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4,5,7", 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.905336+0000", 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024011 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907815+0000", 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024790139999999999 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908076+0000", 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026105799999999999 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908089+0000", 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2847000000000001e-05 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908126+0000", 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7177e-05 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913917+0000", 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057903410000000001 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913979+0000", 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2253000000000002e-05 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.914099+0000", 2024-09-21T19:08:59.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120463 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237756 8.22 8:47738879:::benchmark_data_smithi081_105538_object105524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 422.68411149799999, 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.228195653, 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.425 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237756 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694416+0000", 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694417+0000", 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7199999999999999e-07 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694419+0000", 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.447e-06 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694425+0000", 2024-09-21T19:08:59.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4689999999999996e-06 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.694430+0000", 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9099999999999996e-06 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701670+0000", 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0072406739999999999 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.701679+0000", 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4610000000000008e-06 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908648+0000", 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2069698309999999 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908761+0000", 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011227500000000001 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,3,4,5,7", 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908968+0000", 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020707700000000001 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912465+0000", 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003497207 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912477+0000", 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2086000000000001e-05 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913031+0000", 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055408400000000004 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913427+0000", 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039556599999999997 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913488+0000", 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1543000000000004e-05 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922372+0000", 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008883574 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922440+0000", 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7909000000000003e-05 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922612+0000", 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017206799999999999 2024-09-21T19:08:59.432 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:74496 10.6 10:63523bdd:::benchmark_data_smithi081_155596_object74495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:54.373288+0000", 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 125.005239319, 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5022800119999999, 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.433 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 74496 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.373288+0000", 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.373288+0000", 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.373290+0000", 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9190000000000002e-06 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.373319+0000", 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9152000000000001e-05 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.373331+0000", 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1421e-05 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.373337+0000", 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.426e-06 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.374472+0000", 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011342489999999999 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.374600+0000", 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012876000000000001 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4", 2024-09-21T19:08:59.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.374743+0000", 2024-09-21T19:08:59.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000142917 2024-09-21T19:08:59.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:59.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.379142+0000", 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043992500000000004 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.391813+0000", 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012670314 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.391825+0000", 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1789e-05 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695548+0000", 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.30372389100000002 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695562+0000", 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3944999999999999e-05 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.875464+0000", 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1799017219999999 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.875568+0000", 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104257 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:74512 10.6 10:61861872:::benchmark_data_smithi081_155596_object74511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:54.393856+0000", 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 124.984671838, 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.527160407, 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 74512 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.442 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393856+0000", 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393856+0000", 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393858+0000", 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9810000000000002e-06 2024-09-21T19:08:59.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393884+0000", 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6315000000000001e-05 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393895+0000", 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0434e-05 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393904+0000", 2024-09-21T19:08:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6379999999999999e-06 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393962+0000", 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7457000000000002e-05 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.393972+0000", 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0501e-05 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695590+0000", 2024-09-21T19:08:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.301617841 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695597+0000", 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3499999999999999e-06 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.877939+0000", 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.182341273 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.878202+0000", 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026310500000000002 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.878363+0000", 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016091599999999999 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.885744+0000", 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0073811340000000001 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.885905+0000", 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016080700000000001 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.917606+0000", 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031701181000000002 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.917677+0000", 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1072e-05 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.921016+0000", 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003339402 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:74514 10.6 10:677dcd63:::benchmark_data_smithi081_155596_object74513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:54.397319+0000", 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 124.981208745, 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.52998415, 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.449 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 74514 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397319+0000", 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397319+0000", 2024-09-21T19:08:59.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397320+0000", 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4640000000000001e-06 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397343+0000", 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2362e-05 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397354+0000", 2024-09-21T19:08:59.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1065e-05 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397360+0000", 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9959999999999999e-06 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397403+0000", 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2957999999999998e-05 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.397414+0000", 2024-09-21T19:08:59.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0808999999999999e-05 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695620+0000", 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29820608199999998 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695640+0000", 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0381999999999999e-05 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.880782+0000", 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1851414199999999 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.880895+0000", 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011333899999999999 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.881054+0000", 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000159164 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.885922+0000", 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048683010000000002 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.893388+0000", 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074655310000000001 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.921091+0000", 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.027703347999999999 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.455 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.921154+0000", 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2608999999999996e-05 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.927303+0000", 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061493199999999998 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:74535 10.6 10:61da6fdf:::benchmark_data_smithi081_155596_object74534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:54.426881+0000", 2024-09-21T19:08:59.456 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 124.951646262, 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5047350269999999, 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 74535 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.426881+0000", 2024-09-21T19:08:59.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.426881+0000", 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.426885+0000", 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4130000000000002e-06 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.426917+0000", 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2121e-05 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.426937+0000", 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9925e-05 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.426948+0000", 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0795000000000001e-05 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.427014+0000", 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6335999999999996e-05 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.427024+0000", 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4720000000000001e-06 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695666+0000", 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26864295199999999 2024-09-21T19:08:59.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:54.695673+0000", 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9859999999999997e-06 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.883597+0000", 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1879231589999999 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.883744+0000", 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014776500000000001 2024-09-21T19:08:59.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.883911+0000", 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016646099999999999 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.888334+0000", 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044229940000000004 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.894091+0000", 2024-09-21T19:08:59.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057572120000000003 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.927357+0000", 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.033266032000000001 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.927421+0000", 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4255000000000001e-05 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:55.931616+0000", 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041951489999999996 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:124455 10.6 10:6e7d4293:::benchmark_data_smithi081_155596_object6982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:52.492281+0000", 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 66.886247030000007, 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.115414102, 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.464 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 124455 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.492281+0000", 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.492281+0000", 2024-09-21T19:08:59.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.492282+0000", 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.124e-06 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.492286+0000", 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0090000000000001e-06 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.492295+0000", 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7700000000000007e-06 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.492300+0000", 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9030000000000001e-06 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.493686+0000", 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00138597 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.493833+0000", 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000146725 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.493987+0000", 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015408699999999999 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.501096+0000", 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0071091180000000002 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.507246+0000", 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0061492780000000002 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.604937+0000", 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.097691747999999995 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.604994+0000", 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6552999999999999e-05 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:52.607695+0000", 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027008169999999999 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:130931 10.4 10:21eff86c:::benchmark_data_smithi081_155596_object13458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:59.139029+0000", 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.239498488999999, 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19215132600000001, 2024-09-21T19:08:59.470 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 130931 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139029+0000", 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139029+0000", 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139031+0000", 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9139999999999998e-06 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139038+0000", 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6150000000000002e-06 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139048+0000", 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0547000000000001e-05 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139056+0000", 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4680000000000003e-06 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139173+0000", 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116946 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139287+0000", 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011407 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.139401+0000", 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011425300000000001 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.142196+0000", 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027947060000000001 2024-09-21T19:08:59.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.143623+0000", 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001426966 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331085+0000", 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.187462092 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331140+0000", 2024-09-21T19:08:59.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5547000000000002e-05 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331181+0000", 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0201999999999999e-05 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:130949 10.4 10:2f7658bb:::benchmark_data_smithi081_155596_object13476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:59.155046+0000", 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.223481563, 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.176245864, 2024-09-21T19:08:59.476 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 130949 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155046+0000", 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155046+0000", 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155048+0000", 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9190000000000002e-06 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155054+0000", 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6210000000000001e-06 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155066+0000", 2024-09-21T19:08:59.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1960999999999999e-05 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155072+0000", 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7170000000000002e-06 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155123+0000", 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0207000000000002e-05 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155244+0000", 2024-09-21T19:08:59.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121396 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.155360+0000", 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011556399999999999 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.161995+0000", 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066355729999999996 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.162442+0000", 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044684400000000001 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331219+0000", 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16877719799999999 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331258+0000", 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9215000000000003e-05 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331292+0000", 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3649000000000003e-05 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:130953 10.4 10:20a504af:::benchmark_data_smithi081_155596_object13480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:59.158942+0000", 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.219586114999998, 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17246647600000001, 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.483 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 130953 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.158942+0000", 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.158942+0000", 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.158944+0000", 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9010000000000001e-06 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.158949+0000", 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7100000000000004e-06 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.158972+0000", 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2348e-05 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.158979+0000", 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4039999999999998e-06 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.159025+0000", 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5639999999999997e-05 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.159134+0000", 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108955 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.159238+0000", 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104527 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.162002+0000", 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027634479999999999 2024-09-21T19:08:59.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.162960+0000", 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00095815799999999997 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331327+0000", 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.16836701500000001 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331373+0000", 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6622000000000002e-05 2024-09-21T19:08:59.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331408+0000", 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4748e-05 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:130977 10.4 10:234b92f0:::benchmark_data_smithi081_155596_object13504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:59.206097+0000", 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.172430974000001, 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.125484764, 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.489 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 130977 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206097+0000", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206097+0000", 2024-09-21T19:08:59.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206099+0000", 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5909999999999998e-06 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206106+0000", 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.63e-06 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206119+0000", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2561999999999999e-05 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206126+0000", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3880000000000002e-06 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206185+0000", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8690000000000002e-05 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206312+0000", 2024-09-21T19:08:59.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000127136 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.206446+0000", 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013419800000000001 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.211128+0000", 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046822260000000003 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.211183+0000", 2024-09-21T19:08:59.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4595000000000002e-05 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331450+0000", 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.120266757 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331550+0000", 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100604 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331582+0000", 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1387000000000003e-05 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:130980 10.4 10:25511470:::benchmark_data_smithi081_155596_object13507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:07:59.208105+0000", 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 60.170422537, 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.123583966, 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.495 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 130980 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208105+0000", 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208105+0000", 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208107+0000", 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6819999999999999e-06 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208112+0000", 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9359999999999998e-06 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208123+0000", 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1392999999999999e-05 2024-09-21T19:08:59.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208130+0000", 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6730000000000003e-06 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208677+0000", 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00054730700000000002 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208797+0000", 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119936 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,4", 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.208917+0000", 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120098 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.211169+0000", 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022519699999999998 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.499 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.211228+0000", 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8427999999999997e-05 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331618+0000", 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.120390855 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331658+0000", 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9746000000000002e-05 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:07:59.331689+0000", 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0942000000000003e-05 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:166460 10.6 10:6b51706f:::benchmark_data_smithi081_155596_object48987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:08:38.323120+0000", 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.055408033999999, 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12732571200000001, 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.501 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 166460 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.323120+0000", 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.323120+0000", 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.323122+0000", 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.181e-06 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.323128+0000", 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3829999999999997e-06 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.323139+0000", 2024-09-21T19:08:59.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1117e-05 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.323147+0000", 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8620000000000003e-06 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.419958+0000", 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.096811224000000001 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for rw locks", 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.420063+0000", 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104619 2024-09-21T19:08:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.430997+0000", 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010933651000000001 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.431101+0000", 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103793 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.431246+0000", 2024-09-21T19:08:59.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014582399999999999 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.442084+0000", 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010837698 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.442094+0000", 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6560000000000004e-06 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.448876+0000", 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006782738 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.448924+0000", 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.71e-05 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.450445+0000", 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001521866 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:166463 10.6 10:6cf92eb9:::benchmark_data_smithi081_155596_object48990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:08:38.327930+0000", 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.050597660000001, 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.11619244200000001, 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 166463 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.508 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.327930+0000", 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.327930+0000", 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.327932+0000", 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1059999999999998e-06 2024-09-21T19:08:59.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.327939+0000", 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4110000000000001e-06 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.328033+0000", 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4810999999999997e-05 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.328041+0000", 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5730000000000002e-06 2024-09-21T19:08:59.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.420095+0000", 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.092054325000000006 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.420181+0000", 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5832000000000001e-05 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.420272+0000", 2024-09-21T19:08:59.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1292000000000006e-05 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.430951+0000", 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010678896 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.433958+0000", 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030068930000000001 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.442143+0000", 2024-09-21T19:08:59.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0081847550000000002 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.442199+0000", 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6539999999999997e-05 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.444122+0000", 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019230079999999999 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:166669 10.6 10:6224ab46:::benchmark_data_smithi081_155596_object49196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:08:38.594659+0000", 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.783868867999999, 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.12045307199999999, 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 166669 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:08:59.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.594659+0000", 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.594659+0000", 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.594661+0000", 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1830000000000001e-06 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.594667+0000", 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8050000000000003e-06 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.594680+0000", 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3534e-05 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.594687+0000", 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6789999999999999e-06 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.605545+0000", 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010857663 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for rw locks", 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.605661+0000", 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011592200000000001 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.615664+0000", 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010003532000000001 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.615761+0000", 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6582000000000005e-05 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 4,5", 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.615899+0000", 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013849500000000001 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.627551+0000", 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011652071 2024-09-21T19:08:59.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.627590+0000", 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9133e-05 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.627611+0000", 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0650999999999999e-05 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.627791+0000", 2024-09-21T19:08:59.519 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017964599999999999 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:08:38.715112+0000", 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.087321176 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:08:59.520 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:08:59.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:59 smithi104 ceph-mon[32438]: 6.12 scrub starts 2024-09-21T19:08:59.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:59 smithi104 ceph-mon[32438]: 6.12 scrub ok 2024-09-21T19:08:59.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:59 smithi104 ceph-mon[32438]: pgmap v783: 158 pgs: 4 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 8285/156801 objects degraded (5.284%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T19:08:59.521 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:08:59 smithi104 ceph-mon[32438]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T19:08:59.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:08:59.575 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:08:59.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:08:59.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:08:59.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:08:59.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:08:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:08:59.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:08:59.981 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:08:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:08:59.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:00.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:00.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:00.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:00.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:00.259 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:00.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:00.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:00.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:00.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:00.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:00.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:00 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:00.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:00 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T19:09:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:00 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T19:09:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:00 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T19:09:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:00 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T19:09:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:00 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 8285/156801 objects degraded (5.284%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:00.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:00.620 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 8285/156801 objects degraded (5.284%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T19:09:00.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T19:09:00.622 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T19:09:00.622 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T19:09:00.622 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:00 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 8285/156801 objects degraded (5.284%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:00.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:00.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:00.757 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:00.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:00.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:00.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:00.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:01.017 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:01.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:09:01.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:01.186 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:09:01.200 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:01.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:01.344 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:01.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:01.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:01.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:01 smithi104 ceph-mon[32438]: 6.15 scrub starts 2024-09-21T19:09:01.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:01 smithi104 ceph-mon[32438]: 6.15 scrub ok 2024-09-21T19:09:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:01 smithi104 ceph-mon[32438]: pgmap v785: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1015 B/s rd, 0 B/s wr, 782 op/s; 7750/150183 objects degraded (5.160%); 10 MiB/s, 159 objects/s recovering 2024-09-21T19:09:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:01 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T19:09:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:01 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T19:09:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/653632541' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[39192]: 6.15 scrub starts 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[39192]: 6.15 scrub ok 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[39192]: pgmap v785: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1015 B/s rd, 0 B/s wr, 782 op/s; 7750/150183 objects degraded (5.160%); 10 MiB/s, 159 objects/s recovering 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/653632541' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[29384]: 6.15 scrub starts 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[29384]: 6.15 scrub ok 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[29384]: pgmap v785: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1015 B/s rd, 0 B/s wr, 782 op/s; 7750/150183 objects degraded (5.160%); 10 MiB/s, 159 objects/s recovering 2024-09-21T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/653632541' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:01.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:01.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:01.748 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:01.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:01.889 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:01.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:01.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:02.028 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:02.130 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:02.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:02.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:02.307 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:02.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:02.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:02.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:02 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:09:02.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/653632541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:02.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:02 smithi104 ceph-mon[32438]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T19:09:02.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:02.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:02.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:02 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:09:02.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/653632541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:02.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:02 smithi081 ceph-mon[29384]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T19:09:02.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:02 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:09:02.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/653632541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:02.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:02 smithi081 ceph-mon[39192]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T19:09:02.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:02.687 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:02.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:02.830 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:02.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:02.971 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:02.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:03.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:03.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:03.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:03.460 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:03.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:03 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T19:09:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:03 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T19:09:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:03 smithi104 ceph-mon[32438]: pgmap v787: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 770 op/s; 7750/148131 objects degraded (5.232%); 8.4 MiB/s, 133 objects/s recovering 2024-09-21T19:09:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:03 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T19:09:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:03 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T19:09:03.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[39192]: pgmap v787: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 770 op/s; 7750/148131 objects degraded (5.232%); 8.4 MiB/s, 133 objects/s recovering 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[29384]: pgmap v787: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 770 op/s; 7750/148131 objects degraded (5.232%); 8.4 MiB/s, 133 objects/s recovering 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T19:09:03.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:03 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T19:09:03.592 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:03.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:03.763 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:03.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:03.902 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:03.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:04.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:04.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:04.169 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:04.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:04.306 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:04.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:04.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:04.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:04.585 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:04.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:04.720 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:04.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:09:04.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:04.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:04.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:04.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:05.031 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:05.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:05.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:05.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:05.303 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:05.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:05.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:05.470 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:05 smithi104 ceph-mon[32438]: pgmap v788: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 960 op/s; 7528/140265 objects degraded (5.367%); 9.1 MiB/s, 146 objects/s recovering 2024-09-21T19:09:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:05 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 7528/140265 objects degraded (5.367%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:05 smithi104 ceph-mon[32438]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T19:09:05.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:05.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:05 smithi081 ceph-mon[29384]: pgmap v788: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 960 op/s; 7528/140265 objects degraded (5.367%); 9.1 MiB/s, 146 objects/s recovering 2024-09-21T19:09:05.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:05 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 7528/140265 objects degraded (5.367%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:05.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:05 smithi081 ceph-mon[29384]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T19:09:05.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:05 smithi081 ceph-mon[39192]: pgmap v788: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 960 op/s; 7528/140265 objects degraded (5.367%); 9.1 MiB/s, 146 objects/s recovering 2024-09-21T19:09:05.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:05 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 7528/140265 objects degraded (5.367%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:05.570 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:05 smithi081 ceph-mon[39192]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T19:09:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:05.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:05.715 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.756 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:05.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:05.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:05.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:05.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.042 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:06.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:06.144 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.174 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:06.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:06.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.317 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:06.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:06.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:06.511 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:09:06.511 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:09:06.536 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:09:06.537 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:09:06.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:06.571 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:09:06.571 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:09:06.613 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:09:06.614 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:09:06.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:06.658 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:09:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:09:06.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:06.699 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:09:06.700 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:09:06.701 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:09:06.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:06.741 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:09:06.742 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:09:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.755 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:06.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:06.771 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:09:06.771 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:09:06.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:06.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:06.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:07.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:07.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:07.161 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:07.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:07.303 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:07.361 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:09:07.361 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:09:07.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:07.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:09:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: 5.1a scrub starts 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: 5.1a scrub ok 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: pgmap v790: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 673 B/s rd, 0 B/s wr, 881 op/s; 7291/134583 objects degraded (5.417%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/474469546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:07.473 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:09:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: 5.1a scrub starts 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: 5.1a scrub ok 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: pgmap v790: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 673 B/s rd, 0 B/s wr, 881 op/s; 7291/134583 objects degraded (5.417%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T19:09:07.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T19:09:07.475 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/474469546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:07.475 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:07.475 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:07 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: 5.1a scrub starts 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: 5.1a scrub ok 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: pgmap v790: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 673 B/s rd, 0 B/s wr, 881 op/s; 7291/134583 objects degraded (5.417%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/474469546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:07 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:07.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:07.576 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:07.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.716 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:07.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:07.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:07.857 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:07.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:07.993 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:07.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:08.094 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:08.133 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:08.235 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:08.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:08.369 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:08.403 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:08.479 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:09:08.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:08.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:08 smithi104 ceph-mon[32438]: from='client.26200 v1:172.21.15.81:0/1686120802' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:08.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:08 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:09:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:08 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:09:08.520 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:08 smithi081 ceph-mon[29384]: from='client.26200 v1:172.21.15.81:0/1686120802' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:08.520 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:08 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:09:08.520 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:08 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:09:08.520 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:08 smithi081 ceph-mon[39192]: from='client.26200 v1:172.21.15.81:0/1686120802' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:08.521 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:08 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:09:08.521 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:08 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:09:08.564 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:08.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.697 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:08.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:08.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:08.833 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:08.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:08.963 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":790,"stamp":"2024-09-21T19:09:05.760302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2927499827,"num_objects":44861,"num_object_clones":0,"num_object_copies":134583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44861,"num_whiteouts":0,"num_read":2083,"num_read_kb":1835,"num_write":191002,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16263,"num_bytes_recovered":1050683640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87439,"ondisk_log_size":87439,"up":474,"acting":472,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13095920,"kb_used_data":8733340,"kb_used_omap":18,"kb_used_meta":4362221,"kb_avail":736865296,"statfs":{"total":767960285184,"available":754550063104,"internally_reserved":0,"allocated":8942940160,"data_stored":8937124035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4466915312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,4,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-438566912,"num_objects":-6692,"num_object_clones":0,"num_object_copies":-20076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6692,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6692,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":48758784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.596666"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":627,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.206973+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:09:01.206973+0000","last_peered":"2024-09-21T19:09:01.206973+0000","last_clean":"2024-09-21T19:09:01.206973+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:09:01.206973+0000","last_undegraded":"2024-09-21T19:09:01.206973+0000","last_fullsized":"2024-09-21T19:09:01.206973+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":795,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196733+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:09:01.196733+0000","last_peered":"2024-09-21T19:09:01.196733+0000","last_clean":"2024-09-21T19:09:01.196733+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:09:01.196733+0000","last_undegraded":"2024-09-21T19:09:01.196733+0000","last_fullsized":"2024-09-21T19:09:01.196733+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":724,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196658+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:09:01.196658+0000","last_peered":"2024-09-21T19:09:01.196658+0000","last_clean":"2024-09-21T19:09:01.196658+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:09:01.196658+0000","last_undegraded":"2024-09-21T19:09:01.196658+0000","last_fullsized":"2024-09-21T19:09:01.196658+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:31.228363+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":712,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196007+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:09:01.196007+0000","last_peered":"2024-09-21T19:09:01.196007+0000","last_clean":"2024-09-21T19:09:01.196007+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:09:01.196007+0000","last_undegraded":"2024-09-21T19:09:01.196007+0000","last_fullsized":"2024-09-21T19:09:01.196007+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":721,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200151+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:09:01.200151+0000","last_peered":"2024-09-21T19:09:01.200151+0000","last_clean":"2024-09-21T19:09:01.200151+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:09:01.200151+0000","last_undegraded":"2024-09-21T19:09:01.200151+0000","last_fullsized":"2024-09-21T19:09:01.200151+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":748,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199007+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:09:01.199007+0000","last_peered":"2024-09-21T19:09:01.199007+0000","last_clean":"2024-09-21T19:09:01.199007+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:09:01.199007+0000","last_undegraded":"2024-09-21T19:09:01.199007+0000","last_fullsized":"2024-09-21T19:09:01.199007+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":845,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196424+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:09:01.196424+0000","last_peered":"2024-09-21T19:09:01.196424+0000","last_clean":"2024-09-21T19:09:01.196424+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:09:01.196424+0000","last_undegraded":"2024-09-21T19:09:01.196424+0000","last_fullsized":"2024-09-21T19:09:01.196424+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":732,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196657+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:09:01.196657+0000","last_peered":"2024-09-21T19:09:01.196657+0000","last_clean":"2024-09-21T19:09:01.196657+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:09:01.196657+0000","last_undegraded":"2024-09-21T19:09:01.196657+0000","last_fullsized":"2024-09-21T19:09:01.196657+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":719,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197190+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:09:01.197190+0000","last_peered":"2024-09-21T19:09:01.197190+0000","last_clean":"2024-09-21T19:09:01.197190+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:09:01.197190+0000","last_undegraded":"2024-09-21T19:09:01.197190+0000","last_fullsized":"2024-09-21T19:09:01.197190+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":776,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196301+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:09:01.196301+0000","last_peered":"2024-09-21T19:09:01.196301+0000","last_clean":"2024-09-21T19:09:01.196301+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:09:01.196301+0000","last_undegraded":"2024-09-21T19:09:01.196301+0000","last_fullsized":"2024-09-21T19:09:01.196301+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":672,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.187382+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:09:01.187382+0000","last_peered":"2024-09-21T19:09:01.187382+0000","last_clean":"2024-09-21T19:09:01.187382+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:09:01.187382+0000","last_undegraded":"2024-09-21T19:09:01.187382+0000","last_fullsized":"2024-09-21T19:09:01.187382+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":593,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.204693+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:09:01.204693+0000","last_peered":"2024-09-21T19:09:01.204693+0000","last_clean":"2024-09-21T19:09:01.204693+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:09:01.204693+0000","last_undegraded":"2024-09-21T19:09:01.204693+0000","last_fullsized":"2024-09-21T19:09:01.204693+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":272,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199941+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:09:01.199941+0000","last_peered":"2024-09-21T19:09:01.199941+0000","last_clean":"2024-09-21T19:09:01.199941+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:09:01.199941+0000","last_undegraded":"2024-09-21T19:09:01.199941+0000","last_fullsized":"2024-09-21T19:09:01.199941+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":722,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196125+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:09:01.196125+0000","last_peered":"2024-09-21T19:09:01.196125+0000","last_clean":"2024-09-21T19:09:01.196125+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:09:01.196125+0000","last_undegraded":"2024-09-21T19:09:01.196125+0000","last_fullsized":"2024-09-21T19:09:01.196125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":704,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199761+0000","last_change":"2024-09-21T19:03:52.165260+0000","last_active":"2024-09-21T19:09:01.199761+0000","last_peered":"2024-09-21T19:09:01.199761+0000","last_clean":"2024-09-21T19:09:01.199761+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:09:01.199761+0000","last_undegraded":"2024-09-21T19:09:01.199761+0000","last_fullsized":"2024-09-21T19:09:01.199761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:52.165196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:52.165196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":280,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199793+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:09:01.199793+0000","last_peered":"2024-09-21T19:09:01.199793+0000","last_clean":"2024-09-21T19:09:01.199793+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:09:01.199793+0000","last_undegraded":"2024-09-21T19:09:01.199793+0000","last_fullsized":"2024-09-21T19:09:01.199793+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.570833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":740,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.202590+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:09:01.202590+0000","last_peered":"2024-09-21T19:09:01.202590+0000","last_clean":"2024-09-21T19:09:01.202590+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:09:01.202590+0000","last_undegraded":"2024-09-21T19:09:01.202590+0000","last_fullsized":"2024-09-21T19:09:01.202590+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":609,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.206106+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:09:01.206106+0000","last_peered":"2024-09-21T19:09:01.206106+0000","last_clean":"2024-09-21T19:09:01.206106+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:09:01.206106+0000","last_undegraded":"2024-09-21T19:09:01.206106+0000","last_fullsized":"2024-09-21T19:09:01.206106+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":183,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197492+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:09:01.197492+0000","last_peered":"2024-09-21T19:09:01.197492+0000","last_clean":"2024-09-21T19:09:01.197492+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:09:01.197492+0000","last_undegraded":"2024-09-21T19:09:01.197492+0000","last_fullsized":"2024-09-21T19:09:01.197492+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":727,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197599+0000","last_change":"2024-09-21T19:02:38.750456+0000","last_active":"2024-09-21T19:09:01.197599+0000","last_peered":"2024-09-21T19:09:01.197599+0000","last_clean":"2024-09-21T19:09:01.197599+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:09:01.197599+0000","last_undegraded":"2024-09-21T19:09:01.197599+0000","last_fullsized":"2024-09-21T19:09:01.197599+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.750398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.750398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:02.338670+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":717,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200098+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:09:01.200098+0000","last_peered":"2024-09-21T19:09:01.200098+0000","last_clean":"2024-09-21T19:09:01.200098+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:09:01.200098+0000","last_undegraded":"2024-09-21T19:09:01.200098+0000","last_fullsized":"2024-09-21T19:09:01.200098+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":743,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196420+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:09:01.196420+0000","last_peered":"2024-09-21T19:09:01.196420+0000","last_clean":"2024-09-21T19:09:01.196420+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:09:01.196420+0000","last_undegraded":"2024-09-21T19:09:01.196420+0000","last_fullsized":"2024-09-21T19:09:01.196420+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":788,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198583+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:09:01.198583+0000","last_peered":"2024-09-21T19:09:01.198583+0000","last_clean":"2024-09-21T19:09:01.198583+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:09:01.198583+0000","last_undegraded":"2024-09-21T19:09:01.198583+0000","last_fullsized":"2024-09-21T19:09:01.198583+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":727,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199635+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:09:01.199635+0000","last_peered":"2024-09-21T19:09:01.199635+0000","last_clean":"2024-09-21T19:09:01.199635+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:09:01.199635+0000","last_undegraded":"2024-09-21T19:09:01.199635+0000","last_fullsized":"2024-09-21T19:09:01.199635+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":750,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199514+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:09:01.199514+0000","last_peered":"2024-09-21T19:09:01.199514+0000","last_clean":"2024-09-21T19:09:01.199514+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:09:01.199514+0000","last_undegraded":"2024-09-21T19:09:01.199514+0000","last_fullsized":"2024-09-21T19:09:01.199514+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":749,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196745+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:09:01.196745+0000","last_peered":"2024-09-21T19:09:01.196745+0000","last_clean":"2024-09-21T19:09:01.196745+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:09:01.196745+0000","last_undegraded":"2024-09-21T19:09:01.196745+0000","last_fullsized":"2024-09-21T19:09:01.196745+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":637,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.212247+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:09:01.212247+0000","last_peered":"2024-09-21T19:09:01.212247+0000","last_clean":"2024-09-21T19:09:01.212247+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:09:01.212247+0000","last_undegraded":"2024-09-21T19:09:01.212247+0000","last_fullsized":"2024-09-21T19:09:01.212247+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":804,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199253+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:09:01.199253+0000","last_peered":"2024-09-21T19:09:01.199253+0000","last_clean":"2024-09-21T19:09:01.199253+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:09:01.199253+0000","last_undegraded":"2024-09-21T19:09:01.199253+0000","last_fullsized":"2024-09-21T19:09:01.199253+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"394'12001","reported_seq":12141,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.306784+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:09:01.306784+0000","last_peered":"2024-09-21T19:09:01.306784+0000","last_clean":"2024-09-21T19:09:01.306784+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:09:01.306784+0000","last_undegraded":"2024-09-21T19:09:01.306784+0000","last_fullsized":"2024-09-21T19:09:01.306784+0000","mapping_epoch":330,"log_start":"367'7500","ondisk_log_start":"367'7500","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188678144,"num_objects":2879,"num_object_clones":0,"num_object_copies":8637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":178,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197093+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:09:01.197093+0000","last_peered":"2024-09-21T19:09:01.197093+0000","last_clean":"2024-09-21T19:09:01.197093+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:09:01.197093+0000","last_undegraded":"2024-09-21T19:09:01.197093+0000","last_fullsized":"2024-09-21T19:09:01.197093+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":777,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199714+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:09:01.199714+0000","last_peered":"2024-09-21T19:09:01.199714+0000","last_clean":"2024-09-21T19:09:01.199714+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:09:01.199714+0000","last_undegraded":"2024-09-21T19:09:01.199714+0000","last_fullsized":"2024-09-21T19:09:01.199714+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:35.238469+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":715,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195727+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:09:01.195727+0000","last_peered":"2024-09-21T19:09:01.195727+0000","last_clean":"2024-09-21T19:09:01.195727+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:09:01.195727+0000","last_undegraded":"2024-09-21T19:09:01.195727+0000","last_fullsized":"2024-09-21T19:09:01.195727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":956,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.202718+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:09:01.202718+0000","last_peered":"2024-09-21T19:09:01.202718+0000","last_clean":"2024-09-21T19:09:01.202718+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:09:01.202718+0000","last_undegraded":"2024-09-21T19:09:01.202718+0000","last_fullsized":"2024-09-21T19:09:01.202718+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":404,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197518+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:09:01.197518+0000","last_peered":"2024-09-21T19:09:01.197518+0000","last_clean":"2024-09-21T19:09:01.197518+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:09:01.197518+0000","last_undegraded":"2024-09-21T19:09:01.197518+0000","last_fullsized":"2024-09-21T19:09:01.197518+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"394'11639","reported_seq":11777,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.373498+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:09:01.373498+0000","last_peered":"2024-09-21T19:09:01.373498+0000","last_clean":"2024-09-21T19:09:01.373498+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:09:01.373498+0000","last_undegraded":"2024-09-21T19:09:01.373498+0000","last_fullsized":"2024-09-21T19:09:01.373498+0000","mapping_epoch":330,"log_start":"367'7100","ondisk_log_start":"367'7100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183435264,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11639,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":503,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197100+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:09:01.197100+0000","last_peered":"2024-09-21T19:09:01.197100+0000","last_clean":"2024-09-21T19:09:01.197100+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:09:01.197100+0000","last_undegraded":"2024-09-21T19:09:01.197100+0000","last_fullsized":"2024-09-21T19:09:01.197100+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":811,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195509+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:09:01.195509+0000","last_peered":"2024-09-21T19:09:01.195509+0000","last_clean":"2024-09-21T19:09:01.195509+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:09:01.195509+0000","last_undegraded":"2024-09-21T19:09:01.195509+0000","last_fullsized":"2024-09-21T19:09:01.195509+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":737,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201404+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:09:01.201404+0000","last_peered":"2024-09-21T19:09:01.201404+0000","last_clean":"2024-09-21T19:09:01.201404+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:09:01.201404+0000","last_undegraded":"2024-09-21T19:09:01.201404+0000","last_fullsized":"2024-09-21T19:09:01.201404+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":714,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199904+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:09:01.199904+0000","last_peered":"2024-09-21T19:09:01.199904+0000","last_clean":"2024-09-21T19:09:01.199904+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:09:01.199904+0000","last_undegraded":"2024-09-21T19:09:01.199904+0000","last_fullsized":"2024-09-21T19:09:01.199904+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":731,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196224+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:09:01.196224+0000","last_peered":"2024-09-21T19:09:01.196224+0000","last_clean":"2024-09-21T19:09:01.196224+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:09:01.196224+0000","last_undegraded":"2024-09-21T19:09:01.196224+0000","last_fullsized":"2024-09-21T19:09:01.196224+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"394'11933","reported_seq":12073,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.237336+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:09:04.237336+0000","last_peered":"2024-09-21T19:09:04.237336+0000","last_clean":"2024-09-21T19:09:04.237336+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:09:04.237336+0000","last_undegraded":"2024-09-21T19:09:04.237336+0000","last_fullsized":"2024-09-21T19:09:04.237336+0000","mapping_epoch":330,"log_start":"364'6600","ondisk_log_start":"364'6600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181338112,"num_objects":2767,"num_object_clones":0,"num_object_copies":8301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":494,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200537+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:09:01.200537+0000","last_peered":"2024-09-21T19:09:01.200537+0000","last_clean":"2024-09-21T19:09:01.200537+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:09:01.200537+0000","last_undegraded":"2024-09-21T19:09:01.200537+0000","last_fullsized":"2024-09-21T19:09:01.200537+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":782,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200151+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:09:01.200151+0000","last_peered":"2024-09-21T19:09:01.200151+0000","last_clean":"2024-09-21T19:09:01.200151+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:09:01.200151+0000","last_undegraded":"2024-09-21T19:09:01.200151+0000","last_fullsized":"2024-09-21T19:09:01.200151+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":606,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.212520+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:09:01.212520+0000","last_peered":"2024-09-21T19:09:01.212520+0000","last_clean":"2024-09-21T19:09:01.212520+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:09:01.212520+0000","last_undegraded":"2024-09-21T19:09:01.212520+0000","last_fullsized":"2024-09-21T19:09:01.212520+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":899,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195348+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:09:01.195348+0000","last_peered":"2024-09-21T19:09:01.195348+0000","last_clean":"2024-09-21T19:09:01.195348+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:09:01.195348+0000","last_undegraded":"2024-09-21T19:09:01.195348+0000","last_fullsized":"2024-09-21T19:09:01.195348+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":734,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199424+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:09:01.199424+0000","last_peered":"2024-09-21T19:09:01.199424+0000","last_clean":"2024-09-21T19:09:01.199424+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:09:01.199424+0000","last_undegraded":"2024-09-21T19:09:01.199424+0000","last_fullsized":"2024-09-21T19:09:01.199424+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"394'12059","reported_seq":12199,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.861318+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:09:01.861318+0000","last_peered":"2024-09-21T19:09:01.861318+0000","last_clean":"2024-09-21T19:09:01.861318+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:09:01.861318+0000","last_undegraded":"2024-09-21T19:09:01.861318+0000","last_fullsized":"2024-09-21T19:09:01.861318+0000","mapping_epoch":330,"log_start":"365'7000","ondisk_log_start":"365'7000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5059,"log_dups_size":0,"ondisk_log_size":5059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188678144,"num_objects":2879,"num_object_clones":0,"num_object_copies":8637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":518,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199414+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:09:01.199414+0000","last_peered":"2024-09-21T19:09:01.199414+0000","last_clean":"2024-09-21T19:09:01.199414+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:09:01.199414+0000","last_undegraded":"2024-09-21T19:09:01.199414+0000","last_fullsized":"2024-09-21T19:09:01.199414+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":759,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201591+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:09:01.201591+0000","last_peered":"2024-09-21T19:09:01.201591+0000","last_clean":"2024-09-21T19:09:01.201591+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:09:01.201591+0000","last_undegraded":"2024-09-21T19:09:01.201591+0000","last_fullsized":"2024-09-21T19:09:01.201591+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:33.202656+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":278,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198666+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:09:01.198666+0000","last_peered":"2024-09-21T19:09:01.198666+0000","last_clean":"2024-09-21T19:09:01.198666+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:09:01.198666+0000","last_undegraded":"2024-09-21T19:09:01.198666+0000","last_fullsized":"2024-09-21T19:09:01.198666+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":783,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.206064+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:09:01.206064+0000","last_peered":"2024-09-21T19:09:01.206064+0000","last_clean":"2024-09-21T19:09:01.206064+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:09:01.206064+0000","last_undegraded":"2024-09-21T19:09:01.206064+0000","last_fullsized":"2024-09-21T19:09:01.206064+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":608,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.204772+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:09:01.204772+0000","last_peered":"2024-09-21T19:09:01.204772+0000","last_clean":"2024-09-21T19:09:01.204772+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:09:01.204772+0000","last_undegraded":"2024-09-21T19:09:01.204772+0000","last_fullsized":"2024-09-21T19:09:01.204772+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"394'11923","reported_seq":12064,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.226521+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:09:04.226521+0000","last_peered":"2024-09-21T19:09:04.226521+0000","last_clean":"2024-09-21T19:09:04.226521+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:09:04.226521+0000","last_undegraded":"2024-09-21T19:09:04.226521+0000","last_fullsized":"2024-09-21T19:09:04.226521+0000","mapping_epoch":330,"log_start":"364'6600","ondisk_log_start":"364'6600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355096,"num_objects":2753,"num_object_clones":0,"num_object_copies":8259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11923,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":496,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.192118+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:09:01.192118+0000","last_peered":"2024-09-21T19:09:01.192118+0000","last_clean":"2024-09-21T19:09:01.192118+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:09:01.192118+0000","last_undegraded":"2024-09-21T19:09:01.192118+0000","last_fullsized":"2024-09-21T19:09:01.192118+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":654,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.206178+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:09:01.206178+0000","last_peered":"2024-09-21T19:09:01.206178+0000","last_clean":"2024-09-21T19:09:01.206178+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:09:01.206178+0000","last_undegraded":"2024-09-21T19:09:01.206178+0000","last_fullsized":"2024-09-21T19:09:01.206178+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":747,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197161+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:09:01.197161+0000","last_peered":"2024-09-21T19:09:01.197161+0000","last_clean":"2024-09-21T19:09:01.197161+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:09:01.197161+0000","last_undegraded":"2024-09-21T19:09:01.197161+0000","last_fullsized":"2024-09-21T19:09:01.197161+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:34.228030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":632,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.190420+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:09:01.190420+0000","last_peered":"2024-09-21T19:09:01.190420+0000","last_clean":"2024-09-21T19:09:01.190420+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:09:01.190420+0000","last_undegraded":"2024-09-21T19:09:01.190420+0000","last_fullsized":"2024-09-21T19:09:01.190420+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":735,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196552+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:09:01.196552+0000","last_peered":"2024-09-21T19:09:01.196552+0000","last_clean":"2024-09-21T19:09:01.196552+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:09:01.196552+0000","last_undegraded":"2024-09-21T19:09:01.196552+0000","last_fullsized":"2024-09-21T19:09:01.196552+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.253514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"394'12066","reported_seq":21573,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.278682+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:09:04.278682+0000","last_peered":"2024-09-21T19:09:04.278682+0000","last_clean":"2024-09-21T19:09:04.278682+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:09:04.278682+0000","last_undegraded":"2024-09-21T19:09:04.278682+0000","last_fullsized":"2024-09-21T19:09:04.278682+0000","mapping_epoch":380,"log_start":"365'6800","ondisk_log_start":"365'6800","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452224,"num_objects":2784,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12066,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":472,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.204871+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:09:01.204871+0000","last_peered":"2024-09-21T19:09:01.204871+0000","last_clean":"2024-09-21T19:09:01.204871+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:09:01.204871+0000","last_undegraded":"2024-09-21T19:09:01.204871+0000","last_fullsized":"2024-09-21T19:09:01.204871+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1027,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197074+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:09:01.197074+0000","last_peered":"2024-09-21T19:09:01.197074+0000","last_clean":"2024-09-21T19:09:01.197074+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:09:01.197074+0000","last_undegraded":"2024-09-21T19:09:01.197074+0000","last_fullsized":"2024-09-21T19:09:01.197074+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:45.305870+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":755,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196449+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:09:01.196449+0000","last_peered":"2024-09-21T19:09:01.196449+0000","last_clean":"2024-09-21T19:09:01.196449+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:09:01.196449+0000","last_undegraded":"2024-09-21T19:09:01.196449+0000","last_fullsized":"2024-09-21T19:09:01.196449+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":302,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.205681+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:09:01.205681+0000","last_peered":"2024-09-21T19:09:01.205681+0000","last_clean":"2024-09-21T19:09:01.205681+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:09:01.205681+0000","last_undegraded":"2024-09-21T19:09:01.205681+0000","last_fullsized":"2024-09-21T19:09:01.205681+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:25.256087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":594,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.205989+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:09:01.205989+0000","last_peered":"2024-09-21T19:09:01.205989+0000","last_clean":"2024-09-21T19:09:01.205989+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:09:01.205989+0000","last_undegraded":"2024-09-21T19:09:01.205989+0000","last_fullsized":"2024-09-21T19:09:01.205989+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":703,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198794+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:09:01.198794+0000","last_peered":"2024-09-21T19:09:01.198794+0000","last_clean":"2024-09-21T19:09:01.198794+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:09:01.198794+0000","last_undegraded":"2024-09-21T19:09:01.198794+0000","last_fullsized":"2024-09-21T19:09:01.198794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"394'11809","reported_seq":21230,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.784472+0000","last_change":"2024-09-21T19:08:54.754076+0000","last_active":"2024-09-21T19:09:02.784472+0000","last_peered":"2024-09-21T19:09:02.784472+0000","last_clean":"2024-09-21T19:09:02.784472+0000","last_became_active":"2024-09-21T19:08:54.722178+0000","last_became_peered":"2024-09-21T19:08:54.722178+0000","last_unstale":"2024-09-21T19:09:02.784472+0000","last_undegraded":"2024-09-21T19:09:02.784472+0000","last_fullsized":"2024-09-21T19:09:02.784472+0000","mapping_epoch":389,"log_start":"355'5400","ondisk_log_start":"355'5400","created":330,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":8355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11809,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":303890432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":807,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199174+0000","last_change":"2024-09-21T19:08:53.739015+0000","last_active":"2024-09-21T19:09:01.199174+0000","last_peered":"2024-09-21T19:09:01.199174+0000","last_clean":"2024-09-21T19:09:01.199174+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:09:01.199174+0000","last_undegraded":"2024-09-21T19:09:01.199174+0000","last_fullsized":"2024-09-21T19:09:01.199174+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:08:53.738929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.738929+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:55.112735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":220,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198106+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:09:01.198106+0000","last_peered":"2024-09-21T19:09:01.198106+0000","last_clean":"2024-09-21T19:09:01.198106+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:09:01.198106+0000","last_undegraded":"2024-09-21T19:09:01.198106+0000","last_fullsized":"2024-09-21T19:09:01.198106+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":777,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.141807+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:09:02.141807+0000","last_peered":"2024-09-21T19:09:02.141807+0000","last_clean":"2024-09-21T19:09:02.141807+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:09:02.141807+0000","last_undegraded":"2024-09-21T19:09:02.141807+0000","last_fullsized":"2024-09-21T19:09:02.141807+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":171,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199104+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:09:01.199104+0000","last_peered":"2024-09-21T19:09:01.199104+0000","last_clean":"2024-09-21T19:09:01.199104+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:09:01.199104+0000","last_undegraded":"2024-09-21T19:09:01.199104+0000","last_fullsized":"2024-09-21T19:09:01.199104+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":170,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198191+0000","last_change":"2024-09-21T19:06:54.694760+0000","last_active":"2024-09-21T19:09:01.198191+0000","last_peered":"2024-09-21T19:09:01.198191+0000","last_clean":"2024-09-21T19:09:01.198191+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:09:01.198191+0000","last_undegraded":"2024-09-21T19:09:01.198191+0000","last_fullsized":"2024-09-21T19:09:01.198191+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":724,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200077+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:09:01.200077+0000","last_peered":"2024-09-21T19:09:01.200077+0000","last_clean":"2024-09-21T19:09:01.200077+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:09:01.200077+0000","last_undegraded":"2024-09-21T19:09:01.200077+0000","last_fullsized":"2024-09-21T19:09:01.200077+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"394'11885","reported_seq":12025,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.709210+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:09:02.709210+0000","last_peered":"2024-09-21T19:09:02.709210+0000","last_clean":"2024-09-21T19:09:02.709210+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:09:02.709210+0000","last_undegraded":"2024-09-21T19:09:02.709210+0000","last_fullsized":"2024-09-21T19:09:02.709210+0000","mapping_epoch":330,"log_start":"365'6800","ondisk_log_start":"365'6800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186843136,"num_objects":2851,"num_object_clones":0,"num_object_copies":8553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11885,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":505,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197859+0000","last_change":"2024-09-21T19:07:30.274632+0000","last_active":"2024-09-21T19:09:01.197859+0000","last_peered":"2024-09-21T19:09:01.197859+0000","last_clean":"2024-09-21T19:09:01.197859+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:09:01.197859+0000","last_undegraded":"2024-09-21T19:09:01.197859+0000","last_fullsized":"2024-09-21T19:09:01.197859+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:07:30.274412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:07:30.274412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":716,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.212343+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:09:01.212343+0000","last_peered":"2024-09-21T19:09:01.212343+0000","last_clean":"2024-09-21T19:09:01.212343+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:09:01.212343+0000","last_undegraded":"2024-09-21T19:09:01.212343+0000","last_fullsized":"2024-09-21T19:09:01.212343+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":602,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.205924+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:09:01.205924+0000","last_peered":"2024-09-21T19:09:01.205924+0000","last_clean":"2024-09-21T19:09:01.205924+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:09:01.205924+0000","last_undegraded":"2024-09-21T19:09:01.205924+0000","last_fullsized":"2024-09-21T19:09:01.205924+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1694,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198560+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:09:01.198560+0000","last_peered":"2024-09-21T19:09:01.198560+0000","last_clean":"2024-09-21T19:09:01.198560+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:09:01.198560+0000","last_undegraded":"2024-09-21T19:09:01.198560+0000","last_fullsized":"2024-09-21T19:09:01.198560+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":943,"num_read_kb":938,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":674,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.188887+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:09:01.188887+0000","last_peered":"2024-09-21T19:09:01.188887+0000","last_clean":"2024-09-21T19:09:01.188887+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:09:01.188887+0000","last_undegraded":"2024-09-21T19:09:01.188887+0000","last_fullsized":"2024-09-21T19:09:01.188887+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":729,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196842+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:09:01.196842+0000","last_peered":"2024-09-21T19:09:01.196842+0000","last_clean":"2024-09-21T19:09:01.196842+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:09:01.196842+0000","last_undegraded":"2024-09-21T19:09:01.196842+0000","last_fullsized":"2024-09-21T19:09:01.196842+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:30.010503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"394'11931","reported_seq":12071,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.226756+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:09:04.226756+0000","last_peered":"2024-09-21T19:09:04.226756+0000","last_clean":"2024-09-21T19:09:04.226756+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:09:04.226756+0000","last_undegraded":"2024-09-21T19:09:04.226756+0000","last_fullsized":"2024-09-21T19:09:04.226756+0000","mapping_epoch":330,"log_start":"364'6600","ondisk_log_start":"364'6600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177668096,"num_objects":2711,"num_object_clones":0,"num_object_copies":8133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":895,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.191862+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:09:01.191862+0000","last_peered":"2024-09-21T19:09:01.191862+0000","last_clean":"2024-09-21T19:09:01.191862+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:09:01.191862+0000","last_undegraded":"2024-09-21T19:09:01.191862+0000","last_fullsized":"2024-09-21T19:09:01.191862+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":737,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196758+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:09:01.196758+0000","last_peered":"2024-09-21T19:09:01.196758+0000","last_clean":"2024-09-21T19:09:01.196758+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:09:01.196758+0000","last_undegraded":"2024-09-21T19:09:01.196758+0000","last_fullsized":"2024-09-21T19:09:01.196758+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":813,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198680+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:09:01.198680+0000","last_peered":"2024-09-21T19:09:01.198680+0000","last_clean":"2024-09-21T19:09:01.198680+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:09:01.198680+0000","last_undegraded":"2024-09-21T19:09:01.198680+0000","last_fullsized":"2024-09-21T19:09:01.198680+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":738,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200664+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:09:01.200664+0000","last_peered":"2024-09-21T19:09:01.200664+0000","last_clean":"2024-09-21T19:09:01.200664+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:09:01.200664+0000","last_undegraded":"2024-09-21T19:09:01.200664+0000","last_fullsized":"2024-09-21T19:09:01.200664+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":264,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200016+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:09:01.200016+0000","last_peered":"2024-09-21T19:09:01.200016+0000","last_clean":"2024-09-21T19:09:01.200016+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:09:01.200016+0000","last_undegraded":"2024-09-21T19:09:01.200016+0000","last_fullsized":"2024-09-21T19:09:01.200016+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"394'11821","reported_seq":11961,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.229883+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:09:04.229883+0000","last_peered":"2024-09-21T19:09:04.229883+0000","last_clean":"2024-09-21T19:09:04.229883+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:09:04.229883+0000","last_undegraded":"2024-09-21T19:09:04.229883+0000","last_fullsized":"2024-09-21T19:09:04.229883+0000","mapping_epoch":330,"log_start":"364'6500","ondisk_log_start":"364'6500","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171900928,"num_objects":2623,"num_object_clones":0,"num_object_copies":7869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11821,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":776,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.202773+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:09:01.202773+0000","last_peered":"2024-09-21T19:09:01.202773+0000","last_clean":"2024-09-21T19:09:01.202773+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:09:01.202773+0000","last_undegraded":"2024-09-21T19:09:01.202773+0000","last_fullsized":"2024-09-21T19:09:01.202773+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":744,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200448+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:09:01.200448+0000","last_peered":"2024-09-21T19:09:01.200448+0000","last_clean":"2024-09-21T19:09:01.200448+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:09:01.200448+0000","last_undegraded":"2024-09-21T19:09:01.200448+0000","last_fullsized":"2024-09-21T19:09:01.200448+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":728,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196697+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:09:01.196697+0000","last_peered":"2024-09-21T19:09:01.196697+0000","last_clean":"2024-09-21T19:09:01.196697+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:09:01.196697+0000","last_undegraded":"2024-09-21T19:09:01.196697+0000","last_fullsized":"2024-09-21T19:09:01.196697+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":709,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200483+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:09:01.200483+0000","last_peered":"2024-09-21T19:09:01.200483+0000","last_clean":"2024-09-21T19:09:01.200483+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:09:01.200483+0000","last_undegraded":"2024-09-21T19:09:01.200483+0000","last_fullsized":"2024-09-21T19:09:01.200483+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"394'11881","reported_seq":21355,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.825567+0000","last_change":"2024-09-21T19:08:55.780967+0000","last_active":"2024-09-21T19:09:02.825567+0000","last_peered":"2024-09-21T19:09:02.825567+0000","last_clean":"2024-09-21T19:09:02.825567+0000","last_became_active":"2024-09-21T19:08:55.751488+0000","last_became_peered":"2024-09-21T19:08:55.751488+0000","last_unstale":"2024-09-21T19:09:02.825567+0000","last_undegraded":"2024-09-21T19:09:02.825567+0000","last_fullsized":"2024-09-21T19:09:02.825567+0000","mapping_epoch":390,"log_start":"355'5400","ondisk_log_start":"355'5400","created":330,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6481,"log_dups_size":0,"ondisk_log_size":6481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184352768,"num_objects":2813,"num_object_clones":0,"num_object_copies":8439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11881,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4662,"num_bytes_recovered":305528832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":174,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.193256+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:09:01.193256+0000","last_peered":"2024-09-21T19:09:01.193256+0000","last_clean":"2024-09-21T19:09:01.193256+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:09:01.193256+0000","last_undegraded":"2024-09-21T19:09:01.193256+0000","last_fullsized":"2024-09-21T19:09:01.193256+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":825,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196949+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:09:01.196949+0000","last_peered":"2024-09-21T19:09:01.196949+0000","last_clean":"2024-09-21T19:09:01.196949+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:09:01.196949+0000","last_undegraded":"2024-09-21T19:09:01.196949+0000","last_fullsized":"2024-09-21T19:09:01.196949+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":642,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.190661+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:09:01.190661+0000","last_peered":"2024-09-21T19:09:01.190661+0000","last_clean":"2024-09-21T19:09:01.190661+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:09:01.190661+0000","last_undegraded":"2024-09-21T19:09:01.190661+0000","last_fullsized":"2024-09-21T19:09:01.190661+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":167,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.190118+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:09:01.190118+0000","last_peered":"2024-09-21T19:09:01.190118+0000","last_clean":"2024-09-21T19:09:01.190118+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:09:01.190118+0000","last_undegraded":"2024-09-21T19:09:01.190118+0000","last_fullsized":"2024-09-21T19:09:01.190118+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":167,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197890+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:09:01.197890+0000","last_peered":"2024-09-21T19:09:01.197890+0000","last_clean":"2024-09-21T19:09:01.197890+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:09:01.197890+0000","last_undegraded":"2024-09-21T19:09:01.197890+0000","last_fullsized":"2024-09-21T19:09:01.197890+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"394'11920","reported_seq":12060,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.638290+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:09:02.638290+0000","last_peered":"2024-09-21T19:09:02.638290+0000","last_clean":"2024-09-21T19:09:02.638290+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:09:02.638290+0000","last_undegraded":"2024-09-21T19:09:02.638290+0000","last_fullsized":"2024-09-21T19:09:02.638290+0000","mapping_epoch":330,"log_start":"364'6700","ondisk_log_start":"364'6700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:46.323337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":2816,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11920,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":821,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199132+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:09:01.199132+0000","last_peered":"2024-09-21T19:09:01.199132+0000","last_clean":"2024-09-21T19:09:01.199132+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:09:01.199132+0000","last_undegraded":"2024-09-21T19:09:01.199132+0000","last_fullsized":"2024-09-21T19:09:01.199132+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":751,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199761+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:09:01.199761+0000","last_peered":"2024-09-21T19:09:01.199761+0000","last_clean":"2024-09-21T19:09:01.199761+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:09:01.199761+0000","last_undegraded":"2024-09-21T19:09:01.199761+0000","last_fullsized":"2024-09-21T19:09:01.199761+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":751,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201798+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:09:01.201798+0000","last_peered":"2024-09-21T19:09:01.201798+0000","last_clean":"2024-09-21T19:09:01.201798+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:09:01.201798+0000","last_undegraded":"2024-09-21T19:09:01.201798+0000","last_fullsized":"2024-09-21T19:09:01.201798+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":217,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.202395+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:09:01.202395+0000","last_peered":"2024-09-21T19:09:01.202395+0000","last_clean":"2024-09-21T19:09:01.202395+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:09:01.202395+0000","last_undegraded":"2024-09-21T19:09:01.202395+0000","last_fullsized":"2024-09-21T19:09:01.202395+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":724,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.202533+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:09:01.202533+0000","last_peered":"2024-09-21T19:09:01.202533+0000","last_clean":"2024-09-21T19:09:01.202533+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:09:01.202533+0000","last_undegraded":"2024-09-21T19:09:01.202533+0000","last_fullsized":"2024-09-21T19:09:01.202533+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"394'11651","reported_seq":11791,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.804735+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:09:02.804735+0000","last_peered":"2024-09-21T19:09:02.804735+0000","last_clean":"2024-09-21T19:09:02.804735+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:09:02.804735+0000","last_undegraded":"2024-09-21T19:09:02.804735+0000","last_fullsized":"2024-09-21T19:09:02.804735+0000","mapping_epoch":330,"log_start":"354'5200","ondisk_log_start":"354'5200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6451,"log_dups_size":0,"ondisk_log_size":6451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183173120,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11651,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":612,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.206262+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:09:01.206262+0000","last_peered":"2024-09-21T19:09:01.206262+0000","last_clean":"2024-09-21T19:09:01.206262+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:09:01.206262+0000","last_undegraded":"2024-09-21T19:09:01.206262+0000","last_fullsized":"2024-09-21T19:09:01.206262+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":712,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201616+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:09:01.201616+0000","last_peered":"2024-09-21T19:09:01.201616+0000","last_clean":"2024-09-21T19:09:01.201616+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:09:01.201616+0000","last_undegraded":"2024-09-21T19:09:01.201616+0000","last_fullsized":"2024-09-21T19:09:01.201616+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":159,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197008+0000","last_change":"2024-09-21T19:05:42.909550+0000","last_active":"2024-09-21T19:09:01.197008+0000","last_peered":"2024-09-21T19:09:01.197008+0000","last_clean":"2024-09-21T19:09:01.197008+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:09:01.197008+0000","last_undegraded":"2024-09-21T19:09:01.197008+0000","last_fullsized":"2024-09-21T19:09:01.197008+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:57.305767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.305767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":608,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.205826+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:09:01.205826+0000","last_peered":"2024-09-21T19:09:01.205826+0000","last_clean":"2024-09-21T19:09:01.205826+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:09:01.205826+0000","last_undegraded":"2024-09-21T19:09:01.205826+0000","last_fullsized":"2024-09-21T19:09:01.205826+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"394'11782","reported_seq":11922,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:02.677106+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:09:02.677106+0000","last_peered":"2024-09-21T19:09:02.677106+0000","last_clean":"2024-09-21T19:09:02.677106+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:09:02.677106+0000","last_undegraded":"2024-09-21T19:09:02.677106+0000","last_fullsized":"2024-09-21T19:09:02.677106+0000","mapping_epoch":330,"log_start":"365'6700","ondisk_log_start":"365'6700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5082,"log_dups_size":0,"ondisk_log_size":5082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180092928,"num_objects":2748,"num_object_clones":0,"num_object_copies":8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":661,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.190595+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:09:01.190595+0000","last_peered":"2024-09-21T19:09:01.190595+0000","last_clean":"2024-09-21T19:09:01.190595+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:09:01.190595+0000","last_undegraded":"2024-09-21T19:09:01.190595+0000","last_fullsized":"2024-09-21T19:09:01.190595+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":743,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201028+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:09:01.201028+0000","last_peered":"2024-09-21T19:09:01.201028+0000","last_clean":"2024-09-21T19:09:01.201028+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:09:01.201028+0000","last_undegraded":"2024-09-21T19:09:01.201028+0000","last_fullsized":"2024-09-21T19:09:01.201028+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":178,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198499+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:09:01.198499+0000","last_peered":"2024-09-21T19:09:01.198499+0000","last_clean":"2024-09-21T19:09:01.198499+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:09:01.198499+0000","last_undegraded":"2024-09-21T19:09:01.198499+0000","last_fullsized":"2024-09-21T19:09:01.198499+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":715,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198603+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:09:01.198603+0000","last_peered":"2024-09-21T19:09:01.198603+0000","last_clean":"2024-09-21T19:09:01.198603+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:09:01.198603+0000","last_undegraded":"2024-09-21T19:09:01.198603+0000","last_fullsized":"2024-09-21T19:09:01.198603+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"394'11922","reported_seq":13167,"reported_epoch":394,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:04.258344+0000","last_change":"2024-09-21T19:08:53.601069+0000","last_active":"2024-09-21T19:09:04.258344+0000","last_peered":"2024-09-21T19:09:04.258344+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:09:04.258344+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"364'6600","ondisk_log_start":"364'6600","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5322,"log_dups_size":0,"ondisk_log_size":5322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11922,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":551,"num_bytes_recovered":36110336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-1235}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":780,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200307+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:09:01.200307+0000","last_peered":"2024-09-21T19:09:01.200307+0000","last_clean":"2024-09-21T19:09:01.200307+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:09:01.200307+0000","last_undegraded":"2024-09-21T19:09:01.200307+0000","last_fullsized":"2024-09-21T19:09:01.200307+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":737,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200714+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:09:01.200714+0000","last_peered":"2024-09-21T19:09:01.200714+0000","last_clean":"2024-09-21T19:09:01.200714+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:09:01.200714+0000","last_undegraded":"2024-09-21T19:09:01.200714+0000","last_fullsized":"2024-09-21T19:09:01.200714+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":167,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.193168+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:09:01.193168+0000","last_peered":"2024-09-21T19:09:01.193168+0000","last_clean":"2024-09-21T19:09:01.193168+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:09:01.193168+0000","last_undegraded":"2024-09-21T19:09:01.193168+0000","last_fullsized":"2024-09-21T19:09:01.193168+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:00.385824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":703,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200273+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:09:01.200273+0000","last_peered":"2024-09-21T19:09:01.200273+0000","last_clean":"2024-09-21T19:09:01.200273+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:09:01.200273+0000","last_undegraded":"2024-09-21T19:09:01.200273+0000","last_fullsized":"2024-09-21T19:09:01.200273+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"394'12057","reported_seq":15195,"reported_epoch":394,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:02.693072+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:09:02.693072+0000","last_peered":"2024-09-21T19:09:02.693072+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:09:02.693072+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"365'7000","ondisk_log_start":"365'7000","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186974208,"num_objects":2853,"num_object_clones":0,"num_object_copies":8559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12057,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":98238464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":-397}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":335,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198334+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:09:01.198334+0000","last_peered":"2024-09-21T19:09:01.198334+0000","last_clean":"2024-09-21T19:09:01.198334+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:09:01.198334+0000","last_undegraded":"2024-09-21T19:09:01.198334+0000","last_fullsized":"2024-09-21T19:09:01.198334+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":751,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201052+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:09:01.201052+0000","last_peered":"2024-09-21T19:09:01.201052+0000","last_clean":"2024-09-21T19:09:01.201052+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:09:01.201052+0000","last_undegraded":"2024-09-21T19:09:01.201052+0000","last_fullsized":"2024-09-21T19:09:01.201052+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":213,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195862+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:09:01.195862+0000","last_peered":"2024-09-21T19:09:01.195862+0000","last_clean":"2024-09-21T19:09:01.195862+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:09:01.195862+0000","last_undegraded":"2024-09-21T19:09:01.195862+0000","last_fullsized":"2024-09-21T19:09:01.195862+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":738,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.190474+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:09:01.190474+0000","last_peered":"2024-09-21T19:09:01.190474+0000","last_clean":"2024-09-21T19:09:01.190474+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:09:01.190474+0000","last_undegraded":"2024-09-21T19:09:01.190474+0000","last_fullsized":"2024-09-21T19:09:01.190474+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":673,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.191381+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:09:01.191381+0000","last_peered":"2024-09-21T19:09:01.191381+0000","last_clean":"2024-09-21T19:09:01.191381+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:09:01.191381+0000","last_undegraded":"2024-09-21T19:09:01.191381+0000","last_fullsized":"2024-09-21T19:09:01.191381+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":736,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201272+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:09:01.201272+0000","last_peered":"2024-09-21T19:09:01.201272+0000","last_clean":"2024-09-21T19:09:01.201272+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:09:01.201272+0000","last_undegraded":"2024-09-21T19:09:01.201272+0000","last_fullsized":"2024-09-21T19:09:01.201272+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":272,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198718+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:09:01.198718+0000","last_peered":"2024-09-21T19:09:01.198718+0000","last_clean":"2024-09-21T19:09:01.198718+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:09:01.198718+0000","last_undegraded":"2024-09-21T19:09:01.198718+0000","last_fullsized":"2024-09-21T19:09:01.198718+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":711,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200094+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:09:01.200094+0000","last_peered":"2024-09-21T19:09:01.200094+0000","last_clean":"2024-09-21T19:09:01.200094+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:09:01.200094+0000","last_undegraded":"2024-09-21T19:09:01.200094+0000","last_fullsized":"2024-09-21T19:09:01.200094+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":612,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.212426+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:09:01.212426+0000","last_peered":"2024-09-21T19:09:01.212426+0000","last_clean":"2024-09-21T19:09:01.212426+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:09:01.212426+0000","last_undegraded":"2024-09-21T19:09:01.212426+0000","last_fullsized":"2024-09-21T19:09:01.212426+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":723,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199226+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:09:01.199226+0000","last_peered":"2024-09-21T19:09:01.199226+0000","last_clean":"2024-09-21T19:09:01.199226+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:09:01.199226+0000","last_undegraded":"2024-09-21T19:09:01.199226+0000","last_fullsized":"2024-09-21T19:09:01.199226+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":706,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201065+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:09:01.201065+0000","last_peered":"2024-09-21T19:09:01.201065+0000","last_clean":"2024-09-21T19:09:01.201065+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:09:01.201065+0000","last_undegraded":"2024-09-21T19:09:01.201065+0000","last_fullsized":"2024-09-21T19:09:01.201065+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":167,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.758906+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:09:01.758906+0000","last_peered":"2024-09-21T19:09:01.758906+0000","last_clean":"2024-09-21T19:09:01.758906+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:09:01.758906+0000","last_undegraded":"2024-09-21T19:09:01.758906+0000","last_fullsized":"2024-09-21T19:09:01.758906+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":325,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200188+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:09:01.200188+0000","last_peered":"2024-09-21T19:09:01.200188+0000","last_clean":"2024-09-21T19:09:01.200188+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:09:01.200188+0000","last_undegraded":"2024-09-21T19:09:01.200188+0000","last_fullsized":"2024-09-21T19:09:01.200188+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":609,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.207007+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:09:01.207007+0000","last_peered":"2024-09-21T19:09:01.207007+0000","last_clean":"2024-09-21T19:09:01.207007+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:09:01.207007+0000","last_undegraded":"2024-09-21T19:09:01.207007+0000","last_fullsized":"2024-09-21T19:09:01.207007+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":929,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200695+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:09:01.200695+0000","last_peered":"2024-09-21T19:09:01.200695+0000","last_clean":"2024-09-21T19:09:01.200695+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:09:01.200695+0000","last_undegraded":"2024-09-21T19:09:01.200695+0000","last_fullsized":"2024-09-21T19:09:01.200695+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":726,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196614+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:09:01.196614+0000","last_peered":"2024-09-21T19:09:01.196614+0000","last_clean":"2024-09-21T19:09:01.196614+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:09:01.196614+0000","last_undegraded":"2024-09-21T19:09:01.196614+0000","last_fullsized":"2024-09-21T19:09:01.196614+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":732,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200887+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:09:01.200887+0000","last_peered":"2024-09-21T19:09:01.200887+0000","last_clean":"2024-09-21T19:09:01.200887+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:09:01.200887+0000","last_undegraded":"2024-09-21T19:09:01.200887+0000","last_fullsized":"2024-09-21T19:09:01.200887+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":726,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199684+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:09:01.199684+0000","last_peered":"2024-09-21T19:09:01.199684+0000","last_clean":"2024-09-21T19:09:01.199684+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:09:01.199684+0000","last_undegraded":"2024-09-21T19:09:01.199684+0000","last_fullsized":"2024-09-21T19:09:01.199684+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":379,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197471+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:09:01.197471+0000","last_peered":"2024-09-21T19:09:01.197471+0000","last_clean":"2024-09-21T19:09:01.197471+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:09:01.197471+0000","last_undegraded":"2024-09-21T19:09:01.197471+0000","last_fullsized":"2024-09-21T19:09:01.197471+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":601,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.212668+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:09:01.212668+0000","last_peered":"2024-09-21T19:09:01.212668+0000","last_clean":"2024-09-21T19:09:01.212668+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:09:01.212668+0000","last_undegraded":"2024-09-21T19:09:01.212668+0000","last_fullsized":"2024-09-21T19:09:01.212668+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":805,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196218+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:09:01.196218+0000","last_peered":"2024-09-21T19:09:01.196218+0000","last_clean":"2024-09-21T19:09:01.196218+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:09:01.196218+0000","last_undegraded":"2024-09-21T19:09:01.196218+0000","last_fullsized":"2024-09-21T19:09:01.196218+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":724,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200020+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:09:01.200020+0000","last_peered":"2024-09-21T19:09:01.200020+0000","last_clean":"2024-09-21T19:09:01.200020+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:09:01.200020+0000","last_undegraded":"2024-09-21T19:09:01.200020+0000","last_fullsized":"2024-09-21T19:09:01.200020+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":724,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200714+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:09:01.200714+0000","last_peered":"2024-09-21T19:09:01.200714+0000","last_clean":"2024-09-21T19:09:01.200714+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:09:01.200714+0000","last_undegraded":"2024-09-21T19:09:01.200714+0000","last_fullsized":"2024-09-21T19:09:01.200714+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.268143+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":727,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195972+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:09:01.195972+0000","last_peered":"2024-09-21T19:09:01.195972+0000","last_clean":"2024-09-21T19:09:01.195972+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:09:01.195972+0000","last_undegraded":"2024-09-21T19:09:01.195972+0000","last_fullsized":"2024-09-21T19:09:01.195972+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":803,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.205335+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:09:01.205335+0000","last_peered":"2024-09-21T19:09:01.205335+0000","last_clean":"2024-09-21T19:09:01.205335+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:09:01.205335+0000","last_undegraded":"2024-09-21T19:09:01.205335+0000","last_fullsized":"2024-09-21T19:09:01.205335+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:52.278173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":728,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200900+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:09:01.200900+0000","last_peered":"2024-09-21T19:09:01.200900+0000","last_clean":"2024-09-21T19:09:01.200900+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:09:01.200900+0000","last_undegraded":"2024-09-21T19:09:01.200900+0000","last_fullsized":"2024-09-21T19:09:01.200900+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":728,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.200370+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:09:01.200370+0000","last_peered":"2024-09-21T19:09:01.200370+0000","last_clean":"2024-09-21T19:09:01.200370+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:09:01.200370+0000","last_undegraded":"2024-09-21T19:09:01.200370+0000","last_fullsized":"2024-09-21T19:09:01.200370+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":620,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.204796+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:09:01.204796+0000","last_peered":"2024-09-21T19:09:01.204796+0000","last_clean":"2024-09-21T19:09:01.204796+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:09:01.204796+0000","last_undegraded":"2024-09-21T19:09:01.204796+0000","last_fullsized":"2024-09-21T19:09:01.204796+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":752,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.196175+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:09:01.196175+0000","last_peered":"2024-09-21T19:09:01.196175+0000","last_clean":"2024-09-21T19:09:01.196175+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:09:01.196175+0000","last_undegraded":"2024-09-21T19:09:01.196175+0000","last_fullsized":"2024-09-21T19:09:01.196175+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":665,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.188065+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:09:01.188065+0000","last_peered":"2024-09-21T19:09:01.188065+0000","last_clean":"2024-09-21T19:09:01.188065+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:09:01.188065+0000","last_undegraded":"2024-09-21T19:09:01.188065+0000","last_fullsized":"2024-09-21T19:09:01.188065+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":739,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.198134+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:09:01.198134+0000","last_peered":"2024-09-21T19:09:01.198134+0000","last_clean":"2024-09-21T19:09:01.198134+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:09:01.198134+0000","last_undegraded":"2024-09-21T19:09:01.198134+0000","last_fullsized":"2024-09-21T19:09:01.198134+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":727,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.201718+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:09:01.201718+0000","last_peered":"2024-09-21T19:09:01.201718+0000","last_clean":"2024-09-21T19:09:01.201718+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:09:01.201718+0000","last_undegraded":"2024-09-21T19:09:01.201718+0000","last_fullsized":"2024-09-21T19:09:01.201718+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":751,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195411+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:09:01.195411+0000","last_peered":"2024-09-21T19:09:01.195411+0000","last_clean":"2024-09-21T19:09:01.195411+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:09:01.195411+0000","last_undegraded":"2024-09-21T19:09:01.195411+0000","last_fullsized":"2024-09-21T19:09:01.195411+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":604,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.206843+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:09:01.206843+0000","last_peered":"2024-09-21T19:09:01.206843+0000","last_clean":"2024-09-21T19:09:01.206843+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:09:01.206843+0000","last_undegraded":"2024-09-21T19:09:01.206843+0000","last_fullsized":"2024-09-21T19:09:01.206843+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":640,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.190062+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:09:01.190062+0000","last_peered":"2024-09-21T19:09:01.190062+0000","last_clean":"2024-09-21T19:09:01.190062+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:09:01.190062+0000","last_undegraded":"2024-09-21T19:09:01.190062+0000","last_fullsized":"2024-09-21T19:09:01.190062+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":722,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199916+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:09:01.199916+0000","last_peered":"2024-09-21T19:09:01.199916+0000","last_clean":"2024-09-21T19:09:01.199916+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:09:01.199916+0000","last_undegraded":"2024-09-21T19:09:01.199916+0000","last_fullsized":"2024-09-21T19:09:01.199916+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":728,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.197292+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:09:01.197292+0000","last_peered":"2024-09-21T19:09:01.197292+0000","last_clean":"2024-09-21T19:09:01.197292+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:09:01.197292+0000","last_undegraded":"2024-09-21T19:09:01.197292+0000","last_fullsized":"2024-09-21T19:09:01.197292+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":744,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.195652+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:09:01.195652+0000","last_peered":"2024-09-21T19:09:01.195652+0000","last_clean":"2024-09-21T19:09:01.195652+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:09:01.195652+0000","last_undegraded":"2024-09-21T19:09:01.195652+0000","last_fullsized":"2024-09-21T19:09:01.195652+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":706,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.199319+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:09:01.199319+0000","last_peered":"2024-09-21T19:09:01.199319+0000","last_clean":"2024-09-21T19:09:01.199319+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:09:01.199319+0000","last_undegraded":"2024-09-21T19:09:01.199319+0000","last_fullsized":"2024-09-21T19:09:01.199319+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":624,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T19:09:01.212598+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:09:01.212598+0000","last_peered":"2024-09-21T19:09:01.212598+0000","last_clean":"2024-09-21T19:09:01.212598+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:09:01.212598+0000","last_undegraded":"2024-09-21T19:09:01.212598+0000","last_fullsized":"2024-09-21T19:09:01.212598+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2926903320,"num_objects":44662,"num_object_clones":0,"num_object_copies":133986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190280,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16028,"num_bytes_recovered":1050411008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8844161024,"data_stored":8844148808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85780,"ondisk_log_size":85780,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":943,"num_read_kb":938,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266883,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330220,"kb_used_data":739544,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92414932,"statfs":{"total":95995035648,"available":94632890368,"internally_reserved":0,"allocated":757293056,"data_stored":756570139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":604830154},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464334,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506340,"kb_used_data":1239700,"kb_used_omap":4,"kb_used_meta":266619,"kb_avail":92238812,"statfs":{"total":95995035648,"available":94452543488,"internally_reserved":0,"allocated":1269452800,"data_stored":1268733574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":273018532},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619035,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121184,"kb_used_data":901264,"kb_used_omap":4,"kb_used_meta":219899,"kb_avail":92623968,"statfs":{"total":95995035648,"available":94846943232,"internally_reserved":0,"allocated":922894336,"data_stored":922179636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":225177251},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019089,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2011940,"kb_used_data":1497616,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91733212,"statfs":{"total":95995035648,"available":93934809088,"internally_reserved":0,"allocated":1533558784,"data_stored":1532837333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":526645702},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182614,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1635652,"kb_used_data":910256,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92109500,"statfs":{"total":95995035648,"available":94320128000,"internally_reserved":0,"allocated":932102144,"data_stored":931355994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411548,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1644696,"kb_used_data":1081732,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92100456,"statfs":{"total":95995035648,"available":94310866944,"internally_reserved":0,"allocated":1107693568,"data_stored":1106991161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216656,"kb_used_data":541308,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":92528496,"statfs":{"total":95995035648,"available":94749179904,"internally_reserved":0,"allocated":554299392,"data_stored":553540514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738598,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2629232,"kb_used_data":1821920,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":91115920,"statfs":{"total":95995035648,"available":93302702080,"internally_reserved":0,"allocated":1865646080,"data_stored":1864915684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1852116992,"data_stored":1852112920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":548864000,"data_stored":548864000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1102315520,"data_stored":1102315520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":922681344,"data_stored":922681344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1502416896,"data_stored":1502412824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":915079168,"data_stored":915079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1249710080,"data_stored":1249706008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":750977024,"data_stored":750977024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:09:08.982 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:08.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:09.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:09:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:09:09.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:09.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:09.266 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:09.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:09.403 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:09.423 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:09.484 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:09.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:09.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:09 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T19:09:09.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:09 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T19:09:09.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:09 smithi104 ceph-mon[32438]: pgmap v791: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 7094/127815 objects degraded (5.550%); 4.2 MiB/s, 81 objects/s recovering 2024-09-21T19:09:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:09.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:09.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:09 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T19:09:09.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:09 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T19:09:09.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:09 smithi081 ceph-mon[29384]: pgmap v791: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 7094/127815 objects degraded (5.550%); 4.2 MiB/s, 81 objects/s recovering 2024-09-21T19:09:09.558 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:09 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T19:09:09.558 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:09 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T19:09:09.559 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:09 smithi081 ceph-mon[39192]: pgmap v791: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s; 7094/127815 objects degraded (5.550%); 4.2 MiB/s, 81 objects/s recovering 2024-09-21T19:09:09.652 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:09.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:09.824 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:09.926 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:09.961 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:09.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:10.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:10.097 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:10.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:10.235 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:10.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:10.370 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:10 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:10.370 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:10 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 7094/127815 objects degraded (5.550%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:10.370 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:10 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:10.370 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:10 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 7094/127815 objects degraded (5.550%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:10.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:10.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:10 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:10.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:10 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 7094/127815 objects degraded (5.550%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.519 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:10.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:10.535 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:09:10.535 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:09:10.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:10.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:10.791 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:10.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:10.930 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:10.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:11.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:11.055 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:11.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:11.197 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:11.277 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:09:11.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.344 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:11.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:11.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:09:11.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:09:11.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:11.473 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T19:09:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T19:09:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[29384]: pgmap v792: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 6877/118410 objects degraded (5.808%); 4.0 MiB/s, 101 objects/s recovering 2024-09-21T19:09:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/121703122' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[29384]: from='client.36356 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T19:09:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T19:09:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[39192]: pgmap v792: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 6877/118410 objects degraded (5.808%); 4.0 MiB/s, 101 objects/s recovering 2024-09-21T19:09:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/121703122' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:11 smithi081 ceph-mon[39192]: from='client.36356 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:11.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.647 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:11.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:11.748 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:11.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:11 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T19:09:11.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:11 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T19:09:11.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:11 smithi104 ceph-mon[32438]: pgmap v792: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 6877/118410 objects degraded (5.808%); 4.0 MiB/s, 101 objects/s recovering 2024-09-21T19:09:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/121703122' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:11.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:11 smithi104 ceph-mon[32438]: from='client.36356 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.790 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:11.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:11.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:11.932 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:11.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:11.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:09:12.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:12.088 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:12.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:12.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:12.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:12.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:12.399 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:12.413 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:12.444 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:12.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:12.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:12.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:12 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:12.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:12 smithi081 ceph-mon[29384]: from='client.36356 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:09:12.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:12 smithi081 ceph-mon[29384]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T19:09:12.545 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:12 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:12.546 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:12 smithi081 ceph-mon[39192]: from='client.36356 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:09:12.546 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:12 smithi081 ceph-mon[39192]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T19:09:12.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.667 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:12.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:12.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:12 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:12.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:12 smithi104 ceph-mon[32438]: from='client.36356 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:09:12.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:12 smithi104 ceph-mon[32438]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T19:09:12.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:12.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:12.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:13.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:13.184 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:13.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:13.446 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:09:13.446 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:09:13.446 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:13 smithi081 ceph-mon[39192]: pgmap v794: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 993 op/s; 6877/116421 objects degraded (5.907%); 2.6 MiB/s, 81 objects/s recovering 2024-09-21T19:09:13.447 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:13 smithi081 ceph-mon[29384]: pgmap v794: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 993 op/s; 6877/116421 objects degraded (5.907%); 2.6 MiB/s, 81 objects/s recovering 2024-09-21T19:09:13.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:13.503 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:13.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:13.648 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:13.728 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:13.730 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:09:13.750 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:13.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:13 smithi104 ceph-mon[32438]: pgmap v794: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 993 op/s; 6877/116421 objects degraded (5.907%); 2.6 MiB/s, 81 objects/s recovering 2024-09-21T19:09:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:09:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=sqlstore.transactions t=2024-09-21T19:09:13.358697468Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T19:09:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.783 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:13.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:13.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:13.915 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:13.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:14.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:14.050 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:14.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:14.187 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:14.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:09:14.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:14.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:09:14.335 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:14.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:14.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:14 smithi081 ceph-mon[39192]: from='client.26215 v1:172.21.15.81:0/1123326668' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:14.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:14 smithi081 ceph-mon[39192]: pgmap v795: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 973 op/s; 6649/108228 objects degraded (6.144%); 658 KiB/s, 71 objects/s recovering 2024-09-21T19:09:14.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:14 smithi081 ceph-mon[29384]: from='client.26215 v1:172.21.15.81:0/1123326668' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:14.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:14 smithi081 ceph-mon[29384]: pgmap v795: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 973 op/s; 6649/108228 objects degraded (6.144%); 658 KiB/s, 71 objects/s recovering 2024-09-21T19:09:14.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.669 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:14.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:14 smithi104 ceph-mon[32438]: from='client.26215 v1:172.21.15.81:0/1123326668' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:14.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:14 smithi104 ceph-mon[32438]: pgmap v795: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 973 op/s; 6649/108228 objects degraded (6.144%); 658 KiB/s, 71 objects/s recovering 2024-09-21T19:09:14.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:14.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:14.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:14.941 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:14.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:15.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:15.081 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:15.176 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-21T19:09:11.764840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2530744883,"num_objects":38807,"num_object_clones":0,"num_object_copies":116421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38807,"num_whiteouts":0,"num_read":2088,"num_read_kb":1840,"num_write":197056,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16677,"num_bytes_recovered":1056778488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87293,"ondisk_log_size":87293,"up":474,"acting":472,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11899404,"kb_used_data":7565896,"kb_used_omap":18,"kb_used_meta":4333101,"kb_avail":738061812,"statfs":{"total":767960285184,"available":755775295488,"internally_reserved":0,"allocated":7747477504,"data_stored":7741614057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4437096432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-520880128,"num_objects":-7948,"num_object_clones":0,"num_object_copies":-23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7948,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":651,"num_bytes_recovered":21626880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005460"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":629,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.742649+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:09:04.742649+0000","last_peered":"2024-09-21T19:09:04.742649+0000","last_clean":"2024-09-21T19:09:04.742649+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:09:04.742649+0000","last_undegraded":"2024-09-21T19:09:04.742649+0000","last_fullsized":"2024-09-21T19:09:04.742649+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":799,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.273619+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:09:11.273619+0000","last_peered":"2024-09-21T19:09:11.273619+0000","last_clean":"2024-09-21T19:09:11.273619+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:09:11.273619+0000","last_undegraded":"2024-09-21T19:09:11.273619+0000","last_fullsized":"2024-09-21T19:09:11.273619+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":726,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734151+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:09:04.734151+0000","last_peered":"2024-09-21T19:09:04.734151+0000","last_clean":"2024-09-21T19:09:04.734151+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:09:04.734151+0000","last_undegraded":"2024-09-21T19:09:04.734151+0000","last_fullsized":"2024-09-21T19:09:04.734151+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:31.228363+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":716,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272974+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:09:11.272974+0000","last_peered":"2024-09-21T19:09:11.272974+0000","last_clean":"2024-09-21T19:09:11.272974+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:09:11.272974+0000","last_undegraded":"2024-09-21T19:09:11.272974+0000","last_fullsized":"2024-09-21T19:09:11.272974+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":723,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734440+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:09:04.734440+0000","last_peered":"2024-09-21T19:09:04.734440+0000","last_clean":"2024-09-21T19:09:04.734440+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:09:04.734440+0000","last_undegraded":"2024-09-21T19:09:04.734440+0000","last_fullsized":"2024-09-21T19:09:04.734440+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":750,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736316+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:09:04.736316+0000","last_peered":"2024-09-21T19:09:04.736316+0000","last_clean":"2024-09-21T19:09:04.736316+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:09:04.736316+0000","last_undegraded":"2024-09-21T19:09:04.736316+0000","last_fullsized":"2024-09-21T19:09:04.736316+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":847,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735174+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:09:04.735174+0000","last_peered":"2024-09-21T19:09:04.735174+0000","last_clean":"2024-09-21T19:09:04.735174+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:09:04.735174+0000","last_undegraded":"2024-09-21T19:09:04.735174+0000","last_fullsized":"2024-09-21T19:09:04.735174+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":736,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.273522+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:09:11.273522+0000","last_peered":"2024-09-21T19:09:11.273522+0000","last_clean":"2024-09-21T19:09:11.273522+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:09:11.273522+0000","last_undegraded":"2024-09-21T19:09:11.273522+0000","last_fullsized":"2024-09-21T19:09:11.273522+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":723,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.274424+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:09:11.274424+0000","last_peered":"2024-09-21T19:09:11.274424+0000","last_clean":"2024-09-21T19:09:11.274424+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:09:11.274424+0000","last_undegraded":"2024-09-21T19:09:11.274424+0000","last_fullsized":"2024-09-21T19:09:11.274424+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":780,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.273245+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:09:11.273245+0000","last_peered":"2024-09-21T19:09:11.273245+0000","last_clean":"2024-09-21T19:09:11.273245+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:09:11.273245+0000","last_undegraded":"2024-09-21T19:09:11.273245+0000","last_fullsized":"2024-09-21T19:09:11.273245+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":674,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737438+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:09:04.737438+0000","last_peered":"2024-09-21T19:09:04.737438+0000","last_clean":"2024-09-21T19:09:04.737438+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:09:04.737438+0000","last_undegraded":"2024-09-21T19:09:04.737438+0000","last_fullsized":"2024-09-21T19:09:04.737438+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":595,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.747290+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:09:04.747290+0000","last_peered":"2024-09-21T19:09:04.747290+0000","last_clean":"2024-09-21T19:09:04.747290+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:09:04.747290+0000","last_undegraded":"2024-09-21T19:09:04.747290+0000","last_fullsized":"2024-09-21T19:09:04.747290+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":276,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271708+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:09:11.271708+0000","last_peered":"2024-09-21T19:09:11.271708+0000","last_clean":"2024-09-21T19:09:11.271708+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:09:11.271708+0000","last_undegraded":"2024-09-21T19:09:11.271708+0000","last_fullsized":"2024-09-21T19:09:11.271708+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":726,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271810+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:09:11.271810+0000","last_peered":"2024-09-21T19:09:11.271810+0000","last_clean":"2024-09-21T19:09:11.271810+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:09:11.271810+0000","last_undegraded":"2024-09-21T19:09:11.271810+0000","last_fullsized":"2024-09-21T19:09:11.271810+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":714,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:05.634088+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:09:05.634088+0000","last_peered":"2024-09-21T19:09:05.634088+0000","last_clean":"2024-09-21T19:09:05.634088+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:09:05.634088+0000","last_undegraded":"2024-09-21T19:09:05.634088+0000","last_fullsized":"2024-09-21T19:09:05.634088+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":282,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734008+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:09:04.734008+0000","last_peered":"2024-09-21T19:09:04.734008+0000","last_clean":"2024-09-21T19:09:04.734008+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:09:04.734008+0000","last_undegraded":"2024-09-21T19:09:04.734008+0000","last_fullsized":"2024-09-21T19:09:04.734008+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.570833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":744,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272123+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:09:11.272123+0000","last_peered":"2024-09-21T19:09:11.272123+0000","last_clean":"2024-09-21T19:09:11.272123+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:09:11.272123+0000","last_undegraded":"2024-09-21T19:09:11.272123+0000","last_fullsized":"2024-09-21T19:09:11.272123+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":611,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.743456+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:09:04.743456+0000","last_peered":"2024-09-21T19:09:04.743456+0000","last_clean":"2024-09-21T19:09:04.743456+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:09:04.743456+0000","last_undegraded":"2024-09-21T19:09:04.743456+0000","last_fullsized":"2024-09-21T19:09:04.743456+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":185,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.745746+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:09:04.745746+0000","last_peered":"2024-09-21T19:09:04.745746+0000","last_clean":"2024-09-21T19:09:04.745746+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:09:04.745746+0000","last_undegraded":"2024-09-21T19:09:04.745746+0000","last_fullsized":"2024-09-21T19:09:04.745746+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":737,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:06.107239+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:09:06.107239+0000","last_peered":"2024-09-21T19:09:06.107239+0000","last_clean":"2024-09-21T19:09:06.107239+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:09:06.107239+0000","last_undegraded":"2024-09-21T19:09:06.107239+0000","last_fullsized":"2024-09-21T19:09:06.107239+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:07.738704+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":719,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734805+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:09:04.734805+0000","last_peered":"2024-09-21T19:09:04.734805+0000","last_clean":"2024-09-21T19:09:04.734805+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:09:04.734805+0000","last_undegraded":"2024-09-21T19:09:04.734805+0000","last_fullsized":"2024-09-21T19:09:04.734805+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":747,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.273395+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:09:11.273395+0000","last_peered":"2024-09-21T19:09:11.273395+0000","last_clean":"2024-09-21T19:09:11.273395+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:09:11.273395+0000","last_undegraded":"2024-09-21T19:09:11.273395+0000","last_fullsized":"2024-09-21T19:09:11.273395+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":790,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735902+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:09:04.735902+0000","last_peered":"2024-09-21T19:09:04.735902+0000","last_clean":"2024-09-21T19:09:04.735902+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:09:04.735902+0000","last_undegraded":"2024-09-21T19:09:04.735902+0000","last_fullsized":"2024-09-21T19:09:04.735902+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":731,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271474+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:09:11.271474+0000","last_peered":"2024-09-21T19:09:11.271474+0000","last_clean":"2024-09-21T19:09:11.271474+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:09:11.271474+0000","last_undegraded":"2024-09-21T19:09:11.271474+0000","last_fullsized":"2024-09-21T19:09:11.271474+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":752,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.740385+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:09:04.740385+0000","last_peered":"2024-09-21T19:09:04.740385+0000","last_clean":"2024-09-21T19:09:04.740385+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:09:04.740385+0000","last_undegraded":"2024-09-21T19:09:04.740385+0000","last_fullsized":"2024-09-21T19:09:04.740385+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":751,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734238+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:09:04.734238+0000","last_peered":"2024-09-21T19:09:04.734238+0000","last_clean":"2024-09-21T19:09:04.734238+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:09:04.734238+0000","last_undegraded":"2024-09-21T19:09:04.734238+0000","last_fullsized":"2024-09-21T19:09:04.734238+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":639,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.768957+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:09:04.768957+0000","last_peered":"2024-09-21T19:09:04.768957+0000","last_clean":"2024-09-21T19:09:04.768957+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:09:04.768957+0000","last_undegraded":"2024-09-21T19:09:04.768957+0000","last_fullsized":"2024-09-21T19:09:04.768957+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":807,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737990+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:09:04.737990+0000","last_peered":"2024-09-21T19:09:04.737990+0000","last_clean":"2024-09-21T19:09:04.737990+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:09:04.737990+0000","last_undegraded":"2024-09-21T19:09:04.737990+0000","last_fullsized":"2024-09-21T19:09:04.737990+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"396'12655","reported_seq":12799,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.381429+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:09:11.381429+0000","last_peered":"2024-09-21T19:09:11.381429+0000","last_clean":"2024-09-21T19:09:11.381429+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:09:11.381429+0000","last_undegraded":"2024-09-21T19:09:11.381429+0000","last_fullsized":"2024-09-21T19:09:11.381429+0000","mapping_epoch":330,"log_start":"369'8100","ondisk_log_start":"369'8100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145817600,"num_objects":2225,"num_object_clones":0,"num_object_copies":6675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":180,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.745317+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:09:04.745317+0000","last_peered":"2024-09-21T19:09:04.745317+0000","last_clean":"2024-09-21T19:09:04.745317+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:09:04.745317+0000","last_undegraded":"2024-09-21T19:09:04.745317+0000","last_fullsized":"2024-09-21T19:09:04.745317+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":779,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.740062+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:09:04.740062+0000","last_peered":"2024-09-21T19:09:04.740062+0000","last_clean":"2024-09-21T19:09:04.740062+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:09:04.740062+0000","last_undegraded":"2024-09-21T19:09:04.740062+0000","last_fullsized":"2024-09-21T19:09:04.740062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:35.238469+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":717,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736923+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:09:04.736923+0000","last_peered":"2024-09-21T19:09:04.736923+0000","last_clean":"2024-09-21T19:09:04.736923+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:09:04.736923+0000","last_undegraded":"2024-09-21T19:09:04.736923+0000","last_fullsized":"2024-09-21T19:09:04.736923+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":962,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272203+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:09:11.272203+0000","last_peered":"2024-09-21T19:09:11.272203+0000","last_clean":"2024-09-21T19:09:11.272203+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:09:11.272203+0000","last_undegraded":"2024-09-21T19:09:11.272203+0000","last_fullsized":"2024-09-21T19:09:11.272203+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":406,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.739270+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:09:04.739270+0000","last_peered":"2024-09-21T19:09:04.739270+0000","last_clean":"2024-09-21T19:09:04.739270+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:09:04.739270+0000","last_undegraded":"2024-09-21T19:09:04.739270+0000","last_fullsized":"2024-09-21T19:09:04.739270+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"396'12268","reported_seq":12412,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.393794+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:09:11.393794+0000","last_peered":"2024-09-21T19:09:11.393794+0000","last_clean":"2024-09-21T19:09:11.393794+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:09:11.393794+0000","last_undegraded":"2024-09-21T19:09:11.393794+0000","last_fullsized":"2024-09-21T19:09:11.393794+0000","mapping_epoch":330,"log_start":"368'7700","ondisk_log_start":"368'7700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12268,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":508,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.274453+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:09:11.274453+0000","last_peered":"2024-09-21T19:09:11.274453+0000","last_clean":"2024-09-21T19:09:11.274453+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:09:11.274453+0000","last_undegraded":"2024-09-21T19:09:11.274453+0000","last_fullsized":"2024-09-21T19:09:11.274453+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":813,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736726+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:09:04.736726+0000","last_peered":"2024-09-21T19:09:04.736726+0000","last_clean":"2024-09-21T19:09:04.736726+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:09:04.736726+0000","last_undegraded":"2024-09-21T19:09:04.736726+0000","last_fullsized":"2024-09-21T19:09:04.736726+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735257+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:09:04.735257+0000","last_peered":"2024-09-21T19:09:04.735257+0000","last_clean":"2024-09-21T19:09:04.735257+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:09:04.735257+0000","last_undegraded":"2024-09-21T19:09:04.735257+0000","last_fullsized":"2024-09-21T19:09:04.735257+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":718,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271355+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:09:11.271355+0000","last_peered":"2024-09-21T19:09:11.271355+0000","last_clean":"2024-09-21T19:09:11.271355+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:09:11.271355+0000","last_undegraded":"2024-09-21T19:09:11.271355+0000","last_fullsized":"2024-09-21T19:09:11.271355+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":735,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.273180+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:09:11.273180+0000","last_peered":"2024-09-21T19:09:11.273180+0000","last_clean":"2024-09-21T19:09:11.273180+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:09:11.273180+0000","last_undegraded":"2024-09-21T19:09:11.273180+0000","last_fullsized":"2024-09-21T19:09:11.273180+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"395'12290","reported_seq":12432,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:09.275381+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:09:09.275381+0000","last_peered":"2024-09-21T19:09:09.275381+0000","last_clean":"2024-09-21T19:09:09.275381+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:09:09.275381+0000","last_undegraded":"2024-09-21T19:09:09.275381+0000","last_fullsized":"2024-09-21T19:09:09.275381+0000","mapping_epoch":330,"log_start":"366'7000","ondisk_log_start":"366'7000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157941760,"num_objects":2410,"num_object_clones":0,"num_object_copies":7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":497,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737108+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:09:04.737108+0000","last_peered":"2024-09-21T19:09:04.737108+0000","last_clean":"2024-09-21T19:09:04.737108+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:09:04.737108+0000","last_undegraded":"2024-09-21T19:09:04.737108+0000","last_fullsized":"2024-09-21T19:09:04.737108+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":786,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271478+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:09:11.271478+0000","last_peered":"2024-09-21T19:09:11.271478+0000","last_clean":"2024-09-21T19:09:11.271478+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:09:11.271478+0000","last_undegraded":"2024-09-21T19:09:11.271478+0000","last_fullsized":"2024-09-21T19:09:11.271478+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":608,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.769242+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:09:04.769242+0000","last_peered":"2024-09-21T19:09:04.769242+0000","last_clean":"2024-09-21T19:09:04.769242+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:09:04.769242+0000","last_undegraded":"2024-09-21T19:09:04.769242+0000","last_fullsized":"2024-09-21T19:09:04.769242+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":902,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736591+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:09:04.736591+0000","last_peered":"2024-09-21T19:09:04.736591+0000","last_clean":"2024-09-21T19:09:04.736591+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:09:04.736591+0000","last_undegraded":"2024-09-21T19:09:04.736591+0000","last_fullsized":"2024-09-21T19:09:04.736591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":736,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737194+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:09:04.737194+0000","last_peered":"2024-09-21T19:09:04.737194+0000","last_clean":"2024-09-21T19:09:04.737194+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:09:04.737194+0000","last_undegraded":"2024-09-21T19:09:04.737194+0000","last_fullsized":"2024-09-21T19:09:04.737194+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"395'12343","reported_seq":12485,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:06.820483+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:09:06.820483+0000","last_peered":"2024-09-21T19:09:06.820483+0000","last_clean":"2024-09-21T19:09:06.820483+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:09:06.820483+0000","last_undegraded":"2024-09-21T19:09:06.820483+0000","last_fullsized":"2024-09-21T19:09:06.820483+0000","mapping_epoch":330,"log_start":"367'7300","ondisk_log_start":"367'7300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170065920,"num_objects":2595,"num_object_clones":0,"num_object_copies":7785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12343,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":520,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738132+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:09:04.738132+0000","last_peered":"2024-09-21T19:09:04.738132+0000","last_clean":"2024-09-21T19:09:04.738132+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:09:04.738132+0000","last_undegraded":"2024-09-21T19:09:04.738132+0000","last_fullsized":"2024-09-21T19:09:04.738132+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":761,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738274+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:09:04.738274+0000","last_peered":"2024-09-21T19:09:04.738274+0000","last_clean":"2024-09-21T19:09:04.738274+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:09:04.738274+0000","last_undegraded":"2024-09-21T19:09:04.738274+0000","last_fullsized":"2024-09-21T19:09:04.738274+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:33.202656+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":280,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735987+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:09:04.735987+0000","last_peered":"2024-09-21T19:09:04.735987+0000","last_clean":"2024-09-21T19:09:04.735987+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:09:04.735987+0000","last_undegraded":"2024-09-21T19:09:04.735987+0000","last_fullsized":"2024-09-21T19:09:04.735987+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":786,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.743410+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:09:04.743410+0000","last_peered":"2024-09-21T19:09:04.743410+0000","last_clean":"2024-09-21T19:09:04.743410+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:09:04.743410+0000","last_undegraded":"2024-09-21T19:09:04.743410+0000","last_fullsized":"2024-09-21T19:09:04.743410+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":610,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.747425+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:09:04.747425+0000","last_peered":"2024-09-21T19:09:04.747425+0000","last_clean":"2024-09-21T19:09:04.747425+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:09:04.747425+0000","last_undegraded":"2024-09-21T19:09:04.747425+0000","last_fullsized":"2024-09-21T19:09:04.747425+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"395'12258","reported_seq":12401,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:09.198547+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:09:09.198547+0000","last_peered":"2024-09-21T19:09:09.198547+0000","last_clean":"2024-09-21T19:09:09.198547+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:09:09.198547+0000","last_undegraded":"2024-09-21T19:09:09.198547+0000","last_fullsized":"2024-09-21T19:09:09.198547+0000","mapping_epoch":330,"log_start":"365'6900","ondisk_log_start":"365'6900","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5358,"log_dups_size":0,"ondisk_log_size":5358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158400536,"num_objects":2418,"num_object_clones":0,"num_object_copies":7254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12258,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":498,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.733333+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:09:04.733333+0000","last_peered":"2024-09-21T19:09:04.733333+0000","last_clean":"2024-09-21T19:09:04.733333+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:09:04.733333+0000","last_undegraded":"2024-09-21T19:09:04.733333+0000","last_fullsized":"2024-09-21T19:09:04.733333+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":656,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.743545+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:09:04.743545+0000","last_peered":"2024-09-21T19:09:04.743545+0000","last_clean":"2024-09-21T19:09:04.743545+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:09:04.743545+0000","last_undegraded":"2024-09-21T19:09:04.743545+0000","last_fullsized":"2024-09-21T19:09:04.743545+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":749,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738983+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:09:04.738983+0000","last_peered":"2024-09-21T19:09:04.738983+0000","last_clean":"2024-09-21T19:09:04.738983+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:09:04.738983+0000","last_undegraded":"2024-09-21T19:09:04.738983+0000","last_fullsized":"2024-09-21T19:09:04.738983+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:34.228030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":634,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.733617+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:09:04.733617+0000","last_peered":"2024-09-21T19:09:04.733617+0000","last_clean":"2024-09-21T19:09:04.733617+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:09:04.733617+0000","last_undegraded":"2024-09-21T19:09:04.733617+0000","last_fullsized":"2024-09-21T19:09:04.733617+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":737,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734050+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:09:04.734050+0000","last_peered":"2024-09-21T19:09:04.734050+0000","last_clean":"2024-09-21T19:09:04.734050+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:09:04.734050+0000","last_undegraded":"2024-09-21T19:09:04.734050+0000","last_fullsized":"2024-09-21T19:09:04.734050+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.253514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"395'12464","reported_seq":21973,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:09.214837+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:09:09.214837+0000","last_peered":"2024-09-21T19:09:09.214837+0000","last_clean":"2024-09-21T19:09:09.214837+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:09:09.214837+0000","last_undegraded":"2024-09-21T19:09:09.214837+0000","last_fullsized":"2024-09-21T19:09:09.214837+0000","mapping_epoch":380,"log_start":"367'7200","ondisk_log_start":"367'7200","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156368896,"num_objects":2386,"num_object_clones":0,"num_object_copies":7158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12464,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":474,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.743996+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:09:04.743996+0000","last_peered":"2024-09-21T19:09:04.743996+0000","last_clean":"2024-09-21T19:09:04.743996+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:09:04.743996+0000","last_undegraded":"2024-09-21T19:09:04.743996+0000","last_fullsized":"2024-09-21T19:09:04.743996+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1030,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738898+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:09:04.738898+0000","last_peered":"2024-09-21T19:09:04.738898+0000","last_clean":"2024-09-21T19:09:04.738898+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:09:04.738898+0000","last_undegraded":"2024-09-21T19:09:04.738898+0000","last_fullsized":"2024-09-21T19:09:04.738898+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:45.305870+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":759,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272104+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:09:11.272104+0000","last_peered":"2024-09-21T19:09:11.272104+0000","last_clean":"2024-09-21T19:09:11.272104+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:09:11.272104+0000","last_undegraded":"2024-09-21T19:09:11.272104+0000","last_fullsized":"2024-09-21T19:09:11.272104+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":304,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738567+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:09:04.738567+0000","last_peered":"2024-09-21T19:09:04.738567+0000","last_clean":"2024-09-21T19:09:04.738567+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:09:04.738567+0000","last_undegraded":"2024-09-21T19:09:04.738567+0000","last_fullsized":"2024-09-21T19:09:04.738567+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:25.256087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":596,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.748046+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:09:04.748046+0000","last_peered":"2024-09-21T19:09:04.748046+0000","last_clean":"2024-09-21T19:09:04.748046+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:09:04.748046+0000","last_undegraded":"2024-09-21T19:09:04.748046+0000","last_fullsized":"2024-09-21T19:09:04.748046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":705,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736192+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:09:04.736192+0000","last_peered":"2024-09-21T19:09:04.736192+0000","last_clean":"2024-09-21T19:09:04.736192+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:09:04.736192+0000","last_undegraded":"2024-09-21T19:09:04.736192+0000","last_fullsized":"2024-09-21T19:09:04.736192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"395'12146","reported_seq":21571,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.874196+0000","last_change":"2024-09-21T19:08:54.754076+0000","last_active":"2024-09-21T19:09:07.874196+0000","last_peered":"2024-09-21T19:09:07.874196+0000","last_clean":"2024-09-21T19:09:07.874196+0000","last_became_active":"2024-09-21T19:08:54.722178+0000","last_became_peered":"2024-09-21T19:08:54.722178+0000","last_unstale":"2024-09-21T19:09:07.874196+0000","last_undegraded":"2024-09-21T19:09:07.874196+0000","last_fullsized":"2024-09-21T19:09:07.874196+0000","mapping_epoch":389,"log_start":"359'5700","ondisk_log_start":"359'5700","created":330,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160432128,"num_objects":2448,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":303890432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":810,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737194+0000","last_change":"2024-09-21T19:08:53.739015+0000","last_active":"2024-09-21T19:09:04.737194+0000","last_peered":"2024-09-21T19:09:04.737194+0000","last_clean":"2024-09-21T19:09:04.737194+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:09:04.737194+0000","last_undegraded":"2024-09-21T19:09:04.737194+0000","last_fullsized":"2024-09-21T19:09:04.737194+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:08:53.738929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.738929+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:55.112735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":222,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.739829+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:09:04.739829+0000","last_peered":"2024-09-21T19:09:04.739829+0000","last_clean":"2024-09-21T19:09:04.739829+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:09:04.739829+0000","last_undegraded":"2024-09-21T19:09:04.739829+0000","last_fullsized":"2024-09-21T19:09:04.739829+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":779,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735620+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:09:04.735620+0000","last_peered":"2024-09-21T19:09:04.735620+0000","last_clean":"2024-09-21T19:09:04.735620+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:09:04.735620+0000","last_undegraded":"2024-09-21T19:09:04.735620+0000","last_fullsized":"2024-09-21T19:09:04.735620+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":173,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.747608+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:09:04.747608+0000","last_peered":"2024-09-21T19:09:04.747608+0000","last_clean":"2024-09-21T19:09:04.747608+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:09:04.747608+0000","last_undegraded":"2024-09-21T19:09:04.747608+0000","last_fullsized":"2024-09-21T19:09:04.747608+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":172,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.746280+0000","last_change":"2024-09-21T19:06:54.694760+0000","last_active":"2024-09-21T19:09:04.746280+0000","last_peered":"2024-09-21T19:09:04.746280+0000","last_clean":"2024-09-21T19:09:04.746280+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:09:04.746280+0000","last_undegraded":"2024-09-21T19:09:04.746280+0000","last_fullsized":"2024-09-21T19:09:04.746280+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.530705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.530705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":726,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737738+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:09:04.737738+0000","last_peered":"2024-09-21T19:09:04.737738+0000","last_clean":"2024-09-21T19:09:04.737738+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:09:04.737738+0000","last_undegraded":"2024-09-21T19:09:04.737738+0000","last_fullsized":"2024-09-21T19:09:04.737738+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"395'12220","reported_seq":12362,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.688734+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:09:07.688734+0000","last_peered":"2024-09-21T19:09:07.688734+0000","last_clean":"2024-09-21T19:09:07.688734+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:09:07.688734+0000","last_undegraded":"2024-09-21T19:09:07.688734+0000","last_fullsized":"2024-09-21T19:09:07.688734+0000","mapping_epoch":330,"log_start":"367'7200","ondisk_log_start":"367'7200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164888576,"num_objects":2516,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12220,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":508,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.739275+0000","last_change":"2024-09-21T19:07:30.274632+0000","last_active":"2024-09-21T19:09:04.739275+0000","last_peered":"2024-09-21T19:09:04.739275+0000","last_clean":"2024-09-21T19:09:04.739275+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:09:04.739275+0000","last_undegraded":"2024-09-21T19:09:04.739275+0000","last_fullsized":"2024-09-21T19:09:04.739275+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:07:30.274412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:07:30.274412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":718,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.769058+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:09:04.769058+0000","last_peered":"2024-09-21T19:09:04.769058+0000","last_clean":"2024-09-21T19:09:04.769058+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:09:04.769058+0000","last_undegraded":"2024-09-21T19:09:04.769058+0000","last_fullsized":"2024-09-21T19:09:04.769058+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":604,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.744318+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:09:04.744318+0000","last_peered":"2024-09-21T19:09:04.744318+0000","last_clean":"2024-09-21T19:09:04.744318+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:09:04.744318+0000","last_undegraded":"2024-09-21T19:09:04.744318+0000","last_fullsized":"2024-09-21T19:09:04.744318+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1701,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:05.742865+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:09:05.742865+0000","last_peered":"2024-09-21T19:09:05.742865+0000","last_clean":"2024-09-21T19:09:05.742865+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:09:05.742865+0000","last_undegraded":"2024-09-21T19:09:05.742865+0000","last_fullsized":"2024-09-21T19:09:05.742865+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":948,"num_read_kb":943,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":676,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738739+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:09:04.738739+0000","last_peered":"2024-09-21T19:09:04.738739+0000","last_clean":"2024-09-21T19:09:04.738739+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:09:04.738739+0000","last_undegraded":"2024-09-21T19:09:04.738739+0000","last_fullsized":"2024-09-21T19:09:04.738739+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":731,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738683+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:09:04.738683+0000","last_peered":"2024-09-21T19:09:04.738683+0000","last_clean":"2024-09-21T19:09:04.738683+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:09:04.738683+0000","last_undegraded":"2024-09-21T19:09:04.738683+0000","last_fullsized":"2024-09-21T19:09:04.738683+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:30.010503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"395'12256","reported_seq":12398,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:09.260850+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:09:09.260850+0000","last_peered":"2024-09-21T19:09:09.260850+0000","last_clean":"2024-09-21T19:09:09.260850+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:09:09.260850+0000","last_undegraded":"2024-09-21T19:09:09.260850+0000","last_fullsized":"2024-09-21T19:09:09.260850+0000","mapping_epoch":330,"log_start":"365'6900","ondisk_log_start":"365'6900","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156368896,"num_objects":2386,"num_object_clones":0,"num_object_copies":7158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":898,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.733821+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:09:04.733821+0000","last_peered":"2024-09-21T19:09:04.733821+0000","last_clean":"2024-09-21T19:09:04.733821+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:09:04.733821+0000","last_undegraded":"2024-09-21T19:09:04.733821+0000","last_fullsized":"2024-09-21T19:09:04.733821+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735625+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:09:04.735625+0000","last_peered":"2024-09-21T19:09:04.735625+0000","last_clean":"2024-09-21T19:09:04.735625+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:09:04.735625+0000","last_undegraded":"2024-09-21T19:09:04.735625+0000","last_fullsized":"2024-09-21T19:09:04.735625+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":816,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737282+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:09:04.737282+0000","last_peered":"2024-09-21T19:09:04.737282+0000","last_clean":"2024-09-21T19:09:04.737282+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:09:04.737282+0000","last_undegraded":"2024-09-21T19:09:04.737282+0000","last_fullsized":"2024-09-21T19:09:04.737282+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":740,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.740554+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:09:04.740554+0000","last_peered":"2024-09-21T19:09:04.740554+0000","last_clean":"2024-09-21T19:09:04.740554+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:09:04.740554+0000","last_undegraded":"2024-09-21T19:09:04.740554+0000","last_fullsized":"2024-09-21T19:09:04.740554+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":266,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734718+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:09:04.734718+0000","last_peered":"2024-09-21T19:09:04.734718+0000","last_clean":"2024-09-21T19:09:04.734718+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:09:04.734718+0000","last_undegraded":"2024-09-21T19:09:04.734718+0000","last_fullsized":"2024-09-21T19:09:04.734718+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"395'12142","reported_seq":12283,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:09.276803+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:09:09.276803+0000","last_peered":"2024-09-21T19:09:09.276803+0000","last_clean":"2024-09-21T19:09:09.276803+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:09:09.276803+0000","last_undegraded":"2024-09-21T19:09:09.276803+0000","last_fullsized":"2024-09-21T19:09:09.276803+0000","mapping_epoch":330,"log_start":"365'6800","ondisk_log_start":"365'6800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150863872,"num_objects":2302,"num_object_clones":0,"num_object_copies":6906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":780,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272288+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:09:11.272288+0000","last_peered":"2024-09-21T19:09:11.272288+0000","last_clean":"2024-09-21T19:09:11.272288+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:09:11.272288+0000","last_undegraded":"2024-09-21T19:09:11.272288+0000","last_fullsized":"2024-09-21T19:09:11.272288+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":748,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271745+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:09:11.271745+0000","last_peered":"2024-09-21T19:09:11.271745+0000","last_clean":"2024-09-21T19:09:11.271745+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:09:11.271745+0000","last_undegraded":"2024-09-21T19:09:11.271745+0000","last_fullsized":"2024-09-21T19:09:11.271745+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":732,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272369+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:09:11.272369+0000","last_peered":"2024-09-21T19:09:11.272369+0000","last_clean":"2024-09-21T19:09:11.272369+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:09:11.272369+0000","last_undegraded":"2024-09-21T19:09:11.272369+0000","last_fullsized":"2024-09-21T19:09:11.272369+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":711,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737159+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:09:04.737159+0000","last_peered":"2024-09-21T19:09:04.737159+0000","last_clean":"2024-09-21T19:09:04.737159+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:09:04.737159+0000","last_undegraded":"2024-09-21T19:09:04.737159+0000","last_fullsized":"2024-09-21T19:09:04.737159+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"395'12206","reported_seq":21681,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.876246+0000","last_change":"2024-09-21T19:08:55.780967+0000","last_active":"2024-09-21T19:09:07.876246+0000","last_peered":"2024-09-21T19:09:07.876246+0000","last_clean":"2024-09-21T19:09:07.876246+0000","last_became_active":"2024-09-21T19:08:55.751488+0000","last_became_peered":"2024-09-21T19:08:55.751488+0000","last_unstale":"2024-09-21T19:09:07.876246+0000","last_undegraded":"2024-09-21T19:09:07.876246+0000","last_fullsized":"2024-09-21T19:09:07.876246+0000","mapping_epoch":390,"log_start":"359'5800","ondisk_log_start":"359'5800","created":330,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163053568,"num_objects":2488,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12206,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4662,"num_bytes_recovered":305528832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":176,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.746760+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:09:04.746760+0000","last_peered":"2024-09-21T19:09:04.746760+0000","last_clean":"2024-09-21T19:09:04.746760+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:09:04.746760+0000","last_undegraded":"2024-09-21T19:09:04.746760+0000","last_fullsized":"2024-09-21T19:09:04.746760+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":829,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.274248+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:09:11.274248+0000","last_peered":"2024-09-21T19:09:11.274248+0000","last_clean":"2024-09-21T19:09:11.274248+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:09:11.274248+0000","last_undegraded":"2024-09-21T19:09:11.274248+0000","last_fullsized":"2024-09-21T19:09:11.274248+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":644,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734117+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:09:04.734117+0000","last_peered":"2024-09-21T19:09:04.734117+0000","last_clean":"2024-09-21T19:09:04.734117+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:09:04.734117+0000","last_undegraded":"2024-09-21T19:09:04.734117+0000","last_fullsized":"2024-09-21T19:09:04.734117+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":169,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.732632+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:09:04.732632+0000","last_peered":"2024-09-21T19:09:04.732632+0000","last_clean":"2024-09-21T19:09:04.732632+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:09:04.732632+0000","last_undegraded":"2024-09-21T19:09:04.732632+0000","last_fullsized":"2024-09-21T19:09:04.732632+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":169,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.746387+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:09:04.746387+0000","last_peered":"2024-09-21T19:09:04.746387+0000","last_clean":"2024-09-21T19:09:04.746387+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:09:04.746387+0000","last_undegraded":"2024-09-21T19:09:04.746387+0000","last_fullsized":"2024-09-21T19:09:04.746387+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"395'12247","reported_seq":12389,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.581251+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:09:07.581251+0000","last_peered":"2024-09-21T19:09:07.581251+0000","last_clean":"2024-09-21T19:09:07.581251+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:09:07.581251+0000","last_undegraded":"2024-09-21T19:09:07.581251+0000","last_fullsized":"2024-09-21T19:09:07.581251+0000","mapping_epoch":330,"log_start":"365'7000","ondisk_log_start":"365'7000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:46.323337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163119104,"num_objects":2489,"num_object_clones":0,"num_object_copies":7467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12247,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":823,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737632+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:09:04.737632+0000","last_peered":"2024-09-21T19:09:04.737632+0000","last_clean":"2024-09-21T19:09:04.737632+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:09:04.737632+0000","last_undegraded":"2024-09-21T19:09:04.737632+0000","last_fullsized":"2024-09-21T19:09:04.737632+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":755,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271565+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:09:11.271565+0000","last_peered":"2024-09-21T19:09:11.271565+0000","last_clean":"2024-09-21T19:09:11.271565+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:09:11.271565+0000","last_undegraded":"2024-09-21T19:09:11.271565+0000","last_fullsized":"2024-09-21T19:09:11.271565+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":753,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734987+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:09:04.734987+0000","last_peered":"2024-09-21T19:09:04.734987+0000","last_clean":"2024-09-21T19:09:04.734987+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:09:04.734987+0000","last_undegraded":"2024-09-21T19:09:04.734987+0000","last_fullsized":"2024-09-21T19:09:04.734987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":220,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.746443+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:09:04.746443+0000","last_peered":"2024-09-21T19:09:04.746443+0000","last_clean":"2024-09-21T19:09:04.746443+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:09:04.746443+0000","last_undegraded":"2024-09-21T19:09:04.746443+0000","last_fullsized":"2024-09-21T19:09:04.746443+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":728,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272045+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:09:11.272045+0000","last_peered":"2024-09-21T19:09:11.272045+0000","last_clean":"2024-09-21T19:09:11.272045+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:09:11.272045+0000","last_undegraded":"2024-09-21T19:09:11.272045+0000","last_fullsized":"2024-09-21T19:09:11.272045+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"395'12006","reported_seq":12148,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.879527+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:09:07.879527+0000","last_peered":"2024-09-21T19:09:07.879527+0000","last_clean":"2024-09-21T19:09:07.879527+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:09:07.879527+0000","last_undegraded":"2024-09-21T19:09:07.879527+0000","last_fullsized":"2024-09-21T19:09:07.879527+0000","mapping_epoch":330,"log_start":"359'5600","ondisk_log_start":"359'5600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159907840,"num_objects":2440,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":614,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.743835+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:09:04.743835+0000","last_peered":"2024-09-21T19:09:04.743835+0000","last_clean":"2024-09-21T19:09:04.743835+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:09:04.743835+0000","last_undegraded":"2024-09-21T19:09:04.743835+0000","last_fullsized":"2024-09-21T19:09:04.743835+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":714,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737646+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:09:04.737646+0000","last_peered":"2024-09-21T19:09:04.737646+0000","last_clean":"2024-09-21T19:09:04.737646+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:09:04.737646+0000","last_undegraded":"2024-09-21T19:09:04.737646+0000","last_fullsized":"2024-09-21T19:09:04.737646+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":169,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.710551+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:09:07.710551+0000","last_peered":"2024-09-21T19:09:07.710551+0000","last_clean":"2024-09-21T19:09:07.710551+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:09:07.710551+0000","last_undegraded":"2024-09-21T19:09:07.710551+0000","last_fullsized":"2024-09-21T19:09:07.710551+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:10.305625+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":610,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.744239+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:09:04.744239+0000","last_peered":"2024-09-21T19:09:04.744239+0000","last_clean":"2024-09-21T19:09:04.744239+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:09:04.744239+0000","last_undegraded":"2024-09-21T19:09:04.744239+0000","last_fullsized":"2024-09-21T19:09:04.744239+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"395'12108","reported_seq":12250,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:07.680818+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:09:07.680818+0000","last_peered":"2024-09-21T19:09:07.680818+0000","last_clean":"2024-09-21T19:09:07.680818+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:09:07.680818+0000","last_undegraded":"2024-09-21T19:09:07.680818+0000","last_fullsized":"2024-09-21T19:09:07.680818+0000","mapping_epoch":330,"log_start":"367'7100","ondisk_log_start":"367'7100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158728192,"num_objects":2422,"num_object_clones":0,"num_object_copies":7266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12108,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":663,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.733983+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:09:04.733983+0000","last_peered":"2024-09-21T19:09:04.733983+0000","last_clean":"2024-09-21T19:09:04.733983+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:09:04.733983+0000","last_undegraded":"2024-09-21T19:09:04.733983+0000","last_fullsized":"2024-09-21T19:09:04.733983+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":745,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737989+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:09:04.737989+0000","last_peered":"2024-09-21T19:09:04.737989+0000","last_clean":"2024-09-21T19:09:04.737989+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:09:04.737989+0000","last_undegraded":"2024-09-21T19:09:04.737989+0000","last_fullsized":"2024-09-21T19:09:04.737989+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":180,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.748573+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:09:04.748573+0000","last_peered":"2024-09-21T19:09:04.748573+0000","last_clean":"2024-09-21T19:09:04.748573+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:09:04.748573+0000","last_undegraded":"2024-09-21T19:09:04.748573+0000","last_fullsized":"2024-09-21T19:09:04.748573+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":717,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737154+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:09:04.737154+0000","last_peered":"2024-09-21T19:09:04.737154+0000","last_clean":"2024-09-21T19:09:04.737154+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:09:04.737154+0000","last_undegraded":"2024-09-21T19:09:04.737154+0000","last_fullsized":"2024-09-21T19:09:04.737154+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"395'12304","reported_seq":13985,"reported_epoch":395,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:09.255798+0000","last_change":"2024-09-21T19:08:53.601069+0000","last_active":"2024-09-21T19:09:09.255798+0000","last_peered":"2024-09-21T19:09:09.255798+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:09:09.255798+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"367'7000","ondisk_log_start":"367'7000","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158859264,"num_objects":2424,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12304,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":768,"num_bytes_recovered":37355520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-1400}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":784,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271613+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:09:11.271613+0000","last_peered":"2024-09-21T19:09:11.271613+0000","last_clean":"2024-09-21T19:09:11.271613+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:09:11.271613+0000","last_undegraded":"2024-09-21T19:09:11.271613+0000","last_fullsized":"2024-09-21T19:09:11.271613+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738495+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:09:04.738495+0000","last_peered":"2024-09-21T19:09:04.738495+0000","last_clean":"2024-09-21T19:09:04.738495+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:09:04.738495+0000","last_undegraded":"2024-09-21T19:09:04.738495+0000","last_fullsized":"2024-09-21T19:09:04.738495+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":169,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.746672+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:09:04.746672+0000","last_peered":"2024-09-21T19:09:04.746672+0000","last_clean":"2024-09-21T19:09:04.746672+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:09:04.746672+0000","last_undegraded":"2024-09-21T19:09:04.746672+0000","last_fullsized":"2024-09-21T19:09:04.746672+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:00.385824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":705,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734573+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:09:04.734573+0000","last_peered":"2024-09-21T19:09:04.734573+0000","last_clean":"2024-09-21T19:09:04.734573+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:09:04.734573+0000","last_undegraded":"2024-09-21T19:09:04.734573+0000","last_fullsized":"2024-09-21T19:09:04.734573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"395'12421","reported_seq":15952,"reported_epoch":395,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:07.707717+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:09:07.707717+0000","last_peered":"2024-09-21T19:09:07.707717+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:09:07.707717+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"367'7400","ondisk_log_start":"367'7400","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163119104,"num_objects":2489,"num_object_clones":0,"num_object_copies":7467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12421,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":103088128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":-564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":337,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736825+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:09:04.736825+0000","last_peered":"2024-09-21T19:09:04.736825+0000","last_clean":"2024-09-21T19:09:04.736825+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:09:04.736825+0000","last_undegraded":"2024-09-21T19:09:04.736825+0000","last_fullsized":"2024-09-21T19:09:04.736825+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":753,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738901+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:09:04.738901+0000","last_peered":"2024-09-21T19:09:04.738901+0000","last_clean":"2024-09-21T19:09:04.738901+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:09:04.738901+0000","last_undegraded":"2024-09-21T19:09:04.738901+0000","last_fullsized":"2024-09-21T19:09:04.738901+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":216,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734560+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:09:04.734560+0000","last_peered":"2024-09-21T19:09:04.734560+0000","last_clean":"2024-09-21T19:09:04.734560+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:09:04.734560+0000","last_undegraded":"2024-09-21T19:09:04.734560+0000","last_fullsized":"2024-09-21T19:09:04.734560+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":740,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.733709+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:09:04.733709+0000","last_peered":"2024-09-21T19:09:04.733709+0000","last_clean":"2024-09-21T19:09:04.733709+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:09:04.733709+0000","last_undegraded":"2024-09-21T19:09:04.733709+0000","last_fullsized":"2024-09-21T19:09:04.733709+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":675,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.732478+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:09:04.732478+0000","last_peered":"2024-09-21T19:09:04.732478+0000","last_clean":"2024-09-21T19:09:04.732478+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:09:04.732478+0000","last_undegraded":"2024-09-21T19:09:04.732478+0000","last_fullsized":"2024-09-21T19:09:04.732478+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":738,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.735131+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:09:04.735131+0000","last_peered":"2024-09-21T19:09:04.735131+0000","last_clean":"2024-09-21T19:09:04.735131+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:09:04.735131+0000","last_undegraded":"2024-09-21T19:09:04.735131+0000","last_fullsized":"2024-09-21T19:09:04.735131+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":274,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736126+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:09:04.736126+0000","last_peered":"2024-09-21T19:09:04.736126+0000","last_clean":"2024-09-21T19:09:04.736126+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:09:04.736126+0000","last_undegraded":"2024-09-21T19:09:04.736126+0000","last_fullsized":"2024-09-21T19:09:04.736126+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":713,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737636+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:09:04.737636+0000","last_peered":"2024-09-21T19:09:04.737636+0000","last_clean":"2024-09-21T19:09:04.737636+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:09:04.737636+0000","last_undegraded":"2024-09-21T19:09:04.737636+0000","last_fullsized":"2024-09-21T19:09:04.737636+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":614,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.769150+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:09:04.769150+0000","last_peered":"2024-09-21T19:09:04.769150+0000","last_clean":"2024-09-21T19:09:04.769150+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:09:04.769150+0000","last_undegraded":"2024-09-21T19:09:04.769150+0000","last_fullsized":"2024-09-21T19:09:04.769150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":725,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737748+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:09:04.737748+0000","last_peered":"2024-09-21T19:09:04.737748+0000","last_clean":"2024-09-21T19:09:04.737748+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:09:04.737748+0000","last_undegraded":"2024-09-21T19:09:04.737748+0000","last_fullsized":"2024-09-21T19:09:04.737748+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":708,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734951+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:09:04.734951+0000","last_peered":"2024-09-21T19:09:04.734951+0000","last_clean":"2024-09-21T19:09:04.734951+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:09:04.734951+0000","last_undegraded":"2024-09-21T19:09:04.734951+0000","last_fullsized":"2024-09-21T19:09:04.734951+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":169,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.745833+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:09:04.745833+0000","last_peered":"2024-09-21T19:09:04.745833+0000","last_clean":"2024-09-21T19:09:04.745833+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:09:04.745833+0000","last_undegraded":"2024-09-21T19:09:04.745833+0000","last_fullsized":"2024-09-21T19:09:04.745833+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":327,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737848+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:09:04.737848+0000","last_peered":"2024-09-21T19:09:04.737848+0000","last_clean":"2024-09-21T19:09:04.737848+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:09:04.737848+0000","last_undegraded":"2024-09-21T19:09:04.737848+0000","last_fullsized":"2024-09-21T19:09:04.737848+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":611,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.752062+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:09:04.752062+0000","last_peered":"2024-09-21T19:09:04.752062+0000","last_clean":"2024-09-21T19:09:04.752062+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:09:04.752062+0000","last_undegraded":"2024-09-21T19:09:04.752062+0000","last_fullsized":"2024-09-21T19:09:04.752062+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":932,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737283+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:09:04.737283+0000","last_peered":"2024-09-21T19:09:04.737283+0000","last_clean":"2024-09-21T19:09:04.737283+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:09:04.737283+0000","last_undegraded":"2024-09-21T19:09:04.737283+0000","last_fullsized":"2024-09-21T19:09:04.737283+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":730,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.272289+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:09:11.272289+0000","last_peered":"2024-09-21T19:09:11.272289+0000","last_clean":"2024-09-21T19:09:11.272289+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:09:11.272289+0000","last_undegraded":"2024-09-21T19:09:11.272289+0000","last_fullsized":"2024-09-21T19:09:11.272289+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":734,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737498+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:09:04.737498+0000","last_peered":"2024-09-21T19:09:04.737498+0000","last_clean":"2024-09-21T19:09:04.737498+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:09:04.737498+0000","last_undegraded":"2024-09-21T19:09:04.737498+0000","last_fullsized":"2024-09-21T19:09:04.737498+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":728,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738943+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:09:04.738943+0000","last_peered":"2024-09-21T19:09:04.738943+0000","last_clean":"2024-09-21T19:09:04.738943+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:09:04.738943+0000","last_undegraded":"2024-09-21T19:09:04.738943+0000","last_fullsized":"2024-09-21T19:09:04.738943+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":382,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.739230+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:09:04.739230+0000","last_peered":"2024-09-21T19:09:04.739230+0000","last_clean":"2024-09-21T19:09:04.739230+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:09:04.739230+0000","last_undegraded":"2024-09-21T19:09:04.739230+0000","last_fullsized":"2024-09-21T19:09:04.739230+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":603,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.769416+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:09:04.769416+0000","last_peered":"2024-09-21T19:09:04.769416+0000","last_clean":"2024-09-21T19:09:04.769416+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:09:04.769416+0000","last_undegraded":"2024-09-21T19:09:04.769416+0000","last_fullsized":"2024-09-21T19:09:04.769416+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":809,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271895+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:09:11.271895+0000","last_peered":"2024-09-21T19:09:11.271895+0000","last_clean":"2024-09-21T19:09:11.271895+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:09:11.271895+0000","last_undegraded":"2024-09-21T19:09:11.271895+0000","last_fullsized":"2024-09-21T19:09:11.271895+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":728,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-09-21T19:09:11.271787+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:09:11.271787+0000","last_peered":"2024-09-21T19:09:11.271787+0000","last_clean":"2024-09-21T19:09:11.271787+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:09:11.271787+0000","last_undegraded":"2024-09-21T19:09:11.271787+0000","last_fullsized":"2024-09-21T19:09:11.271787+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":726,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738659+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:09:04.738659+0000","last_peered":"2024-09-21T19:09:04.738659+0000","last_clean":"2024-09-21T19:09:04.738659+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:09:04.738659+0000","last_undegraded":"2024-09-21T19:09:04.738659+0000","last_fullsized":"2024-09-21T19:09:04.738659+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.268143+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":729,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734686+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:09:04.734686+0000","last_peered":"2024-09-21T19:09:04.734686+0000","last_clean":"2024-09-21T19:09:04.734686+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:09:04.734686+0000","last_undegraded":"2024-09-21T19:09:04.734686+0000","last_fullsized":"2024-09-21T19:09:04.734686+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":805,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738416+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:09:04.738416+0000","last_peered":"2024-09-21T19:09:04.738416+0000","last_clean":"2024-09-21T19:09:04.738416+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:09:04.738416+0000","last_undegraded":"2024-09-21T19:09:04.738416+0000","last_fullsized":"2024-09-21T19:09:04.738416+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:52.278173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737885+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:09:04.737885+0000","last_peered":"2024-09-21T19:09:04.737885+0000","last_clean":"2024-09-21T19:09:04.737885+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:09:04.737885+0000","last_undegraded":"2024-09-21T19:09:04.737885+0000","last_fullsized":"2024-09-21T19:09:04.737885+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734731+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:09:04.734731+0000","last_peered":"2024-09-21T19:09:04.734731+0000","last_clean":"2024-09-21T19:09:04.734731+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:09:04.734731+0000","last_undegraded":"2024-09-21T19:09:04.734731+0000","last_fullsized":"2024-09-21T19:09:04.734731+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":622,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.747524+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:09:04.747524+0000","last_peered":"2024-09-21T19:09:04.747524+0000","last_clean":"2024-09-21T19:09:04.747524+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:09:04.747524+0000","last_undegraded":"2024-09-21T19:09:04.747524+0000","last_fullsized":"2024-09-21T19:09:04.747524+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":754,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734905+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:09:04.734905+0000","last_peered":"2024-09-21T19:09:04.734905+0000","last_clean":"2024-09-21T19:09:04.734905+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:09:04.734905+0000","last_undegraded":"2024-09-21T19:09:04.734905+0000","last_fullsized":"2024-09-21T19:09:04.734905+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":667,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738279+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:09:04.738279+0000","last_peered":"2024-09-21T19:09:04.738279+0000","last_clean":"2024-09-21T19:09:04.738279+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:09:04.738279+0000","last_undegraded":"2024-09-21T19:09:04.738279+0000","last_fullsized":"2024-09-21T19:09:04.738279+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":741,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736573+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:09:04.736573+0000","last_peered":"2024-09-21T19:09:04.736573+0000","last_clean":"2024-09-21T19:09:04.736573+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:09:04.736573+0000","last_undegraded":"2024-09-21T19:09:04.736573+0000","last_fullsized":"2024-09-21T19:09:04.736573+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":729,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.734892+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:09:04.734892+0000","last_peered":"2024-09-21T19:09:04.734892+0000","last_clean":"2024-09-21T19:09:04.734892+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:09:04.734892+0000","last_undegraded":"2024-09-21T19:09:04.734892+0000","last_fullsized":"2024-09-21T19:09:04.734892+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":753,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736637+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:09:04.736637+0000","last_peered":"2024-09-21T19:09:04.736637+0000","last_clean":"2024-09-21T19:09:04.736637+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:09:04.736637+0000","last_undegraded":"2024-09-21T19:09:04.736637+0000","last_fullsized":"2024-09-21T19:09:04.736637+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":606,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.742519+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:09:04.742519+0000","last_peered":"2024-09-21T19:09:04.742519+0000","last_clean":"2024-09-21T19:09:04.742519+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:09:04.742519+0000","last_undegraded":"2024-09-21T19:09:04.742519+0000","last_fullsized":"2024-09-21T19:09:04.742519+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":642,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.732325+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:09:04.732325+0000","last_peered":"2024-09-21T19:09:04.732325+0000","last_clean":"2024-09-21T19:09:04.732325+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:09:04.732325+0000","last_undegraded":"2024-09-21T19:09:04.732325+0000","last_fullsized":"2024-09-21T19:09:04.732325+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":724,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.737238+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:09:04.737238+0000","last_peered":"2024-09-21T19:09:04.737238+0000","last_clean":"2024-09-21T19:09:04.737238+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:09:04.737238+0000","last_undegraded":"2024-09-21T19:09:04.737238+0000","last_fullsized":"2024-09-21T19:09:04.737238+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.739079+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:09:04.739079+0000","last_peered":"2024-09-21T19:09:04.739079+0000","last_clean":"2024-09-21T19:09:04.739079+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:09:04.739079+0000","last_undegraded":"2024-09-21T19:09:04.739079+0000","last_fullsized":"2024-09-21T19:09:04.739079+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":746,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.736850+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:09:04.736850+0000","last_peered":"2024-09-21T19:09:04.736850+0000","last_clean":"2024-09-21T19:09:04.736850+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:09:04.736850+0000","last_undegraded":"2024-09-21T19:09:04.736850+0000","last_fullsized":"2024-09-21T19:09:04.736850+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":708,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.738041+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:09:04.738041+0000","last_peered":"2024-09-21T19:09:04.738041+0000","last_clean":"2024-09-21T19:09:04.738041+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:09:04.738041+0000","last_undegraded":"2024-09-21T19:09:04.738041+0000","last_fullsized":"2024-09-21T19:09:04.738041+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":626,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T19:09:04.769332+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:09:04.769332+0000","last_peered":"2024-09-21T19:09:04.769332+0000","last_clean":"2024-09-21T19:09:04.769332+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:09:04.769332+0000","last_undegraded":"2024-09-21T19:09:04.769332+0000","last_fullsized":"2024-09-21T19:09:04.769332+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2530148376,"num_objects":38608,"num_object_clones":0,"num_object_copies":115824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196334,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16442,"num_bytes_recovered":1056505856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7626174464,"data_stored":7626162248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85634,"ondisk_log_size":85634,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":948,"num_read_kb":943,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266884,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1257144,"kb_used_data":662308,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92488008,"statfs":{"total":95995035648,"available":94707720192,"internally_reserved":0,"allocated":678203392,"data_stored":677475217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464335,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1360560,"kb_used_data":1085600,"kb_used_omap":4,"kb_used_meta":274939,"kb_avail":92384592,"statfs":{"total":95995035648,"available":94601822208,"internally_reserved":0,"allocated":1111654400,"data_stored":1110929916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":281538212},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619036,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1018860,"kb_used_data":807196,"kb_used_omap":4,"kb_used_meta":211579,"kb_avail":92726292,"statfs":{"total":95995035648,"available":94951723008,"internally_reserved":0,"allocated":826568704,"data_stored":825848746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":216657571},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019091,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680760,"kb_used_data":1162280,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92064392,"statfs":{"total":95995035648,"available":94273937408,"internally_reserved":0,"allocated":1190174720,"data_stored":1189442753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182615,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1517132,"kb_used_data":800124,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92228020,"statfs":{"total":95995035648,"available":94441492480,"internally_reserved":0,"allocated":819326976,"data_stored":818575568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":734198218},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411549,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508512,"kb_used_data":953936,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92236640,"statfs":{"total":95995035648,"available":94450319360,"internally_reserved":0,"allocated":976830464,"data_stored":976122799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1140632,"kb_used_data":481928,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":92604520,"statfs":{"total":95995035648,"available":94827028480,"internally_reserved":0,"allocated":493494272,"data_stored":492730136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738599,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2415804,"kb_used_data":1612524,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91329348,"statfs":{"total":95995035648,"available":93521252352,"internally_reserved":0,"allocated":1651224576,"data_stored":1650488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1632768000,"data_stored":1632763928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":484966400,"data_stored":484966400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":969277440,"data_stored":969277440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":810156032,"data_stored":810156032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1159598080,"data_stored":1159594008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":808517632,"data_stored":808517632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1089343488,"data_stored":1089339416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":671547392,"data_stored":671547392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:09:15.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T19:09:15.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:15.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:15.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:15.360 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:15.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:15.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:15 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 6649/108228 objects degraded (6.144%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:15.496 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:15 smithi081 ceph-mon[39192]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T19:09:15.497 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:15 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 6649/108228 objects degraded (6.144%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:15.497 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:15 smithi081 ceph-mon[29384]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T19:09:15.514 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:15.616 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:15.648 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:15.750 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:15.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:15 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 6649/108228 objects degraded (6.144%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:15.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:15 smithi104 ceph-mon[32438]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T19:09:15.787 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:15.823 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:15.864 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:09:15.864 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:09:15.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:15.893 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:15.899 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:09:15.899 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:09:15.916 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:15.936 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:09:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:09:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:15.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:15.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:15.967 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:09:15.967 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:09:16.001 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:09:16.001 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:09:16.035 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:09:16.035 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:09:16.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:16.070 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:09:16.070 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:09:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.088 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:16.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:16.103 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:09:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:09:16.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:16.217 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:16.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:16.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:16.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:16.459 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:16.495 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:16.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:16 smithi081 ceph-mon[29384]: pgmap v797: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 6405/102036 objects degraded (6.277%); 152 KiB/s, 86 objects/s recovering 2024-09-21T19:09:16.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3776099009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:16.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:16 smithi081 ceph-mon[39192]: pgmap v797: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 6405/102036 objects degraded (6.277%); 152 KiB/s, 86 objects/s recovering 2024-09-21T19:09:16.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3776099009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:16.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:16.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:16.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:16 smithi104 ceph-mon[32438]: pgmap v797: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 6405/102036 objects degraded (6.277%); 152 KiB/s, 86 objects/s recovering 2024-09-21T19:09:16.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3776099009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:16.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:16.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:16.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:17.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:17.078 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:17.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:17.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:17.316 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:09:17.346 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:17.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:17.482 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:17.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1263336106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:17.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:17 smithi081 ceph-mon[29384]: from='client.36371 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:17.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:09:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:09:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:09:17.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1263336106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:17.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:17 smithi081 ceph-mon[39192]: from='client.36371 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:17.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:17.612 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:17.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:17.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:17.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1263336106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:17.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:17 smithi104 ceph-mon[32438]: from='client.36371 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:17.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:17.885 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:17.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:17.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:18.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:18.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:18.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:18.296 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:18.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:18.435 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:18.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:18.569 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.36077.0:206696 10.3 e398/390)", 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:18.189076+0000", 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001558041, 2024-09-21T19:09:18.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001579109, 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189076+0000", 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189076+0000", 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189077+0000", 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0699999999999999e-06 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189081+0000", 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1699999999999999e-06 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189091+0000", 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8220000000000002e-06 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189097+0000", 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6340000000000003e-06 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:206696 10.3 10:cd6efdd2:::benchmark_data_smithi081_155596_object89223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:18.164716+0000", 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.025917589000000001, 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025986621000000001, 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:18.609 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 206696 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.164716+0000", 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.164716+0000", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.164717+0000", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8500000000000001e-07 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.164719+0000", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3319999999999999e-06 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:18.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.164724+0000", 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8799999999999999e-06 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.164728+0000", 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4309999999999996e-06 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.182633+0000", 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017904982 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.182929+0000", 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029549300000000003 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183287+0000", 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035767699999999998 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187835+0000", 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045484840000000002 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.188380+0000", 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00054524 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:206707 10.3 10:c7ddf04c:::benchmark_data_smithi081_155596_object89234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:18.174525+0000", 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.016108857000000001, 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016250589999999999, 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:18.614 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 206707 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.174525+0000", 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.174525+0000", 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.174527+0000", 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0660000000000002e-06 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.174533+0000", 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6590000000000004e-06 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.174546+0000", 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2866e-05 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.174554+0000", 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1270000000000003e-06 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.188415+0000", 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013861465 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.188525+0000", 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109845 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.188632+0000", 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107428 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:206715 10.5 10:a63e8870:::benchmark_data_smithi081_155596_object89242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T19:09:18.618 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:18.183424+0000", 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0072102590000000001, 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074095000000000003, 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 206715 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183424+0000", 2024-09-21T19:09:18.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183424+0000", 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183425+0000", 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.815e-06 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183431+0000", 2024-09-21T19:09:18.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3530000000000002e-06 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183440+0000", 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5710000000000004e-06 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183447+0000", 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7000000000000002e-06 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183581+0000", 2024-09-21T19:09:18.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013357899999999999 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183685+0000", 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104625 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.183790+0000", 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104867 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:18.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.188156+0000", 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043656370000000003 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.189317+0000", 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001161689 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36077.0:206716 10.f e398/330 10:f3f7b53d:::benchmark_data_smithi081_155596_object89243:head v 398'12905, mlcod=398'12903)", 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:18.187476+0000", 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0031573640000000002, 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003426358, 2024-09-21T19:09:18.623 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187476+0000", 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187476+0000", 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:18.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187478+0000", 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.747e-06 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187486+0000", 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0160000000000008e-06 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187500+0000", 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3618000000000001e-05 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187507+0000", 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2450000000000001e-06 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187627+0000", 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011946399999999999 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:18.187647+0000", 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0679999999999999e-05 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:18.626 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.627 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:18.627 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:09:18.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 5 2024-09-21T19:09:18.627 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:09:18.627 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:18.627 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[39192]: from='client.36371 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:18.627 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[39192]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T19:09:18.627 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[39192]: pgmap v799: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s; 6183/94878 objects degraded (6.517%); 0 B/s, 106 objects/s recovering 2024-09-21T19:09:18.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:18.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[29384]: from='client.36371 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:18.628 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[29384]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T19:09:18.628 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:18 smithi081 ceph-mon[29384]: pgmap v799: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s; 6183/94878 objects degraded (6.517%); 0 B/s, 106 objects/s recovering 2024-09-21T19:09:18.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:18.713 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:18.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:09:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:09:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:18 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:18 smithi104 ceph-mon[32438]: from='client.36371 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:18 smithi104 ceph-mon[32438]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T19:09:18.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:18 smithi104 ceph-mon[32438]: pgmap v799: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.21k op/s; 6183/94878 objects degraded (6.517%); 0 B/s, 106 objects/s recovering 2024-09-21T19:09:18.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:18.850 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:18.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:18.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:19.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:19.138 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:19.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:19.275 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:19.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:19.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:19.550 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[39192]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[29384]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T19:09:19.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:19.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:19.552 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:19.552 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6183/94878 objects degraded (6.517%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1191,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":400,"num_osds":8,"num_up_osds":8,"osd_up_since":1726945614,"num_in_osds":8,"osd_in_since":1726945541,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":156},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":158,"num_pools":7,"num_objects":31626,"data_bytes":2060130867,"bytes_used":10766180352,"bytes_avail":757194104832,"bytes_total":767960285184,"degraded_objects":6183,"degraded_total":94878,"degraded_ratio":0.065167899829254416,"recovering_objects_per_sec":106,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":694,"num_bytes_recovered":0,"num_keys_recovered":0,"read_bytes_sec":1575,"write_bytes_sec":0,"read_op_per_sec":1,"write_op_per_sec":1207},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-21T19:07:53.731883+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T19:09:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:19.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:19.683 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:19.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:09:19.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:19 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:09:19.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:19 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:09:19.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:19 smithi104 ceph-mon[32438]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T19:09:19.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:19.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:19.813 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:19.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:09:19.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:19.960 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:19.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:20.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:20.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:20.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:20.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:20.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:20.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:20.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:20.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.667 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:20.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1266603492' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 6183/94878 objects degraded (6.517%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[29384]: pgmap v803: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 5933/85644 objects degraded (6.928%); 0 B/s, 117 objects/s recovering 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T19:09:20.683 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1266603492' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:09:20.684 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 6183/94878 objects degraded (6.517%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:20.684 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T19:09:20.684 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:20 smithi081 ceph-mon[39192]: pgmap v803: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 5933/85644 objects degraded (6.928%); 0 B/s, 117 objects/s recovering 2024-09-21T19:09:20.731 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T19:09:20.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:09:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:09:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:09:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T19:09:20.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1266603492' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:09:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 6183/94878 objects degraded (6.517%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T19:09:20.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:20 smithi104 ceph-mon[32438]: pgmap v803: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 5933/85644 objects degraded (6.928%); 0 B/s, 117 objects/s recovering 2024-09-21T19:09:20.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.812 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:20.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:20.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:20.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:20.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:21.055 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:21.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:21.227 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:21.285 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:21.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:21.357 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:21.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:21.499 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:21.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:21.636 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:21.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:21.773 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:21.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:21.918 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:21.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:21.966 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:21.976 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:09:21.976 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:09:22.012 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:09:22.013 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:09:22.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:22.042 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:09:22.042 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:09:22.069 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:22.077 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:09:22.077 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:09:22.111 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:09:22.111 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:09:22.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:09:22.140 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:09:22.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:22.174 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:09:22.174 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:09:22.205 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:22.206 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:09:22.207 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:09:22.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:22.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:22.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:22.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:22.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:22.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:22.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:22.896 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:22.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:22.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:23.031 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[39192]: pgmap v804: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 5933/83478 objects degraded (7.107%); 0 B/s, 105 objects/s recovering 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3158813581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[29384]: pgmap v804: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 5933/83478 objects degraded (7.107%); 0 B/s, 105 objects/s recovering 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3158813581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:23.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:23.169 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:23.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:22 smithi104 ceph-mon[32438]: pgmap v804: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 5933/83478 objects degraded (7.107%); 0 B/s, 105 objects/s recovering 2024-09-21T19:09:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3158813581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:23.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:23.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:09:23.299 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:23.401 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:23.432 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:23.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:23.564 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:23.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:23.706 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:23.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:23.844 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:23.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:23.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:23.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:24.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:24.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:24.266 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:24.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:24.406 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:24.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:24.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:24.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:24.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:24.817 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:24.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:24.955 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:24.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:25.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:25.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:25.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:24 smithi104 ceph-mon[32438]: from='client.36389 v1:172.21.15.81:0/2212046688' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:24 smithi104 ceph-mon[32438]: pgmap v805: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 5682/75282 objects degraded (7.548%); 0 B/s, 83 objects/s recovering 2024-09-21T19:09:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:24 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 5682/75282 objects degraded (7.548%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:25.313 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:25.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:24 smithi081 ceph-mon[29384]: from='client.36389 v1:172.21.15.81:0/2212046688' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:25.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:24 smithi081 ceph-mon[29384]: pgmap v805: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 5682/75282 objects degraded (7.548%); 0 B/s, 83 objects/s recovering 2024-09-21T19:09:25.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:24 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 5682/75282 objects degraded (7.548%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:25.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:24 smithi081 ceph-mon[39192]: from='client.36389 v1:172.21.15.81:0/2212046688' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:25.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:24 smithi081 ceph-mon[39192]: pgmap v805: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 5682/75282 objects degraded (7.548%); 0 B/s, 83 objects/s recovering 2024-09-21T19:09:25.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:24 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 5682/75282 objects degraded (7.548%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:25.345 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:25.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:25.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:25.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:25.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:25.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:25.753 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:25.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:25.885 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:25.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:25.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:26.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:26.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:26.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:26.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:26.295 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:26.298 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:09:26.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:26.415 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:26.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:26.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:26.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:26.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:26.718 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:26.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:26.860 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:26.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:26.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:26.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:26 smithi081 ceph-mon[29384]: pgmap v806: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 688 B/s rd, 0 B/s wr, 1.02k op/s; 5401/69042 objects degraded (7.823%); 0 B/s, 105 objects/s recovering 2024-09-21T19:09:26.997 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:26 smithi081 ceph-mon[39192]: pgmap v806: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 688 B/s rd, 0 B/s wr, 1.02k op/s; 5401/69042 objects degraded (7.823%); 0 B/s, 105 objects/s recovering 2024-09-21T19:09:26.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:27.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:27.113 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:27.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:27.151 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:09:27.151 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:09:27.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:26 smithi104 ceph-mon[32438]: pgmap v806: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 688 B/s rd, 0 B/s wr, 1.02k op/s; 5401/69042 objects degraded (7.823%); 0 B/s, 105 objects/s recovering 2024-09-21T19:09:27.273 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:27.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:09:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:09:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:09:27.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:27.411 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:27.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:27.543 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:27.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:27.677 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":804,"stamp":"2024-09-21T19:09:21.769122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1811094067,"num_objects":27826,"num_object_clones":0,"num_object_copies":83478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27826,"num_whiteouts":0,"num_read":2098,"num_read_kb":1850,"num_write":208037,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17621,"num_bytes_recovered":1056778488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87374,"ondisk_log_size":87374,"up":474,"acting":472,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9850720,"kb_used_data":5450688,"kb_used_omap":18,"kb_used_meta":4399725,"kb_avail":740110496,"statfs":{"total":767960285184,"available":757873147904,"internally_reserved":0,"allocated":5581504512,"data_stored":5575504507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4505319408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,4,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-341966848,"num_objects":-5218,"num_object_clones":0,"num_object_copies":-15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.466970"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":635,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317686+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:09:17.317686+0000","last_peered":"2024-09-21T19:09:17.317686+0000","last_clean":"2024-09-21T19:09:17.317686+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:09:17.317686+0000","last_undegraded":"2024-09-21T19:09:17.317686+0000","last_fullsized":"2024-09-21T19:09:17.317686+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":809,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736041+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:09:19.736041+0000","last_peered":"2024-09-21T19:09:19.736041+0000","last_clean":"2024-09-21T19:09:19.736041+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:09:19.736041+0000","last_undegraded":"2024-09-21T19:09:19.736041+0000","last_fullsized":"2024-09-21T19:09:19.736041+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":732,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.305917+0000","last_change":"2024-09-21T19:06:54.694001+0000","last_active":"2024-09-21T19:09:17.305917+0000","last_peered":"2024-09-21T19:09:17.305917+0000","last_clean":"2024-09-21T19:09:17.305917+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:09:17.305917+0000","last_undegraded":"2024-09-21T19:09:17.305917+0000","last_fullsized":"2024-09-21T19:09:17.305917+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:37.095528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:37.095528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:31.228363+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":726,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735770+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:09:19.735770+0000","last_peered":"2024-09-21T19:09:19.735770+0000","last_clean":"2024-09-21T19:09:19.735770+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:09:19.735770+0000","last_undegraded":"2024-09-21T19:09:19.735770+0000","last_fullsized":"2024-09-21T19:09:19.735770+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":729,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.316773+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:09:17.316773+0000","last_peered":"2024-09-21T19:09:17.316773+0000","last_clean":"2024-09-21T19:09:17.316773+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:09:17.316773+0000","last_undegraded":"2024-09-21T19:09:17.316773+0000","last_fullsized":"2024-09-21T19:09:17.316773+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314748+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:09:17.314748+0000","last_peered":"2024-09-21T19:09:17.314748+0000","last_clean":"2024-09-21T19:09:17.314748+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:09:17.314748+0000","last_undegraded":"2024-09-21T19:09:17.314748+0000","last_fullsized":"2024-09-21T19:09:17.314748+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":853,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314207+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:09:17.314207+0000","last_peered":"2024-09-21T19:09:17.314207+0000","last_clean":"2024-09-21T19:09:17.314207+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:09:17.314207+0000","last_undegraded":"2024-09-21T19:09:17.314207+0000","last_fullsized":"2024-09-21T19:09:17.314207+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":746,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735962+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:09:19.735962+0000","last_peered":"2024-09-21T19:09:19.735962+0000","last_clean":"2024-09-21T19:09:19.735962+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:09:19.735962+0000","last_undegraded":"2024-09-21T19:09:19.735962+0000","last_fullsized":"2024-09-21T19:09:19.735962+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":733,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.739182+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:09:19.739182+0000","last_peered":"2024-09-21T19:09:19.739182+0000","last_clean":"2024-09-21T19:09:19.739182+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:09:19.739182+0000","last_undegraded":"2024-09-21T19:09:19.739182+0000","last_fullsized":"2024-09-21T19:09:19.739182+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":790,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736665+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:09:19.736665+0000","last_peered":"2024-09-21T19:09:19.736665+0000","last_clean":"2024-09-21T19:09:19.736665+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:09:19.736665+0000","last_undegraded":"2024-09-21T19:09:19.736665+0000","last_fullsized":"2024-09-21T19:09:19.736665+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":680,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314275+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:09:17.314275+0000","last_peered":"2024-09-21T19:09:17.314275+0000","last_clean":"2024-09-21T19:09:17.314275+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:09:17.314275+0000","last_undegraded":"2024-09-21T19:09:17.314275+0000","last_fullsized":"2024-09-21T19:09:17.314275+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":601,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317545+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:09:17.317545+0000","last_peered":"2024-09-21T19:09:17.317545+0000","last_clean":"2024-09-21T19:09:17.317545+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:09:17.317545+0000","last_undegraded":"2024-09-21T19:09:17.317545+0000","last_fullsized":"2024-09-21T19:09:17.317545+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":286,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737590+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:09:19.737590+0000","last_peered":"2024-09-21T19:09:19.737590+0000","last_clean":"2024-09-21T19:09:19.737590+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:09:19.737590+0000","last_undegraded":"2024-09-21T19:09:19.737590+0000","last_fullsized":"2024-09-21T19:09:19.737590+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":736,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736096+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:09:19.736096+0000","last_peered":"2024-09-21T19:09:19.736096+0000","last_clean":"2024-09-21T19:09:19.736096+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:09:19.736096+0000","last_undegraded":"2024-09-21T19:09:19.736096+0000","last_fullsized":"2024-09-21T19:09:19.736096+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":718,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743334+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:09:14.743334+0000","last_peered":"2024-09-21T19:09:14.743334+0000","last_clean":"2024-09-21T19:09:14.743334+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:09:14.743334+0000","last_undegraded":"2024-09-21T19:09:14.743334+0000","last_fullsized":"2024-09-21T19:09:14.743334+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":288,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.305636+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:09:17.305636+0000","last_peered":"2024-09-21T19:09:17.305636+0000","last_clean":"2024-09-21T19:09:17.305636+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:09:17.305636+0000","last_undegraded":"2024-09-21T19:09:17.305636+0000","last_fullsized":"2024-09-21T19:09:17.305636+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.570833+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":755,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737807+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:09:19.737807+0000","last_peered":"2024-09-21T19:09:19.737807+0000","last_clean":"2024-09-21T19:09:19.737807+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:09:19.737807+0000","last_undegraded":"2024-09-21T19:09:19.737807+0000","last_fullsized":"2024-09-21T19:09:19.737807+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":617,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.327435+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:09:17.327435+0000","last_peered":"2024-09-21T19:09:17.327435+0000","last_clean":"2024-09-21T19:09:17.327435+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:09:17.327435+0000","last_undegraded":"2024-09-21T19:09:17.327435+0000","last_fullsized":"2024-09-21T19:09:17.327435+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":191,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.309842+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:09:17.309842+0000","last_peered":"2024-09-21T19:09:17.309842+0000","last_clean":"2024-09-21T19:09:17.309842+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:09:17.309842+0000","last_undegraded":"2024-09-21T19:09:17.309842+0000","last_fullsized":"2024-09-21T19:09:17.309842+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":743,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306339+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:09:17.306339+0000","last_peered":"2024-09-21T19:09:17.306339+0000","last_clean":"2024-09-21T19:09:17.306339+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:09:17.306339+0000","last_undegraded":"2024-09-21T19:09:17.306339+0000","last_fullsized":"2024-09-21T19:09:17.306339+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:07.738704+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":725,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.315022+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:09:17.315022+0000","last_peered":"2024-09-21T19:09:17.315022+0000","last_clean":"2024-09-21T19:09:17.315022+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:09:17.315022+0000","last_undegraded":"2024-09-21T19:09:17.315022+0000","last_fullsized":"2024-09-21T19:09:17.315022+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":757,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736796+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:09:19.736796+0000","last_peered":"2024-09-21T19:09:19.736796+0000","last_clean":"2024-09-21T19:09:19.736796+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:09:19.736796+0000","last_undegraded":"2024-09-21T19:09:19.736796+0000","last_fullsized":"2024-09-21T19:09:19.736796+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":796,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314806+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:09:17.314806+0000","last_peered":"2024-09-21T19:09:17.314806+0000","last_clean":"2024-09-21T19:09:17.314806+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:09:17.314806+0000","last_undegraded":"2024-09-21T19:09:17.314806+0000","last_fullsized":"2024-09-21T19:09:17.314806+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":741,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737438+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:09:19.737438+0000","last_peered":"2024-09-21T19:09:19.737438+0000","last_clean":"2024-09-21T19:09:19.737438+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:09:19.737438+0000","last_undegraded":"2024-09-21T19:09:19.737438+0000","last_fullsized":"2024-09-21T19:09:19.737438+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":756,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742577+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:09:14.742577+0000","last_peered":"2024-09-21T19:09:14.742577+0000","last_clean":"2024-09-21T19:09:14.742577+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:09:14.742577+0000","last_undegraded":"2024-09-21T19:09:14.742577+0000","last_fullsized":"2024-09-21T19:09:14.742577+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.611692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306002+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:09:17.306002+0000","last_peered":"2024-09-21T19:09:17.306002+0000","last_clean":"2024-09-21T19:09:17.306002+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:09:17.306002+0000","last_undegraded":"2024-09-21T19:09:17.306002+0000","last_fullsized":"2024-09-21T19:09:17.306002+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":645,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.323735+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:09:17.323735+0000","last_peered":"2024-09-21T19:09:17.323735+0000","last_clean":"2024-09-21T19:09:17.323735+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:09:17.323735+0000","last_undegraded":"2024-09-21T19:09:17.323735+0000","last_fullsized":"2024-09-21T19:09:17.323735+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":817,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.347297+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:09:18.347297+0000","last_peered":"2024-09-21T19:09:18.347297+0000","last_clean":"2024-09-21T19:09:18.347297+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:09:18.347297+0000","last_undegraded":"2024-09-21T19:09:18.347297+0000","last_fullsized":"2024-09-21T19:09:18.347297+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"401'13348","reported_seq":13502,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:21.366253+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:09:21.366253+0000","last_peered":"2024-09-21T19:09:21.366253+0000","last_clean":"2024-09-21T19:09:21.366253+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:09:21.366253+0000","last_undegraded":"2024-09-21T19:09:21.366253+0000","last_fullsized":"2024-09-21T19:09:21.366253+0000","mapping_epoch":330,"log_start":"374'8800","ondisk_log_start":"374'8800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100401152,"num_objects":1532,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13348,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":186,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.311170+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:09:17.311170+0000","last_peered":"2024-09-21T19:09:17.311170+0000","last_clean":"2024-09-21T19:09:17.311170+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:09:17.311170+0000","last_undegraded":"2024-09-21T19:09:17.311170+0000","last_fullsized":"2024-09-21T19:09:17.311170+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:13.700777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":785,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.307641+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:09:17.307641+0000","last_peered":"2024-09-21T19:09:17.307641+0000","last_clean":"2024-09-21T19:09:17.307641+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:09:17.307641+0000","last_undegraded":"2024-09-21T19:09:17.307641+0000","last_fullsized":"2024-09-21T19:09:17.307641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:35.238469+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":723,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313588+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:09:17.313588+0000","last_peered":"2024-09-21T19:09:17.313588+0000","last_clean":"2024-09-21T19:09:17.313588+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:09:17.313588+0000","last_undegraded":"2024-09-21T19:09:17.313588+0000","last_fullsized":"2024-09-21T19:09:17.313588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":974,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737857+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:09:19.737857+0000","last_peered":"2024-09-21T19:09:19.737857+0000","last_clean":"2024-09-21T19:09:19.737857+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:09:19.737857+0000","last_undegraded":"2024-09-21T19:09:19.737857+0000","last_fullsized":"2024-09-21T19:09:19.737857+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":412,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306256+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:09:17.306256+0000","last_peered":"2024-09-21T19:09:17.306256+0000","last_clean":"2024-09-21T19:09:17.306256+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:09:17.306256+0000","last_undegraded":"2024-09-21T19:09:17.306256+0000","last_fullsized":"2024-09-21T19:09:17.306256+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:21.288105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"401'12941","reported_seq":13095,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:21.414140+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:09:21.414140+0000","last_peered":"2024-09-21T19:09:21.414140+0000","last_clean":"2024-09-21T19:09:21.414140+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:09:21.414140+0000","last_undegraded":"2024-09-21T19:09:21.414140+0000","last_fullsized":"2024-09-21T19:09:21.414140+0000","mapping_epoch":330,"log_start":"373'8400","ondisk_log_start":"373'8400","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98107392,"num_objects":1497,"num_object_clones":0,"num_object_copies":4491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12941,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":518,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.739091+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:09:19.739091+0000","last_peered":"2024-09-21T19:09:19.739091+0000","last_clean":"2024-09-21T19:09:19.739091+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:09:19.739091+0000","last_undegraded":"2024-09-21T19:09:19.739091+0000","last_fullsized":"2024-09-21T19:09:19.739091+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":820,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313490+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:09:17.313490+0000","last_peered":"2024-09-21T19:09:17.313490+0000","last_clean":"2024-09-21T19:09:17.313490+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:09:17.313490+0000","last_undegraded":"2024-09-21T19:09:17.313490+0000","last_fullsized":"2024-09-21T19:09:17.313490+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":745,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.322148+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:09:17.322148+0000","last_peered":"2024-09-21T19:09:17.322148+0000","last_clean":"2024-09-21T19:09:17.322148+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:09:17.322148+0000","last_undegraded":"2024-09-21T19:09:17.322148+0000","last_fullsized":"2024-09-21T19:09:17.322148+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":728,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735512+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:09:19.735512+0000","last_peered":"2024-09-21T19:09:19.735512+0000","last_clean":"2024-09-21T19:09:19.735512+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:09:19.735512+0000","last_undegraded":"2024-09-21T19:09:19.735512+0000","last_fullsized":"2024-09-21T19:09:19.735512+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":745,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736515+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:09:19.736515+0000","last_peered":"2024-09-21T19:09:19.736515+0000","last_clean":"2024-09-21T19:09:19.736515+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:09:19.736515+0000","last_undegraded":"2024-09-21T19:09:19.736515+0000","last_fullsized":"2024-09-21T19:09:19.736515+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"399'12986","reported_seq":13136,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.244822+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:09:19.244822+0000","last_peered":"2024-09-21T19:09:19.244822+0000","last_clean":"2024-09-21T19:09:19.244822+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:09:19.244822+0000","last_undegraded":"2024-09-21T19:09:19.244822+0000","last_fullsized":"2024-09-21T19:09:19.244822+0000","mapping_epoch":330,"log_start":"367'7700","ondisk_log_start":"367'7700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112328704,"num_objects":1714,"num_object_clones":0,"num_object_copies":5142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12986,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":503,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313243+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:09:17.313243+0000","last_peered":"2024-09-21T19:09:17.313243+0000","last_clean":"2024-09-21T19:09:17.313243+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:09:17.313243+0000","last_undegraded":"2024-09-21T19:09:17.313243+0000","last_fullsized":"2024-09-21T19:09:17.313243+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":796,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735688+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:09:19.735688+0000","last_peered":"2024-09-21T19:09:19.735688+0000","last_clean":"2024-09-21T19:09:19.735688+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:09:19.735688+0000","last_undegraded":"2024-09-21T19:09:19.735688+0000","last_fullsized":"2024-09-21T19:09:19.735688+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":614,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.324046+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:09:17.324046+0000","last_peered":"2024-09-21T19:09:17.324046+0000","last_clean":"2024-09-21T19:09:17.324046+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:09:17.324046+0000","last_undegraded":"2024-09-21T19:09:17.324046+0000","last_fullsized":"2024-09-21T19:09:17.324046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":911,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313363+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:09:17.313363+0000","last_peered":"2024-09-21T19:09:17.313363+0000","last_clean":"2024-09-21T19:09:17.313363+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:09:17.313363+0000","last_undegraded":"2024-09-21T19:09:17.313363+0000","last_fullsized":"2024-09-21T19:09:17.313363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":740,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742353+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:09:14.742353+0000","last_peered":"2024-09-21T19:09:14.742353+0000","last_clean":"2024-09-21T19:09:14.742353+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:09:14.742353+0000","last_undegraded":"2024-09-21T19:09:14.742353+0000","last_fullsized":"2024-09-21T19:09:14.742353+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"397'13047","reported_seq":13193,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:16.845900+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:09:16.845900+0000","last_peered":"2024-09-21T19:09:16.845900+0000","last_clean":"2024-09-21T19:09:16.845900+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:09:16.845900+0000","last_undegraded":"2024-09-21T19:09:16.845900+0000","last_fullsized":"2024-09-21T19:09:16.845900+0000","mapping_epoch":330,"log_start":"368'8000","ondisk_log_start":"368'8000","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5047,"log_dups_size":0,"ondisk_log_size":5047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123928576,"num_objects":1891,"num_object_clones":0,"num_object_copies":5673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13047,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":529,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.347480+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:09:18.347480+0000","last_peered":"2024-09-21T19:09:18.347480+0000","last_clean":"2024-09-21T19:09:18.347480+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:09:18.347480+0000","last_undegraded":"2024-09-21T19:09:18.347480+0000","last_fullsized":"2024-09-21T19:09:18.347480+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":767,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306440+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:09:17.306440+0000","last_peered":"2024-09-21T19:09:17.306440+0000","last_clean":"2024-09-21T19:09:17.306440+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:09:17.306440+0000","last_undegraded":"2024-09-21T19:09:17.306440+0000","last_fullsized":"2024-09-21T19:09:17.306440+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:33.202656+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":286,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314896+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:09:17.314896+0000","last_peered":"2024-09-21T19:09:17.314896+0000","last_clean":"2024-09-21T19:09:17.314896+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:09:17.314896+0000","last_undegraded":"2024-09-21T19:09:17.314896+0000","last_fullsized":"2024-09-21T19:09:17.314896+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":794,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.327383+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:09:17.327383+0000","last_peered":"2024-09-21T19:09:17.327383+0000","last_clean":"2024-09-21T19:09:17.327383+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:09:17.327383+0000","last_undegraded":"2024-09-21T19:09:17.327383+0000","last_fullsized":"2024-09-21T19:09:17.327383+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":616,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317659+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:09:17.317659+0000","last_peered":"2024-09-21T19:09:17.317659+0000","last_clean":"2024-09-21T19:09:17.317659+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:09:17.317659+0000","last_undegraded":"2024-09-21T19:09:17.317659+0000","last_fullsized":"2024-09-21T19:09:17.317659+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"399'12944","reported_seq":13095,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.291983+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:09:19.291983+0000","last_peered":"2024-09-21T19:09:19.291983+0000","last_clean":"2024-09-21T19:09:19.291983+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:09:19.291983+0000","last_undegraded":"2024-09-21T19:09:19.291983+0000","last_fullsized":"2024-09-21T19:09:19.291983+0000","mapping_epoch":330,"log_start":"367'7600","ondisk_log_start":"367'7600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5344,"log_dups_size":0,"ondisk_log_size":5344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113442840,"num_objects":1732,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12944,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":507,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.349363+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:09:18.349363+0000","last_peered":"2024-09-21T19:09:18.349363+0000","last_clean":"2024-09-21T19:09:18.349363+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:09:18.349363+0000","last_undegraded":"2024-09-21T19:09:18.349363+0000","last_fullsized":"2024-09-21T19:09:18.349363+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":662,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.327537+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:09:17.327537+0000","last_peered":"2024-09-21T19:09:17.327537+0000","last_clean":"2024-09-21T19:09:17.327537+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:09:17.327537+0000","last_undegraded":"2024-09-21T19:09:17.327537+0000","last_fullsized":"2024-09-21T19:09:17.327537+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":755,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306099+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:09:17.306099+0000","last_peered":"2024-09-21T19:09:17.306099+0000","last_clean":"2024-09-21T19:09:17.306099+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:09:17.306099+0000","last_undegraded":"2024-09-21T19:09:17.306099+0000","last_fullsized":"2024-09-21T19:09:17.306099+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:34.228030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":642,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.343201+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:09:18.343201+0000","last_peered":"2024-09-21T19:09:18.343201+0000","last_clean":"2024-09-21T19:09:18.343201+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:09:18.343201+0000","last_undegraded":"2024-09-21T19:09:18.343201+0000","last_fullsized":"2024-09-21T19:09:18.343201+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":743,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.305835+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:09:17.305835+0000","last_peered":"2024-09-21T19:09:17.305835+0000","last_clean":"2024-09-21T19:09:17.305835+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:09:17.305835+0000","last_undegraded":"2024-09-21T19:09:17.305835+0000","last_fullsized":"2024-09-21T19:09:17.305835+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.253514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"399'13212","reported_seq":22729,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.292613+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:09:19.292613+0000","last_peered":"2024-09-21T19:09:19.292613+0000","last_clean":"2024-09-21T19:09:19.292613+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:09:19.292613+0000","last_undegraded":"2024-09-21T19:09:19.292613+0000","last_fullsized":"2024-09-21T19:09:19.292613+0000","mapping_epoch":380,"log_start":"367'7900","ondisk_log_start":"367'7900","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":480,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.321897+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:09:17.321897+0000","last_peered":"2024-09-21T19:09:17.321897+0000","last_clean":"2024-09-21T19:09:17.321897+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:09:17.321897+0000","last_undegraded":"2024-09-21T19:09:17.321897+0000","last_fullsized":"2024-09-21T19:09:17.321897+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1038,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.305997+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:09:17.305997+0000","last_peered":"2024-09-21T19:09:17.305997+0000","last_clean":"2024-09-21T19:09:17.305997+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:09:17.305997+0000","last_undegraded":"2024-09-21T19:09:17.305997+0000","last_fullsized":"2024-09-21T19:09:17.305997+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:45.305870+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":769,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736408+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:09:19.736408+0000","last_peered":"2024-09-21T19:09:19.736408+0000","last_clean":"2024-09-21T19:09:19.736408+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:09:19.736408+0000","last_undegraded":"2024-09-21T19:09:19.736408+0000","last_fullsized":"2024-09-21T19:09:19.736408+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":310,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306743+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:09:17.306743+0000","last_peered":"2024-09-21T19:09:17.306743+0000","last_clean":"2024-09-21T19:09:17.306743+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:09:17.306743+0000","last_undegraded":"2024-09-21T19:09:17.306743+0000","last_fullsized":"2024-09-21T19:09:17.306743+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:25.256087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":602,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317990+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:09:17.317990+0000","last_peered":"2024-09-21T19:09:17.317990+0000","last_clean":"2024-09-21T19:09:17.317990+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:09:17.317990+0000","last_undegraded":"2024-09-21T19:09:17.317990+0000","last_fullsized":"2024-09-21T19:09:17.317990+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":711,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314480+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:09:17.314480+0000","last_peered":"2024-09-21T19:09:17.314480+0000","last_clean":"2024-09-21T19:09:17.314480+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:09:17.314480+0000","last_undegraded":"2024-09-21T19:09:17.314480+0000","last_fullsized":"2024-09-21T19:09:17.314480+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"398'12824","reported_seq":22254,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.914421+0000","last_change":"2024-09-21T19:08:54.754076+0000","last_active":"2024-09-21T19:09:17.914421+0000","last_peered":"2024-09-21T19:09:17.914421+0000","last_clean":"2024-09-21T19:09:17.914421+0000","last_became_active":"2024-09-21T19:08:54.722178+0000","last_became_peered":"2024-09-21T19:08:54.722178+0000","last_unstale":"2024-09-21T19:09:17.914421+0000","last_undegraded":"2024-09-21T19:09:17.914421+0000","last_fullsized":"2024-09-21T19:09:17.914421+0000","mapping_epoch":389,"log_start":"363'6400","ondisk_log_start":"363'6400","created":330,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115998720,"num_objects":1770,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12824,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":303890432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":814,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.741865+0000","last_change":"2024-09-21T19:08:53.739015+0000","last_active":"2024-09-21T19:09:14.741865+0000","last_peered":"2024-09-21T19:09:14.741865+0000","last_clean":"2024-09-21T19:09:14.741865+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:09:14.741865+0000","last_undegraded":"2024-09-21T19:09:14.741865+0000","last_fullsized":"2024-09-21T19:09:14.741865+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:08:53.738929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.738929+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:55.112735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":228,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.307222+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:09:17.307222+0000","last_peered":"2024-09-21T19:09:17.307222+0000","last_clean":"2024-09-21T19:09:17.307222+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:09:17.307222+0000","last_undegraded":"2024-09-21T19:09:17.307222+0000","last_fullsized":"2024-09-21T19:09:17.307222+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":786,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.305412+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:09:17.305412+0000","last_peered":"2024-09-21T19:09:17.305412+0000","last_clean":"2024-09-21T19:09:17.305412+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:09:17.305412+0000","last_undegraded":"2024-09-21T19:09:17.305412+0000","last_fullsized":"2024-09-21T19:09:17.305412+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":179,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.309734+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:09:17.309734+0000","last_peered":"2024-09-21T19:09:17.309734+0000","last_clean":"2024-09-21T19:09:17.309734+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:09:17.309734+0000","last_undegraded":"2024-09-21T19:09:17.309734+0000","last_fullsized":"2024-09-21T19:09:17.309734+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":186,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.315586+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:09:17.315586+0000","last_peered":"2024-09-21T19:09:17.315586+0000","last_clean":"2024-09-21T19:09:17.315586+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:09:17.315586+0000","last_undegraded":"2024-09-21T19:09:17.315586+0000","last_fullsized":"2024-09-21T19:09:17.315586+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.374847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":730,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742859+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:09:14.742859+0000","last_peered":"2024-09-21T19:09:14.742859+0000","last_clean":"2024-09-21T19:09:14.742859+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:09:14.742859+0000","last_undegraded":"2024-09-21T19:09:14.742859+0000","last_fullsized":"2024-09-21T19:09:14.742859+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"398'12909","reported_seq":13055,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.730691+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:09:17.730691+0000","last_peered":"2024-09-21T19:09:17.730691+0000","last_clean":"2024-09-21T19:09:17.730691+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:09:17.730691+0000","last_undegraded":"2024-09-21T19:09:17.730691+0000","last_fullsized":"2024-09-21T19:09:17.730691+0000","mapping_epoch":330,"log_start":"368'7900","ondisk_log_start":"368'7900","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":5481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12909,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":522,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.307227+0000","last_change":"2024-09-21T19:09:08.156345+0000","last_active":"2024-09-21T19:09:17.307227+0000","last_peered":"2024-09-21T19:09:17.307227+0000","last_clean":"2024-09-21T19:09:17.307227+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:09:17.307227+0000","last_undegraded":"2024-09-21T19:09:17.307227+0000","last_fullsized":"2024-09-21T19:09:17.307227+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:08.156258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:08.156258+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.996519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":724,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.323848+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:09:17.323848+0000","last_peered":"2024-09-21T19:09:17.323848+0000","last_clean":"2024-09-21T19:09:17.323848+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:09:17.323848+0000","last_undegraded":"2024-09-21T19:09:17.323848+0000","last_fullsized":"2024-09-21T19:09:17.323848+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":610,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.322238+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:09:17.322238+0000","last_peered":"2024-09-21T19:09:17.322238+0000","last_clean":"2024-09-21T19:09:17.322238+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:09:17.322238+0000","last_undegraded":"2024-09-21T19:09:17.322238+0000","last_fullsized":"2024-09-21T19:09:17.322238+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1716,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:15.758812+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:09:15.758812+0000","last_peered":"2024-09-21T19:09:15.758812+0000","last_clean":"2024-09-21T19:09:15.758812+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:09:15.758812+0000","last_undegraded":"2024-09-21T19:09:15.758812+0000","last_fullsized":"2024-09-21T19:09:15.758812+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":958,"num_read_kb":953,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":682,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314843+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:09:17.314843+0000","last_peered":"2024-09-21T19:09:17.314843+0000","last_clean":"2024-09-21T19:09:17.314843+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:09:17.314843+0000","last_undegraded":"2024-09-21T19:09:17.314843+0000","last_fullsized":"2024-09-21T19:09:17.314843+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":737,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306139+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:09:17.306139+0000","last_peered":"2024-09-21T19:09:17.306139+0000","last_clean":"2024-09-21T19:09:17.306139+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:09:17.306139+0000","last_undegraded":"2024-09-21T19:09:17.306139+0000","last_fullsized":"2024-09-21T19:09:17.306139+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:30.010503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"399'12945","reported_seq":13094,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.291183+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:09:19.291183+0000","last_peered":"2024-09-21T19:09:19.291183+0000","last_clean":"2024-09-21T19:09:19.291183+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:09:19.291183+0000","last_undegraded":"2024-09-21T19:09:19.291183+0000","last_fullsized":"2024-09-21T19:09:19.291183+0000","mapping_epoch":330,"log_start":"367'7600","ondisk_log_start":"367'7600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12945,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":908,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.348072+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:09:18.348072+0000","last_peered":"2024-09-21T19:09:18.348072+0000","last_clean":"2024-09-21T19:09:18.348072+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:09:18.348072+0000","last_undegraded":"2024-09-21T19:09:18.348072+0000","last_fullsized":"2024-09-21T19:09:18.348072+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":745,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314604+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:09:17.314604+0000","last_peered":"2024-09-21T19:09:17.314604+0000","last_clean":"2024-09-21T19:09:17.314604+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:09:17.314604+0000","last_undegraded":"2024-09-21T19:09:17.314604+0000","last_fullsized":"2024-09-21T19:09:17.314604+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":820,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.745885+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:09:14.745885+0000","last_peered":"2024-09-21T19:09:14.745885+0000","last_clean":"2024-09-21T19:09:14.745885+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:09:14.745885+0000","last_undegraded":"2024-09-21T19:09:14.745885+0000","last_fullsized":"2024-09-21T19:09:14.745885+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":744,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742730+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:09:14.742730+0000","last_peered":"2024-09-21T19:09:14.742730+0000","last_clean":"2024-09-21T19:09:14.742730+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:09:14.742730+0000","last_undegraded":"2024-09-21T19:09:14.742730+0000","last_fullsized":"2024-09-21T19:09:14.742730+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":272,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314943+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:09:17.314943+0000","last_peered":"2024-09-21T19:09:17.314943+0000","last_clean":"2024-09-21T19:09:17.314943+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:09:17.314943+0000","last_undegraded":"2024-09-21T19:09:17.314943+0000","last_fullsized":"2024-09-21T19:09:17.314943+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"399'12797","reported_seq":12947,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.243568+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:09:19.243568+0000","last_peered":"2024-09-21T19:09:19.243568+0000","last_clean":"2024-09-21T19:09:19.243568+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:09:19.243568+0000","last_undegraded":"2024-09-21T19:09:19.243568+0000","last_fullsized":"2024-09-21T19:09:19.243568+0000","mapping_epoch":330,"log_start":"367'7500","ondisk_log_start":"367'7500","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12797,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":790,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.738047+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:09:19.738047+0000","last_peered":"2024-09-21T19:09:19.738047+0000","last_clean":"2024-09-21T19:09:19.738047+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:09:19.738047+0000","last_undegraded":"2024-09-21T19:09:19.738047+0000","last_fullsized":"2024-09-21T19:09:19.738047+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":758,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736358+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:09:19.736358+0000","last_peered":"2024-09-21T19:09:19.736358+0000","last_clean":"2024-09-21T19:09:19.736358+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:09:19.736358+0000","last_undegraded":"2024-09-21T19:09:19.736358+0000","last_fullsized":"2024-09-21T19:09:19.736358+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":742,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735766+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:09:19.735766+0000","last_peered":"2024-09-21T19:09:19.735766+0000","last_clean":"2024-09-21T19:09:19.735766+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:09:19.735766+0000","last_undegraded":"2024-09-21T19:09:19.735766+0000","last_fullsized":"2024-09-21T19:09:19.735766+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":717,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313334+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:09:17.313334+0000","last_peered":"2024-09-21T19:09:17.313334+0000","last_clean":"2024-09-21T19:09:17.313334+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:09:17.313334+0000","last_undegraded":"2024-09-21T19:09:17.313334+0000","last_fullsized":"2024-09-21T19:09:17.313334+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"398'12847","reported_seq":22329,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.903475+0000","last_change":"2024-09-21T19:08:55.780967+0000","last_active":"2024-09-21T19:09:17.903475+0000","last_peered":"2024-09-21T19:09:17.903475+0000","last_clean":"2024-09-21T19:09:17.903475+0000","last_became_active":"2024-09-21T19:08:55.751488+0000","last_became_peered":"2024-09-21T19:08:55.751488+0000","last_unstale":"2024-09-21T19:09:17.903475+0000","last_undegraded":"2024-09-21T19:09:17.903475+0000","last_fullsized":"2024-09-21T19:09:17.903475+0000","mapping_epoch":390,"log_start":"363'6400","ondisk_log_start":"363'6400","created":330,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4662,"num_bytes_recovered":305528832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":182,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.312815+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:09:17.312815+0000","last_peered":"2024-09-21T19:09:17.312815+0000","last_clean":"2024-09-21T19:09:17.312815+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:09:17.312815+0000","last_undegraded":"2024-09-21T19:09:17.312815+0000","last_fullsized":"2024-09-21T19:09:17.312815+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.739272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":839,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.738944+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:09:19.738944+0000","last_peered":"2024-09-21T19:09:19.738944+0000","last_clean":"2024-09-21T19:09:19.738944+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:09:19.738944+0000","last_undegraded":"2024-09-21T19:09:19.738944+0000","last_fullsized":"2024-09-21T19:09:19.738944+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":653,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.343634+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:09:18.343634+0000","last_peered":"2024-09-21T19:09:18.343634+0000","last_clean":"2024-09-21T19:09:18.343634+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:09:18.343634+0000","last_undegraded":"2024-09-21T19:09:18.343634+0000","last_fullsized":"2024-09-21T19:09:18.343634+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":177,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.344412+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:09:18.344412+0000","last_peered":"2024-09-21T19:09:18.344412+0000","last_clean":"2024-09-21T19:09:18.344412+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:09:18.344412+0000","last_undegraded":"2024-09-21T19:09:18.344412+0000","last_fullsized":"2024-09-21T19:09:18.344412+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":175,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314284+0000","last_change":"2024-09-21T19:06:51.856396+0000","last_active":"2024-09-21T19:09:17.314284+0000","last_peered":"2024-09-21T19:09:17.314284+0000","last_clean":"2024-09-21T19:09:17.314284+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:09:17.314284+0000","last_undegraded":"2024-09-21T19:09:17.314284+0000","last_fullsized":"2024-09-21T19:09:17.314284+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.856212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.856212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"398'12953","reported_seq":13101,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.636493+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:09:17.636493+0000","last_peered":"2024-09-21T19:09:17.636493+0000","last_clean":"2024-09-21T19:09:17.636493+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:09:17.636493+0000","last_undegraded":"2024-09-21T19:09:17.636493+0000","last_fullsized":"2024-09-21T19:09:17.636493+0000","mapping_epoch":330,"log_start":"367'7700","ondisk_log_start":"367'7700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:46.323337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116850688,"num_objects":1783,"num_object_clones":0,"num_object_copies":5349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12953,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":827,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743296+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:09:14.743296+0000","last_peered":"2024-09-21T19:09:14.743296+0000","last_clean":"2024-09-21T19:09:14.743296+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:09:14.743296+0000","last_undegraded":"2024-09-21T19:09:14.743296+0000","last_fullsized":"2024-09-21T19:09:14.743296+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":765,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737487+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:09:19.737487+0000","last_peered":"2024-09-21T19:09:19.737487+0000","last_clean":"2024-09-21T19:09:19.737487+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:09:19.737487+0000","last_undegraded":"2024-09-21T19:09:19.737487+0000","last_fullsized":"2024-09-21T19:09:19.737487+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":759,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.318683+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:09:17.318683+0000","last_peered":"2024-09-21T19:09:17.318683+0000","last_clean":"2024-09-21T19:09:17.318683+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:09:17.318683+0000","last_undegraded":"2024-09-21T19:09:17.318683+0000","last_fullsized":"2024-09-21T19:09:17.318683+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":228,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.314097+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:09:17.314097+0000","last_peered":"2024-09-21T19:09:17.314097+0000","last_clean":"2024-09-21T19:09:17.314097+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:09:17.314097+0000","last_undegraded":"2024-09-21T19:09:17.314097+0000","last_fullsized":"2024-09-21T19:09:17.314097+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":738,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737683+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:09:19.737683+0000","last_peered":"2024-09-21T19:09:19.737683+0000","last_clean":"2024-09-21T19:09:19.737683+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:09:19.737683+0000","last_undegraded":"2024-09-21T19:09:19.737683+0000","last_fullsized":"2024-09-21T19:09:19.737683+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"398'12677","reported_seq":12825,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.899322+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:09:17.899322+0000","last_peered":"2024-09-21T19:09:17.899322+0000","last_clean":"2024-09-21T19:09:17.899322+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:09:17.899322+0000","last_undegraded":"2024-09-21T19:09:17.899322+0000","last_fullsized":"2024-09-21T19:09:17.899322+0000","mapping_epoch":330,"log_start":"363'6200","ondisk_log_start":"363'6200","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115933184,"num_objects":1769,"num_object_clones":0,"num_object_copies":5307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":620,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.327680+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:09:17.327680+0000","last_peered":"2024-09-21T19:09:17.327680+0000","last_clean":"2024-09-21T19:09:17.327680+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:09:17.327680+0000","last_undegraded":"2024-09-21T19:09:17.327680+0000","last_fullsized":"2024-09-21T19:09:17.327680+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":720,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313842+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:09:17.313842+0000","last_peered":"2024-09-21T19:09:17.313842+0000","last_clean":"2024-09-21T19:09:17.313842+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:09:17.313842+0000","last_undegraded":"2024-09-21T19:09:17.313842+0000","last_fullsized":"2024-09-21T19:09:17.313842+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":175,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.311059+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:09:17.311059+0000","last_peered":"2024-09-21T19:09:17.311059+0000","last_clean":"2024-09-21T19:09:17.311059+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:09:17.311059+0000","last_undegraded":"2024-09-21T19:09:17.311059+0000","last_fullsized":"2024-09-21T19:09:17.311059+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:10.305625+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":616,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.322150+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:09:17.322150+0000","last_peered":"2024-09-21T19:09:17.322150+0000","last_clean":"2024-09-21T19:09:17.322150+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:09:17.322150+0000","last_undegraded":"2024-09-21T19:09:17.322150+0000","last_fullsized":"2024-09-21T19:09:17.322150+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"398'12766","reported_seq":12914,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.742119+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:09:17.742119+0000","last_peered":"2024-09-21T19:09:17.742119+0000","last_clean":"2024-09-21T19:09:17.742119+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:09:17.742119+0000","last_undegraded":"2024-09-21T19:09:17.742119+0000","last_fullsized":"2024-09-21T19:09:17.742119+0000","mapping_epoch":330,"log_start":"368'7700","ondisk_log_start":"368'7700","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115605504,"num_objects":1764,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12766,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":672,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.343939+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:09:18.343939+0000","last_peered":"2024-09-21T19:09:18.343939+0000","last_clean":"2024-09-21T19:09:18.343939+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:09:18.343939+0000","last_undegraded":"2024-09-21T19:09:18.343939+0000","last_fullsized":"2024-09-21T19:09:18.343939+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":749,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743240+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:09:14.743240+0000","last_peered":"2024-09-21T19:09:14.743240+0000","last_clean":"2024-09-21T19:09:14.743240+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:09:14.743240+0000","last_undegraded":"2024-09-21T19:09:14.743240+0000","last_fullsized":"2024-09-21T19:09:14.743240+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":186,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.310922+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:09:17.310922+0000","last_peered":"2024-09-21T19:09:17.310922+0000","last_clean":"2024-09-21T19:09:17.310922+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:09:17.310922+0000","last_undegraded":"2024-09-21T19:09:17.310922+0000","last_fullsized":"2024-09-21T19:09:17.310922+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":721,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.745771+0000","last_change":"2024-09-21T19:05:42.904115+0000","last_active":"2024-09-21T19:09:14.745771+0000","last_peered":"2024-09-21T19:09:14.745771+0000","last_clean":"2024-09-21T19:09:14.745771+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:09:14.745771+0000","last_undegraded":"2024-09-21T19:09:14.745771+0000","last_fullsized":"2024-09-21T19:09:14.745771+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:39.134616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.134616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"399'12979","reported_seq":15656,"reported_epoch":399,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:19.267898+0000","last_change":"2024-09-21T19:08:53.601069+0000","last_active":"2024-09-21T19:09:19.267898+0000","last_peered":"2024-09-21T19:09:19.267898+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:09:19.267898+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"367'7700","ondisk_log_start":"367'7700","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114622464,"num_objects":1749,"num_object_clones":0,"num_object_copies":5247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12979,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1262,"num_bytes_recovered":37355520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-1581}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":794,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735836+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:09:19.735836+0000","last_peered":"2024-09-21T19:09:19.735836+0000","last_clean":"2024-09-21T19:09:19.735836+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:09:19.735836+0000","last_undegraded":"2024-09-21T19:09:19.735836+0000","last_fullsized":"2024-09-21T19:09:19.735836+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":743,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743201+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:09:14.743201+0000","last_peered":"2024-09-21T19:09:14.743201+0000","last_clean":"2024-09-21T19:09:14.743201+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:09:14.743201+0000","last_undegraded":"2024-09-21T19:09:14.743201+0000","last_fullsized":"2024-09-21T19:09:14.743201+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":175,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.311302+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:09:17.311302+0000","last_peered":"2024-09-21T19:09:17.311302+0000","last_clean":"2024-09-21T19:09:17.311302+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:09:17.311302+0000","last_undegraded":"2024-09-21T19:09:17.311302+0000","last_fullsized":"2024-09-21T19:09:17.311302+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:00.385824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":711,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.316984+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:09:17.316984+0000","last_peered":"2024-09-21T19:09:17.316984+0000","last_clean":"2024-09-21T19:09:17.316984+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:09:17.316984+0000","last_undegraded":"2024-09-21T19:09:17.316984+0000","last_fullsized":"2024-09-21T19:09:17.316984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"398'13140","reported_seq":17580,"reported_epoch":398,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:17.745268+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:09:17.745268+0000","last_peered":"2024-09-21T19:09:17.745268+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:09:17.745268+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"369'8100","ondisk_log_start":"369'8100","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115998720,"num_objects":1770,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13140,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":103088128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":-833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":341,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.745216+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:09:14.745216+0000","last_peered":"2024-09-21T19:09:14.745216+0000","last_clean":"2024-09-21T19:09:14.745216+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:09:14.745216+0000","last_undegraded":"2024-09-21T19:09:14.745216+0000","last_fullsized":"2024-09-21T19:09:14.745216+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":757,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742804+0000","last_change":"2024-09-21T19:05:51.058465+0000","last_active":"2024-09-21T19:09:14.742804+0000","last_peered":"2024-09-21T19:09:14.742804+0000","last_clean":"2024-09-21T19:09:14.742804+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:09:14.742804+0000","last_undegraded":"2024-09-21T19:09:14.742804+0000","last_fullsized":"2024-09-21T19:09:14.742804+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:05:51.058397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.058397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":224,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313719+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:09:17.313719+0000","last_peered":"2024-09-21T19:09:17.313719+0000","last_clean":"2024-09-21T19:09:17.313719+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:09:17.313719+0000","last_undegraded":"2024-09-21T19:09:17.313719+0000","last_fullsized":"2024-09-21T19:09:17.313719+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":748,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.343297+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:09:18.343297+0000","last_peered":"2024-09-21T19:09:18.343297+0000","last_clean":"2024-09-21T19:09:18.343297+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:09:18.343297+0000","last_undegraded":"2024-09-21T19:09:18.343297+0000","last_fullsized":"2024-09-21T19:09:18.343297+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":683,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.345346+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:09:18.345346+0000","last_peered":"2024-09-21T19:09:18.345346+0000","last_clean":"2024-09-21T19:09:18.345346+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:09:18.345346+0000","last_undegraded":"2024-09-21T19:09:18.345346+0000","last_fullsized":"2024-09-21T19:09:18.345346+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":744,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.320468+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:09:17.320468+0000","last_peered":"2024-09-21T19:09:17.320468+0000","last_clean":"2024-09-21T19:09:17.320468+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:09:17.320468+0000","last_undegraded":"2024-09-21T19:09:17.320468+0000","last_fullsized":"2024-09-21T19:09:17.320468+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":280,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313087+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:09:17.313087+0000","last_peered":"2024-09-21T19:09:17.313087+0000","last_clean":"2024-09-21T19:09:17.313087+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:09:17.313087+0000","last_undegraded":"2024-09-21T19:09:17.313087+0000","last_fullsized":"2024-09-21T19:09:17.313087+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":717,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742772+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:09:14.742772+0000","last_peered":"2024-09-21T19:09:14.742772+0000","last_clean":"2024-09-21T19:09:14.742772+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:09:14.742772+0000","last_undegraded":"2024-09-21T19:09:14.742772+0000","last_fullsized":"2024-09-21T19:09:14.742772+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":620,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.323949+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:09:17.323949+0000","last_peered":"2024-09-21T19:09:17.323949+0000","last_clean":"2024-09-21T19:09:17.323949+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:09:17.323949+0000","last_undegraded":"2024-09-21T19:09:17.323949+0000","last_fullsized":"2024-09-21T19:09:17.323949+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":729,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743516+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:09:14.743516+0000","last_peered":"2024-09-21T19:09:14.743516+0000","last_clean":"2024-09-21T19:09:14.743516+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:09:14.743516+0000","last_undegraded":"2024-09-21T19:09:14.743516+0000","last_fullsized":"2024-09-21T19:09:14.743516+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":714,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.320314+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:09:17.320314+0000","last_peered":"2024-09-21T19:09:17.320314+0000","last_clean":"2024-09-21T19:09:17.320314+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:09:17.320314+0000","last_undegraded":"2024-09-21T19:09:17.320314+0000","last_fullsized":"2024-09-21T19:09:17.320314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":175,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.309924+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:09:17.309924+0000","last_peered":"2024-09-21T19:09:17.309924+0000","last_clean":"2024-09-21T19:09:17.309924+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:09:17.309924+0000","last_undegraded":"2024-09-21T19:09:17.309924+0000","last_fullsized":"2024-09-21T19:09:17.309924+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":331,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743014+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:09:14.743014+0000","last_peered":"2024-09-21T19:09:14.743014+0000","last_clean":"2024-09-21T19:09:14.743014+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:09:14.743014+0000","last_undegraded":"2024-09-21T19:09:14.743014+0000","last_fullsized":"2024-09-21T19:09:14.743014+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":617,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.318163+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:09:17.318163+0000","last_peered":"2024-09-21T19:09:17.318163+0000","last_clean":"2024-09-21T19:09:17.318163+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:09:17.318163+0000","last_undegraded":"2024-09-21T19:09:17.318163+0000","last_fullsized":"2024-09-21T19:09:17.318163+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":940,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313456+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:09:17.313456+0000","last_peered":"2024-09-21T19:09:17.313456+0000","last_clean":"2024-09-21T19:09:17.313456+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:09:17.313456+0000","last_undegraded":"2024-09-21T19:09:17.313456+0000","last_fullsized":"2024-09-21T19:09:17.313456+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":740,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.735673+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:09:19.735673+0000","last_peered":"2024-09-21T19:09:19.735673+0000","last_clean":"2024-09-21T19:09:19.735673+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:09:19.735673+0000","last_undegraded":"2024-09-21T19:09:19.735673+0000","last_fullsized":"2024-09-21T19:09:19.735673+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":740,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313660+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:09:17.313660+0000","last_peered":"2024-09-21T19:09:17.313660+0000","last_clean":"2024-09-21T19:09:17.313660+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:09:17.313660+0000","last_undegraded":"2024-09-21T19:09:17.313660+0000","last_fullsized":"2024-09-21T19:09:17.313660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":732,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.742676+0000","last_change":"2024-09-21T19:05:42.903922+0000","last_active":"2024-09-21T19:09:14.742676+0000","last_peered":"2024-09-21T19:09:14.742676+0000","last_clean":"2024-09-21T19:09:14.742676+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:09:14.742676+0000","last_undegraded":"2024-09-21T19:09:14.742676+0000","last_fullsized":"2024-09-21T19:09:14.742676+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:37.202174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.202174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":390,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306200+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:09:17.306200+0000","last_peered":"2024-09-21T19:09:17.306200+0000","last_clean":"2024-09-21T19:09:17.306200+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:09:17.306200+0000","last_undegraded":"2024-09-21T19:09:17.306200+0000","last_fullsized":"2024-09-21T19:09:17.306200+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":609,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.324247+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:09:17.324247+0000","last_peered":"2024-09-21T19:09:17.324247+0000","last_clean":"2024-09-21T19:09:17.324247+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:09:17.324247+0000","last_undegraded":"2024-09-21T19:09:17.324247+0000","last_fullsized":"2024-09-21T19:09:17.324247+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":819,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.736184+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:09:19.736184+0000","last_peered":"2024-09-21T19:09:19.736184+0000","last_clean":"2024-09-21T19:09:19.736184+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:09:19.736184+0000","last_undegraded":"2024-09-21T19:09:19.736184+0000","last_fullsized":"2024-09-21T19:09:19.736184+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":738,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-21T19:09:19.737599+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:09:19.737599+0000","last_peered":"2024-09-21T19:09:19.737599+0000","last_clean":"2024-09-21T19:09:19.737599+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:09:19.737599+0000","last_undegraded":"2024-09-21T19:09:19.737599+0000","last_fullsized":"2024-09-21T19:09:19.737599+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":732,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306296+0000","last_change":"2024-09-21T19:06:54.699978+0000","last_active":"2024-09-21T19:09:17.306296+0000","last_peered":"2024-09-21T19:09:17.306296+0000","last_clean":"2024-09-21T19:09:17.306296+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:09:17.306296+0000","last_undegraded":"2024-09-21T19:09:17.306296+0000","last_fullsized":"2024-09-21T19:09:17.306296+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:33.010157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.010157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:29.268143+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313815+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:09:17.313815+0000","last_peered":"2024-09-21T19:09:17.313815+0000","last_clean":"2024-09-21T19:09:17.313815+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:09:17.313815+0000","last_undegraded":"2024-09-21T19:09:17.313815+0000","last_fullsized":"2024-09-21T19:09:17.313815+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":811,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.306597+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:09:17.306597+0000","last_peered":"2024-09-21T19:09:17.306597+0000","last_clean":"2024-09-21T19:09:17.306597+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:09:17.306597+0000","last_undegraded":"2024-09-21T19:09:17.306597+0000","last_fullsized":"2024-09-21T19:09:17.306597+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:52.278173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":734,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.743103+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:09:14.743103+0000","last_peered":"2024-09-21T19:09:14.743103+0000","last_clean":"2024-09-21T19:09:14.743103+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:09:14.743103+0000","last_undegraded":"2024-09-21T19:09:14.743103+0000","last_fullsized":"2024-09-21T19:09:14.743103+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":736,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317150+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:09:17.317150+0000","last_peered":"2024-09-21T19:09:17.317150+0000","last_clean":"2024-09-21T19:09:17.317150+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:09:17.317150+0000","last_undegraded":"2024-09-21T19:09:17.317150+0000","last_fullsized":"2024-09-21T19:09:17.317150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":628,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317681+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:09:17.317681+0000","last_peered":"2024-09-21T19:09:17.317681+0000","last_clean":"2024-09-21T19:09:17.317681+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:09:17.317681+0000","last_undegraded":"2024-09-21T19:09:17.317681+0000","last_fullsized":"2024-09-21T19:09:17.317681+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":760,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313984+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:09:17.313984+0000","last_peered":"2024-09-21T19:09:17.313984+0000","last_clean":"2024-09-21T19:09:17.313984+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:09:17.313984+0000","last_undegraded":"2024-09-21T19:09:17.313984+0000","last_fullsized":"2024-09-21T19:09:17.313984+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":674,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.320682+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:09:17.320682+0000","last_peered":"2024-09-21T19:09:17.320682+0000","last_clean":"2024-09-21T19:09:17.320682+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:09:17.320682+0000","last_undegraded":"2024-09-21T19:09:17.320682+0000","last_fullsized":"2024-09-21T19:09:17.320682+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":745,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.744147+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:09:14.744147+0000","last_peered":"2024-09-21T19:09:14.744147+0000","last_clean":"2024-09-21T19:09:14.744147+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:09:14.744147+0000","last_undegraded":"2024-09-21T19:09:14.744147+0000","last_fullsized":"2024-09-21T19:09:14.744147+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317320+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:09:17.317320+0000","last_peered":"2024-09-21T19:09:17.317320+0000","last_clean":"2024-09-21T19:09:17.317320+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:09:17.317320+0000","last_undegraded":"2024-09-21T19:09:17.317320+0000","last_fullsized":"2024-09-21T19:09:17.317320+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":759,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313414+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:09:17.313414+0000","last_peered":"2024-09-21T19:09:17.313414+0000","last_clean":"2024-09-21T19:09:17.313414+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:09:17.313414+0000","last_undegraded":"2024-09-21T19:09:17.313414+0000","last_fullsized":"2024-09-21T19:09:17.313414+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":612,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.317549+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:09:17.317549+0000","last_peered":"2024-09-21T19:09:17.317549+0000","last_clean":"2024-09-21T19:09:17.317549+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:09:17.317549+0000","last_undegraded":"2024-09-21T19:09:17.317549+0000","last_fullsized":"2024-09-21T19:09:17.317549+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":650,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.342196+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:09:18.342196+0000","last_peered":"2024-09-21T19:09:18.342196+0000","last_clean":"2024-09-21T19:09:18.342196+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:09:18.342196+0000","last_undegraded":"2024-09-21T19:09:18.342196+0000","last_fullsized":"2024-09-21T19:09:18.342196+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":728,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-09-21T19:09:14.741974+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:09:14.741974+0000","last_peered":"2024-09-21T19:09:14.741974+0000","last_clean":"2024-09-21T19:09:14.741974+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:09:14.741974+0000","last_undegraded":"2024-09-21T19:09:14.741974+0000","last_fullsized":"2024-09-21T19:09:14.741974+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":736,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.305986+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:09:17.305986+0000","last_peered":"2024-09-21T19:09:17.305986+0000","last_clean":"2024-09-21T19:09:17.305986+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:09:17.305986+0000","last_undegraded":"2024-09-21T19:09:17.305986+0000","last_fullsized":"2024-09-21T19:09:17.305986+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":752,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.313566+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:09:17.313566+0000","last_peered":"2024-09-21T19:09:17.313566+0000","last_clean":"2024-09-21T19:09:17.313566+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:09:17.313566+0000","last_undegraded":"2024-09-21T19:09:17.313566+0000","last_fullsized":"2024-09-21T19:09:17.313566+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":716,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T19:09:18.347390+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:09:18.347390+0000","last_peered":"2024-09-21T19:09:18.347390+0000","last_clean":"2024-09-21T19:09:18.347390+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:09:18.347390+0000","last_undegraded":"2024-09-21T19:09:18.347390+0000","last_fullsized":"2024-09-21T19:09:18.347390+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":632,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T19:09:17.324144+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:09:17.324144+0000","last_peered":"2024-09-21T19:09:17.324144+0000","last_clean":"2024-09-21T19:09:17.324144+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:09:17.324144+0000","last_undegraded":"2024-09-21T19:09:17.324144+0000","last_fullsized":"2024-09-21T19:09:17.324144+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1810497560,"num_objects":27627,"num_object_clones":0,"num_object_copies":82881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207315,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17386,"num_bytes_recovered":1056505856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5465321472,"data_stored":5465309256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85715,"ondisk_log_size":85715,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":958,"num_read_kb":953,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266886,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1075980,"kb_used_data":485308,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92669172,"statfs":{"total":95995035648,"available":94893232128,"internally_reserved":0,"allocated":496955392,"data_stored":496216701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":604830154},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464337,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052776,"kb_used_data":773584,"kb_used_omap":4,"kb_used_meta":279099,"kb_avail":92692376,"statfs":{"total":95995035648,"available":94916993024,"internally_reserved":0,"allocated":792150016,"data_stored":791404550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":285798052},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619038,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":835156,"kb_used_data":583296,"kb_used_omap":4,"kb_used_meta":251835,"kb_avail":92909996,"statfs":{"total":95995035648,"available":95139835904,"internally_reserved":0,"allocated":597295104,"data_stored":596559372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":257879715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019093,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1331784,"kb_used_data":813296,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92413368,"statfs":{"total":95995035648,"available":94631288832,"internally_reserved":0,"allocated":832815104,"data_stored":832051689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182617,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315060,"kb_used_data":584160,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92430092,"statfs":{"total":95995035648,"available":94648414208,"internally_reserved":0,"allocated":598179840,"data_stored":597412658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":748419530},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411551,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1244360,"kb_used_data":693940,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92500792,"statfs":{"total":95995035648,"available":94720811008,"internally_reserved":0,"allocated":710594560,"data_stored":709871121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019252,"kb_used_data":352224,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92725900,"statfs":{"total":95995035648,"available":94951321600,"internally_reserved":0,"allocated":360677376,"data_stored":359902724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738601,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1976352,"kb_used_data":1164880,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91768800,"statfs":{"total":95995035648,"available":93971251200,"internally_reserved":0,"allocated":1192837120,"data_stored":1192085692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1175719936,"data_stored":1175715864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":350224384,"data_stored":350224384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":702939136,"data_stored":702939136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":588906496,"data_stored":588906496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":804655104,"data_stored":804651032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":579993600,"data_stored":579993600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":772214784,"data_stored":772210712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":490668032,"data_stored":490668032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:09:27.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:27.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:27.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:27.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:27.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:28.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:27 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:09:28.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:27 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:09:28.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:27 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:09:28.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:27 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:09:28.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:28.088 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:28.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:28.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:09:28.222 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:28.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:27 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:09:28.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:27 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:09:28.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:28.366 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:28.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:09:28.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:28.502 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:28.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:28.639 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:28.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.772 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:28.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:28.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:09:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:09:28.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:28.901 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:28.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:28.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:28 smithi081 ceph-mon[29384]: pgmap v807: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 830 op/s; 5401/64806 objects degraded (8.334%); 0 B/s, 63 objects/s recovering 2024-09-21T19:09:28.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:28 smithi081 ceph-mon[29384]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T19:09:28.967 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:28 smithi081 ceph-mon[39192]: pgmap v807: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 830 op/s; 5401/64806 objects degraded (8.334%); 0 B/s, 63 objects/s recovering 2024-09-21T19:09:28.967 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:28 smithi081 ceph-mon[39192]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T19:09:29.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.031 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:29.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:29.034 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:29.132 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:29.170 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:28 smithi104 ceph-mon[32438]: pgmap v807: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 830 op/s; 5401/64806 objects degraded (8.334%); 0 B/s, 63 objects/s recovering 2024-09-21T19:09:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:28 smithi104 ceph-mon[32438]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T19:09:29.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:29.311 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:29.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.452 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:29.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:29.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.586 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:29.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:29.668 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:29.682 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:09:29.682 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:09:29.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:29.720 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:09:29.721 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:09:29.737 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:09:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.739 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:29.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:29.777 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:09:29.777 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:09:29.814 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:09:29.815 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:09:29.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:29.848 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:09:29.848 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:09:29.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:09:29.880 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:09:29.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:29.893 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:29.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:29.910 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:09:29.910 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:09:29.943 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:09:29.943 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:09:29.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.032 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:30.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:30.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3622463773' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2636141184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 5401/64806 objects degraded (8.334%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3622463773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[39192]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3622463773' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:09:30.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2636141184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:30.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 5401/64806 objects degraded (8.334%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:30.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:30.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3622463773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:09:30.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:29 smithi081 ceph-mon[29384]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T19:09:30.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:30.168 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3622463773' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2636141184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 5401/64806 objects degraded (8.334%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3622463773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:09:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:29 smithi104 ceph-mon[32438]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T19:09:30.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.307 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:30.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:30.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:30.443 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:30.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:30.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:30.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:30.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:30.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:09:30.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:30.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:30.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:30.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:30.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:31.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:30 smithi081 ceph-mon[39192]: pgmap v810: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.0 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 4870/51912 objects degraded (9.381%); 0 B/s, 132 objects/s recovering 2024-09-21T19:09:31.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:30 smithi081 ceph-mon[29384]: pgmap v810: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.0 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 4870/51912 objects degraded (9.381%); 0 B/s, 132 objects/s recovering 2024-09-21T19:09:31.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:31.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:31.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:31.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:30 smithi104 ceph-mon[32438]: pgmap v810: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.0 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 4870/51912 objects degraded (9.381%); 0 B/s, 132 objects/s recovering 2024-09-21T19:09:31.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.389 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:31.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:31.490 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:31.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:31.662 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:31.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:31.802 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:31.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:31.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:31.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:32.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.076 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:32.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:32.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:32.217 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:31 smithi104 ceph-mon[32438]: 3.f scrub starts 2024-09-21T19:09:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:31 smithi104 ceph-mon[32438]: 3.f scrub ok 2024-09-21T19:09:32.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:31 smithi081 ceph-mon[39192]: 3.f scrub starts 2024-09-21T19:09:32.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:31 smithi081 ceph-mon[39192]: 3.f scrub ok 2024-09-21T19:09:32.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:31 smithi081 ceph-mon[29384]: 3.f scrub starts 2024-09-21T19:09:32.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:31 smithi081 ceph-mon[29384]: 3.f scrub ok 2024-09-21T19:09:32.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:235880 8.e e235/172 8:7f5b8aa9:::benchmark_data_smithi081_105538_object103648:head v 235'14796, mlcod=235'14793)", 2024-09-21T19:09:32.367 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.660995+0000", 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 457.98078783, 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.215059784, 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.660995+0000", 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.660995+0000", 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.660996+0000", 2024-09-21T19:09:32.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2780000000000001e-06 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661004+0000", 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.537e-06 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661014+0000", 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8509999999999994e-06 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661021+0000", 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1049999999999997e-06 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.673549+0000", 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012528169 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.673572+0000", 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2654e-05 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876055+0000", 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20248319000000001 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237698 8.1d 8:bb593120:::benchmark_data_smithi081_105538_object105466:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.696353+0000", 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 454.94543039600001, 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2170390849999999, 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:09:32.372 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237698 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696353+0000", 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696353+0000", 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696354+0000", 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7489999999999999e-06 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696359+0000", 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9300000000000002e-06 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696369+0000", 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5850000000000006e-06 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696378+0000", 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8780000000000004e-06 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696533+0000", 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015554699999999999 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696542+0000", 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5779999999999999e-06 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907858+0000", 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2113164430000001 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907991+0000", 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000132287 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,1,3,5,7", 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908107+0000", 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011627400000000001 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.909917+0000", 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001810617 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.909964+0000", 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6118999999999997e-05 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.911547+0000", 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001583751 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912015+0000", 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000468159 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912881+0000", 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000865985 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913148+0000", 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026665800000000002 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913220+0000", 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1758e-05 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913392+0000", 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000171767 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244006 9.6 9:6e3a6723:::benchmark_data_smithi081_131274_object116151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:32.379 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 223.69393860700001, 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0195242950000001, 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244006 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947847+0000", 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3649999999999998e-06 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.381 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947854+0000", 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5519999999999999e-06 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947872+0000", 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999000000000002e-05 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947880+0000", 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4200000000000001e-06 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948077+0000", 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019761400000000001 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948118+0000", 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0926000000000001e-05 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959355+0000", 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.011236561 2024-09-21T19:09:32.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959470+0000", 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115578 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959574+0000", 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010335700000000001 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962650+0000", 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030760499999999999 2024-09-21T19:09:32.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962661+0000", 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1022000000000001e-05 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967081+0000", 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044202369999999996 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967163+0000", 2024-09-21T19:09:32.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1880999999999998e-05 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967369+0000", 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020573299999999999 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244020 9.6 9:649892a9:::benchmark_data_smithi081_131274_object116165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:09:32.386 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 223.685424923, 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0117105959999999, 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244020 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:09:32.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956361+0000", 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2320000000000001e-06 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956369+0000", 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7789999999999996e-06 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956389+0000", 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9944000000000001e-05 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956398+0000", 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1600000000000004e-06 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956553+0000", 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015503399999999999 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956577+0000", 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3506000000000002e-05 2024-09-21T19:09:32.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964359+0000", 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0077825030000001 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964473+0000", 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113613 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:09:32.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964573+0000", 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9993000000000003e-05 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967381+0000", 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028085699999999998 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967707+0000", 2024-09-21T19:09:32.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000325555 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967927+0000", 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021985700000000001 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967974+0000", 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.761e-05 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.968069+0000", 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4240000000000006e-05 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:31267 10.9 10:9783b47f:::benchmark_data_smithi081_155596_object31266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.394 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:26.607735+0000", 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 185.03404823100001, 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20489924400000001, 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 31267 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607735+0000", 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607735+0000", 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607737+0000", 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3290000000000001e-06 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607761+0000", 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4369999999999999e-05 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607773+0000", 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1241e-05 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607780+0000", 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7300000000000005e-06 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607945+0000", 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016431700000000001 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.608072+0000", 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012773 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 7", 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.608196+0000", 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012397600000000001 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609756+0000", 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001559653 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812319+0000", 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20256344600000001 2024-09-21T19:09:32.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812379+0000", 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9110000000000002e-05 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812634+0000", 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025534199999999998 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:31274 10.9 10:924e9fbb:::benchmark_data_smithi081_155596_object31273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:26.609958+0000", 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 185.03182493200001, 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20294334, 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 31274 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609958+0000", 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609958+0000", 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609960+0000", 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.34e-06 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609983+0000", 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2238999999999998e-05 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609994+0000", 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0981e-05 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610000+0000", 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.815e-06 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610407+0000", 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040638000000000002 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610538+0000", 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013086900000000001 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 7", 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610658+0000", 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012067599999999999 2024-09-21T19:09:32.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.810099+0000", 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19944103199999999 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812689+0000", 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025899830000000001 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812831+0000", 2024-09-21T19:09:32.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014130699999999999 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812901+0000", 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0717999999999996e-05 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33317 10.5 10:a1fd2c68:::benchmark_data_smithi081_155596_object33316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.080876+0000", 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.56090736600001, 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24413483999999999, 2024-09-21T19:09:32.407 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33317 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080876+0000", 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080876+0000", 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080877+0000", 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5179999999999999e-06 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080901+0000", 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4229000000000001e-05 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080915+0000", 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4166e-05 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080935+0000", 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9604000000000001e-05 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081048+0000", 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112958 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081171+0000", 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123039 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081439+0000", 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000268244 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084329+0000", 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028894839999999999 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084352+0000", 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3459999999999999e-05 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324916+0000", 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24056408900000001 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324971+0000", 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4605999999999999e-05 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325010+0000", 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9443000000000002e-05 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33320 10.5 10:a8d5ea65:::benchmark_data_smithi081_155596_object33319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.081030+0000", 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.56075294999999, 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.244095749, 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.413 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33320 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081030+0000", 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081030+0000", 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081031+0000", 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3239999999999999e-06 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081054+0000", 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2931000000000001e-05 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081064+0000", 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9839999999999996e-06 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081071+0000", 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7710000000000001e-06 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.082782+0000", 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017113300000000001 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.082905+0000", 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012306900000000001 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083057+0000", 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015153500000000001 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084421+0000", 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001363813 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086857+0000", 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002436485 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325056+0000", 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23819896199999999 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325096+0000", 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9486000000000001e-05 2024-09-21T19:09:32.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325126+0000", 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0059000000000001e-05 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33324 10.3 10:c24b9b15:::benchmark_data_smithi081_155596_object33323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.083022+0000", 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.55876119000001, 2024-09-21T19:09:32.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.241109666, 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33324 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083022+0000", 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083022+0000", 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083023+0000", 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6109999999999999e-06 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083045+0000", 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1503000000000001e-05 2024-09-21T19:09:32.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083055+0000", 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0385e-05 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083062+0000", 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7009999999999999e-06 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083114+0000", 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2135999999999997e-05 2024-09-21T19:09:32.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083224+0000", 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010973599999999999 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3", 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083351+0000", 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012762400000000001 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086269+0000", 2024-09-21T19:09:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029178239999999999 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.323973+0000", 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23770374 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324045+0000", 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2119999999999994e-05 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324131+0000", 2024-09-21T19:09:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6285999999999993e-05 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33326 10.3 10:c4df0ca1:::benchmark_data_smithi081_155596_object33325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.083993+0000", 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.55778998700001, 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24031893300000001, 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.425 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33326 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083993+0000", 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083993+0000", 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083994+0000", 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4980000000000001e-06 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084018+0000", 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3116999999999999e-05 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084030+0000", 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2183e-05 2024-09-21T19:09:32.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084038+0000", 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7980000000000006e-06 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084782+0000", 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00074450700000000005 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084920+0000", 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000137849 2024-09-21T19:09:32.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3", 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085101+0000", 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018122099999999999 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086382+0000", 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001280757 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324184+0000", 2024-09-21T19:09:32.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.237802455 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324237+0000", 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2840000000000002e-05 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324312+0000", 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4708000000000004e-05 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33329 10.5 10:a18a2d61:::benchmark_data_smithi081_155596_object33328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.084952+0000", 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.55683128199999, 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24032500900000001, 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33329 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084952+0000", 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084952+0000", 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084953+0000", 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.296e-06 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084979+0000", 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6511000000000001e-05 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084991+0000", 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1089e-05 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084998+0000", 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4120000000000004e-06 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085089+0000", 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1058999999999997e-05 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085203+0000", 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114437 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085332+0000", 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012876900000000001 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086575+0000", 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001242504 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086939+0000", 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036457500000000001 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325201+0000", 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23826132699999999 2024-09-21T19:09:32.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325246+0000", 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5204000000000003e-05 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325277+0000", 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0825999999999999e-05 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33342 10.5 10:a1946825:::benchmark_data_smithi081_155596_object33341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.092684+0000", 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.549099066, 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23319526600000001, 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33342 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092684+0000", 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092684+0000", 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092685+0000", 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1990000000000001e-06 2024-09-21T19:09:32.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092704+0000", 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9352999999999999e-05 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092714+0000", 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7180000000000008e-06 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092719+0000", 2024-09-21T19:09:32.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.955e-06 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092761+0000", 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2271000000000002e-05 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092882+0000", 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012021 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.093025+0000", 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000143141 2024-09-21T19:09:32.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094974+0000", 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019494709999999999 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094998+0000", 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3421000000000002e-05 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325787+0000", 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23078942699999999 2024-09-21T19:09:32.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325841+0000", 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3797e-05 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325879+0000", 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8303000000000002e-05 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.442 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33344 10.5 10:af6f1865:::benchmark_data_smithi081_155596_object33343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.092888+0000", 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 183.54889505700001, 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.233110239, 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33344 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092888+0000", 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092888+0000", 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092889+0000", 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.04e-06 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092914+0000", 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5072999999999999e-05 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092923+0000", 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8769999999999999e-06 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092931+0000", 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.909e-06 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094112+0000", 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011809800000000001 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094190+0000", 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8523999999999996e-05 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094270+0000", 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9764000000000004e-05 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.095095+0000", 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082455899999999999 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.095392+0000", 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029711899999999999 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325932+0000", 2024-09-21T19:09:32.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23053996800000001 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325965+0000", 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3707000000000001e-05 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325998+0000", 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2719000000000003e-05 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189643 10.3 10:cbf6bec4:::benchmark_data_smithi081_155596_object72170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.352828378000002, 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.28335258899999999, 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189643 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288956+0000", 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.674e-06 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288961+0000", 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0039999999999999e-06 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288973+0000", 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2011000000000001e-05 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288980+0000", 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1729999999999998e-06 2024-09-21T19:09:32.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289033+0000", 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2995999999999999e-05 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289155+0000", 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121433 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289279+0000", 2024-09-21T19:09:32.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012419300000000001 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.292626+0000", 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033469849999999998 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.295941+0000", 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033146389999999999 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572053+0000", 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27611242699999999 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572259+0000", 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020604900000000001 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572307+0000", 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8004999999999999e-05 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189647 10.5 10:a120596a:::benchmark_data_smithi081_155596_object72174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.348199014999999, 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29500822199999999, 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189647 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.455 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293586+0000", 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8470000000000001e-06 2024-09-21T19:09:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293592+0000", 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5810000000000002e-06 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293603+0000", 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0728e-05 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293610+0000", 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2250000000000003e-06 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293666+0000", 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5977999999999998e-05 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293807+0000", 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014065599999999999 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293923+0000", 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011639800000000001 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.296808+0000", 2024-09-21T19:09:32.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028842630000000002 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.297628+0000", 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082086500000000005 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588498+0000", 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29086982900000002 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588546+0000", 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7862999999999999e-05 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588592+0000", 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5989000000000003e-05 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.461 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189666 10.5 10:afed0067:::benchmark_data_smithi081_155596_object72193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.328231959, 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27515819499999999, 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.462 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189666 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313553+0000", 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0530000000000001e-06 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313559+0000", 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3489999999999997e-06 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313573+0000", 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.324e-05 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313581+0000", 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5320000000000006e-06 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313644+0000", 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3293000000000005e-05 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.465 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313760+0000", 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011513799999999999 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313872+0000", 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011291 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.316982+0000", 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031098860000000001 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.317682+0000", 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000699711 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588631+0000", 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27094870199999999 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588676+0000", 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5027000000000001e-05 2024-09-21T19:09:32.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588709+0000", 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3353999999999999e-05 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189670 10.3 10:c9bbc12c:::benchmark_data_smithi081_155596_object72197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318154+0000", 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.323628709000001, 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26539114400000002, 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.468 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189670 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318154+0000", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318154+0000", 2024-09-21T19:09:32.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318156+0000", 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318162+0000", 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8619999999999999e-06 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318177+0000", 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5451000000000002e-05 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318185+0000", 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2309999999999999e-06 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318245+0000", 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9849000000000002e-05 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318368+0000", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123416 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318444+0000", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6353000000000006e-05 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321387+0000", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002943153 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321453+0000", 2024-09-21T19:09:32.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5414999999999996e-05 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583333+0000", 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.261880057 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583423+0000", 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0205999999999997e-05 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583545+0000", 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012219100000000001 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189672 10.5 10:a358f405:::benchmark_data_smithi081_155596_object72199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.323170744999999, 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27021128500000002, 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.474 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189672 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318613+0000", 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1829999999999999e-06 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318618+0000", 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3819999999999997e-06 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318630+0000", 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2054e-05 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318637+0000", 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0690000000000004e-06 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318711+0000", 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4373000000000001e-05 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318818+0000", 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010637999999999999 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318920+0000", 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010200300000000001 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.324913+0000", 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005993651 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.324929+0000", 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5424999999999999e-05 2024-09-21T19:09:32.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588745+0000", 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26381601999999998 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588790+0000", 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5621000000000002e-05 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588823+0000", 2024-09-21T19:09:32.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3124000000000003e-05 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189673 10.5 10:a287fbc8:::benchmark_data_smithi081_155596_object72200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.323016213999999, 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27468967399999999, 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.480 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189673 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318768+0000", 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318773+0000", 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6940000000000003e-06 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318783+0000", 2024-09-21T19:09:32.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0121e-05 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318790+0000", 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1210000000000001e-06 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321882+0000", 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030923249999999999 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321979+0000", 2024-09-21T19:09:32.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6842000000000006e-05 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.322109+0000", 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012934000000000001 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.325016+0000", 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002907251 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.325126+0000", 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110315 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593359+0000", 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26823260999999998 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593411+0000", 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2719000000000001e-05 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593456+0000", 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4946e-05 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189677 10.3 10:c4e3b9d2:::benchmark_data_smithi081_155596_object72204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.331628+0000", 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.310154504, 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.25206215199999998, 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:32.486 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189677 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331628+0000", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331628+0000", 2024-09-21T19:09:32.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:32.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.488 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:32.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331630+0000", 2024-09-21T19:09:32.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7850000000000001e-06 2024-09-21T19:09:32.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:32.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331635+0000", 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7609999999999998e-06 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331645+0000", 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0312000000000001e-05 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331651+0000", 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6119999999999998e-06 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331699+0000", 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7834000000000003e-05 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331778+0000", 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9080999999999997e-05 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331852+0000", 2024-09-21T19:09:32.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4070000000000001e-05 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.334467+0000", 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002614894 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.334579+0000", 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112322 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:32.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583607+0000", 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.249028095 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583656+0000", 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8356999999999998e-05 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583691+0000", 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5029e-05 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.493 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.494 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:32.494 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:32.494 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:09:32.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:32.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:32.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:32.669 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:32.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:32.807 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:32.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:32.950 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:32.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:33.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:33.089 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:33.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:33.214 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:32 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T19:09:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:32 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T19:09:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:32 smithi104 ceph-mon[32438]: pgmap v811: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 4870/49896 objects degraded (9.760%); 0 B/s, 101 objects/s recovering 2024-09-21T19:09:33.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:32 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T19:09:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:32 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T19:09:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:32 smithi081 ceph-mon[39192]: pgmap v811: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 4870/49896 objects degraded (9.760%); 0 B/s, 101 objects/s recovering 2024-09-21T19:09:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:32 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T19:09:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:32 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T19:09:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:32 smithi081 ceph-mon[29384]: pgmap v811: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 4870/49896 objects degraded (9.760%); 0 B/s, 101 objects/s recovering 2024-09-21T19:09:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:33.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:33.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:33.474 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:33.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:33.615 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:33.718 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.757 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:33.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:33.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:33.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:33.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:33.993 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:34.039 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:34.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:34.181 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:33 smithi104 ceph-mon[32438]: 3.13 scrub starts 2024-09-21T19:09:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:33 smithi104 ceph-mon[32438]: 3.13 scrub ok 2024-09-21T19:09:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:33 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T19:09:34.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:33 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T19:09:34.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[29384]: 3.13 scrub starts 2024-09-21T19:09:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[29384]: 3.13 scrub ok 2024-09-21T19:09:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T19:09:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T19:09:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[39192]: 3.13 scrub starts 2024-09-21T19:09:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[39192]: 3.13 scrub ok 2024-09-21T19:09:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T19:09:34.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:33 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T19:09:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:34.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:34.459 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:34.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:34.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:34.716 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:34.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:34.852 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:34.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:34.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:35.088 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:35.120 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:35.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:34 smithi104 ceph-mon[32438]: pgmap v812: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 855 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 4572/41634 objects degraded (10.981%); 0 B/s, 103 objects/s recovering 2024-09-21T19:09:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:34 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 4572/41634 objects degraded (10.981%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:35.260 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:34 smithi081 ceph-mon[29384]: pgmap v812: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 855 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 4572/41634 objects degraded (10.981%); 0 B/s, 103 objects/s recovering 2024-09-21T19:09:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:34 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 4572/41634 objects degraded (10.981%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:34 smithi081 ceph-mon[39192]: pgmap v812: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 855 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 4572/41634 objects degraded (10.981%); 0 B/s, 103 objects/s recovering 2024-09-21T19:09:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:34 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 4572/41634 objects degraded (10.981%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:35.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:35.397 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:35.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:35.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:35.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:35.676 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:35.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:35.809 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:35.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:35.947 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:35.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:36.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:36.081 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:36.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:36.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:36.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:36.488 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:36.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:36.629 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:36.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:36.764 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:36.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:36.901 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:36.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:37.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.036 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:37.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:37.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:37.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:37.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:36 smithi081 ceph-mon[29384]: 6.8 scrub starts 2024-09-21T19:09:37.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:36 smithi081 ceph-mon[29384]: 6.8 scrub ok 2024-09-21T19:09:37.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:36 smithi081 ceph-mon[29384]: pgmap v813: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 725 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s; 4272/35355 objects degraded (12.083%); 0 B/s, 141 objects/s recovering 2024-09-21T19:09:37.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:36 smithi081 ceph-mon[39192]: 6.8 scrub starts 2024-09-21T19:09:37.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:36 smithi081 ceph-mon[39192]: 6.8 scrub ok 2024-09-21T19:09:37.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:36 smithi081 ceph-mon[39192]: pgmap v813: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 725 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s; 4272/35355 objects degraded (12.083%); 0 B/s, 141 objects/s recovering 2024-09-21T19:09:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:36 smithi104 ceph-mon[32438]: 6.8 scrub starts 2024-09-21T19:09:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:36 smithi104 ceph-mon[32438]: 6.8 scrub ok 2024-09-21T19:09:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:36 smithi104 ceph-mon[32438]: pgmap v813: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 725 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s; 4272/35355 objects degraded (12.083%); 0 B/s, 141 objects/s recovering 2024-09-21T19:09:37.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:37.304 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:37.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:37.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:37.541 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:37.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:37.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:09:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:09:37.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:37.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:37.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:37.834 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:37.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:37.969 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:37.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:38.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:38.111 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:38.213 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:38.248 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:38.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:37 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T19:09:38.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:37 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T19:09:38.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:38.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T19:09:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T19:09:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T19:09:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T19:09:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:38.349 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:38.391 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:38.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:38.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:38.658 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:38.760 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:38.793 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:38.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:38.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:09:38.931 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:38.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:39.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:39.075 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:39.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:39.211 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:09:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:09:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T19:09:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T19:09:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-mon[32438]: pgmap v814: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 631 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 4272/30873 objects degraded (13.837%); 0 B/s, 87 objects/s recovering 2024-09-21T19:09:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:38 smithi104 ceph-mon[32438]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T19:09:39.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:09:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:09:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T19:09:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T19:09:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[29384]: pgmap v814: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 631 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 4272/30873 objects degraded (13.837%); 0 B/s, 87 objects/s recovering 2024-09-21T19:09:39.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[29384]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T19:09:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:09:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:09:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T19:09:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T19:09:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[39192]: pgmap v814: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 631 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 4272/30873 objects degraded (13.837%); 0 B/s, 87 objects/s recovering 2024-09-21T19:09:39.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:38 smithi081 ceph-mon[39192]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T19:09:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.350 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:39.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:39.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.492 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:39.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:39.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:39.625 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:39.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:39.761 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:39.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:09:39.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:39.899 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:39.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:40.001 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:40.036 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: 10.c scrub starts 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: 10.c scrub ok 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 4272/30873 objects degraded (13.837%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[39192]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: 10.c scrub starts 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: 10.c scrub ok 2024-09-21T19:09:40.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T19:09:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T19:09:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 4272/30873 objects degraded (13.837%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:40.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:39 smithi081 ceph-mon[29384]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T19:09:40.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:40.168 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: 10.c scrub starts 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: 10.c scrub ok 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 4272/30873 objects degraded (13.837%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:39 smithi104 ceph-mon[32438]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T19:09:40.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:40.305 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:40.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:40.456 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:40.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:40.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:40.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.718 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:40.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:40.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:40.840 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:40.941 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:40.977 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:40.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:41.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:41.115 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:41.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:41.248 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:41.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:40 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T19:09:41.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:40 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T19:09:41.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:40 smithi104 ceph-mon[32438]: pgmap v817: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 358 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 3699/17742 objects degraded (20.849%); 0 B/s, 146 objects/s recovering 2024-09-21T19:09:41.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:40 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T19:09:41.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:40 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T19:09:41.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:40 smithi081 ceph-mon[29384]: pgmap v817: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 358 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 3699/17742 objects degraded (20.849%); 0 B/s, 146 objects/s recovering 2024-09-21T19:09:41.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:40 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T19:09:41.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:40 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T19:09:41.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:40 smithi081 ceph-mon[39192]: pgmap v817: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 358 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 3699/17742 objects degraded (20.849%); 0 B/s, 146 objects/s recovering 2024-09-21T19:09:41.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:41.390 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:41.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:09:41.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:41.536 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:41.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.682 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:41.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:41.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:41.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:41.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:41.952 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:41.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:42.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:42.083 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:42.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:42.217 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:42.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.351 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:42.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:42.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:09:42.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:42.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:42.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.658 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:42.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:42.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:42.795 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:42.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:42.924 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:42.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:43.025 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:43.075 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:43.121 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:43.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:09:43.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:09:43.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[29384]: pgmap v818: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 313 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 3699/15618 objects degraded (23.684%); 0 B/s, 109 objects/s recovering 2024-09-21T19:09:43.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[29384]: 3.4 scrub starts 2024-09-21T19:09:43.122 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[29384]: 3.4 scrub ok 2024-09-21T19:09:43.122 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:09:43.122 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:09:43.122 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[39192]: pgmap v818: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 313 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 3699/15618 objects degraded (23.684%); 0 B/s, 109 objects/s recovering 2024-09-21T19:09:43.122 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[39192]: 3.4 scrub starts 2024-09-21T19:09:43.122 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:42 smithi081 ceph-mon[39192]: 3.4 scrub ok 2024-09-21T19:09:43.128 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:09:43.149 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Removed 117470 objects 2024-09-21T19:09:43.149 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Clean up completed and total clean up time :117.927 2024-09-21T19:09:43.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:43.201 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:42 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:09:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:42 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:09:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:42 smithi104 ceph-mon[32438]: pgmap v818: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 313 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 3699/15618 objects degraded (23.684%); 0 B/s, 109 objects/s recovering 2024-09-21T19:09:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:42 smithi104 ceph-mon[32438]: 3.4 scrub starts 2024-09-21T19:09:43.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:42 smithi104 ceph-mon[32438]: 3.4 scrub ok 2024-09-21T19:09:43.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.326 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:43.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:43.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.449 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:43.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:43.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:43.571 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:43.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:43.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:43.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:43.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:43.946 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:43.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:44.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:44.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:44.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:44.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:43 smithi104 ceph-mon[32438]: from='client.36401 v1:172.21.15.81:0/4045817878' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:44.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:44.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:43 smithi081 ceph-mon[29384]: from='client.36401 v1:172.21.15.81:0/4045817878' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:43 smithi081 ceph-mon[39192]: from='client.36401 v1:172.21.15.81:0/4045817878' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:44.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:44.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:44.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:44.474 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":818,"stamp":"2024-09-21T19:09:41.776643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328669747,"num_objects":5206,"num_object_clones":0,"num_object_copies":15618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":2118,"num_read_kb":1870,"num_write":230657,"num_write_kb":7518126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19855,"num_bytes_recovered":1056778488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87294,"ondisk_log_size":87294,"up":474,"acting":472,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5573676,"kb_used_data":1116844,"kb_used_omap":18,"kb_used_meta":4456621,"kb_avail":744387540,"statfs":{"total":767960285184,"available":762252840960,"internally_reserved":0,"allocated":1143648256,"data_stored":1137433823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4563580912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,5,9,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-568328192,"num_objects":-8672,"num_object_clones":0,"num_object_copies":-26016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8672,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8672,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003334"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":645,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.765122+0000","last_change":"2024-09-21T19:06:54.705441+0000","last_active":"2024-09-21T19:09:29.765122+0000","last_peered":"2024-09-21T19:09:29.765122+0000","last_clean":"2024-09-21T19:09:29.765122+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:09:29.765122+0000","last_undegraded":"2024-09-21T19:09:29.765122+0000","last_fullsized":"2024-09-21T19:09:29.765122+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:35.573978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:35.573978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":817,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.740460+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:09:39.740460+0000","last_peered":"2024-09-21T19:09:39.740460+0000","last_clean":"2024-09-21T19:09:39.740460+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:09:39.740460+0000","last_undegraded":"2024-09-21T19:09:39.740460+0000","last_fullsized":"2024-09-21T19:09:39.740460+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":750,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.069611+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:09:37.069611+0000","last_peered":"2024-09-21T19:09:37.069611+0000","last_clean":"2024-09-21T19:09:37.069611+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:09:37.069611+0000","last_undegraded":"2024-09-21T19:09:37.069611+0000","last_fullsized":"2024-09-21T19:09:37.069611+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.809763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":734,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.740889+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:09:39.740889+0000","last_peered":"2024-09-21T19:09:39.740889+0000","last_clean":"2024-09-21T19:09:39.740889+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:09:39.740889+0000","last_undegraded":"2024-09-21T19:09:39.740889+0000","last_fullsized":"2024-09-21T19:09:39.740889+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":741,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.902017+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:09:37.902017+0000","last_peered":"2024-09-21T19:09:37.902017+0000","last_clean":"2024-09-21T19:09:37.902017+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:09:37.902017+0000","last_undegraded":"2024-09-21T19:09:37.902017+0000","last_fullsized":"2024-09-21T19:09:37.902017+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":769,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905595+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:09:37.905595+0000","last_peered":"2024-09-21T19:09:37.905595+0000","last_clean":"2024-09-21T19:09:37.905595+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:09:37.905595+0000","last_undegraded":"2024-09-21T19:09:37.905595+0000","last_fullsized":"2024-09-21T19:09:37.905595+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":865,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.902835+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:09:37.902835+0000","last_peered":"2024-09-21T19:09:37.902835+0000","last_clean":"2024-09-21T19:09:37.902835+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:09:37.902835+0000","last_undegraded":"2024-09-21T19:09:37.902835+0000","last_fullsized":"2024-09-21T19:09:37.902835+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":754,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.740366+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:09:39.740366+0000","last_peered":"2024-09-21T19:09:39.740366+0000","last_clean":"2024-09-21T19:09:39.740366+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:09:39.740366+0000","last_undegraded":"2024-09-21T19:09:39.740366+0000","last_fullsized":"2024-09-21T19:09:39.740366+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":741,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.743358+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:09:39.743358+0000","last_peered":"2024-09-21T19:09:39.743358+0000","last_clean":"2024-09-21T19:09:39.743358+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:09:39.743358+0000","last_undegraded":"2024-09-21T19:09:39.743358+0000","last_fullsized":"2024-09-21T19:09:39.743358+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":798,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.741332+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:09:39.741332+0000","last_peered":"2024-09-21T19:09:39.741332+0000","last_clean":"2024-09-21T19:09:39.741332+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:09:39.741332+0000","last_undegraded":"2024-09-21T19:09:39.741332+0000","last_fullsized":"2024-09-21T19:09:39.741332+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":693,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.908183+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:09:37.908183+0000","last_peered":"2024-09-21T19:09:37.908183+0000","last_clean":"2024-09-21T19:09:37.908183+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:09:37.908183+0000","last_undegraded":"2024-09-21T19:09:37.908183+0000","last_fullsized":"2024-09-21T19:09:37.908183+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":611,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.748906+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:09:29.748906+0000","last_peered":"2024-09-21T19:09:29.748906+0000","last_clean":"2024-09-21T19:09:29.748906+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:09:29.748906+0000","last_undegraded":"2024-09-21T19:09:29.748906+0000","last_fullsized":"2024-09-21T19:09:29.748906+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":294,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.745777+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:09:39.745777+0000","last_peered":"2024-09-21T19:09:39.745777+0000","last_clean":"2024-09-21T19:09:39.745777+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:09:39.745777+0000","last_undegraded":"2024-09-21T19:09:39.745777+0000","last_fullsized":"2024-09-21T19:09:39.745777+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":744,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.740667+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:09:39.740667+0000","last_peered":"2024-09-21T19:09:39.740667+0000","last_clean":"2024-09-21T19:09:39.740667+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:09:39.740667+0000","last_undegraded":"2024-09-21T19:09:39.740667+0000","last_fullsized":"2024-09-21T19:09:39.740667+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":730,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.744879+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:09:29.744879+0000","last_peered":"2024-09-21T19:09:29.744879+0000","last_clean":"2024-09-21T19:09:29.744879+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:09:29.744879+0000","last_undegraded":"2024-09-21T19:09:29.744879+0000","last_fullsized":"2024-09-21T19:09:29.744879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":298,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.748276+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:09:29.748276+0000","last_peered":"2024-09-21T19:09:29.748276+0000","last_clean":"2024-09-21T19:09:29.748276+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:09:29.748276+0000","last_undegraded":"2024-09-21T19:09:29.748276+0000","last_fullsized":"2024-09-21T19:09:29.748276+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":763,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.749673+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:09:39.749673+0000","last_peered":"2024-09-21T19:09:39.749673+0000","last_clean":"2024-09-21T19:09:39.749673+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:09:39.749673+0000","last_undegraded":"2024-09-21T19:09:39.749673+0000","last_fullsized":"2024-09-21T19:09:39.749673+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":627,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.755896+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:09:29.755896+0000","last_peered":"2024-09-21T19:09:29.755896+0000","last_clean":"2024-09-21T19:09:29.755896+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:09:29.755896+0000","last_undegraded":"2024-09-21T19:09:29.755896+0000","last_fullsized":"2024-09-21T19:09:29.755896+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":203,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909888+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:09:37.909888+0000","last_peered":"2024-09-21T19:09:37.909888+0000","last_clean":"2024-09-21T19:09:37.909888+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:09:37.909888+0000","last_undegraded":"2024-09-21T19:09:37.909888+0000","last_fullsized":"2024-09-21T19:09:37.909888+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:24.726173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":753,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757640+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:09:29.757640+0000","last_peered":"2024-09-21T19:09:29.757640+0000","last_clean":"2024-09-21T19:09:29.757640+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:09:29.757640+0000","last_undegraded":"2024-09-21T19:09:29.757640+0000","last_fullsized":"2024-09-21T19:09:29.757640+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:07.738704+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":737,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905928+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:09:37.905928+0000","last_peered":"2024-09-21T19:09:37.905928+0000","last_clean":"2024-09-21T19:09:37.905928+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:09:37.905928+0000","last_undegraded":"2024-09-21T19:09:37.905928+0000","last_fullsized":"2024-09-21T19:09:37.905928+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":765,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.741465+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:09:39.741465+0000","last_peered":"2024-09-21T19:09:39.741465+0000","last_clean":"2024-09-21T19:09:39.741465+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:09:39.741465+0000","last_undegraded":"2024-09-21T19:09:39.741465+0000","last_fullsized":"2024-09-21T19:09:39.741465+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":808,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905206+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:09:37.905206+0000","last_peered":"2024-09-21T19:09:37.905206+0000","last_clean":"2024-09-21T19:09:37.905206+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:09:37.905206+0000","last_undegraded":"2024-09-21T19:09:37.905206+0000","last_fullsized":"2024-09-21T19:09:37.905206+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":749,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.745567+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:09:39.745567+0000","last_peered":"2024-09-21T19:09:39.745567+0000","last_clean":"2024-09-21T19:09:39.745567+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:09:39.745567+0000","last_undegraded":"2024-09-21T19:09:39.745567+0000","last_fullsized":"2024-09-21T19:09:39.745567+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":768,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745624+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:09:29.745624+0000","last_peered":"2024-09-21T19:09:29.745624+0000","last_clean":"2024-09-21T19:09:29.745624+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:09:29.745624+0000","last_undegraded":"2024-09-21T19:09:29.745624+0000","last_fullsized":"2024-09-21T19:09:29.745624+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.611692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":767,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756736+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:09:29.756736+0000","last_peered":"2024-09-21T19:09:29.756736+0000","last_clean":"2024-09-21T19:09:29.756736+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:09:29.756736+0000","last_undegraded":"2024-09-21T19:09:29.756736+0000","last_fullsized":"2024-09-21T19:09:29.756736+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":655,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756486+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:09:29.756486+0000","last_peered":"2024-09-21T19:09:29.756486+0000","last_clean":"2024-09-21T19:09:29.756486+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:09:29.756486+0000","last_undegraded":"2024-09-21T19:09:29.756486+0000","last_fullsized":"2024-09-21T19:09:29.756486+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":831,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909823+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:09:37.909823+0000","last_peered":"2024-09-21T19:09:37.909823+0000","last_clean":"2024-09-21T19:09:37.909823+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:09:37.909823+0000","last_undegraded":"2024-09-21T19:09:37.909823+0000","last_fullsized":"2024-09-21T19:09:37.909823+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"405'14782","reported_seq":14944,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:41.432407+0000","last_change":"2024-09-21T19:06:09.007701+0000","last_active":"2024-09-21T19:09:41.432407+0000","last_peered":"2024-09-21T19:09:41.432407+0000","last_clean":"2024-09-21T19:09:41.432407+0000","last_became_active":"2024-09-21T19:06:09.007396+0000","last_became_peered":"2024-09-21T19:06:09.007396+0000","last_unstale":"2024-09-21T19:09:41.432407+0000","last_undegraded":"2024-09-21T19:09:41.432407+0000","last_fullsized":"2024-09-21T19:09:41.432407+0000","mapping_epoch":330,"log_start":"383'10300","ondisk_log_start":"383'10300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14782,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":198,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909708+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:09:37.909708+0000","last_peered":"2024-09-21T19:09:37.909708+0000","last_clean":"2024-09-21T19:09:37.909708+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:09:37.909708+0000","last_undegraded":"2024-09-21T19:09:37.909708+0000","last_fullsized":"2024-09-21T19:09:37.909708+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:13.700777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":795,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757911+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:09:29.757911+0000","last_peered":"2024-09-21T19:09:29.757911+0000","last_clean":"2024-09-21T19:09:29.757911+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:09:29.757911+0000","last_undegraded":"2024-09-21T19:09:29.757911+0000","last_fullsized":"2024-09-21T19:09:29.757911+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:35.238469+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":735,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903705+0000","last_change":"2024-09-21T19:00:01.925227+0000","last_active":"2024-09-21T19:09:37.903705+0000","last_peered":"2024-09-21T19:09:37.903705+0000","last_clean":"2024-09-21T19:09:37.903705+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:09:37.903705+0000","last_undegraded":"2024-09-21T19:09:37.903705+0000","last_fullsized":"2024-09-21T19:09:37.903705+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:00:01.925163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:00:01.925163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":986,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.749753+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:09:39.749753+0000","last_peered":"2024-09-21T19:09:39.749753+0000","last_clean":"2024-09-21T19:09:39.749753+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:09:39.749753+0000","last_undegraded":"2024-09-21T19:09:39.749753+0000","last_fullsized":"2024-09-21T19:09:39.749753+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":422,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757561+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:09:29.757561+0000","last_peered":"2024-09-21T19:09:29.757561+0000","last_clean":"2024-09-21T19:09:29.757561+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:09:29.757561+0000","last_undegraded":"2024-09-21T19:09:29.757561+0000","last_fullsized":"2024-09-21T19:09:29.757561+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:22.031875+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"405'14321","reported_seq":14483,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:41.438825+0000","last_change":"2024-09-21T19:06:09.005253+0000","last_active":"2024-09-21T19:09:41.438825+0000","last_peered":"2024-09-21T19:09:41.438825+0000","last_clean":"2024-09-21T19:09:41.438825+0000","last_became_active":"2024-09-21T19:06:09.004912+0000","last_became_peered":"2024-09-21T19:06:09.004912+0000","last_unstale":"2024-09-21T19:09:41.438825+0000","last_undegraded":"2024-09-21T19:09:41.438825+0000","last_fullsized":"2024-09-21T19:09:41.438825+0000","mapping_epoch":330,"log_start":"383'9800","ondisk_log_start":"383'9800","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14321,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":526,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.743275+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:09:39.743275+0000","last_peered":"2024-09-21T19:09:39.743275+0000","last_clean":"2024-09-21T19:09:39.743275+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:09:39.743275+0000","last_undegraded":"2024-09-21T19:09:39.743275+0000","last_fullsized":"2024-09-21T19:09:39.743275+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":832,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903514+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:09:37.903514+0000","last_peered":"2024-09-21T19:09:37.903514+0000","last_clean":"2024-09-21T19:09:37.903514+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:09:37.903514+0000","last_undegraded":"2024-09-21T19:09:37.903514+0000","last_fullsized":"2024-09-21T19:09:37.903514+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":757,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.906358+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:09:37.906358+0000","last_peered":"2024-09-21T19:09:37.906358+0000","last_clean":"2024-09-21T19:09:37.906358+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:09:37.906358+0000","last_undegraded":"2024-09-21T19:09:37.906358+0000","last_fullsized":"2024-09-21T19:09:37.906358+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":736,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.742557+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:09:39.742557+0000","last_peered":"2024-09-21T19:09:39.742557+0000","last_clean":"2024-09-21T19:09:39.742557+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:09:39.742557+0000","last_undegraded":"2024-09-21T19:09:39.742557+0000","last_fullsized":"2024-09-21T19:09:39.742557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":753,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.741201+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:09:39.741201+0000","last_peered":"2024-09-21T19:09:39.741201+0000","last_clean":"2024-09-21T19:09:39.741201+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:09:39.741201+0000","last_undegraded":"2024-09-21T19:09:39.741201+0000","last_fullsized":"2024-09-21T19:09:39.741201+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"404'14441","reported_seq":14601,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.349783+0000","last_change":"2024-09-21T19:06:09.005368+0000","last_active":"2024-09-21T19:09:39.349783+0000","last_peered":"2024-09-21T19:09:39.349783+0000","last_clean":"2024-09-21T19:09:39.349783+0000","last_became_active":"2024-09-21T19:06:09.005049+0000","last_became_peered":"2024-09-21T19:06:09.005049+0000","last_unstale":"2024-09-21T19:09:39.349783+0000","last_undegraded":"2024-09-21T19:09:39.349783+0000","last_fullsized":"2024-09-21T19:09:39.349783+0000","mapping_epoch":330,"log_start":"377'9100","ondisk_log_start":"377'9100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14441,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":515,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.906685+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:09:37.906685+0000","last_peered":"2024-09-21T19:09:37.906685+0000","last_clean":"2024-09-21T19:09:37.906685+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:09:37.906685+0000","last_undegraded":"2024-09-21T19:09:37.906685+0000","last_fullsized":"2024-09-21T19:09:37.906685+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":804,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.742687+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:09:39.742687+0000","last_peered":"2024-09-21T19:09:39.742687+0000","last_clean":"2024-09-21T19:09:39.742687+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:09:39.742687+0000","last_undegraded":"2024-09-21T19:09:39.742687+0000","last_fullsized":"2024-09-21T19:09:39.742687+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":624,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756785+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:09:29.756785+0000","last_peered":"2024-09-21T19:09:29.756785+0000","last_clean":"2024-09-21T19:09:29.756785+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:09:29.756785+0000","last_undegraded":"2024-09-21T19:09:29.756785+0000","last_fullsized":"2024-09-21T19:09:29.756785+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":927,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903767+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:09:37.903767+0000","last_peered":"2024-09-21T19:09:37.903767+0000","last_clean":"2024-09-21T19:09:37.903767+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:09:37.903767+0000","last_undegraded":"2024-09-21T19:09:37.903767+0000","last_fullsized":"2024-09-21T19:09:37.903767+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":752,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745119+0000","last_change":"2024-09-21T19:06:45.353676+0000","last_active":"2024-09-21T19:09:29.745119+0000","last_peered":"2024-09-21T19:09:29.745119+0000","last_clean":"2024-09-21T19:09:29.745119+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:09:29.745119+0000","last_undegraded":"2024-09-21T19:09:29.745119+0000","last_fullsized":"2024-09-21T19:09:29.745119+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:45.353608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:45.353608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"403'14458","reported_seq":14616,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:36.861711+0000","last_change":"2024-09-21T19:06:09.007378+0000","last_active":"2024-09-21T19:09:36.861711+0000","last_peered":"2024-09-21T19:09:36.861711+0000","last_clean":"2024-09-21T19:09:36.861711+0000","last_became_active":"2024-09-21T19:06:09.007020+0000","last_became_peered":"2024-09-21T19:06:09.007020+0000","last_unstale":"2024-09-21T19:09:36.861711+0000","last_undegraded":"2024-09-21T19:09:36.861711+0000","last_fullsized":"2024-09-21T19:09:36.861711+0000","mapping_epoch":330,"log_start":"379'9400","ondisk_log_start":"379'9400","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":480,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14458,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":539,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910067+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:09:37.910067+0000","last_peered":"2024-09-21T19:09:37.910067+0000","last_clean":"2024-09-21T19:09:37.910067+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:09:37.910067+0000","last_undegraded":"2024-09-21T19:09:37.910067+0000","last_fullsized":"2024-09-21T19:09:37.910067+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":777,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.748870+0000","last_change":"2024-09-21T19:06:54.705154+0000","last_active":"2024-09-21T19:09:29.748870+0000","last_peered":"2024-09-21T19:09:29.748870+0000","last_clean":"2024-09-21T19:09:29.748870+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:09:29.748870+0000","last_undegraded":"2024-09-21T19:09:29.748870+0000","last_fullsized":"2024-09-21T19:09:29.748870+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:59.218403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.218403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:33.202656+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":299,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905207+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:09:37.905207+0000","last_peered":"2024-09-21T19:09:37.905207+0000","last_clean":"2024-09-21T19:09:37.905207+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:09:37.905207+0000","last_undegraded":"2024-09-21T19:09:37.905207+0000","last_fullsized":"2024-09-21T19:09:37.905207+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":808,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:32.651007+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:09:32.651007+0000","last_peered":"2024-09-21T19:09:32.651007+0000","last_clean":"2024-09-21T19:09:32.651007+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:09:32.651007+0000","last_undegraded":"2024-09-21T19:09:32.651007+0000","last_fullsized":"2024-09-21T19:09:32.651007+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":626,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.749047+0000","last_change":"2024-09-21T19:05:42.901144+0000","last_active":"2024-09-21T19:09:29.749047+0000","last_peered":"2024-09-21T19:09:29.749047+0000","last_clean":"2024-09-21T19:09:29.749047+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:09:29.749047+0000","last_undegraded":"2024-09-21T19:09:29.749047+0000","last_fullsized":"2024-09-21T19:09:29.749047+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:18.444620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:18.444620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"404'14408","reported_seq":14569,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.359654+0000","last_change":"2024-09-21T19:06:09.006383+0000","last_active":"2024-09-21T19:09:39.359654+0000","last_peered":"2024-09-21T19:09:39.359654+0000","last_clean":"2024-09-21T19:09:39.359654+0000","last_became_active":"2024-09-21T19:06:09.006115+0000","last_became_peered":"2024-09-21T19:06:09.006115+0000","last_unstale":"2024-09-21T19:09:39.359654+0000","last_undegraded":"2024-09-21T19:09:39.359654+0000","last_fullsized":"2024-09-21T19:09:39.359654+0000","mapping_epoch":330,"log_start":"378'9100","ondisk_log_start":"378'9100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498136,"num_objects":268,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14408,"num_write_kb":469569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":517,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910207+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:09:37.910207+0000","last_peered":"2024-09-21T19:09:37.910207+0000","last_clean":"2024-09-21T19:09:37.910207+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:09:37.910207+0000","last_undegraded":"2024-09-21T19:09:37.910207+0000","last_fullsized":"2024-09-21T19:09:37.910207+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":672,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.755989+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:09:29.755989+0000","last_peered":"2024-09-21T19:09:29.755989+0000","last_clean":"2024-09-21T19:09:29.755989+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:09:29.755989+0000","last_undegraded":"2024-09-21T19:09:29.755989+0000","last_fullsized":"2024-09-21T19:09:29.755989+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":765,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757217+0000","last_change":"2024-09-21T19:06:54.701552+0000","last_active":"2024-09-21T19:09:29.757217+0000","last_peered":"2024-09-21T19:09:29.757217+0000","last_clean":"2024-09-21T19:09:29.757217+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:09:29.757217+0000","last_undegraded":"2024-09-21T19:09:29.757217+0000","last_fullsized":"2024-09-21T19:09:29.757217+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:28.333589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.333589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:34.228030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":652,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910644+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:09:37.910644+0000","last_peered":"2024-09-21T19:09:37.910644+0000","last_clean":"2024-09-21T19:09:37.910644+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:09:37.910644+0000","last_undegraded":"2024-09-21T19:09:37.910644+0000","last_fullsized":"2024-09-21T19:09:37.910644+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":753,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756545+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:09:29.756545+0000","last_peered":"2024-09-21T19:09:29.756545+0000","last_clean":"2024-09-21T19:09:29.756545+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:09:29.756545+0000","last_undegraded":"2024-09-21T19:09:29.756545+0000","last_fullsized":"2024-09-21T19:09:29.756545+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.253514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"404'14586","reported_seq":24112,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.364412+0000","last_change":"2024-09-21T19:08:19.768786+0000","last_active":"2024-09-21T19:09:39.364412+0000","last_peered":"2024-09-21T19:09:39.364412+0000","last_clean":"2024-09-21T19:09:39.364412+0000","last_became_active":"2024-09-21T19:08:19.754770+0000","last_became_peered":"2024-09-21T19:08:19.754770+0000","last_unstale":"2024-09-21T19:09:39.364412+0000","last_undegraded":"2024-09-21T19:09:39.364412+0000","last_fullsized":"2024-09-21T19:09:39.364412+0000","mapping_epoch":380,"log_start":"379'9300","ondisk_log_start":"379'9300","created":330,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":264,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14586,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":491,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757226+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:09:29.757226+0000","last_peered":"2024-09-21T19:09:29.757226+0000","last_clean":"2024-09-21T19:09:29.757226+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:09:29.757226+0000","last_undegraded":"2024-09-21T19:09:29.757226+0000","last_fullsized":"2024-09-21T19:09:29.757226+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1052,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:32.650925+0000","last_change":"2024-09-21T19:05:42.931008+0000","last_active":"2024-09-21T19:09:32.650925+0000","last_peered":"2024-09-21T19:09:32.650925+0000","last_clean":"2024-09-21T19:09:32.650925+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:09:32.650925+0000","last_undegraded":"2024-09-21T19:09:32.650925+0000","last_fullsized":"2024-09-21T19:09:32.650925+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T19:04:26.993716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.993716+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:45.305870+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":777,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.741004+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:09:39.741004+0000","last_peered":"2024-09-21T19:09:39.741004+0000","last_clean":"2024-09-21T19:09:39.741004+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:09:39.741004+0000","last_undegraded":"2024-09-21T19:09:39.741004+0000","last_fullsized":"2024-09-21T19:09:39.741004+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":321,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.750151+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:09:29.750151+0000","last_peered":"2024-09-21T19:09:29.750151+0000","last_clean":"2024-09-21T19:09:29.750151+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:09:29.750151+0000","last_undegraded":"2024-09-21T19:09:29.750151+0000","last_fullsized":"2024-09-21T19:09:29.750151+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":612,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.755843+0000","last_change":"2024-09-21T19:02:38.618578+0000","last_active":"2024-09-21T19:09:29.755843+0000","last_peered":"2024-09-21T19:09:29.755843+0000","last_clean":"2024-09-21T19:09:29.755843+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:09:29.755843+0000","last_undegraded":"2024-09-21T19:09:29.755843+0000","last_fullsized":"2024-09-21T19:09:29.755843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:38.618523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:38.618523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":723,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905346+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:09:37.905346+0000","last_peered":"2024-09-21T19:09:37.905346+0000","last_clean":"2024-09-21T19:09:37.905346+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:09:37.905346+0000","last_undegraded":"2024-09-21T19:09:37.905346+0000","last_fullsized":"2024-09-21T19:09:37.905346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"403'14243","reported_seq":23686,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.912538+0000","last_change":"2024-09-21T19:08:54.754076+0000","last_active":"2024-09-21T19:09:37.912538+0000","last_peered":"2024-09-21T19:09:37.912538+0000","last_clean":"2024-09-21T19:09:37.912538+0000","last_became_active":"2024-09-21T19:08:54.722178+0000","last_became_peered":"2024-09-21T19:08:54.722178+0000","last_unstale":"2024-09-21T19:09:37.912538+0000","last_undegraded":"2024-09-21T19:09:37.912538+0000","last_fullsized":"2024-09-21T19:09:37.912538+0000","mapping_epoch":389,"log_start":"368'7800","ondisk_log_start":"368'7800","created":330,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.699127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14243,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":303890432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":826,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.744696+0000","last_change":"2024-09-21T19:08:53.739015+0000","last_active":"2024-09-21T19:09:29.744696+0000","last_peered":"2024-09-21T19:09:29.744696+0000","last_clean":"2024-09-21T19:09:29.744696+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:09:29.744696+0000","last_undegraded":"2024-09-21T19:09:29.744696+0000","last_fullsized":"2024-09-21T19:09:29.744696+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:08:53.738929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.738929+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:55.112735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":239,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757670+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:09:29.757670+0000","last_peered":"2024-09-21T19:09:29.757670+0000","last_clean":"2024-09-21T19:09:29.757670+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:09:29.757670+0000","last_undegraded":"2024-09-21T19:09:29.757670+0000","last_fullsized":"2024-09-21T19:09:29.757670+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:44.105971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":796,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.747987+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:09:29.747987+0000","last_peered":"2024-09-21T19:09:29.747987+0000","last_clean":"2024-09-21T19:09:29.747987+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:09:29.747987+0000","last_undegraded":"2024-09-21T19:09:29.747987+0000","last_fullsized":"2024-09-21T19:09:29.747987+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":191,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.912393+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:09:37.912393+0000","last_peered":"2024-09-21T19:09:37.912393+0000","last_clean":"2024-09-21T19:09:37.912393+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:09:37.912393+0000","last_undegraded":"2024-09-21T19:09:37.912393+0000","last_fullsized":"2024-09-21T19:09:37.912393+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.753652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":198,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910096+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:09:37.910096+0000","last_peered":"2024-09-21T19:09:37.910096+0000","last_clean":"2024-09-21T19:09:37.910096+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:09:37.910096+0000","last_undegraded":"2024-09-21T19:09:37.910096+0000","last_fullsized":"2024-09-21T19:09:37.910096+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.374847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":742,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745842+0000","last_change":"2024-09-21T19:06:09.161799+0000","last_active":"2024-09-21T19:09:29.745842+0000","last_peered":"2024-09-21T19:09:29.745842+0000","last_clean":"2024-09-21T19:09:29.745842+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:09:29.745842+0000","last_undegraded":"2024-09-21T19:09:29.745842+0000","last_fullsized":"2024-09-21T19:09:29.745842+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:06:09.161753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.161753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"403'14357","reported_seq":14515,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.790709+0000","last_change":"2024-09-21T19:06:09.006583+0000","last_active":"2024-09-21T19:09:37.790709+0000","last_peered":"2024-09-21T19:09:37.790709+0000","last_clean":"2024-09-21T19:09:37.790709+0000","last_became_active":"2024-09-21T19:06:09.006308+0000","last_became_peered":"2024-09-21T19:06:09.006308+0000","last_unstale":"2024-09-21T19:09:37.790709+0000","last_undegraded":"2024-09-21T19:09:37.790709+0000","last_fullsized":"2024-09-21T19:09:37.790709+0000","mapping_epoch":330,"log_start":"379'9300","ondisk_log_start":"379'9300","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24838144,"num_objects":379,"num_object_clones":0,"num_object_copies":1137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":532,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.751176+0000","last_change":"2024-09-21T19:09:08.156345+0000","last_active":"2024-09-21T19:09:29.751176+0000","last_peered":"2024-09-21T19:09:29.751176+0000","last_clean":"2024-09-21T19:09:29.751176+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:09:29.751176+0000","last_undegraded":"2024-09-21T19:09:29.751176+0000","last_fullsized":"2024-09-21T19:09:29.751176+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:08.156258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:08.156258+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.996519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":734,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756579+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:09:29.756579+0000","last_peered":"2024-09-21T19:09:29.756579+0000","last_clean":"2024-09-21T19:09:29.756579+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:09:29.756579+0000","last_undegraded":"2024-09-21T19:09:29.756579+0000","last_fullsized":"2024-09-21T19:09:29.756579+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":620,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757610+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:09:29.757610+0000","last_peered":"2024-09-21T19:09:29.757610+0000","last_clean":"2024-09-21T19:09:29.757610+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:09:29.757610+0000","last_undegraded":"2024-09-21T19:09:29.757610+0000","last_fullsized":"2024-09-21T19:09:29.757610+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1748,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:35.797828+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:09:35.797828+0000","last_peered":"2024-09-21T19:09:35.797828+0000","last_clean":"2024-09-21T19:09:35.797828+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:09:35.797828+0000","last_undegraded":"2024-09-21T19:09:35.797828+0000","last_fullsized":"2024-09-21T19:09:35.797828+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.664705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":978,"num_read_kb":973,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":694,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.914920+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:09:37.914920+0000","last_peered":"2024-09-21T19:09:37.914920+0000","last_clean":"2024-09-21T19:09:37.914920+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:09:37.914920+0000","last_undegraded":"2024-09-21T19:09:37.914920+0000","last_fullsized":"2024-09-21T19:09:37.914920+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":747,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756840+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:09:29.756840+0000","last_peered":"2024-09-21T19:09:29.756840+0000","last_clean":"2024-09-21T19:09:29.756840+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:09:29.756840+0000","last_undegraded":"2024-09-21T19:09:29.756840+0000","last_fullsized":"2024-09-21T19:09:29.756840+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:30.010503+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"404'14377","reported_seq":14537,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.350806+0000","last_change":"2024-09-21T19:06:09.006804+0000","last_active":"2024-09-21T19:09:39.350806+0000","last_peered":"2024-09-21T19:09:39.350806+0000","last_clean":"2024-09-21T19:09:39.350806+0000","last_became_active":"2024-09-21T19:06:09.006268+0000","last_became_peered":"2024-09-21T19:06:09.006268+0000","last_unstale":"2024-09-21T19:09:39.350806+0000","last_undegraded":"2024-09-21T19:09:39.350806+0000","last_fullsized":"2024-09-21T19:09:39.350806+0000","mapping_epoch":330,"log_start":"378'9100","ondisk_log_start":"378'9100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14377,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":922,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910142+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:09:37.910142+0000","last_peered":"2024-09-21T19:09:37.910142+0000","last_clean":"2024-09-21T19:09:37.910142+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:09:37.910142+0000","last_undegraded":"2024-09-21T19:09:37.910142+0000","last_fullsized":"2024-09-21T19:09:37.910142+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":757,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903261+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:09:37.903261+0000","last_peered":"2024-09-21T19:09:37.903261+0000","last_clean":"2024-09-21T19:09:37.903261+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:09:37.903261+0000","last_undegraded":"2024-09-21T19:09:37.903261+0000","last_fullsized":"2024-09-21T19:09:37.903261+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":832,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745952+0000","last_change":"2024-09-21T19:06:41.271146+0000","last_active":"2024-09-21T19:09:29.745952+0000","last_peered":"2024-09-21T19:09:29.745952+0000","last_clean":"2024-09-21T19:09:29.745952+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:09:29.745952+0000","last_undegraded":"2024-09-21T19:09:29.745952+0000","last_fullsized":"2024-09-21T19:09:29.745952+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:41.271082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:41.271082+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":756,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745717+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:09:29.745717+0000","last_peered":"2024-09-21T19:09:29.745717+0000","last_clean":"2024-09-21T19:09:29.745717+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:09:29.745717+0000","last_undegraded":"2024-09-21T19:09:29.745717+0000","last_fullsized":"2024-09-21T19:09:29.745717+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:15.623405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":284,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905850+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:09:37.905850+0000","last_peered":"2024-09-21T19:09:37.905850+0000","last_clean":"2024-09-21T19:09:37.905850+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:09:37.905850+0000","last_undegraded":"2024-09-21T19:09:37.905850+0000","last_fullsized":"2024-09-21T19:09:37.905850+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"404'14179","reported_seq":14339,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.334502+0000","last_change":"2024-09-21T19:06:09.002428+0000","last_active":"2024-09-21T19:09:39.334502+0000","last_peered":"2024-09-21T19:09:39.334502+0000","last_clean":"2024-09-21T19:09:39.334502+0000","last_became_active":"2024-09-21T19:06:09.002111+0000","last_became_peered":"2024-09-21T19:06:09.002111+0000","last_unstale":"2024-09-21T19:09:39.334502+0000","last_undegraded":"2024-09-21T19:09:39.334502+0000","last_fullsized":"2024-09-21T19:09:39.334502+0000","mapping_epoch":330,"log_start":"377'8900","ondisk_log_start":"377'8900","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14179,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":798,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.749836+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:09:39.749836+0000","last_peered":"2024-09-21T19:09:39.749836+0000","last_clean":"2024-09-21T19:09:39.749836+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:09:39.749836+0000","last_undegraded":"2024-09-21T19:09:39.749836+0000","last_fullsized":"2024-09-21T19:09:39.749836+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":766,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.742963+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:09:39.742963+0000","last_peered":"2024-09-21T19:09:39.742963+0000","last_clean":"2024-09-21T19:09:39.742963+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:09:39.742963+0000","last_undegraded":"2024-09-21T19:09:39.742963+0000","last_fullsized":"2024-09-21T19:09:39.742963+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":750,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.742105+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:09:39.742105+0000","last_peered":"2024-09-21T19:09:39.742105+0000","last_clean":"2024-09-21T19:09:39.742105+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:09:39.742105+0000","last_undegraded":"2024-09-21T19:09:39.742105+0000","last_fullsized":"2024-09-21T19:09:39.742105+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":729,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.906770+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:09:37.906770+0000","last_peered":"2024-09-21T19:09:37.906770+0000","last_clean":"2024-09-21T19:09:37.906770+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:09:37.906770+0000","last_undegraded":"2024-09-21T19:09:37.906770+0000","last_fullsized":"2024-09-21T19:09:37.906770+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"403'14272","reported_seq":23766,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910280+0000","last_change":"2024-09-21T19:08:55.780967+0000","last_active":"2024-09-21T19:09:37.910280+0000","last_peered":"2024-09-21T19:09:37.910280+0000","last_clean":"2024-09-21T19:09:37.910280+0000","last_became_active":"2024-09-21T19:08:55.751488+0000","last_became_peered":"2024-09-21T19:08:55.751488+0000","last_unstale":"2024-09-21T19:09:37.910280+0000","last_undegraded":"2024-09-21T19:09:37.910280+0000","last_fullsized":"2024-09-21T19:09:37.910280+0000","mapping_epoch":390,"log_start":"367'7800","ondisk_log_start":"367'7800","created":330,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.721260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27656192,"num_objects":422,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14272,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4662,"num_bytes_recovered":305528832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":194,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910105+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:09:37.910105+0000","last_peered":"2024-09-21T19:09:37.910105+0000","last_clean":"2024-09-21T19:09:37.910105+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:09:37.910105+0000","last_undegraded":"2024-09-21T19:09:37.910105+0000","last_fullsized":"2024-09-21T19:09:37.910105+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.739272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":847,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.743142+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:09:39.743142+0000","last_peered":"2024-09-21T19:09:39.743142+0000","last_clean":"2024-09-21T19:09:39.743142+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:09:39.743142+0000","last_undegraded":"2024-09-21T19:09:39.743142+0000","last_fullsized":"2024-09-21T19:09:39.743142+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":663,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.911157+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:09:37.911157+0000","last_peered":"2024-09-21T19:09:37.911157+0000","last_clean":"2024-09-21T19:09:37.911157+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:09:37.911157+0000","last_undegraded":"2024-09-21T19:09:37.911157+0000","last_fullsized":"2024-09-21T19:09:37.911157+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":187,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909645+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:09:37.909645+0000","last_peered":"2024-09-21T19:09:37.909645+0000","last_clean":"2024-09-21T19:09:37.909645+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:09:37.909645+0000","last_undegraded":"2024-09-21T19:09:37.909645+0000","last_fullsized":"2024-09-21T19:09:37.909645+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":195,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910103+0000","last_change":"2024-09-21T19:09:31.740952+0000","last_active":"2024-09-21T19:09:37.910103+0000","last_peered":"2024-09-21T19:09:37.910103+0000","last_clean":"2024-09-21T19:09:37.910103+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:09:37.910103+0000","last_undegraded":"2024-09-21T19:09:37.910103+0000","last_fullsized":"2024-09-21T19:09:37.910103+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:31.740891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:31.740891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:40.289541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"403'14368","reported_seq":14526,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.643128+0000","last_change":"2024-09-21T19:06:09.006007+0000","last_active":"2024-09-21T19:09:37.643128+0000","last_peered":"2024-09-21T19:09:37.643128+0000","last_clean":"2024-09-21T19:09:37.643128+0000","last_became_active":"2024-09-21T19:06:09.005695+0000","last_became_peered":"2024-09-21T19:06:09.005695+0000","last_unstale":"2024-09-21T19:09:37.643128+0000","last_undegraded":"2024-09-21T19:09:37.643128+0000","last_fullsized":"2024-09-21T19:09:37.643128+0000","mapping_epoch":330,"log_start":"377'9100","ondisk_log_start":"377'9100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:46.323337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":368,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14368,"num_write_kb":471552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":840,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745036+0000","last_change":"2024-09-21T19:06:20.264841+0000","last_active":"2024-09-21T19:09:29.745036+0000","last_peered":"2024-09-21T19:09:29.745036+0000","last_clean":"2024-09-21T19:09:29.745036+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:09:29.745036+0000","last_undegraded":"2024-09-21T19:09:29.745036+0000","last_fullsized":"2024-09-21T19:09:29.745036+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:06:20.264571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.264571+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":773,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.745650+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:09:39.745650+0000","last_peered":"2024-09-21T19:09:39.745650+0000","last_clean":"2024-09-21T19:09:39.745650+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:09:39.745650+0000","last_undegraded":"2024-09-21T19:09:39.745650+0000","last_fullsized":"2024-09-21T19:09:39.745650+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":771,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.904943+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:09:37.904943+0000","last_peered":"2024-09-21T19:09:37.904943+0000","last_clean":"2024-09-21T19:09:37.904943+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:09:37.904943+0000","last_undegraded":"2024-09-21T19:09:37.904943+0000","last_fullsized":"2024-09-21T19:09:37.904943+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":244,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909932+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:09:37.909932+0000","last_peered":"2024-09-21T19:09:37.909932+0000","last_clean":"2024-09-21T19:09:37.909932+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:09:37.909932+0000","last_undegraded":"2024-09-21T19:09:37.909932+0000","last_fullsized":"2024-09-21T19:09:37.909932+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:19.687093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":746,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.749470+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:09:39.749470+0000","last_peered":"2024-09-21T19:09:39.749470+0000","last_clean":"2024-09-21T19:09:39.749470+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:09:39.749470+0000","last_undegraded":"2024-09-21T19:09:39.749470+0000","last_fullsized":"2024-09-21T19:09:39.749470+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"404'14061","reported_seq":14221,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.921680+0000","last_change":"2024-09-21T19:06:09.005666+0000","last_active":"2024-09-21T19:09:37.921680+0000","last_peered":"2024-09-21T19:09:37.921680+0000","last_clean":"2024-09-21T19:09:37.921680+0000","last_became_active":"2024-09-21T19:06:09.004811+0000","last_became_peered":"2024-09-21T19:06:09.004811+0000","last_unstale":"2024-09-21T19:09:37.921680+0000","last_undegraded":"2024-09-21T19:09:37.921680+0000","last_fullsized":"2024-09-21T19:09:37.921680+0000","mapping_epoch":330,"log_start":"367'7600","ondisk_log_start":"367'7600","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":6461,"log_dups_size":0,"ondisk_log_size":6461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.729694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14061,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":631,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756068+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:09:29.756068+0000","last_peered":"2024-09-21T19:09:29.756068+0000","last_clean":"2024-09-21T19:09:29.756068+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:09:29.756068+0000","last_undegraded":"2024-09-21T19:09:29.756068+0000","last_fullsized":"2024-09-21T19:09:29.756068+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":732,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.907255+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:09:37.907255+0000","last_peered":"2024-09-21T19:09:37.907255+0000","last_clean":"2024-09-21T19:09:37.907255+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:09:37.907255+0000","last_undegraded":"2024-09-21T19:09:37.907255+0000","last_fullsized":"2024-09-21T19:09:37.907255+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":187,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909625+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:09:37.909625+0000","last_peered":"2024-09-21T19:09:37.909625+0000","last_clean":"2024-09-21T19:09:37.909625+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:09:37.909625+0000","last_undegraded":"2024-09-21T19:09:37.909625+0000","last_fullsized":"2024-09-21T19:09:37.909625+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:10.305625+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":626,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757484+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:09:29.757484+0000","last_peered":"2024-09-21T19:09:29.757484+0000","last_clean":"2024-09-21T19:09:29.757484+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:09:29.757484+0000","last_undegraded":"2024-09-21T19:09:29.757484+0000","last_fullsized":"2024-09-21T19:09:29.757484+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"403'14145","reported_seq":14303,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.782902+0000","last_change":"2024-09-21T19:06:09.007368+0000","last_active":"2024-09-21T19:09:37.782902+0000","last_peered":"2024-09-21T19:09:37.782902+0000","last_clean":"2024-09-21T19:09:37.782902+0000","last_became_active":"2024-09-21T19:06:09.007102+0000","last_became_peered":"2024-09-21T19:06:09.007102+0000","last_unstale":"2024-09-21T19:09:37.782902+0000","last_undegraded":"2024-09-21T19:09:37.782902+0000","last_fullsized":"2024-09-21T19:09:37.782902+0000","mapping_epoch":330,"log_start":"379'9100","ondisk_log_start":"379'9100","created":330,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":682,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910993+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:09:37.910993+0000","last_peered":"2024-09-21T19:09:37.910993+0000","last_clean":"2024-09-21T19:09:37.910993+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:09:37.910993+0000","last_undegraded":"2024-09-21T19:09:37.910993+0000","last_fullsized":"2024-09-21T19:09:37.910993+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":761,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.746033+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:09:29.746033+0000","last_peered":"2024-09-21T19:09:29.746033+0000","last_clean":"2024-09-21T19:09:29.746033+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:09:29.746033+0000","last_undegraded":"2024-09-21T19:09:29.746033+0000","last_fullsized":"2024-09-21T19:09:29.746033+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.612669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":198,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910415+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:09:37.910415+0000","last_peered":"2024-09-21T19:09:37.910415+0000","last_clean":"2024-09-21T19:09:37.910415+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:09:37.910415+0000","last_undegraded":"2024-09-21T19:09:37.910415+0000","last_fullsized":"2024-09-21T19:09:37.910415+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":741,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:35.677140+0000","last_change":"2024-09-21T19:09:35.677140+0000","last_active":"2024-09-21T19:09:35.677140+0000","last_peered":"2024-09-21T19:09:35.677140+0000","last_clean":"2024-09-21T19:09:35.677140+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:09:35.677140+0000","last_undegraded":"2024-09-21T19:09:35.677140+0000","last_fullsized":"2024-09-21T19:09:35.677140+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:35.677043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:35.677043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:50.521879+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"404'14418","reported_seq":19395,"reported_epoch":404,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:39.374629+0000","last_change":"2024-09-21T19:08:53.601069+0000","last_active":"2024-09-21T19:09:39.374629+0000","last_peered":"2024-09-21T19:09:39.374629+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.832203+0000","last_became_peered":"2024-09-21T19:06:55.832203+0000","last_unstale":"2024-09-21T19:09:39.374629+0000","last_undegraded":"2024-09-21T19:06:55.705814+0000","last_fullsized":"2024-09-21T19:06:55.682723+0000","mapping_epoch":349,"log_start":"377'9100","ondisk_log_start":"377'9100","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14418,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2407,"num_bytes_recovered":37355520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-1875}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":802,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.742824+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:09:39.742824+0000","last_peered":"2024-09-21T19:09:39.742824+0000","last_clean":"2024-09-21T19:09:39.742824+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:09:39.742824+0000","last_undegraded":"2024-09-21T19:09:39.742824+0000","last_fullsized":"2024-09-21T19:09:39.742824+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":755,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.746231+0000","last_change":"2024-09-21T19:06:21.298040+0000","last_active":"2024-09-21T19:09:29.746231+0000","last_peered":"2024-09-21T19:09:29.746231+0000","last_clean":"2024-09-21T19:09:29.746231+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:09:29.746231+0000","last_undegraded":"2024-09-21T19:09:29.746231+0000","last_fullsized":"2024-09-21T19:09:29.746231+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:21.297961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.297961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":187,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910023+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:09:37.910023+0000","last_peered":"2024-09-21T19:09:37.910023+0000","last_clean":"2024-09-21T19:09:37.910023+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:09:37.910023+0000","last_undegraded":"2024-09-21T19:09:37.910023+0000","last_fullsized":"2024-09-21T19:09:37.910023+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:00.385824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":723,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.902103+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:09:37.902103+0000","last_peered":"2024-09-21T19:09:37.902103+0000","last_clean":"2024-09-21T19:09:37.902103+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:09:37.902103+0000","last_undegraded":"2024-09-21T19:09:37.902103+0000","last_fullsized":"2024-09-21T19:09:37.902103+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"403'14519","reported_seq":21147,"reported_epoch":403,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T19:09:37.788758+0000","last_change":"2024-09-21T19:08:18.645441+0000","last_active":"2024-09-21T19:09:37.788758+0000","last_peered":"2024-09-21T19:09:37.788758+0000","last_clean":"2024-09-21T19:06:07.982998+0000","last_became_active":"2024-09-21T19:06:55.869794+0000","last_became_peered":"2024-09-21T19:06:55.869794+0000","last_unstale":"2024-09-21T19:09:37.788758+0000","last_undegraded":"2024-09-21T19:06:55.707620+0000","last_fullsized":"2024-09-21T19:06:55.687264+0000","mapping_epoch":349,"log_start":"379'9500","ondisk_log_start":"379'9500","created":330,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:06:07.982998+0000","last_clean_scrub_stamp":"2024-09-21T19:06:07.982998+0000","objects_scrubbed":0,"log_size":5019,"log_dups_size":0,"ondisk_log_size":5019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14519,"num_write_kb":477120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3235,"num_bytes_recovered":103088128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":-1123}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":353,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745489+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:09:29.745489+0000","last_peered":"2024-09-21T19:09:29.745489+0000","last_clean":"2024-09-21T19:09:29.745489+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:09:29.745489+0000","last_undegraded":"2024-09-21T19:09:29.745489+0000","last_fullsized":"2024-09-21T19:09:29.745489+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:17.636097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":777,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:30.608553+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:09:30.608553+0000","last_peered":"2024-09-21T19:09:30.608553+0000","last_clean":"2024-09-21T19:09:30.608553+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:09:30.608553+0000","last_undegraded":"2024-09-21T19:09:30.608553+0000","last_fullsized":"2024-09-21T19:09:30.608553+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:31.474930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":240,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.902304+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:09:37.902304+0000","last_peered":"2024-09-21T19:09:37.902304+0000","last_clean":"2024-09-21T19:09:37.902304+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:09:37.902304+0000","last_undegraded":"2024-09-21T19:09:37.902304+0000","last_fullsized":"2024-09-21T19:09:37.902304+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":759,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910767+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:09:37.910767+0000","last_peered":"2024-09-21T19:09:37.910767+0000","last_clean":"2024-09-21T19:09:37.910767+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:09:37.910767+0000","last_undegraded":"2024-09-21T19:09:37.910767+0000","last_fullsized":"2024-09-21T19:09:37.910767+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":693,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910708+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:09:37.910708+0000","last_peered":"2024-09-21T19:09:37.910708+0000","last_clean":"2024-09-21T19:09:37.910708+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:09:37.910708+0000","last_undegraded":"2024-09-21T19:09:37.910708+0000","last_fullsized":"2024-09-21T19:09:37.910708+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":756,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.906238+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:09:37.906238+0000","last_peered":"2024-09-21T19:09:37.906238+0000","last_clean":"2024-09-21T19:09:37.906238+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:09:37.906238+0000","last_undegraded":"2024-09-21T19:09:37.906238+0000","last_fullsized":"2024-09-21T19:09:37.906238+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":292,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.905267+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:09:37.905267+0000","last_peered":"2024-09-21T19:09:37.905267+0000","last_clean":"2024-09-21T19:09:37.905267+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:09:37.905267+0000","last_undegraded":"2024-09-21T19:09:37.905267+0000","last_fullsized":"2024-09-21T19:09:37.905267+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":729,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745780+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:09:29.745780+0000","last_peered":"2024-09-21T19:09:29.745780+0000","last_clean":"2024-09-21T19:09:29.745780+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:09:29.745780+0000","last_undegraded":"2024-09-21T19:09:29.745780+0000","last_fullsized":"2024-09-21T19:09:29.745780+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":630,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756670+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:09:29.756670+0000","last_peered":"2024-09-21T19:09:29.756670+0000","last_clean":"2024-09-21T19:09:29.756670+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:09:29.756670+0000","last_undegraded":"2024-09-21T19:09:29.756670+0000","last_fullsized":"2024-09-21T19:09:29.756670+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":742,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745214+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:09:29.745214+0000","last_peered":"2024-09-21T19:09:29.745214+0000","last_clean":"2024-09-21T19:09:29.745214+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:09:29.745214+0000","last_undegraded":"2024-09-21T19:09:29.745214+0000","last_fullsized":"2024-09-21T19:09:29.745214+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:16.645472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":726,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.906064+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:09:37.906064+0000","last_peered":"2024-09-21T19:09:37.906064+0000","last_clean":"2024-09-21T19:09:37.906064+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:09:37.906064+0000","last_undegraded":"2024-09-21T19:09:37.906064+0000","last_fullsized":"2024-09-21T19:09:37.906064+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":187,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909999+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:09:37.909999+0000","last_peered":"2024-09-21T19:09:37.909999+0000","last_clean":"2024-09-21T19:09:37.909999+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:09:37.909999+0000","last_undegraded":"2024-09-21T19:09:37.909999+0000","last_fullsized":"2024-09-21T19:09:37.909999+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":343,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745936+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:09:29.745936+0000","last_peered":"2024-09-21T19:09:29.745936+0000","last_clean":"2024-09-21T19:09:29.745936+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:09:29.745936+0000","last_undegraded":"2024-09-21T19:09:29.745936+0000","last_fullsized":"2024-09-21T19:09:29.745936+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:29.639722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":627,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.755449+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:09:29.755449+0000","last_peered":"2024-09-21T19:09:29.755449+0000","last_clean":"2024-09-21T19:09:29.755449+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:09:29.755449+0000","last_undegraded":"2024-09-21T19:09:29.755449+0000","last_fullsized":"2024-09-21T19:09:29.755449+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":957,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.906850+0000","last_change":"2024-09-21T19:05:42.919182+0000","last_active":"2024-09-21T19:09:37.906850+0000","last_peered":"2024-09-21T19:09:37.906850+0000","last_clean":"2024-09-21T19:09:37.906850+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:09:37.906850+0000","last_undegraded":"2024-09-21T19:09:37.906850+0000","last_fullsized":"2024-09-21T19:09:37.906850+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:00:19.852021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:00:19.852021+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":748,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.741214+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:09:39.741214+0000","last_peered":"2024-09-21T19:09:39.741214+0000","last_clean":"2024-09-21T19:09:39.741214+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:09:39.741214+0000","last_undegraded":"2024-09-21T19:09:39.741214+0000","last_fullsized":"2024-09-21T19:09:39.741214+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":753,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.907066+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:09:37.907066+0000","last_peered":"2024-09-21T19:09:37.907066+0000","last_clean":"2024-09-21T19:09:37.907066+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:09:37.907066+0000","last_undegraded":"2024-09-21T19:09:37.907066+0000","last_fullsized":"2024-09-21T19:09:37.907066+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":753,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:32.615127+0000","last_change":"2024-09-21T19:09:32.615127+0000","last_active":"2024-09-21T19:09:32.615127+0000","last_peered":"2024-09-21T19:09:32.615127+0000","last_clean":"2024-09-21T19:09:32.615127+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:09:32.615127+0000","last_undegraded":"2024-09-21T19:09:32.615127+0000","last_fullsized":"2024-09-21T19:09:32.615127+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:32.615041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:32.615041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.657148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":404,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:32.650896+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:09:32.650896+0000","last_peered":"2024-09-21T19:09:32.650896+0000","last_clean":"2024-09-21T19:09:32.650896+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:09:32.650896+0000","last_undegraded":"2024-09-21T19:09:32.650896+0000","last_fullsized":"2024-09-21T19:09:32.650896+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:41.102423+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":619,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757140+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:09:29.757140+0000","last_peered":"2024-09-21T19:09:29.757140+0000","last_clean":"2024-09-21T19:09:29.757140+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:09:29.757140+0000","last_undegraded":"2024-09-21T19:09:29.757140+0000","last_fullsized":"2024-09-21T19:09:29.757140+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":828,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.740759+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:09:39.740759+0000","last_peered":"2024-09-21T19:09:39.740759+0000","last_clean":"2024-09-21T19:09:39.740759+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:09:39.740759+0000","last_undegraded":"2024-09-21T19:09:39.740759+0000","last_fullsized":"2024-09-21T19:09:39.740759+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":747,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-21T19:09:39.745849+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:09:39.745849+0000","last_peered":"2024-09-21T19:09:39.745849+0000","last_clean":"2024-09-21T19:09:39.745849+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:09:39.745849+0000","last_undegraded":"2024-09-21T19:09:39.745849+0000","last_fullsized":"2024-09-21T19:09:39.745849+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":751,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:33.104038+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:09:33.104038+0000","last_peered":"2024-09-21T19:09:33.104038+0000","last_clean":"2024-09-21T19:09:33.104038+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:09:33.104038+0000","last_undegraded":"2024-09-21T19:09:33.104038+0000","last_fullsized":"2024-09-21T19:09:33.104038+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:37.041685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":748,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.902484+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:09:37.902484+0000","last_peered":"2024-09-21T19:09:37.902484+0000","last_clean":"2024-09-21T19:09:37.902484+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:09:37.902484+0000","last_undegraded":"2024-09-21T19:09:37.902484+0000","last_fullsized":"2024-09-21T19:09:37.902484+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":822,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.749589+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:09:29.749589+0000","last_peered":"2024-09-21T19:09:29.749589+0000","last_clean":"2024-09-21T19:09:29.749589+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:09:29.749589+0000","last_undegraded":"2024-09-21T19:09:29.749589+0000","last_fullsized":"2024-09-21T19:09:29.749589+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:52.278173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":746,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745945+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:09:29.745945+0000","last_peered":"2024-09-21T19:09:29.745945+0000","last_clean":"2024-09-21T19:09:29.745945+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:09:29.745945+0000","last_undegraded":"2024-09-21T19:09:29.745945+0000","last_fullsized":"2024-09-21T19:09:29.745945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":749,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903884+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:09:37.903884+0000","last_peered":"2024-09-21T19:09:37.903884+0000","last_clean":"2024-09-21T19:09:37.903884+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:09:37.903884+0000","last_undegraded":"2024-09-21T19:09:37.903884+0000","last_fullsized":"2024-09-21T19:09:37.903884+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":638,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.749166+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:09:29.749166+0000","last_peered":"2024-09-21T19:09:29.749166+0000","last_clean":"2024-09-21T19:09:29.749166+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:09:29.749166+0000","last_undegraded":"2024-09-21T19:09:29.749166+0000","last_fullsized":"2024-09-21T19:09:29.749166+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":773,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.902620+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:09:37.902620+0000","last_peered":"2024-09-21T19:09:37.902620+0000","last_clean":"2024-09-21T19:09:37.902620+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:09:37.902620+0000","last_undegraded":"2024-09-21T19:09:37.902620+0000","last_fullsized":"2024-09-21T19:09:37.902620+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":686,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.910237+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:09:37.910237+0000","last_peered":"2024-09-21T19:09:37.910237+0000","last_clean":"2024-09-21T19:09:37.910237+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:09:37.910237+0000","last_undegraded":"2024-09-21T19:09:37.910237+0000","last_fullsized":"2024-09-21T19:09:37.910237+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":758,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.745299+0000","last_change":"2024-09-21T19:06:48.347823+0000","last_active":"2024-09-21T19:09:29.745299+0000","last_peered":"2024-09-21T19:09:29.745299+0000","last_clean":"2024-09-21T19:09:29.745299+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:09:29.745299+0000","last_undegraded":"2024-09-21T19:09:29.745299+0000","last_fullsized":"2024-09-21T19:09:29.745299+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.347750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.347750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":748,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.904891+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:09:37.904891+0000","last_peered":"2024-09-21T19:09:37.904891+0000","last_clean":"2024-09-21T19:09:37.904891+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:09:37.904891+0000","last_undegraded":"2024-09-21T19:09:37.904891+0000","last_fullsized":"2024-09-21T19:09:37.904891+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":772,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903426+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:09:37.903426+0000","last_peered":"2024-09-21T19:09:37.903426+0000","last_clean":"2024-09-21T19:09:37.903426+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:09:37.903426+0000","last_undegraded":"2024-09-21T19:09:37.903426+0000","last_fullsized":"2024-09-21T19:09:37.903426+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":622,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.764984+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:09:29.764984+0000","last_peered":"2024-09-21T19:09:29.764984+0000","last_clean":"2024-09-21T19:09:29.764984+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:09:29.764984+0000","last_undegraded":"2024-09-21T19:09:29.764984+0000","last_fullsized":"2024-09-21T19:09:29.764984+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":660,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909328+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:09:37.909328+0000","last_peered":"2024-09-21T19:09:37.909328+0000","last_clean":"2024-09-21T19:09:37.909328+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:09:37.909328+0000","last_undegraded":"2024-09-21T19:09:37.909328+0000","last_fullsized":"2024-09-21T19:09:37.909328+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":741,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.744836+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:09:29.744836+0000","last_peered":"2024-09-21T19:09:29.744836+0000","last_clean":"2024-09-21T19:09:29.744836+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:09:29.744836+0000","last_undegraded":"2024-09-21T19:09:29.744836+0000","last_fullsized":"2024-09-21T19:09:29.744836+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":746,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.757318+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:09:29.757318+0000","last_peered":"2024-09-21T19:09:29.757318+0000","last_clean":"2024-09-21T19:09:29.757318+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:09:29.757318+0000","last_undegraded":"2024-09-21T19:09:29.757318+0000","last_fullsized":"2024-09-21T19:09:29.757318+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":764,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.903639+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:09:37.903639+0000","last_peered":"2024-09-21T19:09:37.903639+0000","last_clean":"2024-09-21T19:09:37.903639+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:09:37.903639+0000","last_undegraded":"2024-09-21T19:09:37.903639+0000","last_fullsized":"2024-09-21T19:09:37.903639+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":726,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-21T19:09:37.909968+0000","last_change":"2024-09-21T19:06:54.695058+0000","last_active":"2024-09-21T19:09:37.909968+0000","last_peered":"2024-09-21T19:09:37.909968+0000","last_clean":"2024-09-21T19:09:37.909968+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:09:37.909968+0000","last_undegraded":"2024-09-21T19:09:37.909968+0000","last_fullsized":"2024-09-21T19:09:37.909968+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:03:17.580427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:17.580427+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":642,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-21T19:09:29.756872+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:09:29.756872+0000","last_peered":"2024-09-21T19:09:29.756872+0000","last_clean":"2024-09-21T19:09:29.756872+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:09:29.756872+0000","last_undegraded":"2024-09-21T19:09:29.756872+0000","last_fullsized":"2024-09-21T19:09:29.756872+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":328073240,"num_objects":5007,"num_object_clones":0,"num_object_copies":15021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229935,"num_write_kb":7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19620,"num_bytes_recovered":1056505856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021194240,"data_stored":1021182024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85635,"ondisk_log_size":85635,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":978,"num_read_kb":973,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266890,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":715028,"kb_used_data":120196,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93030124,"statfs":{"total":95995035648,"available":95262846976,"internally_reserved":0,"allocated":123080704,"data_stored":122310615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464341,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435548,"kb_used_data":138380,"kb_used_omap":4,"kb_used_meta":297147,"kb_avail":93309604,"statfs":{"total":95995035648,"available":95549034496,"internally_reserved":0,"allocated":141701120,"data_stored":140929514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":304279204},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619042,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":379916,"kb_used_data":132220,"kb_used_omap":4,"kb_used_meta":247675,"kb_avail":93365236,"statfs":{"total":95995035648,"available":95606001664,"internally_reserved":0,"allocated":135393280,"data_stored":134631408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":253619875},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019097,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636464,"kb_used_data":91616,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93108688,"statfs":{"total":95995035648,"available":95343296512,"internally_reserved":0,"allocated":93814784,"data_stored":93030437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182621,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874860,"kb_used_data":131484,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92870292,"statfs":{"total":95995035648,"available":95099179008,"internally_reserved":0,"allocated":134639616,"data_stored":133846294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411555,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":698816,"kb_used_data":148400,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93046336,"statfs":{"total":95995035648,"available":95279448064,"internally_reserved":0,"allocated":151961600,"data_stored":151212021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751160,"kb_used_data":84136,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92993992,"statfs":{"total":95995035648,"available":95225847808,"internally_reserved":0,"allocated":86155264,"data_stored":85349214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081884,"kb_used_data":270412,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92663268,"statfs":{"total":95995035648,"available":94887186432,"internally_reserved":0,"allocated":276901888,"data_stored":276124320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249499648,"data_stored":249495576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75431936,"data_stored":75431936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147324928,"data_stored":147324928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128385024,"data_stored":128385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61280256,"data_stored":61276184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":120520704,"data_stored":120520704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":123211776,"data_stored":123207704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":115539968,"data_stored":115539968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:09:44.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T19:09:44.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:44.564 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:44.665 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:44.697 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:09:44.697 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:09:44.697 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-09-21T19:09:44.697 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:09:44.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:44.812 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:44.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:44.938 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:44.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:44.939 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:45.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:45.061 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:45.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:44 smithi081 ceph-mon[39192]: pgmap v819: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 129 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 3371/6786 objects degraded (49.676%); 0 B/s, 112 objects/s recovering 2024-09-21T19:09:45.064 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:44 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 3371/6786 objects degraded (49.676%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:45.064 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:44 smithi081 ceph-mon[29384]: pgmap v819: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 129 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 3371/6786 objects degraded (49.676%); 0 B/s, 112 objects/s recovering 2024-09-21T19:09:45.064 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:44 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 3371/6786 objects degraded (49.676%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:45.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:45.185 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:44 smithi104 ceph-mon[32438]: pgmap v819: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 129 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 3371/6786 objects degraded (49.676%); 0 B/s, 112 objects/s recovering 2024-09-21T19:09:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:44 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 3371/6786 objects degraded (49.676%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:45.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:45.410 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:45.432 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:45.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:45.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:09:45.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:45.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:45.678 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:45.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:45.801 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:45.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:45.943 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:45.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:46.045 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:46.143 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:46.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:46.191 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:46.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:46.317 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:46.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:46.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:46.582 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:46.683 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:46.705 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:46.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:46.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:46.862 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:09:46.862 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:09:46.878 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:09:46.879 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:09:46.894 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:09:46.894 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:09:46.910 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:09:46.910 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:09:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:09:46.929 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:46.930 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:09:46.930 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:09:46.949 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:09:46.949 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:09:46.952 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:46.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:46.972 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:09:46.972 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:09:46.989 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:09:46.989 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:09:47.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:47.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[39192]: pgmap v820: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 28 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 2845/1893 objects degraded (150.291%); 0 B/s, 178 objects/s recovering 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2028795249' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[39192]: from='client.36407 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2617519781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[29384]: pgmap v820: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 28 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 2845/1893 objects degraded (150.291%); 0 B/s, 178 objects/s recovering 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2028795249' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[29384]: from='client.36407 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:47.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2617519781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:47.076 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:47.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:47.200 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:46 smithi104 ceph-mon[32438]: pgmap v820: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 28 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 2845/1893 objects degraded (150.291%); 0 B/s, 178 objects/s recovering 2024-09-21T19:09:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2028795249' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:46 smithi104 ceph-mon[32438]: from='client.36407 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:09:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2617519781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:47.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:47.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:47.326 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:47.326 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2845/1893 objects degraded (150.291%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1218,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":406,"num_osds":8,"num_up_osds":8,"osd_up_since":1726945614,"num_in_osds":8,"osd_in_since":1726945541,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":156},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":158,"num_pools":7,"num_objects":631,"data_bytes":28908059,"bytes_used":4794814464,"bytes_avail":763165470720,"bytes_total":767960285184,"degraded_objects":2845,"degraded_total":1893,"degraded_ratio":1.5029054410987852,"recovering_objects_per_sec":178,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1427,"num_bytes_recovered":0,"num_keys_recovered":0,"read_bytes_sec":639,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1206},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":16,"modified":"2024-09-21T19:09:37.775080+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T19:09:47.422 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:09:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:09:47.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:47.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:47.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:47.571 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:47.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:47.694 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:47.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:47.818 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:47.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:47.947 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:47.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:48.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:09:48.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:09:48.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:09:48.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: from='client.36407 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:48.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: 6.0 scrub starts 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: 6.0 scrub ok 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3109411869' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: from='client.36407 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: 6.0 scrub starts 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: 6.0 scrub ok 2024-09-21T19:09:48.040 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:47 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3109411869' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:09:48.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:48.071 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:48.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:48.195 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:48.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:09:48.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:09:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:09:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: from='client.36407 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:09:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T19:09:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: 6.0 scrub starts 2024-09-21T19:09:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: 6.0 scrub ok 2024-09-21T19:09:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:47 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3109411869' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:09:48.297 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:48.318 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:48.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:48.442 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:48.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:48.565 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:48.572 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T19:09:48.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T19:09:48.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:48.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:48.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:48.814 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T19:09:48.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:09:48.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:48.936 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:48.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:48.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:09:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:09:49.038 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:49.060 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:49.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:48 smithi081 ceph-mon[29384]: pgmap v822: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 7.2 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 698 op/s; 2845/915 objects degraded (310.929%); 0 B/s, 106 objects/s recovering 2024-09-21T19:09:49.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:48 smithi081 ceph-mon[29384]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T19:09:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:48 smithi081 ceph-mon[39192]: pgmap v822: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 7.2 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 698 op/s; 2845/915 objects degraded (310.929%); 0 B/s, 106 objects/s recovering 2024-09-21T19:09:49.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:48 smithi081 ceph-mon[39192]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T19:09:49.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:49.182 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:49.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:48 smithi104 ceph-mon[32438]: pgmap v822: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 156 active+clean; 7.2 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 698 op/s; 2845/915 objects degraded (310.929%); 0 B/s, 106 objects/s recovering 2024-09-21T19:09:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:48 smithi104 ceph-mon[32438]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T19:09:49.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:49.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:49.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:49.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:49.553 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:49.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:49.677 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:49.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:49.802 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:49.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:09:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:49.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:49.925 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:49.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:50.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:50.048 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:50.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:50.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: 6.3 scrub starts 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: 6.3 scrub ok 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: Health check update: Degraded data redundancy: 2845/915 objects degraded (310.929%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:49 smithi104 ceph-mon[32438]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T19:09:50.273 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:50.294 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: 6.3 scrub starts 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: 6.3 scrub ok 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: Health check update: Degraded data redundancy: 2845/915 objects degraded (310.929%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:50.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[29384]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: 6.3 scrub starts 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: 6.3 scrub ok 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: Health check update: Degraded data redundancy: 2845/915 objects degraded (310.929%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T19:09:50.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:49 smithi081 ceph-mon[39192]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T19:09:50.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:50.422 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:50.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:50.548 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:50.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:50.674 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:50.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:50.798 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:50.899 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:50.922 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:50.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:51.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:51.045 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:51.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:51.172 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:51.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[39192]: pgmap v826: 158 pgs: 2 peering, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 109 op/s; 0 B/s, 711 objects/s recovering 2024-09-21T19:09:51.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[39192]: 3.1f scrub starts 2024-09-21T19:09:51.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[39192]: 3.1f scrub ok 2024-09-21T19:09:51.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[39192]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2845/915 objects degraded (310.929%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T19:09:51.238 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:09:51.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[29384]: pgmap v826: 158 pgs: 2 peering, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 109 op/s; 0 B/s, 711 objects/s recovering 2024-09-21T19:09:51.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[29384]: 3.1f scrub starts 2024-09-21T19:09:51.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[29384]: 3.1f scrub ok 2024-09-21T19:09:51.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[29384]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2845/915 objects degraded (310.929%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T19:09:51.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:50 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:09:51.239 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:09:51.239 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:09:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:50 smithi104 ceph-mon[32438]: pgmap v826: 158 pgs: 2 peering, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 109 op/s; 0 B/s, 711 objects/s recovering 2024-09-21T19:09:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:50 smithi104 ceph-mon[32438]: 3.1f scrub starts 2024-09-21T19:09:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:50 smithi104 ceph-mon[32438]: 3.1f scrub ok 2024-09-21T19:09:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:50 smithi104 ceph-mon[32438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2845/915 objects degraded (310.929%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T19:09:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:50 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:09:51.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:51.399 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.423 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:51.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:51.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:51.550 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:51.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:51.777 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:51.803 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:51.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:09:51.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:51.938 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:51.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:52.012 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:52.028 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:52.038 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:52.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 3.8 scrub starts 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 3.8 scrub ok 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 5.c scrub starts 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 5.c scrub ok 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 5.5 scrub starts 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 5.5 scrub ok 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 4.4 scrub starts 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[39192]: 4.4 scrub ok 2024-09-21T19:09:52.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 3.8 scrub starts 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 3.8 scrub ok 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 5.c scrub starts 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 5.c scrub ok 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 5.5 scrub starts 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 5.5 scrub ok 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 4.4 scrub starts 2024-09-21T19:09:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:51 smithi081 ceph-mon[29384]: 4.4 scrub ok 2024-09-21T19:09:52.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:52.193 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:52.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 3.8 scrub starts 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 3.8 scrub ok 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 5.c scrub starts 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 5.c scrub ok 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 5.5 scrub starts 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 5.5 scrub ok 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 4.4 scrub starts 2024-09-21T19:09:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:51 smithi104 ceph-mon[32438]: 4.4 scrub ok 2024-09-21T19:09:52.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:52.316 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:52.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:52.541 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:52.562 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:52.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:52.686 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:52.788 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:52.811 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:52.910 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:09:52.910 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:09:52.913 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:52.917 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:09:52.917 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:09:52.937 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:52.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:52.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:52.943 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:09:52.943 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:09:52.954 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:52.961 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:09:52.961 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:09:52.980 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:09:52.980 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:09:52.999 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:09:52.999 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:09:53.016 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:09:53.016 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:09:53.032 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:09:53.033 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:09:53.038 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:53.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:53.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:53.185 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:53.200 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_3' removed 2024-09-21T19:09:53.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:53.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: 6.17 scrub starts 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: 6.17 scrub ok 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: pgmap v827: 158 pgs: 2 peering, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 26 op/s; 0 B/s, 585 objects/s recovering 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2902697686' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:09:53.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: from='client.36425 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2218792790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: 6.17 scrub starts 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: 6.17 scrub ok 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: pgmap v827: 158 pgs: 2 peering, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 26 op/s; 0 B/s, 585 objects/s recovering 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2902697686' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: from='client.36425 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:09:53.474 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2218792790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: 6.17 scrub starts 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: 6.17 scrub ok 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: pgmap v827: 158 pgs: 2 peering, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 26 op/s; 0 B/s, 585 objects/s recovering 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2902697686' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: from='client.36425 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:09:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2218792790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:09:53.551 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:53.571 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:53.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:53.694 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:53.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:53.816 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:53.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:53.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:53.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:53.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:09:54.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:54.064 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:54.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:54.186 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235867 8.1b 8:df932d6c:::benchmark_data_smithi081_105538_object103635:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.677779+0000", 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 480.52118122799999, 2024-09-21T19:09:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0885322630000001, 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235867 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677779+0000", 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677779+0000", 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677781+0000", 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5160000000000001e-06 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677788+0000", 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5189999999999998e-06 2024-09-21T19:09:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677808+0000", 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0095e-05 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677817+0000", 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7430000000000008e-06 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.854891+0000", 2024-09-21T19:09:54.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.177073809 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.854897+0000", 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7730000000000002e-06 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.766207+0000", 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91130918400000005 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.766311+0000", 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010462400000000001 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235890 8.20 8:05cf56f2:::benchmark_data_smithi081_105538_object103658:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.677831+0000", 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 480.52112867, 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.074999805, 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.208 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235890 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677831+0000", 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677831+0000", 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677832+0000", 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7599999999999996e-07 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677836+0000", 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.365e-06 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677841+0000", 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8409999999999996e-06 2024-09-21T19:09:54.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677847+0000", 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0479999999999998e-06 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.794917+0000", 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.117070297 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.794925+0000", 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8129999999999996e-06 2024-09-21T19:09:54.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.738124+0000", 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.943199016 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.742476+0000", 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043526110000000002 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.742673+0000", 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019624200000000001 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746583+0000", 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039104150000000004 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.746691+0000", 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108418 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752730+0000", 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060383679999999997 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752787+0000", 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7544000000000003e-05 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752831+0000", 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4051000000000002e-05 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235894 8.1b 8:da874d6a:::benchmark_data_smithi081_105538_object103662:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.677854+0000", 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 480.52110635399998, 2024-09-21T19:09:54.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0956787429999999, 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235894 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677854+0000", 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677854+0000", 2024-09-21T19:09:54.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677854+0000", 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1699999999999998e-07 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677857+0000", 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4959999999999999e-06 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677861+0000", 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6639999999999997e-06 2024-09-21T19:09:54.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.677866+0000", 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5090000000000004e-06 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.854915+0000", 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.17704895100000001 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.854920+0000", 2024-09-21T19:09:54.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9579999999999998e-06 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.766717+0000", 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91179701700000004 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.766821+0000", 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010409899999999999 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:09:54.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.767043+0000", 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022221099999999999 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.769671+0000", 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026275690000000002 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.769863+0000", 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019175299999999999 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.773439+0000", 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035760219999999999 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.773497+0000", 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8173e-05 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.773533+0000", 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5803999999999998e-05 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:17514 9.1 9:8c9b4ade:::benchmark_data_smithi081_131274_object17513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:02:45.917038+0000", 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 428.28192252700001, 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.038878545, 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 17514 2024-09-21T19:09:54.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917038+0000", 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917038+0000", 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917038+0000", 2024-09-21T19:09:54.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8800000000000002e-07 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917057+0000", 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8357999999999999e-05 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917062+0000", 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4190000000000001e-06 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917067+0000", 2024-09-21T19:09:54.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7620000000000003e-06 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917402+0000", 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00033530499999999999 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.917411+0000", 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0529999999999996e-06 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.952248+0000", 2024-09-21T19:09:54.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0348364379999999 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.952378+0000", 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000130796 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 6,7", 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.952533+0000", 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015451199999999999 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955760+0000", 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003227129 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955808+0000", 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8374999999999997e-05 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955848+0000", 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9452000000000001e-05 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955892+0000", 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3893000000000001e-05 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955916+0000", 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4465e-05 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:17517 9.1 9:8a57b481:::benchmark_data_smithi081_131274_object17516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:02:45.918937+0000", 2024-09-21T19:09:54.228 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 428.28002366700002, 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.039404985, 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 17517 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.918937+0000", 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.918937+0000", 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.918938+0000", 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7549999999999999e-06 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.918961+0000", 2024-09-21T19:09:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2843000000000001e-05 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.918972+0000", 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0431000000000001e-05 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.918978+0000", 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.224e-06 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.919065+0000", 2024-09-21T19:09:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7168000000000003e-05 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.919073+0000", 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4339999999999992e-06 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.953931+0000", 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.034857895 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.954055+0000", 2024-09-21T19:09:54.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012361499999999999 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 6,7", 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.954154+0000", 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8781000000000002e-05 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955951+0000", 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017970180000000001 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958224+0000", 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002273153 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958262+0000", 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8037000000000002e-05 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958306+0000", 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4419999999999998e-05 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958341+0000", 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5210999999999999e-05 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:17524 9.1 9:821879e8:::benchmark_data_smithi081_131274_object17523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:02:45.923898+0000", 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 428.27506171700003, 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0345943049999999, 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 17524 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.923898+0000", 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.923898+0000", 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.923899+0000", 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.33e-07 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.923957+0000", 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7575999999999999e-05 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.923965+0000", 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.422e-06 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.923981+0000", 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5568000000000001e-05 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.924047+0000", 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6191000000000004e-05 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.924057+0000", 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0127e-05 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.954846+0000", 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0307884409999999 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.954967+0000", 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121416 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 6,7", 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955105+0000", 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138209 2024-09-21T19:09:54.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958117+0000", 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030115469999999998 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958375+0000", 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025860800000000001 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958416+0000", 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0553000000000003e-05 2024-09-21T19:09:54.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958455+0000", 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8862999999999997e-05 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958493+0000", 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7851000000000001e-05 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.241 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:17539 9.1 9:8e5ea8f2:::benchmark_data_smithi081_131274_object17538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:02:45.928505+0000", 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 428.27045471500003, 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0322582849999999, 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 17539 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928505+0000", 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928505+0000", 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928506+0000", 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9899999999999999e-07 2024-09-21T19:09:54.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928528+0000", 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2144999999999999e-05 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928536+0000", 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6459999999999993e-06 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928542+0000", 2024-09-21T19:09:54.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2589999999999997e-06 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928823+0000", 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028052900000000001 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.928833+0000", 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0524e-05 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.955978+0000", 2024-09-21T19:09:54.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.027144684 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.956098+0000", 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119497 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 6,7", 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.956270+0000", 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000172898 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958234+0000", 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019634159999999999 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958533+0000", 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029894200000000001 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.960661+0000", 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021278960000000002 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.960721+0000", 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9888e-05 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.960764+0000", 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3161999999999998e-05 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:17562 9.1 9:844a1e25:::benchmark_data_smithi081_131274_object17561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T19:09:54.248 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:02:45.935522+0000", 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 428.26343807400002, 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0276195370000001, 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 17562 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935522+0000", 2024-09-21T19:09:54.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935522+0000", 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935523+0000", 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.384e-06 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935542+0000", 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8388000000000001e-05 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935553+0000", 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1592e-05 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935562+0000", 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1440000000000003e-06 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935636+0000", 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3973999999999993e-05 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.935665+0000", 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9767000000000001e-05 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.956818+0000", 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.021152286 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.957419+0000", 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000601105 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 6,7", 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.957549+0000", 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013026800000000001 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958389+0000", 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00084024100000000003 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.960614+0000", 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022250939999999999 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.963038+0000", 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024233589999999999 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.963098+0000", 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0038999999999999e-05 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.963142+0000", 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3896000000000001e-05 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.255 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:17568 9.1 9:8a093eca:::benchmark_data_smithi081_131274_object17567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T19:09:54.256 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:02:45.938473+0000", 2024-09-21T19:09:54.256 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 428.26048701100001, 2024-09-21T19:09:54.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0248265430000001, 2024-09-21T19:09:54.256 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.256 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 17568 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938473+0000", 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938473+0000", 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938475+0000", 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3489999999999999e-06 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938492+0000", 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7478000000000001e-05 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938501+0000", 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9020000000000005e-06 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938506+0000", 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1170000000000003e-06 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938550+0000", 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4156999999999998e-05 2024-09-21T19:09:54.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:45.938556+0000", 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8619999999999999e-06 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958561+0000", 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.020004994 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958662+0000", 2024-09-21T19:09:54.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010105799999999999 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 6,7", 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.958787+0000", 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012514500000000001 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.960812+0000", 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020252460000000001 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.962984+0000", 2024-09-21T19:09:54.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021719230000000001 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.963214+0000", 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000229967 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.963264+0000", 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9236000000000001e-05 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:02:46.963300+0000", 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6109000000000003e-05 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219766 9.2 9:4fcff26f:::benchmark_data_smithi081_131274_object91911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.749045+0000", 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 268.449915309, 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0377334439999999, 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:09:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219766 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749045+0000", 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749045+0000", 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749046+0000", 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.511e-06 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749051+0000", 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2490000000000003e-06 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749058+0000", 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6520000000000006e-06 2024-09-21T19:09:54.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749064+0000", 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5629999999999999e-06 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.751794+0000", 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027304389999999999 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.751874+0000", 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9385e-05 2024-09-21T19:09:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,7", 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.751994+0000", 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120357 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.754636+0000", 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026419350000000002 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.754934+0000", 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029846499999999998 2024-09-21T19:09:54.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.765492+0000", 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010557676 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.765502+0000", 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5240000000000006e-06 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.786702+0000", 2024-09-21T19:09:54.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.021200541 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.786778+0000", 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6147000000000001e-05 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219801 9.2 9:490b2d1b:::benchmark_data_smithi081_131274_object91946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.783370+0000", 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 268.41559003100002, 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0096375900000001, 2024-09-21T19:09:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219801 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783370+0000", 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783370+0000", 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783372+0000", 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.035e-06 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783378+0000", 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783392+0000", 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3685e-05 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783404+0000", 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1819e-05 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783474+0000", 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0130999999999997e-05 2024-09-21T19:09:54.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783482+0000", 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6300000000000004e-06 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.787211+0000", 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0037284470000001 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.787309+0000", 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8555999999999996e-05 2024-09-21T19:09:54.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.787420+0000", 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110451 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792822+0000", 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005402095 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792917+0000", 2024-09-21T19:09:54.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5041e-05 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792929+0000", 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2169e-05 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792972+0000", 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3120000000000001e-05 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793008+0000", 2024-09-21T19:09:54.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5379999999999997e-05 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219859 9.2 9:4f58cd03:::benchmark_data_smithi081_131274_object92004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.818793+0000", 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 268.38016667699998, 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97437190200000001, 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.276 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219859 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818793+0000", 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818793+0000", 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818795+0000", 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6470000000000001e-06 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818800+0000", 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0520000000000001e-06 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818815+0000", 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4905e-05 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818823+0000", 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8299999999999996e-06 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818877+0000", 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4301000000000001e-05 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.818884+0000", 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1930000000000004e-06 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.789148+0000", 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97026384300000001 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.789251+0000", 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010316599999999999 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.789369+0000", 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117205 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792863+0000", 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034940879999999998 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793027+0000", 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000164334 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793081+0000", 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3737999999999999e-05 2024-09-21T19:09:54.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793138+0000", 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7361000000000002e-05 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793165+0000", 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7239000000000001e-05 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219867 9.2 9:45f6c1df:::benchmark_data_smithi081_131274_object92012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.824212+0000", 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 268.37474791599999, 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.970874291, 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219867 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824212+0000", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824212+0000", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824214+0000", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.962e-06 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824220+0000", 2024-09-21T19:09:54.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7649999999999999e-06 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824231+0000", 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.064e-05 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824237+0000", 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.81e-06 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824291+0000", 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3186999999999997e-05 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.824298+0000", 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7149999999999998e-06 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.791324+0000", 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96702565500000004 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.791392+0000", 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8133000000000006e-05 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.791467+0000", 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4801000000000004e-05 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793204+0000", 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001736654 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793213+0000", 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0219999999999993e-06 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.794993+0000", 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017808609999999999 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795049+0000", 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5928999999999999e-05 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795087+0000", 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7156999999999997e-05 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219870 9.2 9:4d69cc28:::benchmark_data_smithi081_131274_object92015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.826593+0000", 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 268.37236715199998, 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97429603899999995, 2024-09-21T19:09:54.289 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219870 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826593+0000", 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826593+0000", 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826595+0000", 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9470000000000002e-06 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826600+0000", 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2989999999999996e-06 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826610+0000", 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0003999999999999e-05 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826617+0000", 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4509999999999997e-06 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826807+0000", 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000190032 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.826814+0000", 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.515e-06 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793238+0000", 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96642404999999998 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793311+0000", 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2719000000000006e-05 2024-09-21T19:09:54.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793417+0000", 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105795 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.797080+0000", 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036633600000000001 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.797093+0000", 2024-09-21T19:09:54.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.258e-05 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.800819+0000", 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037261640000000001 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.800850+0000", 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.065e-05 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.800889+0000", 2024-09-21T19:09:54.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9473e-05 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:243993 9.2 9:4d91acac:::benchmark_data_smithi081_131274_object116138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.939817+0000", 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.259143034, 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0272507099999999, 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.296 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243993 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939817+0000", 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939817+0000", 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939819+0000", 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7969999999999999e-06 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939824+0000", 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9339999999999997e-06 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939836+0000", 2024-09-21T19:09:54.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2277e-05 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939845+0000", 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3799999999999994e-06 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939906+0000", 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1457999999999993e-05 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939915+0000", 2024-09-21T19:09:54.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6319999999999997e-06 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.956102+0000", 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.016187784 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.956213+0000", 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011089 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.956332+0000", 2024-09-21T19:09:54.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118491 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.961700+0000", 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053683489999999997 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966900+0000", 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051996409999999996 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966975+0000", 2024-09-21T19:09:54.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4986000000000003e-05 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967026+0000", 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1731999999999999e-05 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967068+0000", 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1359000000000002e-05 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244009 9.2 9:43e65944:::benchmark_data_smithi081_131274_object116154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.948852+0000", 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.25010794100001, 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0234964339999999, 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244009 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948852+0000", 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948852+0000", 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948854+0000", 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.407e-06 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948858+0000", 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3919999999999996e-06 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948868+0000", 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8320000000000001e-06 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948873+0000", 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7469999999999996e-06 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948922+0000", 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9359e-05 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948927+0000", 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.49e-06 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959174+0000", 2024-09-21T19:09:54.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.01024611 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959284+0000", 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110189 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959396+0000", 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112038 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966862+0000", 2024-09-21T19:09:54.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074661299999999996 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966940+0000", 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8193000000000002e-05 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972236+0000", 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005295803 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972287+0000", 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0955999999999998e-05 2024-09-21T19:09:54.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972349+0000", 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1787999999999999e-05 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244035 9.2 9:4644a85f:::benchmark_data_smithi081_131274_object116180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.965060+0000", 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.23390041299999, 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.007491846, 2024-09-21T19:09:54.309 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244035 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965060+0000", 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965060+0000", 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965061+0000", 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.249e-06 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965065+0000", 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1069999999999998e-06 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965075+0000", 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8849999999999994e-06 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965081+0000", 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4330000000000001e-06 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965132+0000", 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0871999999999997e-05 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965140+0000", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5619999999999998e-06 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.961731+0000", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.99659121699999997 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.961830+0000", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8666999999999996e-05 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:09:54.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962122+0000", 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029182999999999998 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966911+0000", 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047890169999999996 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972395+0000", 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054844339999999998 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972469+0000", 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4361000000000002e-05 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972515+0000", 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5766999999999997e-05 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972552+0000", 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6445000000000001e-05 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244062 9.2 9:4e3e292f:::benchmark_data_smithi081_131274_object116207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.989381+0000", 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.209578677, 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98591870999999998, 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244062 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989381+0000", 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989381+0000", 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989383+0000", 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.206e-06 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989386+0000", 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7110000000000001e-06 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989395+0000", 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.259e-06 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989400+0000", 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1950000000000002e-06 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.999601+0000", 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010201412 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.999608+0000", 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2019999999999998e-06 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964532+0000", 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96492351600000004 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964625+0000", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2520000000000002e-05 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964724+0000", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9300000000000001e-05 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972430+0000", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0077064070000000002 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972591+0000", 2024-09-21T19:09:54.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016030600000000001 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975215+0000", 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026247470000000002 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975259+0000", 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3557000000000003e-05 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975300+0000", 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1372000000000003e-05 2024-09-21T19:09:54.321 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244068 9.2 9:4f59cf3c:::benchmark_data_smithi081_131274_object116213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:48.006259+0000", 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.19270141600001, 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96913308399999998, 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.322 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244068 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006259+0000", 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006259+0000", 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006260+0000", 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5030000000000001e-06 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006264+0000", 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7390000000000001e-06 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006273+0000", 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2080000000000006e-06 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006280+0000", 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6229999999999999e-06 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006328+0000", 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7936999999999999e-05 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.006337+0000", 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8380000000000008e-06 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967128+0000", 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96079151799999996 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967247+0000", 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011862999999999999 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967358+0000", 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011162699999999999 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972632+0000", 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0052740499999999997 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975196+0000", 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002563802 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975308+0000", 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011225799999999999 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975359+0000", 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0087999999999999e-05 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975392+0000", 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3263000000000003e-05 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244072 9.2 9:4b86ab9c:::benchmark_data_smithi081_131274_object116217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:48.011542+0000", 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 246.187417673, 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.016618896, 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:09:54.328 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244072 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011542+0000", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011542+0000", 2024-09-21T19:09:54.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011544+0000", 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3459999999999999e-06 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011548+0000", 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9339999999999999e-06 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011556+0000", 2024-09-21T19:09:54.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3450000000000006e-06 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011561+0000", 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6229999999999996e-06 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011599+0000", 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8726000000000002e-05 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:09:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011605+0000", 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0749999999999997e-06 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.969646+0000", 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.958041052 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.969767+0000", 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012104600000000001 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.969864+0000", 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6974999999999999e-05 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975402+0000", 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055379100000000001 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.027823+0000", 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052421855000000003 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.333 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.027886+0000", 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2778000000000001e-05 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.028117+0000", 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023073200000000001 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.028161+0000", 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4499000000000002e-05 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.334 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.335 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:09:54.335 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:09:54.335 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:09:54.335 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:54.356 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:54.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: 5.14 scrub starts 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: 5.14 scrub ok 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: 4.16 scrub starts 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: 4.16 scrub ok 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: from='client.36425 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:54 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: 5.14 scrub starts 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: 5.14 scrub ok 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: 4.16 scrub starts 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: 4.16 scrub ok 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: from='client.36425 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T19:09:54.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: 5.14 scrub starts 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: 5.14 scrub ok 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: 4.16 scrub starts 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: 4.16 scrub ok 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: from='client.36425 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:09:54.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:54 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:54.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:54.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:54.614 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:09:54.615 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:09:54.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:54.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:54.848 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:54.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:54.970 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:54.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:55.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:55.132 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:55.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:55.220 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:55.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.345 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:55.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:55.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:55.468 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 1.5 scrub starts 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 1.5 scrub ok 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 6.7 scrub starts 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 6.7 scrub ok 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: pgmap v829: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-09-21T19:09:55.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T19:09:55.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:09:55.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:09:55.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:55.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T19:09:55.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:55 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 1.5 scrub starts 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 1.5 scrub ok 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 6.7 scrub starts 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 6.7 scrub ok 2024-09-21T19:09:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: pgmap v829: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 1.5 scrub starts 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 1.5 scrub ok 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 6.7 scrub starts 2024-09-21T19:09:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 6.7 scrub ok 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: pgmap v829: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T19:09:55.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:55 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T19:09:55.570 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:55.591 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:55.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:55.715 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:55.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:09:55.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:55.838 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:55.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:55.961 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:55.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:56.016 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:09:56.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:56.084 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:56.186 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:56.194 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-09-21T19:09:53.780140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2132,"num_read_kb":1884,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":272632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4491912,"kb_used_data":15652,"kb_used_omap":18,"kb_used_meta":4476077,"kb_avail":745469304,"statfs":{"total":767960285184,"available":763360567296,"internally_reserved":0,"allocated":16027648,"data_stored":9598916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4583503856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2949120,"num_objects":-45,"num_object_clones":0,"num_object_copies":-135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-45,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-234897,"num_write_kb":-7518081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-21660,"num_bytes_recovered":-1056505856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.850741"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":665,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:50.182643+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:09:50.182643+0000","last_peered":"2024-09-21T19:09:50.182643+0000","last_clean":"2024-09-21T19:09:50.182643+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:09:50.182643+0000","last_undegraded":"2024-09-21T19:09:50.182643+0000","last_fullsized":"2024-09-21T19:09:50.182643+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:10.748546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":825,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746755+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:09:49.746755+0000","last_peered":"2024-09-21T19:09:49.746755+0000","last_clean":"2024-09-21T19:09:49.746755+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:09:49.746755+0000","last_undegraded":"2024-09-21T19:09:49.746755+0000","last_fullsized":"2024-09-21T19:09:49.746755+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":762,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737557+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:09:49.737557+0000","last_peered":"2024-09-21T19:09:49.737557+0000","last_clean":"2024-09-21T19:09:49.737557+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:09:49.737557+0000","last_undegraded":"2024-09-21T19:09:49.737557+0000","last_fullsized":"2024-09-21T19:09:49.737557+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.809763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":742,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747546+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:09:49.747546+0000","last_peered":"2024-09-21T19:09:49.747546+0000","last_clean":"2024-09-21T19:09:49.747546+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:09:49.747546+0000","last_undegraded":"2024-09-21T19:09:49.747546+0000","last_fullsized":"2024-09-21T19:09:49.747546+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":751,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738288+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:09:49.738288+0000","last_peered":"2024-09-21T19:09:49.738288+0000","last_clean":"2024-09-21T19:09:49.738288+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:09:49.738288+0000","last_undegraded":"2024-09-21T19:09:49.738288+0000","last_fullsized":"2024-09-21T19:09:49.738288+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":779,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740296+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:09:49.740296+0000","last_peered":"2024-09-21T19:09:49.740296+0000","last_clean":"2024-09-21T19:09:49.740296+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:09:49.740296+0000","last_undegraded":"2024-09-21T19:09:49.740296+0000","last_fullsized":"2024-09-21T19:09:49.740296+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":875,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738016+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:09:49.738016+0000","last_peered":"2024-09-21T19:09:49.738016+0000","last_clean":"2024-09-21T19:09:49.738016+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:09:49.738016+0000","last_undegraded":"2024-09-21T19:09:49.738016+0000","last_fullsized":"2024-09-21T19:09:49.738016+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":762,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746656+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:09:49.746656+0000","last_peered":"2024-09-21T19:09:49.746656+0000","last_clean":"2024-09-21T19:09:49.746656+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:09:49.746656+0000","last_undegraded":"2024-09-21T19:09:49.746656+0000","last_fullsized":"2024-09-21T19:09:49.746656+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":749,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748694+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:09:49.748694+0000","last_peered":"2024-09-21T19:09:49.748694+0000","last_clean":"2024-09-21T19:09:49.748694+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:09:49.748694+0000","last_undegraded":"2024-09-21T19:09:49.748694+0000","last_fullsized":"2024-09-21T19:09:49.748694+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":806,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747860+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:09:49.747860+0000","last_peered":"2024-09-21T19:09:49.747860+0000","last_clean":"2024-09-21T19:09:49.747860+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:09:49.747860+0000","last_undegraded":"2024-09-21T19:09:49.747860+0000","last_fullsized":"2024-09-21T19:09:49.747860+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":703,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741047+0000","last_change":"2024-09-21T19:06:54.696557+0000","last_active":"2024-09-21T19:09:49.741047+0000","last_peered":"2024-09-21T19:09:49.741047+0000","last_clean":"2024-09-21T19:09:49.741047+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:09:49.741047+0000","last_undegraded":"2024-09-21T19:09:49.741047+0000","last_fullsized":"2024-09-21T19:09:49.741047+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:08.212068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:08.212068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":623,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740381+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:09:49.740381+0000","last_peered":"2024-09-21T19:09:49.740381+0000","last_clean":"2024-09-21T19:09:49.740381+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:09:49.740381+0000","last_undegraded":"2024-09-21T19:09:49.740381+0000","last_fullsized":"2024-09-21T19:09:49.740381+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":302,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748877+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:09:49.748877+0000","last_peered":"2024-09-21T19:09:49.748877+0000","last_clean":"2024-09-21T19:09:49.748877+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:09:49.748877+0000","last_undegraded":"2024-09-21T19:09:49.748877+0000","last_fullsized":"2024-09-21T19:09:49.748877+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":752,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747000+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:09:49.747000+0000","last_peered":"2024-09-21T19:09:49.747000+0000","last_clean":"2024-09-21T19:09:49.747000+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:09:49.747000+0000","last_undegraded":"2024-09-21T19:09:49.747000+0000","last_fullsized":"2024-09-21T19:09:49.747000+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":742,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747630+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:09:49.747630+0000","last_peered":"2024-09-21T19:09:49.747630+0000","last_clean":"2024-09-21T19:09:49.747630+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:09:49.747630+0000","last_undegraded":"2024-09-21T19:09:49.747630+0000","last_fullsized":"2024-09-21T19:09:49.747630+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":310,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737334+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:09:49.737334+0000","last_peered":"2024-09-21T19:09:49.737334+0000","last_clean":"2024-09-21T19:09:49.737334+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:09:49.737334+0000","last_undegraded":"2024-09-21T19:09:49.737334+0000","last_fullsized":"2024-09-21T19:09:49.737334+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":771,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748432+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:09:49.748432+0000","last_peered":"2024-09-21T19:09:49.748432+0000","last_clean":"2024-09-21T19:09:49.748432+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:09:49.748432+0000","last_undegraded":"2024-09-21T19:09:49.748432+0000","last_fullsized":"2024-09-21T19:09:49.748432+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":639,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741369+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:09:49.741369+0000","last_peered":"2024-09-21T19:09:49.741369+0000","last_clean":"2024-09-21T19:09:49.741369+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:09:49.741369+0000","last_undegraded":"2024-09-21T19:09:49.741369+0000","last_fullsized":"2024-09-21T19:09:49.741369+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":213,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747663+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:09:49.747663+0000","last_peered":"2024-09-21T19:09:49.747663+0000","last_clean":"2024-09-21T19:09:49.747663+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:09:49.747663+0000","last_undegraded":"2024-09-21T19:09:49.747663+0000","last_fullsized":"2024-09-21T19:09:49.747663+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:24.726173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":765,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738291+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:09:49.738291+0000","last_peered":"2024-09-21T19:09:49.738291+0000","last_clean":"2024-09-21T19:09:49.738291+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:09:49.738291+0000","last_undegraded":"2024-09-21T19:09:49.738291+0000","last_fullsized":"2024-09-21T19:09:49.738291+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:07.738704+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":747,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738730+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:09:49.738730+0000","last_peered":"2024-09-21T19:09:49.738730+0000","last_clean":"2024-09-21T19:09:49.738730+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:09:49.738730+0000","last_undegraded":"2024-09-21T19:09:49.738730+0000","last_fullsized":"2024-09-21T19:09:49.738730+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":773,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748000+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:09:49.748000+0000","last_peered":"2024-09-21T19:09:49.748000+0000","last_clean":"2024-09-21T19:09:49.748000+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:09:49.748000+0000","last_undegraded":"2024-09-21T19:09:49.748000+0000","last_fullsized":"2024-09-21T19:09:49.748000+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":818,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738350+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:09:49.738350+0000","last_peered":"2024-09-21T19:09:49.738350+0000","last_clean":"2024-09-21T19:09:49.738350+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:09:49.738350+0000","last_undegraded":"2024-09-21T19:09:49.738350+0000","last_fullsized":"2024-09-21T19:09:49.738350+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":757,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748686+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:09:49.748686+0000","last_peered":"2024-09-21T19:09:49.748686+0000","last_clean":"2024-09-21T19:09:49.748686+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:09:49.748686+0000","last_undegraded":"2024-09-21T19:09:49.748686+0000","last_fullsized":"2024-09-21T19:09:49.748686+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":780,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748039+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:09:49.748039+0000","last_peered":"2024-09-21T19:09:49.748039+0000","last_clean":"2024-09-21T19:09:49.748039+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:09:49.748039+0000","last_undegraded":"2024-09-21T19:09:49.748039+0000","last_fullsized":"2024-09-21T19:09:49.748039+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.611692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":779,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737646+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:09:49.737646+0000","last_peered":"2024-09-21T19:09:49.737646+0000","last_clean":"2024-09-21T19:09:49.737646+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:09:49.737646+0000","last_undegraded":"2024-09-21T19:09:49.737646+0000","last_fullsized":"2024-09-21T19:09:49.737646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":668,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741850+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:09:49.741850+0000","last_peered":"2024-09-21T19:09:49.741850+0000","last_clean":"2024-09-21T19:09:49.741850+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:09:49.741850+0000","last_undegraded":"2024-09-21T19:09:49.741850+0000","last_fullsized":"2024-09-21T19:09:49.741850+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":841,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.937582+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:09:48.937582+0000","last_peered":"2024-09-21T19:09:48.937582+0000","last_clean":"2024-09-21T19:09:48.937582+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:09:48.937582+0000","last_undegraded":"2024-09-21T19:09:48.937582+0000","last_fullsized":"2024-09-21T19:09:48.937582+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":208,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747354+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:09:49.747354+0000","last_peered":"2024-09-21T19:09:49.747354+0000","last_clean":"2024-09-21T19:09:49.747354+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:09:49.747354+0000","last_undegraded":"2024-09-21T19:09:49.747354+0000","last_fullsized":"2024-09-21T19:09:49.747354+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:13.700777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":807,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739402+0000","last_change":"2024-09-21T19:06:48.298726+0000","last_active":"2024-09-21T19:09:49.739402+0000","last_peered":"2024-09-21T19:09:49.739402+0000","last_clean":"2024-09-21T19:09:49.739402+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:09:49.739402+0000","last_undegraded":"2024-09-21T19:09:49.739402+0000","last_fullsized":"2024-09-21T19:09:49.739402+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:48.298519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.298519+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":753,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:50.708334+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:09:50.708334+0000","last_peered":"2024-09-21T19:09:50.708334+0000","last_clean":"2024-09-21T19:09:50.708334+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:09:50.708334+0000","last_undegraded":"2024-09-21T19:09:50.708334+0000","last_fullsized":"2024-09-21T19:09:50.708334+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.830842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":996,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748520+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:09:49.748520+0000","last_peered":"2024-09-21T19:09:49.748520+0000","last_clean":"2024-09-21T19:09:49.748520+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:09:49.748520+0000","last_undegraded":"2024-09-21T19:09:49.748520+0000","last_fullsized":"2024-09-21T19:09:49.748520+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":434,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738028+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:09:49.738028+0000","last_peered":"2024-09-21T19:09:49.738028+0000","last_clean":"2024-09-21T19:09:49.738028+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:09:49.738028+0000","last_undegraded":"2024-09-21T19:09:49.738028+0000","last_fullsized":"2024-09-21T19:09:49.738028+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:22.031875+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":534,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748596+0000","last_change":"2024-09-21T19:05:42.888622+0000","last_active":"2024-09-21T19:09:49.748596+0000","last_peered":"2024-09-21T19:09:49.748596+0000","last_clean":"2024-09-21T19:09:49.748596+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:09:49.748596+0000","last_undegraded":"2024-09-21T19:09:49.748596+0000","last_fullsized":"2024-09-21T19:09:49.748596+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:02:45.696468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:02:45.696468+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":842,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737109+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:09:49.737109+0000","last_peered":"2024-09-21T19:09:49.737109+0000","last_clean":"2024-09-21T19:09:49.737109+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:09:49.737109+0000","last_undegraded":"2024-09-21T19:09:49.737109+0000","last_fullsized":"2024-09-21T19:09:49.737109+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":767,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739275+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:09:49.739275+0000","last_peered":"2024-09-21T19:09:49.739275+0000","last_clean":"2024-09-21T19:09:49.739275+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:09:49.739275+0000","last_undegraded":"2024-09-21T19:09:49.739275+0000","last_fullsized":"2024-09-21T19:09:49.739275+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":745,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746814+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:09:49.746814+0000","last_peered":"2024-09-21T19:09:49.746814+0000","last_clean":"2024-09-21T19:09:49.746814+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:09:49.746814+0000","last_undegraded":"2024-09-21T19:09:49.746814+0000","last_fullsized":"2024-09-21T19:09:49.746814+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":762,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747814+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:09:49.747814+0000","last_peered":"2024-09-21T19:09:49.747814+0000","last_clean":"2024-09-21T19:09:49.747814+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:09:49.747814+0000","last_undegraded":"2024-09-21T19:09:49.747814+0000","last_fullsized":"2024-09-21T19:09:49.747814+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":525,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738537+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:09:49.738537+0000","last_peered":"2024-09-21T19:09:49.738537+0000","last_clean":"2024-09-21T19:09:49.738537+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:09:49.738537+0000","last_undegraded":"2024-09-21T19:09:49.738537+0000","last_fullsized":"2024-09-21T19:09:49.738537+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":813,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746909+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:09:49.746909+0000","last_peered":"2024-09-21T19:09:49.746909+0000","last_clean":"2024-09-21T19:09:49.746909+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:09:49.746909+0000","last_undegraded":"2024-09-21T19:09:49.746909+0000","last_fullsized":"2024-09-21T19:09:49.746909+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":636,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.742170+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:09:49.742170+0000","last_peered":"2024-09-21T19:09:49.742170+0000","last_clean":"2024-09-21T19:09:49.742170+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:09:49.742170+0000","last_undegraded":"2024-09-21T19:09:49.742170+0000","last_fullsized":"2024-09-21T19:09:49.742170+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":940,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737056+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:09:49.737056+0000","last_peered":"2024-09-21T19:09:49.737056+0000","last_clean":"2024-09-21T19:09:49.737056+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:09:49.737056+0000","last_undegraded":"2024-09-21T19:09:49.737056+0000","last_fullsized":"2024-09-21T19:09:49.737056+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":772,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747969+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:09:49.747969+0000","last_peered":"2024-09-21T19:09:49.747969+0000","last_clean":"2024-09-21T19:09:49.747969+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:09:49.747969+0000","last_undegraded":"2024-09-21T19:09:49.747969+0000","last_fullsized":"2024-09-21T19:09:49.747969+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.662781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":547,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.937690+0000","last_change":"2024-09-21T19:03:55.615837+0000","last_active":"2024-09-21T19:09:48.937690+0000","last_peered":"2024-09-21T19:09:48.937690+0000","last_clean":"2024-09-21T19:09:48.937690+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:09:48.937690+0000","last_undegraded":"2024-09-21T19:09:48.937690+0000","last_fullsized":"2024-09-21T19:09:48.937690+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:55.615772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.615772+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":798,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738080+0000","last_change":"2024-09-21T19:09:39.038158+0000","last_active":"2024-09-21T19:09:49.738080+0000","last_peered":"2024-09-21T19:09:49.738080+0000","last_clean":"2024-09-21T19:09:49.738080+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:09:49.738080+0000","last_undegraded":"2024-09-21T19:09:49.738080+0000","last_fullsized":"2024-09-21T19:09:49.738080+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:09:39.038064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.038064+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.698592+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":309,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738474+0000","last_change":"2024-09-21T19:06:54.691513+0000","last_active":"2024-09-21T19:09:49.738474+0000","last_peered":"2024-09-21T19:09:49.738474+0000","last_clean":"2024-09-21T19:09:49.738474+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:09:49.738474+0000","last_undegraded":"2024-09-21T19:09:49.738474+0000","last_fullsized":"2024-09-21T19:09:49.738474+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:34.156633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:34.156633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":823,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741314+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:09:49.741314+0000","last_peered":"2024-09-21T19:09:49.741314+0000","last_clean":"2024-09-21T19:09:49.741314+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:09:49.741314+0000","last_undegraded":"2024-09-21T19:09:49.741314+0000","last_fullsized":"2024-09-21T19:09:49.741314+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":647,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740563+0000","last_change":"2024-09-21T19:09:47.129430+0000","last_active":"2024-09-21T19:09:49.740563+0000","last_peered":"2024-09-21T19:09:49.740563+0000","last_clean":"2024-09-21T19:09:49.740563+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:09:49.740563+0000","last_undegraded":"2024-09-21T19:09:49.740563+0000","last_fullsized":"2024-09-21T19:09:49.740563+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:47.129379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:47.129379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.229738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":525,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.934392+0000","last_change":"2024-09-21T19:03:57.653062+0000","last_active":"2024-09-21T19:09:48.934392+0000","last_peered":"2024-09-21T19:09:48.934392+0000","last_clean":"2024-09-21T19:09:48.934392+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:09:48.934392+0000","last_undegraded":"2024-09-21T19:09:48.934392+0000","last_fullsized":"2024-09-21T19:09:48.934392+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:03:57.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:03:57.652944+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":684,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741456+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:09:49.741456+0000","last_peered":"2024-09-21T19:09:49.741456+0000","last_clean":"2024-09-21T19:09:49.741456+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:09:49.741456+0000","last_undegraded":"2024-09-21T19:09:49.741456+0000","last_fullsized":"2024-09-21T19:09:49.741456+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":786,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737913+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:09:49.737913+0000","last_peered":"2024-09-21T19:09:49.737913+0000","last_clean":"2024-09-21T19:09:49.737913+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:09:49.737913+0000","last_undegraded":"2024-09-21T19:09:49.737913+0000","last_fullsized":"2024-09-21T19:09:49.737913+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.451996+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":660,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.933769+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:09:48.933769+0000","last_peered":"2024-09-21T19:09:48.933769+0000","last_clean":"2024-09-21T19:09:48.933769+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:09:48.933769+0000","last_undegraded":"2024-09-21T19:09:48.933769+0000","last_fullsized":"2024-09-21T19:09:48.933769+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":765,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737450+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:09:49.737450+0000","last_peered":"2024-09-21T19:09:49.737450+0000","last_clean":"2024-09-21T19:09:49.737450+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:09:49.737450+0000","last_undegraded":"2024-09-21T19:09:49.737450+0000","last_fullsized":"2024-09-21T19:09:49.737450+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":503,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740564+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:09:49.740564+0000","last_peered":"2024-09-21T19:09:49.740564+0000","last_clean":"2024-09-21T19:09:49.740564+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:09:49.740564+0000","last_undegraded":"2024-09-21T19:09:49.740564+0000","last_fullsized":"2024-09-21T19:09:49.740564+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1076,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:52.655060+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:09:52.655060+0000","last_peered":"2024-09-21T19:09:52.655060+0000","last_clean":"2024-09-21T19:09:52.655060+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:09:52.655060+0000","last_undegraded":"2024-09-21T19:09:52.655060+0000","last_fullsized":"2024-09-21T19:09:52.655060+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.313979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":786,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747373+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:09:49.747373+0000","last_peered":"2024-09-21T19:09:49.747373+0000","last_clean":"2024-09-21T19:09:49.747373+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:09:49.747373+0000","last_undegraded":"2024-09-21T19:09:49.747373+0000","last_fullsized":"2024-09-21T19:09:49.747373+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":333,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738345+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:09:49.738345+0000","last_peered":"2024-09-21T19:09:49.738345+0000","last_clean":"2024-09-21T19:09:49.738345+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:09:49.738345+0000","last_undegraded":"2024-09-21T19:09:49.738345+0000","last_fullsized":"2024-09-21T19:09:49.738345+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":632,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:51.157757+0000","last_change":"2024-09-21T19:09:51.157757+0000","last_active":"2024-09-21T19:09:51.157757+0000","last_peered":"2024-09-21T19:09:51.157757+0000","last_clean":"2024-09-21T19:09:51.157757+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:09:51.157757+0000","last_undegraded":"2024-09-21T19:09:51.157757+0000","last_fullsized":"2024-09-21T19:09:51.157757+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.157711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.157711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:54.408363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":733,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740012+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:09:49.740012+0000","last_peered":"2024-09-21T19:09:49.740012+0000","last_clean":"2024-09-21T19:09:49.740012+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:09:49.740012+0000","last_undegraded":"2024-09-21T19:09:49.740012+0000","last_fullsized":"2024-09-21T19:09:49.740012+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":838,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747401+0000","last_change":"2024-09-21T19:08:53.739015+0000","last_active":"2024-09-21T19:09:49.747401+0000","last_peered":"2024-09-21T19:09:49.747401+0000","last_clean":"2024-09-21T19:09:49.747401+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:09:49.747401+0000","last_undegraded":"2024-09-21T19:09:49.747401+0000","last_fullsized":"2024-09-21T19:09:49.747401+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:08:53.738929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.738929+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:55.112735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":251,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739052+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:09:49.739052+0000","last_peered":"2024-09-21T19:09:49.739052+0000","last_clean":"2024-09-21T19:09:49.739052+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:09:49.739052+0000","last_undegraded":"2024-09-21T19:09:49.739052+0000","last_fullsized":"2024-09-21T19:09:49.739052+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":808,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737908+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:09:49.737908+0000","last_peered":"2024-09-21T19:09:49.737908+0000","last_clean":"2024-09-21T19:09:49.737908+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:09:49.737908+0000","last_undegraded":"2024-09-21T19:09:49.737908+0000","last_fullsized":"2024-09-21T19:09:49.737908+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":202,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747233+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:09:49.747233+0000","last_peered":"2024-09-21T19:09:49.747233+0000","last_clean":"2024-09-21T19:09:49.747233+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:09:49.747233+0000","last_undegraded":"2024-09-21T19:09:49.747233+0000","last_fullsized":"2024-09-21T19:09:49.747233+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.753652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":208,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747960+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:09:49.747960+0000","last_peered":"2024-09-21T19:09:49.747960+0000","last_clean":"2024-09-21T19:09:49.747960+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:09:49.747960+0000","last_undegraded":"2024-09-21T19:09:49.747960+0000","last_fullsized":"2024-09-21T19:09:49.747960+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.374847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":762,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746907+0000","last_change":"2024-09-21T19:09:37.588463+0000","last_active":"2024-09-21T19:09:49.746907+0000","last_peered":"2024-09-21T19:09:49.746907+0000","last_clean":"2024-09-21T19:09:49.746907+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:09:49.746907+0000","last_undegraded":"2024-09-21T19:09:49.746907+0000","last_fullsized":"2024-09-21T19:09:49.746907+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:09:37.588390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.588390+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.015172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":544,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739828+0000","last_change":"2024-09-21T19:09:08.156345+0000","last_active":"2024-09-21T19:09:49.739828+0000","last_peered":"2024-09-21T19:09:49.739828+0000","last_clean":"2024-09-21T19:09:49.739828+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:09:49.739828+0000","last_undegraded":"2024-09-21T19:09:49.739828+0000","last_fullsized":"2024-09-21T19:09:49.739828+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:08.156258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:08.156258+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.996519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":746,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.742012+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:09:49.742012+0000","last_peered":"2024-09-21T19:09:49.742012+0000","last_clean":"2024-09-21T19:09:49.742012+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:09:49.742012+0000","last_undegraded":"2024-09-21T19:09:49.742012+0000","last_fullsized":"2024-09-21T19:09:49.742012+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":632,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741065+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:09:49.741065+0000","last_peered":"2024-09-21T19:09:49.741065+0000","last_clean":"2024-09-21T19:09:49.741065+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:09:49.741065+0000","last_undegraded":"2024-09-21T19:09:49.741065+0000","last_fullsized":"2024-09-21T19:09:49.741065+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1775,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:50.816076+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:09:50.816076+0000","last_peered":"2024-09-21T19:09:50.816076+0000","last_clean":"2024-09-21T19:09:50.816076+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:09:50.816076+0000","last_undegraded":"2024-09-21T19:09:50.816076+0000","last_fullsized":"2024-09-21T19:09:50.816076+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.664705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":993,"num_read_kb":988,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":704,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741831+0000","last_change":"2024-09-21T19:06:54.695751+0000","last_active":"2024-09-21T19:09:49.741831+0000","last_peered":"2024-09-21T19:09:49.741831+0000","last_clean":"2024-09-21T19:09:49.741831+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:09:49.741831+0000","last_undegraded":"2024-09-21T19:09:49.741831+0000","last_fullsized":"2024-09-21T19:09:49.741831+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:06.242382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:06.242382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":759,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737478+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:09:49.737478+0000","last_peered":"2024-09-21T19:09:49.737478+0000","last_clean":"2024-09-21T19:09:49.737478+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:09:49.737478+0000","last_undegraded":"2024-09-21T19:09:49.737478+0000","last_fullsized":"2024-09-21T19:09:49.737478+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":932,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.934149+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:09:48.934149+0000","last_peered":"2024-09-21T19:09:48.934149+0000","last_clean":"2024-09-21T19:09:48.934149+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:09:48.934149+0000","last_undegraded":"2024-09-21T19:09:48.934149+0000","last_fullsized":"2024-09-21T19:09:48.934149+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":768,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740207+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:09:49.740207+0000","last_peered":"2024-09-21T19:09:49.740207+0000","last_clean":"2024-09-21T19:09:49.740207+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:09:49.740207+0000","last_undegraded":"2024-09-21T19:09:49.740207+0000","last_fullsized":"2024-09-21T19:09:49.740207+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":852,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748885+0000","last_change":"2024-09-21T19:09:46.543580+0000","last_active":"2024-09-21T19:09:49.748885+0000","last_peered":"2024-09-21T19:09:49.748885+0000","last_clean":"2024-09-21T19:09:49.748885+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:09:49.748885+0000","last_undegraded":"2024-09-21T19:09:49.748885+0000","last_fullsized":"2024-09-21T19:09:49.748885+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:46.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:46.543525+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.937610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":768,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748114+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:09:49.748114+0000","last_peered":"2024-09-21T19:09:49.748114+0000","last_clean":"2024-09-21T19:09:49.748114+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:09:49.748114+0000","last_undegraded":"2024-09-21T19:09:49.748114+0000","last_fullsized":"2024-09-21T19:09:49.748114+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:15.623405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":295,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738637+0000","last_change":"2024-09-21T19:06:54.708291+0000","last_active":"2024-09-21T19:09:49.738637+0000","last_peered":"2024-09-21T19:09:49.738637+0000","last_clean":"2024-09-21T19:09:49.738637+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:09:49.738637+0000","last_undegraded":"2024-09-21T19:09:49.738637+0000","last_fullsized":"2024-09-21T19:09:49.738637+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:01:10.038257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:01:10.038257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":806,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.749084+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:09:49.749084+0000","last_peered":"2024-09-21T19:09:49.749084+0000","last_clean":"2024-09-21T19:09:49.749084+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:09:49.749084+0000","last_undegraded":"2024-09-21T19:09:49.749084+0000","last_fullsized":"2024-09-21T19:09:49.749084+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":774,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748312+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:09:49.748312+0000","last_peered":"2024-09-21T19:09:49.748312+0000","last_clean":"2024-09-21T19:09:49.748312+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:09:49.748312+0000","last_undegraded":"2024-09-21T19:09:49.748312+0000","last_fullsized":"2024-09-21T19:09:49.748312+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.749277+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:09:49.749277+0000","last_peered":"2024-09-21T19:09:49.749277+0000","last_clean":"2024-09-21T19:09:49.749277+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:09:49.749277+0000","last_undegraded":"2024-09-21T19:09:49.749277+0000","last_fullsized":"2024-09-21T19:09:49.749277+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":739,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738655+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:09:49.738655+0000","last_peered":"2024-09-21T19:09:49.738655+0000","last_clean":"2024-09-21T19:09:49.738655+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:09:49.738655+0000","last_undegraded":"2024-09-21T19:09:49.738655+0000","last_fullsized":"2024-09-21T19:09:49.738655+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":204,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746969+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:09:49.746969+0000","last_peered":"2024-09-21T19:09:49.746969+0000","last_clean":"2024-09-21T19:09:49.746969+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:09:49.746969+0000","last_undegraded":"2024-09-21T19:09:49.746969+0000","last_fullsized":"2024-09-21T19:09:49.746969+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.739272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":855,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748436+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:09:49.748436+0000","last_peered":"2024-09-21T19:09:49.748436+0000","last_clean":"2024-09-21T19:09:49.748436+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:09:49.748436+0000","last_undegraded":"2024-09-21T19:09:49.748436+0000","last_fullsized":"2024-09-21T19:09:49.748436+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":671,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.934038+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:09:48.934038+0000","last_peered":"2024-09-21T19:09:48.934038+0000","last_clean":"2024-09-21T19:09:48.934038+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:09:48.934038+0000","last_undegraded":"2024-09-21T19:09:48.934038+0000","last_fullsized":"2024-09-21T19:09:48.934038+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":195,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.933944+0000","last_change":"2024-09-21T19:05:37.837352+0000","last_active":"2024-09-21T19:09:48.933944+0000","last_peered":"2024-09-21T19:09:48.933944+0000","last_clean":"2024-09-21T19:09:48.933944+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:09:48.933944+0000","last_undegraded":"2024-09-21T19:09:48.933944+0000","last_fullsized":"2024-09-21T19:09:48.933944+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:42.617402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:42.617402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":205,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748006+0000","last_change":"2024-09-21T19:09:31.740952+0000","last_active":"2024-09-21T19:09:49.748006+0000","last_peered":"2024-09-21T19:09:49.748006+0000","last_clean":"2024-09-21T19:09:49.748006+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:09:49.748006+0000","last_undegraded":"2024-09-21T19:09:49.748006+0000","last_fullsized":"2024-09-21T19:09:49.748006+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:31.740891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:31.740891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:40.289541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":860,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748508+0000","last_change":"2024-09-21T19:09:39.566152+0000","last_active":"2024-09-21T19:09:49.748508+0000","last_peered":"2024-09-21T19:09:49.748508+0000","last_clean":"2024-09-21T19:09:49.748508+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:09:49.748508+0000","last_undegraded":"2024-09-21T19:09:49.748508+0000","last_fullsized":"2024-09-21T19:09:49.748508+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:39.565921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.565921+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:44.360587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":781,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748778+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:09:49.748778+0000","last_peered":"2024-09-21T19:09:49.748778+0000","last_clean":"2024-09-21T19:09:49.748778+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:09:49.748778+0000","last_undegraded":"2024-09-21T19:09:49.748778+0000","last_fullsized":"2024-09-21T19:09:49.748778+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":781,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738881+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:09:49.738881+0000","last_peered":"2024-09-21T19:09:49.738881+0000","last_clean":"2024-09-21T19:09:49.738881+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:09:49.738881+0000","last_undegraded":"2024-09-21T19:09:49.738881+0000","last_fullsized":"2024-09-21T19:09:49.738881+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":257,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748125+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:09:49.748125+0000","last_peered":"2024-09-21T19:09:49.748125+0000","last_clean":"2024-09-21T19:09:49.748125+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:09:49.748125+0000","last_undegraded":"2024-09-21T19:09:49.748125+0000","last_fullsized":"2024-09-21T19:09:49.748125+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:19.687093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":754,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748339+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:09:49.748339+0000","last_peered":"2024-09-21T19:09:49.748339+0000","last_clean":"2024-09-21T19:09:49.748339+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:09:49.748339+0000","last_undegraded":"2024-09-21T19:09:49.748339+0000","last_fullsized":"2024-09-21T19:09:49.748339+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":643,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741571+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:09:49.741571+0000","last_peered":"2024-09-21T19:09:49.741571+0000","last_clean":"2024-09-21T19:09:49.741571+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:09:49.741571+0000","last_undegraded":"2024-09-21T19:09:49.741571+0000","last_fullsized":"2024-09-21T19:09:49.741571+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":742,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739550+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:09:49.739550+0000","last_peered":"2024-09-21T19:09:49.739550+0000","last_clean":"2024-09-21T19:09:49.739550+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:09:49.739550+0000","last_undegraded":"2024-09-21T19:09:49.739550+0000","last_fullsized":"2024-09-21T19:09:49.739550+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":197,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747253+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:09:49.747253+0000","last_peered":"2024-09-21T19:09:49.747253+0000","last_clean":"2024-09-21T19:09:49.747253+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:09:49.747253+0000","last_undegraded":"2024-09-21T19:09:49.747253+0000","last_fullsized":"2024-09-21T19:09:49.747253+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:10.305625+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":638,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740947+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:09:49.740947+0000","last_peered":"2024-09-21T19:09:49.740947+0000","last_clean":"2024-09-21T19:09:49.740947+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:09:49.740947+0000","last_undegraded":"2024-09-21T19:09:49.740947+0000","last_fullsized":"2024-09-21T19:09:49.740947+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":690,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.933953+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:09:48.933953+0000","last_peered":"2024-09-21T19:09:48.933953+0000","last_clean":"2024-09-21T19:09:48.933953+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:09:48.933953+0000","last_undegraded":"2024-09-21T19:09:48.933953+0000","last_fullsized":"2024-09-21T19:09:48.933953+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":773,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747318+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:09:49.747318+0000","last_peered":"2024-09-21T19:09:49.747318+0000","last_clean":"2024-09-21T19:09:49.747318+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:09:49.747318+0000","last_undegraded":"2024-09-21T19:09:49.747318+0000","last_fullsized":"2024-09-21T19:09:49.747318+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.612669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":208,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747743+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:09:49.747743+0000","last_peered":"2024-09-21T19:09:49.747743+0000","last_clean":"2024-09-21T19:09:49.747743+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:09:49.747743+0000","last_undegraded":"2024-09-21T19:09:49.747743+0000","last_fullsized":"2024-09-21T19:09:49.747743+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":753,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748820+0000","last_change":"2024-09-21T19:09:35.677140+0000","last_active":"2024-09-21T19:09:49.748820+0000","last_peered":"2024-09-21T19:09:49.748820+0000","last_clean":"2024-09-21T19:09:49.748820+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:09:49.748820+0000","last_undegraded":"2024-09-21T19:09:49.748820+0000","last_fullsized":"2024-09-21T19:09:49.748820+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:35.677043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:35.677043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:50.521879+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":810,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747052+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:09:49.747052+0000","last_peered":"2024-09-21T19:09:49.747052+0000","last_clean":"2024-09-21T19:09:49.747052+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:09:49.747052+0000","last_undegraded":"2024-09-21T19:09:49.747052+0000","last_fullsized":"2024-09-21T19:09:49.747052+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":775,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747313+0000","last_change":"2024-09-21T19:09:41.535785+0000","last_active":"2024-09-21T19:09:49.747313+0000","last_peered":"2024-09-21T19:09:49.747313+0000","last_clean":"2024-09-21T19:09:49.747313+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:09:49.747313+0000","last_undegraded":"2024-09-21T19:09:49.747313+0000","last_fullsized":"2024-09-21T19:09:49.747313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:41.535714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:41.535714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:45.775155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":197,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.746957+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:09:49.746957+0000","last_peered":"2024-09-21T19:09:49.746957+0000","last_clean":"2024-09-21T19:09:49.746957+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:09:49.746957+0000","last_undegraded":"2024-09-21T19:09:49.746957+0000","last_fullsized":"2024-09-21T19:09:49.746957+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:00.385824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":733,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738387+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:09:49.738387+0000","last_peered":"2024-09-21T19:09:49.738387+0000","last_clean":"2024-09-21T19:09:49.738387+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:09:49.738387+0000","last_undegraded":"2024-09-21T19:09:49.738387+0000","last_fullsized":"2024-09-21T19:09:49.738387+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":366,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748587+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:09:49.748587+0000","last_peered":"2024-09-21T19:09:49.748587+0000","last_clean":"2024-09-21T19:09:49.748587+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:09:49.748587+0000","last_undegraded":"2024-09-21T19:09:49.748587+0000","last_fullsized":"2024-09-21T19:09:49.748587+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:17.636097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":789,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747789+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:09:49.747789+0000","last_peered":"2024-09-21T19:09:49.747789+0000","last_clean":"2024-09-21T19:09:49.747789+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:09:49.747789+0000","last_undegraded":"2024-09-21T19:09:49.747789+0000","last_fullsized":"2024-09-21T19:09:49.747789+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:31.474930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":253,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737245+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:09:49.737245+0000","last_peered":"2024-09-21T19:09:49.737245+0000","last_clean":"2024-09-21T19:09:49.737245+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:09:49.737245+0000","last_undegraded":"2024-09-21T19:09:49.737245+0000","last_fullsized":"2024-09-21T19:09:49.737245+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":767,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.933767+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:09:48.933767+0000","last_peered":"2024-09-21T19:09:48.933767+0000","last_clean":"2024-09-21T19:09:48.933767+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:09:48.933767+0000","last_undegraded":"2024-09-21T19:09:48.933767+0000","last_fullsized":"2024-09-21T19:09:48.933767+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":702,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.933577+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:09:48.933577+0000","last_peered":"2024-09-21T19:09:48.933577+0000","last_clean":"2024-09-21T19:09:48.933577+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:09:48.933577+0000","last_undegraded":"2024-09-21T19:09:48.933577+0000","last_fullsized":"2024-09-21T19:09:48.933577+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":766,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739109+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:09:49.739109+0000","last_peered":"2024-09-21T19:09:49.739109+0000","last_clean":"2024-09-21T19:09:49.739109+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:09:49.739109+0000","last_undegraded":"2024-09-21T19:09:49.739109+0000","last_fullsized":"2024-09-21T19:09:49.739109+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":302,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739922+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:09:49.739922+0000","last_peered":"2024-09-21T19:09:49.739922+0000","last_clean":"2024-09-21T19:09:49.739922+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:09:49.739922+0000","last_undegraded":"2024-09-21T19:09:49.739922+0000","last_fullsized":"2024-09-21T19:09:49.739922+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":741,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747405+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:09:49.747405+0000","last_peered":"2024-09-21T19:09:49.747405+0000","last_clean":"2024-09-21T19:09:49.747405+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:09:49.747405+0000","last_undegraded":"2024-09-21T19:09:49.747405+0000","last_fullsized":"2024-09-21T19:09:49.747405+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":642,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741949+0000","last_change":"2024-09-21T19:02:39.608305+0000","last_active":"2024-09-21T19:09:49.741949+0000","last_peered":"2024-09-21T19:09:49.741949+0000","last_clean":"2024-09-21T19:09:49.741949+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:09:49.741949+0000","last_undegraded":"2024-09-21T19:09:49.741949+0000","last_fullsized":"2024-09-21T19:09:49.741949+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-21T19:02:39.608224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:02:39.608224+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":754,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748713+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:09:49.748713+0000","last_peered":"2024-09-21T19:09:49.748713+0000","last_clean":"2024-09-21T19:09:49.748713+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:09:49.748713+0000","last_undegraded":"2024-09-21T19:09:49.748713+0000","last_fullsized":"2024-09-21T19:09:49.748713+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:16.645472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":736,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738880+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:09:49.738880+0000","last_peered":"2024-09-21T19:09:49.738880+0000","last_clean":"2024-09-21T19:09:49.738880+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:09:49.738880+0000","last_undegraded":"2024-09-21T19:09:49.738880+0000","last_fullsized":"2024-09-21T19:09:49.738880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":197,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747661+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:09:49.747661+0000","last_peered":"2024-09-21T19:09:49.747661+0000","last_clean":"2024-09-21T19:09:49.747661+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:09:49.747661+0000","last_undegraded":"2024-09-21T19:09:49.747661+0000","last_fullsized":"2024-09-21T19:09:49.747661+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":355,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747073+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:09:49.747073+0000","last_peered":"2024-09-21T19:09:49.747073+0000","last_clean":"2024-09-21T19:09:49.747073+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:09:49.747073+0000","last_undegraded":"2024-09-21T19:09:49.747073+0000","last_fullsized":"2024-09-21T19:09:49.747073+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:29.639722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":640,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741226+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:09:49.741226+0000","last_peered":"2024-09-21T19:09:49.741226+0000","last_clean":"2024-09-21T19:09:49.741226+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:09:49.741226+0000","last_undegraded":"2024-09-21T19:09:49.741226+0000","last_fullsized":"2024-09-21T19:09:49.741226+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":978,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:52.767913+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:09:52.767913+0000","last_peered":"2024-09-21T19:09:52.767913+0000","last_clean":"2024-09-21T19:09:52.767913+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:09:52.767913+0000","last_undegraded":"2024-09-21T19:09:52.767913+0000","last_fullsized":"2024-09-21T19:09:52.767913+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:12.049319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":756,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.749277+0000","last_change":"2024-09-21T19:06:54.699473+0000","last_active":"2024-09-21T19:09:49.749277+0000","last_peered":"2024-09-21T19:09:49.749277+0000","last_clean":"2024-09-21T19:09:49.749277+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:09:49.749277+0000","last_undegraded":"2024-09-21T19:09:49.749277+0000","last_fullsized":"2024-09-21T19:09:49.749277+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:36.651381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:36.651381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":763,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.739233+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:09:49.739233+0000","last_peered":"2024-09-21T19:09:49.739233+0000","last_clean":"2024-09-21T19:09:49.739233+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:09:49.739233+0000","last_undegraded":"2024-09-21T19:09:49.739233+0000","last_fullsized":"2024-09-21T19:09:49.739233+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":765,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748274+0000","last_change":"2024-09-21T19:09:32.615127+0000","last_active":"2024-09-21T19:09:49.748274+0000","last_peered":"2024-09-21T19:09:49.748274+0000","last_clean":"2024-09-21T19:09:49.748274+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:09:49.748274+0000","last_undegraded":"2024-09-21T19:09:49.748274+0000","last_fullsized":"2024-09-21T19:09:49.748274+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:32.615041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:32.615041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.657148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":419,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737973+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:09:49.737973+0000","last_peered":"2024-09-21T19:09:49.737973+0000","last_clean":"2024-09-21T19:09:49.737973+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:09:49.737973+0000","last_undegraded":"2024-09-21T19:09:49.737973+0000","last_fullsized":"2024-09-21T19:09:49.737973+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":631,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.742375+0000","last_change":"2024-09-21T19:02:47.717512+0000","last_active":"2024-09-21T19:09:49.742375+0000","last_peered":"2024-09-21T19:09:49.742375+0000","last_clean":"2024-09-21T19:09:49.742375+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:09:49.742375+0000","last_undegraded":"2024-09-21T19:09:49.742375+0000","last_fullsized":"2024-09-21T19:09:49.742375+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:47.717433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:47.717433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":836,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747094+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:09:49.747094+0000","last_peered":"2024-09-21T19:09:49.747094+0000","last_clean":"2024-09-21T19:09:49.747094+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:09:49.747094+0000","last_undegraded":"2024-09-21T19:09:49.747094+0000","last_fullsized":"2024-09-21T19:09:49.747094+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":755,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.748959+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:09:49.748959+0000","last_peered":"2024-09-21T19:09:49.748959+0000","last_clean":"2024-09-21T19:09:49.748959+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:09:49.748959+0000","last_undegraded":"2024-09-21T19:09:49.748959+0000","last_fullsized":"2024-09-21T19:09:49.748959+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":763,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737871+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:09:49.737871+0000","last_peered":"2024-09-21T19:09:49.737871+0000","last_clean":"2024-09-21T19:09:49.737871+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:09:49.737871+0000","last_undegraded":"2024-09-21T19:09:49.737871+0000","last_fullsized":"2024-09-21T19:09:49.737871+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:37.041685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737095+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:09:49.737095+0000","last_peered":"2024-09-21T19:09:49.737095+0000","last_clean":"2024-09-21T19:09:49.737095+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:09:49.737095+0000","last_undegraded":"2024-09-21T19:09:49.737095+0000","last_fullsized":"2024-09-21T19:09:49.737095+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":834,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738189+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:09:49.738189+0000","last_peered":"2024-09-21T19:09:49.738189+0000","last_clean":"2024-09-21T19:09:49.738189+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:09:49.738189+0000","last_undegraded":"2024-09-21T19:09:49.738189+0000","last_fullsized":"2024-09-21T19:09:49.738189+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:09:52.278173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747171+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:09:49.747171+0000","last_peered":"2024-09-21T19:09:49.747171+0000","last_clean":"2024-09-21T19:09:49.747171+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:09:49.747171+0000","last_undegraded":"2024-09-21T19:09:49.747171+0000","last_fullsized":"2024-09-21T19:09:49.747171+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":759,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738654+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:09:49.738654+0000","last_peered":"2024-09-21T19:09:49.738654+0000","last_clean":"2024-09-21T19:09:49.738654+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:09:49.738654+0000","last_undegraded":"2024-09-21T19:09:49.738654+0000","last_fullsized":"2024-09-21T19:09:49.738654+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":651,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.740587+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:09:49.740587+0000","last_peered":"2024-09-21T19:09:49.740587+0000","last_clean":"2024-09-21T19:09:49.740587+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:09:49.740587+0000","last_undegraded":"2024-09-21T19:09:49.740587+0000","last_fullsized":"2024-09-21T19:09:49.740587+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":783,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737301+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:09:49.737301+0000","last_peered":"2024-09-21T19:09:49.737301+0000","last_clean":"2024-09-21T19:09:49.737301+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:09:49.737301+0000","last_undegraded":"2024-09-21T19:09:49.737301+0000","last_fullsized":"2024-09-21T19:09:49.737301+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":696,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741594+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:09:49.741594+0000","last_peered":"2024-09-21T19:09:49.741594+0000","last_clean":"2024-09-21T19:09:49.741594+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:09:49.741594+0000","last_undegraded":"2024-09-21T19:09:49.741594+0000","last_fullsized":"2024-09-21T19:09:49.741594+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":778,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:51.462137+0000","last_change":"2024-09-21T19:09:51.462137+0000","last_active":"2024-09-21T19:09:51.462137+0000","last_peered":"2024-09-21T19:09:51.462137+0000","last_clean":"2024-09-21T19:09:51.462137+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:09:51.462137+0000","last_undegraded":"2024-09-21T19:09:51.462137+0000","last_fullsized":"2024-09-21T19:09:51.462137+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.462077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.604947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.738779+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:09:49.738779+0000","last_peered":"2024-09-21T19:09:49.738779+0000","last_clean":"2024-09-21T19:09:49.738779+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:09:49.738779+0000","last_undegraded":"2024-09-21T19:09:49.738779+0000","last_fullsized":"2024-09-21T19:09:49.738779+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":782,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737035+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:09:49.737035+0000","last_peered":"2024-09-21T19:09:49.737035+0000","last_clean":"2024-09-21T19:09:49.737035+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:09:49.737035+0000","last_undegraded":"2024-09-21T19:09:49.737035+0000","last_fullsized":"2024-09-21T19:09:49.737035+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":634,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.741998+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:09:49.741998+0000","last_peered":"2024-09-21T19:09:49.741998+0000","last_clean":"2024-09-21T19:09:49.741998+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:09:49.741998+0000","last_undegraded":"2024-09-21T19:09:49.741998+0000","last_fullsized":"2024-09-21T19:09:49.741998+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":669,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.933503+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:09:48.933503+0000","last_peered":"2024-09-21T19:09:48.933503+0000","last_clean":"2024-09-21T19:09:48.933503+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:09:48.933503+0000","last_undegraded":"2024-09-21T19:09:48.933503+0000","last_fullsized":"2024-09-21T19:09:48.933503+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":753,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.747538+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:09:49.747538+0000","last_peered":"2024-09-21T19:09:49.747538+0000","last_clean":"2024-09-21T19:09:49.747538+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:09:49.747538+0000","last_undegraded":"2024-09-21T19:09:49.747538+0000","last_fullsized":"2024-09-21T19:09:49.747538+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737437+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:09:49.737437+0000","last_peered":"2024-09-21T19:09:49.737437+0000","last_clean":"2024-09-21T19:09:49.737437+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:09:49.737437+0000","last_undegraded":"2024-09-21T19:09:49.737437+0000","last_fullsized":"2024-09-21T19:09:49.737437+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":774,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.737207+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:09:49.737207+0000","last_peered":"2024-09-21T19:09:49.737207+0000","last_clean":"2024-09-21T19:09:49.737207+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:09:49.737207+0000","last_undegraded":"2024-09-21T19:09:49.737207+0000","last_fullsized":"2024-09-21T19:09:49.737207+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":742,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-21T19:09:48.937613+0000","last_change":"2024-09-21T19:09:48.894774+0000","last_active":"2024-09-21T19:09:48.937613+0000","last_peered":"2024-09-21T19:09:48.937613+0000","last_clean":"2024-09-21T19:09:48.937613+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:09:48.937613+0000","last_undegraded":"2024-09-21T19:09:48.937613+0000","last_fullsized":"2024-09-21T19:09:48.937613+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:09:48.894708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.894708+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.651349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":655,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-21T19:09:49.742324+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:09:49.742324+0000","last_peered":"2024-09-21T19:09:49.742324+0000","last_clean":"2024-09-21T19:09:49.742324+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:09:49.742324+0000","last_undegraded":"2024-09-21T19:09:49.742324+0000","last_fullsized":"2024-09-21T19:09:49.742324+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":993,"num_read_kb":988,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266893,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":588580,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93156572,"statfs":{"total":95995035648,"available":95392329728,"internally_reserved":0,"allocated":2109440,"data_stored":1307753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":600570314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464343,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290560,"kb_used_data":1708,"kb_used_omap":4,"kb_used_meta":288827,"kb_avail":93454592,"statfs":{"total":95995035648,"available":95697502208,"internally_reserved":0,"allocated":1748992,"data_stored":965663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":295759524},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619045,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":272224,"kb_used_data":2308,"kb_used_omap":4,"kb_used_meta":269883,"kb_avail":93472928,"statfs":{"total":95995035648,"available":95716278272,"internally_reserved":0,"allocated":2363392,"data_stored":1569922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":276360867},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019099,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538272,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93206880,"statfs":{"total":95995035648,"available":95443845120,"internally_reserved":0,"allocated":1781760,"data_stored":980353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182624,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737404,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93007748,"statfs":{"total":95995035648,"available":95239933952,"internally_reserved":0,"allocated":2396160,"data_stored":1571240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560712,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93184440,"statfs":{"total":95995035648,"available":95420866560,"internally_reserved":0,"allocated":2023424,"data_stored":1242247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":677120,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":93068032,"statfs":{"total":95995035648,"available":95301664768,"internally_reserved":0,"allocated":1818624,"data_stored":980976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827040,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918112,"statfs":{"total":95995035648,"available":95148146688,"internally_reserved":0,"allocated":1785856,"data_stored":980762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:09:56.210 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:56.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: from='client.36437 v1:172.21.15.81:0/2501723884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: 3.5 scrub starts 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: 3.5 scrub ok 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: 1.7 scrub starts 2024-09-21T19:09:56.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[29384]: 1.7 scrub ok 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: from='client.36437 v1:172.21.15.81:0/2501723884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: 3.5 scrub starts 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: 3.5 scrub ok 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: 1.7 scrub starts 2024-09-21T19:09:56.424 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:56 smithi081 ceph-mon[39192]: 1.7 scrub ok 2024-09-21T19:09:56.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: from='client.36437 v1:172.21.15.81:0/2501723884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: 3.5 scrub starts 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: 3.5 scrub ok 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: 1.7 scrub starts 2024-09-21T19:09:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:56 smithi104 ceph-mon[32438]: 1.7 scrub ok 2024-09-21T19:09:56.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:56.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:56.674 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:09:56.674 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:09:56.674 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T19:09:56.674 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:09:56.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:56.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:56.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.827 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:56.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:56.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:56.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:56.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:57.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:57.073 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:57.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.202 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:57.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:57.213 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:09:57.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:57.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:57.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: 1.2 scrub starts 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: 1.2 scrub ok 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: pgmap v831: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 op/s 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: 6.16 scrub starts 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: 6.16 scrub ok 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T19:09:57.416 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1523250400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:57.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:09:57.450 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:09:57.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:57.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:09:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: 1.2 scrub starts 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: 1.2 scrub ok 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: pgmap v831: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 op/s 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: 6.16 scrub starts 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: 6.16 scrub ok 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T19:09:57.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1523250400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: 1.2 scrub starts 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: 1.2 scrub ok 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: pgmap v831: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 op/s 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: 6.16 scrub starts 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: 6.16 scrub ok 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T19:09:57.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1523250400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:09:57.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:57.643 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:09:57.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:57.696 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:57.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:57.819 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:57.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:57.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:58.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:58.067 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:58.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:58.190 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:58.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:58.315 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:58.319 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:09:58.319 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:09:58.319 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:09:58.319 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:09:58.320 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:09:58.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:09:58.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:09:58.467 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T19:09:58.467 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T19:09:58.467 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T19:09:58.468 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T19:09:58.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:58 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:09:58.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:58 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:09:58.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:58 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:58.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:58 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1523250400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:09:58.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:58 smithi104 ceph-mon[32438]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T19:09:58.529 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:09:58.529 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1523250400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[29384]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1523250400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:09:58.530 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:58 smithi081 ceph-mon[39192]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T19:09:58.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:58.562 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:58.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:58.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:58.757 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:58.786 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:58.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:58.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:59.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:09:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:09:59.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:09:59.055 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:59.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:09:59.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:09:59.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:59.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:59.202 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:09:59.213 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:09:59.280 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:09:59.301 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:09:59.312 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:09:59.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:09:59.417 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:09:59 smithi104 ceph-mon[32438]: pgmap v834: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:09:59.426 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:09:59.512 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:09:59 smithi081 ceph-mon[29384]: pgmap v834: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:09:59.512 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:09:59 smithi081 ceph-mon[39192]: pgmap v834: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:09:59.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:09:59.552 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:09:59.583 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:09:59.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:09:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.676 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:09:59.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:09:59.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235896 8.22 8:45adf4b4:::benchmark_data_smithi081_105538_object103664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 486.102713467, 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039943074, 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.781 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235896 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675143+0000", 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.729e-06 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675150+0000", 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6820000000000003e-06 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675168+0000", 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7875000000000001e-05 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675178+0000", 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5610000000000004e-06 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731748+0000", 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056570691999999999 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731754+0000", 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.976e-06 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.700981+0000", 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96922663099999995 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701110+0000", 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864500000000001 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701320+0000", 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021059200000000001 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702216+0000", 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089577400000000001 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702245+0000", 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8608000000000001e-05 2024-09-21T19:09:59.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.714979+0000", 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012734354999999999 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715038+0000", 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8588999999999998e-05 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715083+0000", 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5364999999999997e-05 2024-09-21T19:09:59.787 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235898 8.2 8:4337194e:::benchmark_data_smithi081_105538_object103666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 486.10266008899998, 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0181600850000001, 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.788 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235898 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675194+0000", 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1699999999999997e-07 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675197+0000", 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675203+0000", 2024-09-21T19:09:59.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2369999999999997e-06 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675208+0000", 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9989999999999999e-06 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731592+0000", 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056383961000000003 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731604+0000", 2024-09-21T19:09:59.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2278e-05 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685684+0000", 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95407945800000005 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685861+0000", 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017756400000000001 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.686082+0000", 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022110200000000001 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.687697+0000", 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016145230000000001 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.690382+0000", 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026848850000000001 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693261+0000", 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879291 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693314+0000", 2024-09-21T19:09:59.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2845e-05 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693353+0000", 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9369000000000002e-05 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237679 8.1b 8:deadaa1e:::benchmark_data_smithi081_105538_object105447:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:09:59.795 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 484.09320780899998, 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0964987129999999, 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237679 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684647+0000", 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.153e-06 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684651+0000", 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3370000000000001e-06 2024-09-21T19:09:59.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684660+0000", 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6470000000000004e-06 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684667+0000", 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684783+0000", 2024-09-21T19:09:59.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011565600000000001 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684791+0000", 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7809999999999994e-06 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691384+0000", 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0065922899999999 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691392+0000", 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.636e-06 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781063+0000", 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0896703569999999 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781144+0000", 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1546999999999997e-05 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237686 8.1b 8:df3678b5:::benchmark_data_smithi081_105538_object105454:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 484.093168752, 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.173105171, 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237686 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1299999999999999e-07 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684688+0000", 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5830000000000001e-06 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684693+0000", 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5910000000000003e-06 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684699+0000", 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684822+0000", 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012324499999999999 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684830+0000", 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6750000000000002e-06 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691420+0000", 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006590262 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691426+0000", 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4559999999999997e-06 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781412+0000", 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.089985687 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for missing object", 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781478+0000", 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6272000000000005e-05 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857711+0000", 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076232366999999995 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857790+0000", 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9029999999999994e-05 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.806 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237687 8.1b 8:dc29a777:::benchmark_data_smithi081_105538_object105455:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 484.09314841399998, 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1710659049999999, 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237687 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4899999999999995e-07 2024-09-21T19:09:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684708+0000", 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1339999999999998e-06 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684713+0000", 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9749999999999998e-06 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684734+0000", 2024-09-21T19:09:59.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1206e-05 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684861+0000", 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012699799999999999 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684870+0000", 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9770000000000006e-06 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691453+0000", 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006583418 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691459+0000", 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2580000000000001e-06 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781524+0000", 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0900641010000001 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781643+0000", 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119735 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781902+0000", 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025915999999999998 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784476+0000", 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025738229999999998 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784488+0000", 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1401e-05 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784513+0000", 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5462000000000001e-05 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855646+0000", 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071132476 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.813 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855686+0000", 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0157999999999997e-05 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855771+0000", 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5074000000000006e-05 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237708 8.1b 8:dbdbeb85:::benchmark_data_smithi081_105538_object105476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:09:59.814 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 484.073460818, 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.151669386, 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237708 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704394+0000", 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.855e-06 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704399+0000", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9699999999999998e-06 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704413+0000", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3560000000000001e-05 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704421+0000", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6399999999999997e-06 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704473+0000", 2024-09-21T19:09:59.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.27e-05 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704483+0000", 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4520000000000003e-06 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691484+0000", 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98700145800000005 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691491+0000", 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782825+0000", 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0913340810000001 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782935+0000", 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110647 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:09:59.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.783135+0000", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020000399999999999 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784491+0000", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013554319999999999 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784543+0000", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2609999999999999e-05 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784566+0000", 2024-09-21T19:09:59.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2945000000000001e-05 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855871+0000", 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071304949000000006 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855929+0000", 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7924999999999999e-05 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.856062+0000", 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013279700000000001 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237711 8.1b 8:dd96cf97:::benchmark_data_smithi081_105538_object105479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 484.07126123099999, 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1506975869999998, 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.822 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237711 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706594+0000", 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9099999999999999e-06 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706599+0000", 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0259999999999998e-06 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706618+0000", 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8499e-05 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706633+0000", 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5931999999999999e-05 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706751+0000", 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117395 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706762+0000", 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1316e-05 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691770+0000", 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.985007619 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691776+0000", 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5490000000000003e-06 2024-09-21T19:09:59.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784627+0000", 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0928504649999999 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784699+0000", 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2539000000000005e-05 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784817+0000", 2024-09-21T19:09:59.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118029 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785805+0000", 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098732000000000008 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785853+0000", 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8554000000000003e-05 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785945+0000", 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1835000000000004e-05 2024-09-21T19:09:59.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857195+0000", 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071249589000000002 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857226+0000", 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1705999999999998e-05 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857290+0000", 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3304000000000002e-05 2024-09-21T19:09:59.829 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2026 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94695648315648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 482.50546342500002, 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46135494399999999, 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:09:59.830 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2026 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9019999999999994e-06 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272421+0000", 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7499999999999998e-05 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0607e-05 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272504+0000", 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2151999999999996e-05 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272515+0000", 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1241e-05 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702573+0000", 2024-09-21T19:09:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.430058259 2024-09-21T19:09:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.834 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702717+0000", 2024-09-21T19:09:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.835 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:59.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:09:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143931 2024-09-21T19:09:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.733745+0000", 2024-09-21T19:09:59.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031027852000000002 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2027 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 482.50539572600002, 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.464594917, 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.837 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2027 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4000000000000001e-07 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272462+0000", 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4199999999999999e-06 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272469+0000", 2024-09-21T19:09:59.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272474+0000", 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4249999999999997e-06 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272539+0000", 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5066000000000003e-05 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272549+0000", 2024-09-21T19:09:59.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8709999999999992e-06 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704278+0000", 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43172896500000002 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704401+0000", 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012265999999999999 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.737053+0000", 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.032651612000000003 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2036 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 482.224035786, 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15105808700000001, 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.842 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2036 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:09:59.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0399999999999996e-07 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553821+0000", 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6769999999999999e-06 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553827+0000", 2024-09-21T19:09:59.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553831+0000", 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6550000000000003e-06 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553873+0000", 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1241e-05 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553879+0000", 2024-09-21T19:09:59.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5429999999999999e-06 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704657+0000", 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150777833 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704730+0000", 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3398000000000004e-05 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704876+0000", 2024-09-21T19:09:59.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014533600000000001 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76067 9.c e280/254 9:3b8d2f8b:::benchmark_data_smithi081_131274_object76066:head v 280'4754, mlcod=280'4748)", 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 396.432467782, 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23228328300000001, 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345387+0000", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5460000000000001e-06 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570553+0000", 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.22516550099999999 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570563+0000", 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0163000000000001e-05 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570569+0000", 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7660000000000004e-06 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570644+0000", 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5710999999999994e-05 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570678+0000", 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3608000000000001e-05 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577637+0000", 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069593909999999997 2024-09-21T19:09:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577669+0000", 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1597000000000003e-05 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29874 10.1 10:8233bd87:::benchmark_data_smithi081_155596_object29873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.816351+0000", 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 213.96150200299999, 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.117039637, 2024-09-21T19:09:59.851 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29874 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816351+0000", 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816351+0000", 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816352+0000", 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2999999999999999e-07 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816387+0000", 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5271999999999997e-05 2024-09-21T19:09:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816395+0000", 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1890000000000001e-06 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816400+0000", 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5899999999999998e-06 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816490+0000", 2024-09-21T19:09:59.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9506000000000007e-05 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816621+0000", 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000131712 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.817621+0000", 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00099927000000000002 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.820057+0000", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024366959999999999 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823504+0000", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034463850000000002 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823623+0000", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118751 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823676+0000", 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3559e-05 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933391+0000", 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109714877 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29876 10.1 10:8bb58828:::benchmark_data_smithi081_155596_object29875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.816825+0000", 2024-09-21T19:09:59.857 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 213.96102877800001, 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.116730545, 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29876 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816825+0000", 2024-09-21T19:09:59.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816825+0000", 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816826+0000", 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.42e-06 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816858+0000", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1702999999999998e-05 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816868+0000", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0596e-05 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.816875+0000", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9920000000000001e-06 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.818840+0000", 2024-09-21T19:09:59.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019643870000000002 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.818946+0000", 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106789 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.819064+0000", 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117364 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.821350+0000", 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022863219999999999 2024-09-21T19:09:59.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.823572+0000", 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022222549999999998 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933459+0000", 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.109886758 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933506+0000", 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7225999999999998e-05 2024-09-21T19:09:59.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933555+0000", 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8733000000000002e-05 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29887 10.1 10:85f2786f:::benchmark_data_smithi081_155596_object29886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 213.953747862, 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12143838799999999, 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.863 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29887 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824107+0000", 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.767e-06 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824151+0000", 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4229e-05 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933322+0000", 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10917025 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933345+0000", 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3669999999999999e-05 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934070+0000", 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072502099999999996 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934269+0000", 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867600000000001 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934418+0000", 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014940099999999999 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.940623+0000", 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062042729999999997 2024-09-21T19:09:59.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.940648+0000", 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5347999999999999e-05 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945384+0000", 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047363209999999999 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945492+0000", 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107899 2024-09-21T19:09:59.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945544+0000", 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1533000000000003e-05 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31276 10.e 10:78850574:::benchmark_data_smithi081_155596_object31275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 213.16653153999999, 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20212575599999999, 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.869 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31276 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611324+0000", 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9740000000000001e-06 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611355+0000", 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1634999999999998e-05 2024-09-21T19:09:59.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611367+0000", 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1613999999999999e-05 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611374+0000", 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8029999999999999e-06 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611557+0000", 2024-09-21T19:09:59.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018305700000000001 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611683+0000", 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012569999999999999 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611833+0000", 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015060799999999999 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.613723+0000", 2024-09-21T19:09:59.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00188988 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614412+0000", 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068917700000000002 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813312+0000", 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198899348 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.874 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813403+0000", 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1162000000000006e-05 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813448+0000", 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4798e-05 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31282 10.e 10:7508a8e6:::benchmark_data_smithi081_155596_object31281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:09:59.875 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 213.16349718000001, 2024-09-21T19:09:59.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19924375599999999, 2024-09-21T19:09:59.876 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.876 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.876 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.877 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.877 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31282 2024-09-21T19:09:59.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614359+0000", 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4099999999999998e-06 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614395+0000", 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6782999999999997e-05 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614409+0000", 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3772e-05 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614417+0000", 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9030000000000004e-06 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614476+0000", 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8980000000000001e-05 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614640+0000", 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016440500000000001 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614800+0000", 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015944200000000001 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.615994+0000", 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001193784 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.616143+0000", 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000149013 2024-09-21T19:09:59.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813485+0000", 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.197342778 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813565+0000", 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9160999999999996e-05 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813600+0000", 2024-09-21T19:09:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5324999999999998e-05 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31755 10.1 10:87c30842:::benchmark_data_smithi081_155596_object31754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 212.76757022699999, 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.163083229, 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.884 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31755 2024-09-21T19:09:59.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.884 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010285+0000", 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8959999999999999e-06 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010383+0000", 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7940000000000001e-05 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010393+0000", 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0414999999999999e-05 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010409+0000", 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5746000000000001e-05 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010487+0000", 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8318000000000005e-05 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010631+0000", 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143502 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010769+0000", 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013836 2024-09-21T19:09:59.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.015045+0000", 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042753640000000002 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.015059+0000", 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4182000000000001e-05 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173282+0000", 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15822359799999999 2024-09-21T19:09:59.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173328+0000", 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5524e-05 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173366+0000", 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8383999999999997e-05 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:221681 10.e 10:79878b7c:::benchmark_data_smithi081_155596_object104208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:09:31.602507+0000", 2024-09-21T19:09:59.889 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 28.175346328, 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.111802384, 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 221681 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602507+0000", 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602507+0000", 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602509+0000", 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.793e-06 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602513+0000", 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6700000000000002e-06 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602522+0000", 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8829999999999995e-06 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602534+0000", 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2067e-05 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602589+0000", 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4703000000000001e-05 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602689+0000", 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9698999999999995e-05 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.602787+0000", 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8395999999999998e-05 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.606009+0000", 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032213260000000001 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.606243+0000", 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023472200000000001 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.714166+0000", 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.107922531 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.714263+0000", 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7499999999999998e-05 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.714309+0000", 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6094000000000003e-05 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:221686 10.e 10:7fb366c1:::benchmark_data_smithi081_155596_object104213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:09:31.607760+0000", 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 28.170093186999999, 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.106796924, 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.895 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 221686 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607760+0000", 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607760+0000", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607761+0000", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3209999999999999e-06 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607766+0000", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7330000000000003e-06 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607776+0000", 2024-09-21T19:09:59.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0105999999999999e-05 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607782+0000", 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4020000000000001e-06 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607825+0000", 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3182000000000001e-05 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607900+0000", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4696000000000004e-05 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.607968+0000", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8351999999999999e-05 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.609856+0000", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001888238 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.611087+0000", 2024-09-21T19:09:59.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00123127 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.714344+0000", 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.103256308 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.714515+0000", 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000171335 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:31.714557+0000", 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1980999999999997e-05 2024-09-21T19:09:59.900 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:223429 10.1 10:87d6488d:::benchmark_data_smithi081_155596_object105956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:09:33.138181+0000", 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.639671947, 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.110049286, 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:09:59.901 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 223429 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138181+0000", 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138181+0000", 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:09:59.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138183+0000", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.719e-06 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138188+0000", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7700000000000001e-06 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138199+0000", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1384e-05 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138206+0000", 2024-09-21T19:09:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.153e-06 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138260+0000", 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3454e-05 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138364+0000", 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104349 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.138739+0000", 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000374483 2024-09-21T19:09:59.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.142052+0000", 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033128179999999999 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.147723+0000", 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056716680000000004 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.248130+0000", 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10040727000000001 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.248185+0000", 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4809000000000003e-05 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:33.248231+0000", 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5408999999999998e-05 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:09:59.906 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:09:59.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:09:59.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:09:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:09:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:09:59.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:00.058 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:00.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:00.200 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:00.235 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_4' created 2024-09-21T19:10:00.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1198934144' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[29384]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[29384]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1198934144' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[39192]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-21T19:10:00.293 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:00 smithi081 ceph-mon[39192]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T19:10:00.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:00.316 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:10:00.318 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:10:00.328 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:00.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.452 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:00.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:00 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1198934144' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T19:10:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:00 smithi104 ceph-mon[32438]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-21T19:10:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:00 smithi104 ceph-mon[32438]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T19:10:00.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:00.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:00.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:10:00.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.698 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:00.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:00.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:00.820 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:00.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:00.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:00.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:01.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:01.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:01.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:01.188 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:01.249 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:01 smithi104 ceph-mon[32438]: pgmap v835: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:10:01.250 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1198934144' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T19:10:01.250 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:01 smithi104 ceph-mon[32438]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T19:10:01.251 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":836,"stamp":"2024-09-21T19:10:00.236697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2137,"num_read_kb":1889,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":272632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4532596,"kb_used_data":15996,"kb_used_omap":18,"kb_used_meta":4516397,"kb_avail":745428620,"statfs":{"total":767960285184,"available":763318906880,"internally_reserved":0,"allocated":16379904,"data_stored":9774699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4624791536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.496848"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":672,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254744+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:09:57.254744+0000","last_peered":"2024-09-21T19:09:57.254744+0000","last_clean":"2024-09-21T19:09:57.254744+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:09:57.254744+0000","last_undegraded":"2024-09-21T19:09:57.254744+0000","last_fullsized":"2024-09-21T19:09:57.254744+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:10.748546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":831,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215422+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:09:56.215422+0000","last_peered":"2024-09-21T19:09:56.215422+0000","last_clean":"2024-09-21T19:09:56.215422+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:09:56.215422+0000","last_undegraded":"2024-09-21T19:09:56.215422+0000","last_fullsized":"2024-09-21T19:09:56.215422+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":770,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215745+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:09:57.215745+0000","last_peered":"2024-09-21T19:09:57.215745+0000","last_clean":"2024-09-21T19:09:57.215745+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:09:57.215745+0000","last_undegraded":"2024-09-21T19:09:57.215745+0000","last_fullsized":"2024-09-21T19:09:57.215745+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.809763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215587+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:09:56.215587+0000","last_peered":"2024-09-21T19:09:56.215587+0000","last_clean":"2024-09-21T19:09:56.215587+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:09:56.215587+0000","last_undegraded":"2024-09-21T19:09:56.215587+0000","last_fullsized":"2024-09-21T19:09:56.215587+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":759,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216819+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:09:57.216819+0000","last_peered":"2024-09-21T19:09:57.216819+0000","last_clean":"2024-09-21T19:09:57.216819+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:09:57.216819+0000","last_undegraded":"2024-09-21T19:09:57.216819+0000","last_fullsized":"2024-09-21T19:09:57.216819+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":787,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216230+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:09:57.216230+0000","last_peered":"2024-09-21T19:09:57.216230+0000","last_clean":"2024-09-21T19:09:57.216230+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:09:57.216230+0000","last_undegraded":"2024-09-21T19:09:57.216230+0000","last_fullsized":"2024-09-21T19:09:57.216230+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":883,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215484+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:09:57.215484+0000","last_peered":"2024-09-21T19:09:57.215484+0000","last_clean":"2024-09-21T19:09:57.215484+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:09:57.215484+0000","last_undegraded":"2024-09-21T19:09:57.215484+0000","last_fullsized":"2024-09-21T19:09:57.215484+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":768,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215402+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:09:56.215402+0000","last_peered":"2024-09-21T19:09:56.215402+0000","last_clean":"2024-09-21T19:09:56.215402+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:09:56.215402+0000","last_undegraded":"2024-09-21T19:09:56.215402+0000","last_fullsized":"2024-09-21T19:09:56.215402+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":755,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216284+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:09:56.216284+0000","last_peered":"2024-09-21T19:09:56.216284+0000","last_clean":"2024-09-21T19:09:56.216284+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:09:56.216284+0000","last_undegraded":"2024-09-21T19:09:56.216284+0000","last_fullsized":"2024-09-21T19:09:56.216284+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":812,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215827+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:09:56.215827+0000","last_peered":"2024-09-21T19:09:56.215827+0000","last_clean":"2024-09-21T19:09:56.215827+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:09:56.215827+0000","last_undegraded":"2024-09-21T19:09:56.215827+0000","last_fullsized":"2024-09-21T19:09:56.215827+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":719,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624344+0000","last_change":"2024-09-21T19:09:56.743173+0000","last_active":"2024-09-21T19:09:57.624344+0000","last_peered":"2024-09-21T19:09:57.624344+0000","last_clean":"2024-09-21T19:09:57.624344+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:09:57.624344+0000","last_undegraded":"2024-09-21T19:09:57.624344+0000","last_fullsized":"2024-09-21T19:09:57.624344+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.743123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.743123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.761375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":630,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254107+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:09:57.254107+0000","last_peered":"2024-09-21T19:09:57.254107+0000","last_clean":"2024-09-21T19:09:57.254107+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:09:57.254107+0000","last_undegraded":"2024-09-21T19:09:57.254107+0000","last_fullsized":"2024-09-21T19:09:57.254107+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":308,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216072+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:09:56.216072+0000","last_peered":"2024-09-21T19:09:56.216072+0000","last_clean":"2024-09-21T19:09:56.216072+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:09:56.216072+0000","last_undegraded":"2024-09-21T19:09:56.216072+0000","last_fullsized":"2024-09-21T19:09:56.216072+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":758,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215661+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:09:56.215661+0000","last_peered":"2024-09-21T19:09:56.215661+0000","last_clean":"2024-09-21T19:09:56.215661+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:09:56.215661+0000","last_undegraded":"2024-09-21T19:09:56.215661+0000","last_fullsized":"2024-09-21T19:09:56.215661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":746,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947167+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:09:54.947167+0000","last_peered":"2024-09-21T19:09:54.947167+0000","last_clean":"2024-09-21T19:09:54.947167+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:09:54.947167+0000","last_undegraded":"2024-09-21T19:09:54.947167+0000","last_fullsized":"2024-09-21T19:09:54.947167+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":318,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215553+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:09:57.215553+0000","last_peered":"2024-09-21T19:09:57.215553+0000","last_clean":"2024-09-21T19:09:57.215553+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:09:57.215553+0000","last_undegraded":"2024-09-21T19:09:57.215553+0000","last_fullsized":"2024-09-21T19:09:57.215553+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":777,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216115+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:09:56.216115+0000","last_peered":"2024-09-21T19:09:56.216115+0000","last_clean":"2024-09-21T19:09:56.216115+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:09:56.216115+0000","last_undegraded":"2024-09-21T19:09:56.216115+0000","last_fullsized":"2024-09-21T19:09:56.216115+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":646,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254838+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:09:57.254838+0000","last_peered":"2024-09-21T19:09:57.254838+0000","last_clean":"2024-09-21T19:09:57.254838+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:09:57.254838+0000","last_undegraded":"2024-09-21T19:09:57.254838+0000","last_fullsized":"2024-09-21T19:09:57.254838+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":219,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.625081+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:09:57.625081+0000","last_peered":"2024-09-21T19:09:57.625081+0000","last_clean":"2024-09-21T19:09:57.625081+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:09:57.625081+0000","last_undegraded":"2024-09-21T19:09:57.625081+0000","last_fullsized":"2024-09-21T19:09:57.625081+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:24.726173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":773,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216061+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:09:57.216061+0000","last_peered":"2024-09-21T19:09:57.216061+0000","last_clean":"2024-09-21T19:09:57.216061+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:09:57.216061+0000","last_undegraded":"2024-09-21T19:09:57.216061+0000","last_fullsized":"2024-09-21T19:09:57.216061+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:07.738704+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":756,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216622+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:09:57.216622+0000","last_peered":"2024-09-21T19:09:57.216622+0000","last_clean":"2024-09-21T19:09:57.216622+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:09:57.216622+0000","last_undegraded":"2024-09-21T19:09:57.216622+0000","last_fullsized":"2024-09-21T19:09:57.216622+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":779,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215939+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:09:56.215939+0000","last_peered":"2024-09-21T19:09:56.215939+0000","last_clean":"2024-09-21T19:09:56.215939+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:09:56.215939+0000","last_undegraded":"2024-09-21T19:09:56.215939+0000","last_fullsized":"2024-09-21T19:09:56.215939+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":826,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217225+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:09:57.217225+0000","last_peered":"2024-09-21T19:09:57.217225+0000","last_clean":"2024-09-21T19:09:57.217225+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:09:57.217225+0000","last_undegraded":"2024-09-21T19:09:57.217225+0000","last_fullsized":"2024-09-21T19:09:57.217225+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":763,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215843+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:09:56.215843+0000","last_peered":"2024-09-21T19:09:56.215843+0000","last_clean":"2024-09-21T19:09:56.215843+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:09:56.215843+0000","last_undegraded":"2024-09-21T19:09:56.215843+0000","last_fullsized":"2024-09-21T19:09:56.215843+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":784,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947649+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:09:54.947649+0000","last_peered":"2024-09-21T19:09:54.947649+0000","last_clean":"2024-09-21T19:09:54.947649+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:09:54.947649+0000","last_undegraded":"2024-09-21T19:09:54.947649+0000","last_fullsized":"2024-09-21T19:09:54.947649+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.611692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":787,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215820+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:09:57.215820+0000","last_peered":"2024-09-21T19:09:57.215820+0000","last_clean":"2024-09-21T19:09:57.215820+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:09:57.215820+0000","last_undegraded":"2024-09-21T19:09:57.215820+0000","last_fullsized":"2024-09-21T19:09:57.215820+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":675,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254580+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:09:57.254580+0000","last_peered":"2024-09-21T19:09:57.254580+0000","last_clean":"2024-09-21T19:09:57.254580+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:09:57.254580+0000","last_undegraded":"2024-09-21T19:09:57.254580+0000","last_fullsized":"2024-09-21T19:09:57.254580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":853,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355204+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:09:58.355204+0000","last_peered":"2024-09-21T19:09:58.355204+0000","last_clean":"2024-09-21T19:09:58.355204+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:09:58.355204+0000","last_undegraded":"2024-09-21T19:09:58.355204+0000","last_fullsized":"2024-09-21T19:09:58.355204+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":214,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624850+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:09:57.624850+0000","last_peered":"2024-09-21T19:09:57.624850+0000","last_clean":"2024-09-21T19:09:57.624850+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:09:57.624850+0000","last_undegraded":"2024-09-21T19:09:57.624850+0000","last_fullsized":"2024-09-21T19:09:57.624850+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:13.700777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":823,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216724+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:09:57.216724+0000","last_peered":"2024-09-21T19:09:57.216724+0000","last_clean":"2024-09-21T19:09:57.216724+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:09:57.216724+0000","last_undegraded":"2024-09-21T19:09:57.216724+0000","last_fullsized":"2024-09-21T19:09:57.216724+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:13.885784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":761,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215818+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:09:57.215818+0000","last_peered":"2024-09-21T19:09:57.215818+0000","last_clean":"2024-09-21T19:09:57.215818+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:09:57.215818+0000","last_undegraded":"2024-09-21T19:09:57.215818+0000","last_fullsized":"2024-09-21T19:09:57.215818+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.830842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1003,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216193+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:09:56.216193+0000","last_peered":"2024-09-21T19:09:56.216193+0000","last_clean":"2024-09-21T19:09:56.216193+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:09:56.216193+0000","last_undegraded":"2024-09-21T19:09:56.216193+0000","last_fullsized":"2024-09-21T19:09:56.216193+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":443,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216019+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:09:57.216019+0000","last_peered":"2024-09-21T19:09:57.216019+0000","last_clean":"2024-09-21T19:09:57.216019+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:09:57.216019+0000","last_undegraded":"2024-09-21T19:09:57.216019+0000","last_fullsized":"2024-09-21T19:09:57.216019+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:22.031875+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":548,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216223+0000","last_change":"2024-09-21T19:09:53.595884+0000","last_active":"2024-09-21T19:09:56.216223+0000","last_peered":"2024-09-21T19:09:56.216223+0000","last_clean":"2024-09-21T19:09:56.216223+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:09:56.216223+0000","last_undegraded":"2024-09-21T19:09:56.216223+0000","last_fullsized":"2024-09-21T19:09:56.216223+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:15.381125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":850,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215630+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:09:57.215630+0000","last_peered":"2024-09-21T19:09:57.215630+0000","last_clean":"2024-09-21T19:09:57.215630+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:09:57.215630+0000","last_undegraded":"2024-09-21T19:09:57.215630+0000","last_fullsized":"2024-09-21T19:09:57.215630+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":775,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217003+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:09:57.217003+0000","last_peered":"2024-09-21T19:09:57.217003+0000","last_clean":"2024-09-21T19:09:57.217003+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:09:57.217003+0000","last_undegraded":"2024-09-21T19:09:57.217003+0000","last_fullsized":"2024-09-21T19:09:57.217003+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":751,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215457+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:09:56.215457+0000","last_peered":"2024-09-21T19:09:56.215457+0000","last_clean":"2024-09-21T19:09:56.215457+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:09:56.215457+0000","last_undegraded":"2024-09-21T19:09:56.215457+0000","last_fullsized":"2024-09-21T19:09:56.215457+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":768,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215801+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:09:56.215801+0000","last_peered":"2024-09-21T19:09:56.215801+0000","last_clean":"2024-09-21T19:09:56.215801+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:09:56.215801+0000","last_undegraded":"2024-09-21T19:09:56.215801+0000","last_fullsized":"2024-09-21T19:09:56.215801+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":533,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217216+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:09:57.217216+0000","last_peered":"2024-09-21T19:09:57.217216+0000","last_clean":"2024-09-21T19:09:57.217216+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:09:57.217216+0000","last_undegraded":"2024-09-21T19:09:57.217216+0000","last_fullsized":"2024-09-21T19:09:57.217216+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":819,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215518+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:09:56.215518+0000","last_peered":"2024-09-21T19:09:56.215518+0000","last_clean":"2024-09-21T19:09:56.215518+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:09:56.215518+0000","last_undegraded":"2024-09-21T19:09:56.215518+0000","last_fullsized":"2024-09-21T19:09:56.215518+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":643,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254803+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:09:57.254803+0000","last_peered":"2024-09-21T19:09:57.254803+0000","last_clean":"2024-09-21T19:09:57.254803+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:09:57.254803+0000","last_undegraded":"2024-09-21T19:09:57.254803+0000","last_fullsized":"2024-09-21T19:09:57.254803+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":949,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216345+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:09:57.216345+0000","last_peered":"2024-09-21T19:09:57.216345+0000","last_clean":"2024-09-21T19:09:57.216345+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:09:57.216345+0000","last_undegraded":"2024-09-21T19:09:57.216345+0000","last_fullsized":"2024-09-21T19:09:57.216345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":777,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947119+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:09:54.947119+0000","last_peered":"2024-09-21T19:09:54.947119+0000","last_clean":"2024-09-21T19:09:54.947119+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:09:54.947119+0000","last_undegraded":"2024-09-21T19:09:54.947119+0000","last_fullsized":"2024-09-21T19:09:54.947119+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.662781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":565,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355316+0000","last_change":"2024-09-21T19:09:55.907018+0000","last_active":"2024-09-21T19:09:58.355316+0000","last_peered":"2024-09-21T19:09:58.355316+0000","last_clean":"2024-09-21T19:09:58.355316+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:09:58.355316+0000","last_undegraded":"2024-09-21T19:09:58.355316+0000","last_fullsized":"2024-09-21T19:09:58.355316+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:11.373203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":806,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216162+0000","last_change":"2024-09-21T19:09:39.038158+0000","last_active":"2024-09-21T19:09:57.216162+0000","last_peered":"2024-09-21T19:09:57.216162+0000","last_clean":"2024-09-21T19:09:57.216162+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:09:57.216162+0000","last_undegraded":"2024-09-21T19:09:57.216162+0000","last_fullsized":"2024-09-21T19:09:57.216162+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:09:39.038064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.038064+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.698592+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":325,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217312+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:09:57.217312+0000","last_peered":"2024-09-21T19:09:57.217312+0000","last_clean":"2024-09-21T19:09:57.217312+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:09:57.217312+0000","last_undegraded":"2024-09-21T19:09:57.217312+0000","last_fullsized":"2024-09-21T19:09:57.217312+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.609167+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":831,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254693+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:09:57.254693+0000","last_peered":"2024-09-21T19:09:57.254693+0000","last_clean":"2024-09-21T19:09:57.254693+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:09:57.254693+0000","last_undegraded":"2024-09-21T19:09:57.254693+0000","last_fullsized":"2024-09-21T19:09:57.254693+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":654,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254351+0000","last_change":"2024-09-21T19:09:47.129430+0000","last_active":"2024-09-21T19:09:57.254351+0000","last_peered":"2024-09-21T19:09:57.254351+0000","last_clean":"2024-09-21T19:09:57.254351+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:09:57.254351+0000","last_undegraded":"2024-09-21T19:09:57.254351+0000","last_fullsized":"2024-09-21T19:09:57.254351+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:47.129379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:47.129379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.229738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":543,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355080+0000","last_change":"2024-09-21T19:09:54.937870+0000","last_active":"2024-09-21T19:09:58.355080+0000","last_peered":"2024-09-21T19:09:58.355080+0000","last_clean":"2024-09-21T19:09:58.355080+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:09:58.355080+0000","last_undegraded":"2024-09-21T19:09:58.355080+0000","last_fullsized":"2024-09-21T19:09:58.355080+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:57.511376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":691,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254813+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:09:57.254813+0000","last_peered":"2024-09-21T19:09:57.254813+0000","last_clean":"2024-09-21T19:09:57.254813+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:09:57.254813+0000","last_undegraded":"2024-09-21T19:09:57.254813+0000","last_fullsized":"2024-09-21T19:09:57.254813+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":794,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215686+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:09:57.215686+0000","last_peered":"2024-09-21T19:09:57.215686+0000","last_clean":"2024-09-21T19:09:57.215686+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:09:57.215686+0000","last_undegraded":"2024-09-21T19:09:57.215686+0000","last_fullsized":"2024-09-21T19:09:57.215686+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.451996+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":670,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.354895+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:09:58.354895+0000","last_peered":"2024-09-21T19:09:58.354895+0000","last_clean":"2024-09-21T19:09:58.354895+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:09:58.354895+0000","last_undegraded":"2024-09-21T19:09:58.354895+0000","last_fullsized":"2024-09-21T19:09:58.354895+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":774,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215702+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:09:57.215702+0000","last_peered":"2024-09-21T19:09:57.215702+0000","last_clean":"2024-09-21T19:09:57.215702+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:09:57.215702+0000","last_undegraded":"2024-09-21T19:09:57.215702+0000","last_fullsized":"2024-09-21T19:09:57.215702+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":510,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254026+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:09:57.254026+0000","last_peered":"2024-09-21T19:09:57.254026+0000","last_clean":"2024-09-21T19:09:57.254026+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:09:57.254026+0000","last_undegraded":"2024-09-21T19:09:57.254026+0000","last_fullsized":"2024-09-21T19:09:57.254026+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1085,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215596+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:09:57.215596+0000","last_peered":"2024-09-21T19:09:57.215596+0000","last_clean":"2024-09-21T19:09:57.215596+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:09:57.215596+0000","last_undegraded":"2024-09-21T19:09:57.215596+0000","last_fullsized":"2024-09-21T19:09:57.215596+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.313979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":792,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215963+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:09:56.215963+0000","last_peered":"2024-09-21T19:09:56.215963+0000","last_clean":"2024-09-21T19:09:56.215963+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:09:56.215963+0000","last_undegraded":"2024-09-21T19:09:56.215963+0000","last_fullsized":"2024-09-21T19:09:56.215963+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":341,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216418+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:09:57.216418+0000","last_peered":"2024-09-21T19:09:57.216418+0000","last_clean":"2024-09-21T19:09:57.216418+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:09:57.216418+0000","last_undegraded":"2024-09-21T19:09:57.216418+0000","last_fullsized":"2024-09-21T19:09:57.216418+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":639,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254578+0000","last_change":"2024-09-21T19:09:51.157757+0000","last_active":"2024-09-21T19:09:57.254578+0000","last_peered":"2024-09-21T19:09:57.254578+0000","last_clean":"2024-09-21T19:09:57.254578+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:09:57.254578+0000","last_undegraded":"2024-09-21T19:09:57.254578+0000","last_fullsized":"2024-09-21T19:09:57.254578+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.157711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.157711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:54.408363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":741,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216024+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:09:57.216024+0000","last_peered":"2024-09-21T19:09:57.216024+0000","last_clean":"2024-09-21T19:09:57.216024+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:09:57.216024+0000","last_undegraded":"2024-09-21T19:09:57.216024+0000","last_fullsized":"2024-09-21T19:09:57.216024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":850,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:55.484547+0000","last_change":"2024-09-21T19:09:55.484547+0000","last_active":"2024-09-21T19:09:55.484547+0000","last_peered":"2024-09-21T19:09:55.484547+0000","last_clean":"2024-09-21T19:09:55.484547+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:09:55.484547+0000","last_undegraded":"2024-09-21T19:09:55.484547+0000","last_fullsized":"2024-09-21T19:09:55.484547+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:15.423137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":259,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215058+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:09:57.215058+0000","last_peered":"2024-09-21T19:09:57.215058+0000","last_clean":"2024-09-21T19:09:57.215058+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:09:57.215058+0000","last_undegraded":"2024-09-21T19:09:57.215058+0000","last_fullsized":"2024-09-21T19:09:57.215058+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":816,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.214975+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:09:57.214975+0000","last_peered":"2024-09-21T19:09:57.214975+0000","last_clean":"2024-09-21T19:09:57.214975+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:09:57.214975+0000","last_undegraded":"2024-09-21T19:09:57.214975+0000","last_fullsized":"2024-09-21T19:09:57.214975+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":208,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624794+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:09:57.624794+0000","last_peered":"2024-09-21T19:09:57.624794+0000","last_clean":"2024-09-21T19:09:57.624794+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:09:57.624794+0000","last_undegraded":"2024-09-21T19:09:57.624794+0000","last_fullsized":"2024-09-21T19:09:57.624794+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.753652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":214,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.625029+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:09:57.625029+0000","last_peered":"2024-09-21T19:09:57.625029+0000","last_clean":"2024-09-21T19:09:57.625029+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:09:57.625029+0000","last_undegraded":"2024-09-21T19:09:57.625029+0000","last_fullsized":"2024-09-21T19:09:57.625029+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.374847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":767,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947671+0000","last_change":"2024-09-21T19:09:37.588463+0000","last_active":"2024-09-21T19:09:54.947671+0000","last_peered":"2024-09-21T19:09:54.947671+0000","last_clean":"2024-09-21T19:09:54.947671+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:09:54.947671+0000","last_undegraded":"2024-09-21T19:09:54.947671+0000","last_fullsized":"2024-09-21T19:09:54.947671+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:09:37.588390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.588390+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.015172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":552,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216417+0000","last_change":"2024-09-21T19:09:08.156345+0000","last_active":"2024-09-21T19:09:57.216417+0000","last_peered":"2024-09-21T19:09:57.216417+0000","last_clean":"2024-09-21T19:09:57.216417+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:09:57.216417+0000","last_undegraded":"2024-09-21T19:09:57.216417+0000","last_fullsized":"2024-09-21T19:09:57.216417+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:08.156258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:08.156258+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.996519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":753,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254678+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:09:57.254678+0000","last_peered":"2024-09-21T19:09:57.254678+0000","last_clean":"2024-09-21T19:09:57.254678+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:09:57.254678+0000","last_undegraded":"2024-09-21T19:09:57.254678+0000","last_fullsized":"2024-09-21T19:09:57.254678+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":639,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254440+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:09:57.254440+0000","last_peered":"2024-09-21T19:09:57.254440+0000","last_clean":"2024-09-21T19:09:57.254440+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:09:57.254440+0000","last_undegraded":"2024-09-21T19:09:57.254440+0000","last_fullsized":"2024-09-21T19:09:57.254440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1784,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:55.821397+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:09:55.821397+0000","last_peered":"2024-09-21T19:09:55.821397+0000","last_clean":"2024-09-21T19:09:55.821397+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:09:55.821397+0000","last_undegraded":"2024-09-21T19:09:55.821397+0000","last_fullsized":"2024-09-21T19:09:55.821397+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.664705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":998,"num_read_kb":993,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":720,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624958+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:09:57.624958+0000","last_peered":"2024-09-21T19:09:57.624958+0000","last_clean":"2024-09-21T19:09:57.624958+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:09:57.624958+0000","last_undegraded":"2024-09-21T19:09:57.624958+0000","last_fullsized":"2024-09-21T19:09:57.624958+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.747406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":768,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215394+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:09:57.215394+0000","last_peered":"2024-09-21T19:09:57.215394+0000","last_clean":"2024-09-21T19:09:57.215394+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:09:57.215394+0000","last_undegraded":"2024-09-21T19:09:57.215394+0000","last_fullsized":"2024-09-21T19:09:57.215394+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":944,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355071+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:09:58.355071+0000","last_peered":"2024-09-21T19:09:58.355071+0000","last_clean":"2024-09-21T19:09:58.355071+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:09:58.355071+0000","last_undegraded":"2024-09-21T19:09:58.355071+0000","last_fullsized":"2024-09-21T19:09:58.355071+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":776,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215327+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:09:57.215327+0000","last_peered":"2024-09-21T19:09:57.215327+0000","last_clean":"2024-09-21T19:09:57.215327+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:09:57.215327+0000","last_undegraded":"2024-09-21T19:09:57.215327+0000","last_fullsized":"2024-09-21T19:09:57.215327+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":856,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947297+0000","last_change":"2024-09-21T19:09:46.543580+0000","last_active":"2024-09-21T19:09:54.947297+0000","last_peered":"2024-09-21T19:09:54.947297+0000","last_clean":"2024-09-21T19:09:54.947297+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:09:54.947297+0000","last_undegraded":"2024-09-21T19:09:54.947297+0000","last_fullsized":"2024-09-21T19:09:54.947297+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:46.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:46.543525+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.937610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":773,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947796+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:09:54.947796+0000","last_peered":"2024-09-21T19:09:54.947796+0000","last_clean":"2024-09-21T19:09:54.947796+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:09:54.947796+0000","last_undegraded":"2024-09-21T19:09:54.947796+0000","last_fullsized":"2024-09-21T19:09:54.947796+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:15.623405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":311,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216507+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:09:57.216507+0000","last_peered":"2024-09-21T19:09:57.216507+0000","last_clean":"2024-09-21T19:09:57.216507+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:09:57.216507+0000","last_undegraded":"2024-09-21T19:09:57.216507+0000","last_fullsized":"2024-09-21T19:09:57.216507+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.170603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":812,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216230+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:09:56.216230+0000","last_peered":"2024-09-21T19:09:56.216230+0000","last_clean":"2024-09-21T19:09:56.216230+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:09:56.216230+0000","last_undegraded":"2024-09-21T19:09:56.216230+0000","last_fullsized":"2024-09-21T19:09:56.216230+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":780,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215718+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:09:56.215718+0000","last_peered":"2024-09-21T19:09:56.215718+0000","last_clean":"2024-09-21T19:09:56.215718+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:09:56.215718+0000","last_undegraded":"2024-09-21T19:09:56.215718+0000","last_fullsized":"2024-09-21T19:09:56.215718+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":764,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216293+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:09:56.216293+0000","last_peered":"2024-09-21T19:09:56.216293+0000","last_clean":"2024-09-21T19:09:56.216293+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:09:56.216293+0000","last_undegraded":"2024-09-21T19:09:56.216293+0000","last_fullsized":"2024-09-21T19:09:56.216293+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":747,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217306+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:09:57.217306+0000","last_peered":"2024-09-21T19:09:57.217306+0000","last_clean":"2024-09-21T19:09:57.217306+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:09:57.217306+0000","last_undegraded":"2024-09-21T19:09:57.217306+0000","last_fullsized":"2024-09-21T19:09:57.217306+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":210,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624731+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:09:57.624731+0000","last_peered":"2024-09-21T19:09:57.624731+0000","last_clean":"2024-09-21T19:09:57.624731+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:09:57.624731+0000","last_undegraded":"2024-09-21T19:09:57.624731+0000","last_fullsized":"2024-09-21T19:09:57.624731+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.739272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":861,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216070+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:09:56.216070+0000","last_peered":"2024-09-21T19:09:56.216070+0000","last_clean":"2024-09-21T19:09:56.216070+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:09:56.216070+0000","last_undegraded":"2024-09-21T19:09:56.216070+0000","last_fullsized":"2024-09-21T19:09:56.216070+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":681,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355457+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:09:58.355457+0000","last_peered":"2024-09-21T19:09:58.355457+0000","last_clean":"2024-09-21T19:09:58.355457+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:09:58.355457+0000","last_undegraded":"2024-09-21T19:09:58.355457+0000","last_fullsized":"2024-09-21T19:09:58.355457+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":213,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355559+0000","last_change":"2024-09-21T19:09:50.927165+0000","last_active":"2024-09-21T19:09:58.355559+0000","last_peered":"2024-09-21T19:09:58.355559+0000","last_clean":"2024-09-21T19:09:58.355559+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:09:58.355559+0000","last_undegraded":"2024-09-21T19:09:58.355559+0000","last_fullsized":"2024-09-21T19:09:58.355559+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.927103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.927103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.477371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":211,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.625083+0000","last_change":"2024-09-21T19:09:31.740952+0000","last_active":"2024-09-21T19:09:57.625083+0000","last_peered":"2024-09-21T19:09:57.625083+0000","last_clean":"2024-09-21T19:09:57.625083+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:09:57.625083+0000","last_undegraded":"2024-09-21T19:09:57.625083+0000","last_fullsized":"2024-09-21T19:09:57.625083+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:31.740891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:31.740891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:40.289541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":864,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947361+0000","last_change":"2024-09-21T19:09:39.566152+0000","last_active":"2024-09-21T19:09:54.947361+0000","last_peered":"2024-09-21T19:09:54.947361+0000","last_clean":"2024-09-21T19:09:54.947361+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:09:54.947361+0000","last_undegraded":"2024-09-21T19:09:54.947361+0000","last_fullsized":"2024-09-21T19:09:54.947361+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:39.565921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.565921+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:44.360587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":787,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215929+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:09:56.215929+0000","last_peered":"2024-09-21T19:09:56.215929+0000","last_clean":"2024-09-21T19:09:56.215929+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:09:56.215929+0000","last_undegraded":"2024-09-21T19:09:56.215929+0000","last_fullsized":"2024-09-21T19:09:56.215929+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":789,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217080+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:09:57.217080+0000","last_peered":"2024-09-21T19:09:57.217080+0000","last_clean":"2024-09-21T19:09:57.217080+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:09:57.217080+0000","last_undegraded":"2024-09-21T19:09:57.217080+0000","last_fullsized":"2024-09-21T19:09:57.217080+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":264,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.625070+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:09:57.625070+0000","last_peered":"2024-09-21T19:09:57.625070+0000","last_clean":"2024-09-21T19:09:57.625070+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:09:57.625070+0000","last_undegraded":"2024-09-21T19:09:57.625070+0000","last_fullsized":"2024-09-21T19:09:57.625070+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:19.687093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":760,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216094+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:09:56.216094+0000","last_peered":"2024-09-21T19:09:56.216094+0000","last_clean":"2024-09-21T19:09:56.216094+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:09:56.216094+0000","last_undegraded":"2024-09-21T19:09:56.216094+0000","last_fullsized":"2024-09-21T19:09:56.216094+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":650,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254893+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:09:57.254893+0000","last_peered":"2024-09-21T19:09:57.254893+0000","last_clean":"2024-09-21T19:09:57.254893+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:09:57.254893+0000","last_undegraded":"2024-09-21T19:09:57.254893+0000","last_fullsized":"2024-09-21T19:09:57.254893+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":751,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217764+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:09:57.217764+0000","last_peered":"2024-09-21T19:09:57.217764+0000","last_clean":"2024-09-21T19:09:57.217764+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:09:57.217764+0000","last_undegraded":"2024-09-21T19:09:57.217764+0000","last_fullsized":"2024-09-21T19:09:57.217764+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":203,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624911+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:09:57.624911+0000","last_peered":"2024-09-21T19:09:57.624911+0000","last_clean":"2024-09-21T19:09:57.624911+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:09:57.624911+0000","last_undegraded":"2024-09-21T19:09:57.624911+0000","last_fullsized":"2024-09-21T19:09:57.624911+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:10.305625+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":645,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254294+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:09:57.254294+0000","last_peered":"2024-09-21T19:09:57.254294+0000","last_clean":"2024-09-21T19:09:57.254294+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:09:57.254294+0000","last_undegraded":"2024-09-21T19:09:57.254294+0000","last_fullsized":"2024-09-21T19:09:57.254294+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":700,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355370+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:09:58.355370+0000","last_peered":"2024-09-21T19:09:58.355370+0000","last_clean":"2024-09-21T19:09:58.355370+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:09:58.355370+0000","last_undegraded":"2024-09-21T19:09:58.355370+0000","last_fullsized":"2024-09-21T19:09:58.355370+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":777,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947938+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:09:54.947938+0000","last_peered":"2024-09-21T19:09:54.947938+0000","last_clean":"2024-09-21T19:09:54.947938+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:09:54.947938+0000","last_undegraded":"2024-09-21T19:09:54.947938+0000","last_fullsized":"2024-09-21T19:09:54.947938+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.612669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":214,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624892+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:09:57.624892+0000","last_peered":"2024-09-21T19:09:57.624892+0000","last_clean":"2024-09-21T19:09:57.624892+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:09:57.624892+0000","last_undegraded":"2024-09-21T19:09:57.624892+0000","last_fullsized":"2024-09-21T19:09:57.624892+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":757,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947205+0000","last_change":"2024-09-21T19:09:35.677140+0000","last_active":"2024-09-21T19:09:54.947205+0000","last_peered":"2024-09-21T19:09:54.947205+0000","last_clean":"2024-09-21T19:09:54.947205+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:09:54.947205+0000","last_undegraded":"2024-09-21T19:09:54.947205+0000","last_fullsized":"2024-09-21T19:09:54.947205+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:35.677043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:35.677043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:50.521879+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":816,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215621+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:09:56.215621+0000","last_peered":"2024-09-21T19:09:56.215621+0000","last_clean":"2024-09-21T19:09:56.215621+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:09:56.215621+0000","last_undegraded":"2024-09-21T19:09:56.215621+0000","last_fullsized":"2024-09-21T19:09:56.215621+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":779,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.948269+0000","last_change":"2024-09-21T19:09:41.535785+0000","last_active":"2024-09-21T19:09:54.948269+0000","last_peered":"2024-09-21T19:09:54.948269+0000","last_clean":"2024-09-21T19:09:54.948269+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:09:54.948269+0000","last_undegraded":"2024-09-21T19:09:54.948269+0000","last_fullsized":"2024-09-21T19:09:54.948269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:41.535714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:41.535714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:45.775155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":203,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624651+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:09:57.624651+0000","last_peered":"2024-09-21T19:09:57.624651+0000","last_clean":"2024-09-21T19:09:57.624651+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:09:57.624651+0000","last_undegraded":"2024-09-21T19:09:57.624651+0000","last_fullsized":"2024-09-21T19:09:57.624651+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:00.385824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":741,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216914+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:09:57.216914+0000","last_peered":"2024-09-21T19:09:57.216914+0000","last_clean":"2024-09-21T19:09:57.216914+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:09:57.216914+0000","last_undegraded":"2024-09-21T19:09:57.216914+0000","last_fullsized":"2024-09-21T19:09:57.216914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T19:10:00.224748+0000","last_change":"2024-09-21T19:10:00.224748+0000","last_active":"2024-09-21T19:10:00.224748+0000","last_peered":"2024-09-21T19:10:00.224748+0000","last_clean":"2024-09-21T19:10:00.224748+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T19:10:00.224748+0000","last_undegraded":"2024-09-21T19:10:00.224748+0000","last_fullsized":"2024-09-21T19:10:00.224748+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":370,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.946941+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:09:54.946941+0000","last_peered":"2024-09-21T19:09:54.946941+0000","last_clean":"2024-09-21T19:09:54.946941+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:09:54.946941+0000","last_undegraded":"2024-09-21T19:09:54.946941+0000","last_fullsized":"2024-09-21T19:09:54.946941+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:17.636097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":793,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.948299+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:09:54.948299+0000","last_peered":"2024-09-21T19:09:54.948299+0000","last_clean":"2024-09-21T19:09:54.948299+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:09:54.948299+0000","last_undegraded":"2024-09-21T19:09:54.948299+0000","last_fullsized":"2024-09-21T19:09:54.948299+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:31.474930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":263,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.656091+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:09:57.656091+0000","last_peered":"2024-09-21T19:09:57.656091+0000","last_clean":"2024-09-21T19:09:57.656091+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:09:57.656091+0000","last_undegraded":"2024-09-21T19:09:57.656091+0000","last_fullsized":"2024-09-21T19:09:57.656091+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":777,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355207+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:09:58.355207+0000","last_peered":"2024-09-21T19:09:58.355207+0000","last_clean":"2024-09-21T19:09:58.355207+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:09:58.355207+0000","last_undegraded":"2024-09-21T19:09:58.355207+0000","last_fullsized":"2024-09-21T19:09:58.355207+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":712,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.354738+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:09:58.354738+0000","last_peered":"2024-09-21T19:09:58.354738+0000","last_clean":"2024-09-21T19:09:58.354738+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:09:58.354738+0000","last_undegraded":"2024-09-21T19:09:58.354738+0000","last_fullsized":"2024-09-21T19:09:58.354738+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":774,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216856+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:09:57.216856+0000","last_peered":"2024-09-21T19:09:57.216856+0000","last_clean":"2024-09-21T19:09:57.216856+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:09:57.216856+0000","last_undegraded":"2024-09-21T19:09:57.216856+0000","last_fullsized":"2024-09-21T19:09:57.216856+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":310,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215958+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:09:57.215958+0000","last_peered":"2024-09-21T19:09:57.215958+0000","last_clean":"2024-09-21T19:09:57.215958+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:09:57.215958+0000","last_undegraded":"2024-09-21T19:09:57.215958+0000","last_fullsized":"2024-09-21T19:09:57.215958+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":745,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947517+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:09:54.947517+0000","last_peered":"2024-09-21T19:09:54.947517+0000","last_clean":"2024-09-21T19:09:54.947517+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:09:54.947517+0000","last_undegraded":"2024-09-21T19:09:54.947517+0000","last_fullsized":"2024-09-21T19:09:54.947517+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":657,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254720+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:09:57.254720+0000","last_peered":"2024-09-21T19:09:57.254720+0000","last_clean":"2024-09-21T19:09:57.254720+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:09:57.254720+0000","last_undegraded":"2024-09-21T19:09:57.254720+0000","last_fullsized":"2024-09-21T19:09:57.254720+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.161530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":758,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947527+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:09:54.947527+0000","last_peered":"2024-09-21T19:09:54.947527+0000","last_clean":"2024-09-21T19:09:54.947527+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:09:54.947527+0000","last_undegraded":"2024-09-21T19:09:54.947527+0000","last_fullsized":"2024-09-21T19:09:54.947527+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:16.645472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":744,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216663+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:09:57.216663+0000","last_peered":"2024-09-21T19:09:57.216663+0000","last_clean":"2024-09-21T19:09:57.216663+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:09:57.216663+0000","last_undegraded":"2024-09-21T19:09:57.216663+0000","last_fullsized":"2024-09-21T19:09:57.216663+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":203,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.625125+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:09:57.625125+0000","last_peered":"2024-09-21T19:09:57.625125+0000","last_clean":"2024-09-21T19:09:57.625125+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:09:57.625125+0000","last_undegraded":"2024-09-21T19:09:57.625125+0000","last_fullsized":"2024-09-21T19:09:57.625125+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":359,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947781+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:09:54.947781+0000","last_peered":"2024-09-21T19:09:54.947781+0000","last_clean":"2024-09-21T19:09:54.947781+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:09:54.947781+0000","last_undegraded":"2024-09-21T19:09:54.947781+0000","last_fullsized":"2024-09-21T19:09:54.947781+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:29.639722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":647,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254387+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:09:57.254387+0000","last_peered":"2024-09-21T19:09:57.254387+0000","last_clean":"2024-09-21T19:09:57.254387+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:09:57.254387+0000","last_undegraded":"2024-09-21T19:09:57.254387+0000","last_fullsized":"2024-09-21T19:09:57.254387+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":987,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217437+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:09:57.217437+0000","last_peered":"2024-09-21T19:09:57.217437+0000","last_clean":"2024-09-21T19:09:57.217437+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:09:57.217437+0000","last_undegraded":"2024-09-21T19:09:57.217437+0000","last_fullsized":"2024-09-21T19:09:57.217437+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:12.049319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":770,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216257+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:09:56.216257+0000","last_peered":"2024-09-21T19:09:56.216257+0000","last_clean":"2024-09-21T19:09:56.216257+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:09:56.216257+0000","last_undegraded":"2024-09-21T19:09:56.216257+0000","last_fullsized":"2024-09-21T19:09:56.216257+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.177889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":771,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217654+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:09:57.217654+0000","last_peered":"2024-09-21T19:09:57.217654+0000","last_clean":"2024-09-21T19:09:57.217654+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:09:57.217654+0000","last_undegraded":"2024-09-21T19:09:57.217654+0000","last_fullsized":"2024-09-21T19:09:57.217654+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":769,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947348+0000","last_change":"2024-09-21T19:09:32.615127+0000","last_active":"2024-09-21T19:09:54.947348+0000","last_peered":"2024-09-21T19:09:54.947348+0000","last_clean":"2024-09-21T19:09:54.947348+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:09:54.947348+0000","last_undegraded":"2024-09-21T19:09:54.947348+0000","last_fullsized":"2024-09-21T19:09:54.947348+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:32.615041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:32.615041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.657148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":428,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215938+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:09:57.215938+0000","last_peered":"2024-09-21T19:09:57.215938+0000","last_clean":"2024-09-21T19:09:57.215938+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:09:57.215938+0000","last_undegraded":"2024-09-21T19:09:57.215938+0000","last_fullsized":"2024-09-21T19:09:57.215938+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":646,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254923+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:09:57.254923+0000","last_peered":"2024-09-21T19:09:57.254923+0000","last_clean":"2024-09-21T19:09:57.254923+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:09:57.254923+0000","last_undegraded":"2024-09-21T19:09:57.254923+0000","last_fullsized":"2024-09-21T19:09:57.254923+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:18.380729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":842,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.215703+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:09:56.215703+0000","last_peered":"2024-09-21T19:09:56.215703+0000","last_clean":"2024-09-21T19:09:56.215703+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:09:56.215703+0000","last_undegraded":"2024-09-21T19:09:56.215703+0000","last_fullsized":"2024-09-21T19:09:56.215703+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":761,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T19:09:56.216155+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:09:56.216155+0000","last_peered":"2024-09-21T19:09:56.216155+0000","last_clean":"2024-09-21T19:09:56.216155+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:09:56.216155+0000","last_undegraded":"2024-09-21T19:09:56.216155+0000","last_fullsized":"2024-09-21T19:09:56.216155+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":771,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216030+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:09:57.216030+0000","last_peered":"2024-09-21T19:09:57.216030+0000","last_clean":"2024-09-21T19:09:57.216030+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:09:57.216030+0000","last_undegraded":"2024-09-21T19:09:57.216030+0000","last_fullsized":"2024-09-21T19:09:57.216030+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:37.041685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":766,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215054+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:09:57.215054+0000","last_peered":"2024-09-21T19:09:57.215054+0000","last_clean":"2024-09-21T19:09:57.215054+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:09:57.215054+0000","last_undegraded":"2024-09-21T19:09:57.215054+0000","last_fullsized":"2024-09-21T19:09:57.215054+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":842,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216289+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:09:57.216289+0000","last_peered":"2024-09-21T19:09:57.216289+0000","last_clean":"2024-09-21T19:09:57.216289+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:09:57.216289+0000","last_undegraded":"2024-09-21T19:09:57.216289+0000","last_fullsized":"2024-09-21T19:09:57.216289+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":762,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947791+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:09:54.947791+0000","last_peered":"2024-09-21T19:09:54.947791+0000","last_clean":"2024-09-21T19:09:54.947791+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:09:54.947791+0000","last_undegraded":"2024-09-21T19:09:54.947791+0000","last_fullsized":"2024-09-21T19:09:54.947791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":767,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.216974+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:09:57.216974+0000","last_peered":"2024-09-21T19:09:57.216974+0000","last_clean":"2024-09-21T19:09:57.216974+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:09:57.216974+0000","last_undegraded":"2024-09-21T19:09:57.216974+0000","last_fullsized":"2024-09-21T19:09:57.216974+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":658,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254528+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:09:57.254528+0000","last_peered":"2024-09-21T19:09:57.254528+0000","last_clean":"2024-09-21T19:09:57.254528+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:09:57.254528+0000","last_undegraded":"2024-09-21T19:09:57.254528+0000","last_fullsized":"2024-09-21T19:09:57.254528+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":791,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215241+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:09:57.215241+0000","last_peered":"2024-09-21T19:09:57.215241+0000","last_clean":"2024-09-21T19:09:57.215241+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:09:57.215241+0000","last_undegraded":"2024-09-21T19:09:57.215241+0000","last_fullsized":"2024-09-21T19:09:57.215241+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":704,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.624908+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:09:57.624908+0000","last_peered":"2024-09-21T19:09:57.624908+0000","last_clean":"2024-09-21T19:09:57.624908+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:09:57.624908+0000","last_undegraded":"2024-09-21T19:09:57.624908+0000","last_fullsized":"2024-09-21T19:09:57.624908+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":782,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.946834+0000","last_change":"2024-09-21T19:09:51.462137+0000","last_active":"2024-09-21T19:09:54.946834+0000","last_peered":"2024-09-21T19:09:54.946834+0000","last_clean":"2024-09-21T19:09:54.946834+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:09:54.946834+0000","last_undegraded":"2024-09-21T19:09:54.946834+0000","last_fullsized":"2024-09-21T19:09:54.946834+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.462077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.604947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":766,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.217105+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:09:57.217105+0000","last_peered":"2024-09-21T19:09:57.217105+0000","last_clean":"2024-09-21T19:09:57.217105+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:09:57.217105+0000","last_undegraded":"2024-09-21T19:09:57.217105+0000","last_fullsized":"2024-09-21T19:09:57.217105+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":790,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215542+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:09:57.215542+0000","last_peered":"2024-09-21T19:09:57.215542+0000","last_clean":"2024-09-21T19:09:57.215542+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:09:57.215542+0000","last_undegraded":"2024-09-21T19:09:57.215542+0000","last_fullsized":"2024-09-21T19:09:57.215542+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":641,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254647+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:09:57.254647+0000","last_peered":"2024-09-21T19:09:57.254647+0000","last_clean":"2024-09-21T19:09:57.254647+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:09:57.254647+0000","last_undegraded":"2024-09-21T19:09:57.254647+0000","last_fullsized":"2024-09-21T19:09:57.254647+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":679,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.354785+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:09:58.354785+0000","last_peered":"2024-09-21T19:09:58.354785+0000","last_clean":"2024-09-21T19:09:58.354785+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:09:58.354785+0000","last_undegraded":"2024-09-21T19:09:58.354785+0000","last_fullsized":"2024-09-21T19:09:58.354785+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":757,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T19:09:54.947075+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:09:54.947075+0000","last_peered":"2024-09-21T19:09:54.947075+0000","last_clean":"2024-09-21T19:09:54.947075+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:09:54.947075+0000","last_undegraded":"2024-09-21T19:09:54.947075+0000","last_fullsized":"2024-09-21T19:09:54.947075+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":766,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215771+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:09:57.215771+0000","last_peered":"2024-09-21T19:09:57.215771+0000","last_clean":"2024-09-21T19:09:57.215771+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:09:57.215771+0000","last_undegraded":"2024-09-21T19:09:57.215771+0000","last_fullsized":"2024-09-21T19:09:57.215771+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":782,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.215753+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:09:57.215753+0000","last_peered":"2024-09-21T19:09:57.215753+0000","last_clean":"2024-09-21T19:09:57.215753+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:09:57.215753+0000","last_undegraded":"2024-09-21T19:09:57.215753+0000","last_fullsized":"2024-09-21T19:09:57.215753+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":752,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:58.355210+0000","last_change":"2024-09-21T19:09:48.894774+0000","last_active":"2024-09-21T19:09:58.355210+0000","last_peered":"2024-09-21T19:09:58.355210+0000","last_clean":"2024-09-21T19:09:58.355210+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:09:58.355210+0000","last_undegraded":"2024-09-21T19:09:58.355210+0000","last_fullsized":"2024-09-21T19:09:58.355210+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:09:48.894708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.894708+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.651349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":662,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-21T19:09:57.254869+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:09:57.254869+0000","last_peered":"2024-09-21T19:09:57.254869+0000","last_clean":"2024-09-21T19:09:57.254869+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:09:57.254869+0000","last_undegraded":"2024-09-21T19:09:57.254869+0000","last_fullsized":"2024-09-21T19:09:57.254869+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":998,"num_read_kb":993,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266894,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":606652,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93138500,"statfs":{"total":95995035648,"available":95373824000,"internally_reserved":0,"allocated":2138112,"data_stored":1321340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464345,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290640,"kb_used_data":1784,"kb_used_omap":4,"kb_used_meta":288827,"kb_avail":93454512,"statfs":{"total":95995035648,"available":95697420288,"internally_reserved":0,"allocated":1826816,"data_stored":1006495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":295759524},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619046,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":272272,"kb_used_data":2360,"kb_used_omap":4,"kb_used_meta":269883,"kb_avail":93472880,"statfs":{"total":95995035648,"available":95716229120,"internally_reserved":0,"allocated":2416640,"data_stored":1596773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":276360867},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019100,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538304,"kb_used_data":1768,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93206848,"statfs":{"total":95995035648,"available":95443812352,"internally_reserved":0,"allocated":1810432,"data_stored":993940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182625,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737460,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93007692,"statfs":{"total":95995035648,"available":95239876608,"internally_reserved":0,"allocated":2449408,"data_stored":1598091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411559,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560756,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93184396,"statfs":{"total":95995035648,"available":95420821504,"internally_reserved":0,"allocated":2064384,"data_stored":1262491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699440,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":697598,"kb_avail":93045712,"statfs":{"total":95995035648,"available":95278809088,"internally_reserved":0,"allocated":1859584,"data_stored":1001220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":714340805},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738609,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827072,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918080,"statfs":{"total":95995035648,"available":95148113920,"internally_reserved":0,"allocated":1814528,"data_stored":994349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:10:01.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T19:10:01.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:01.303 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:10:01.303 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:10:01.303 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T19:10:01.303 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:10:01.322 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:01.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:01.445 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:01.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:01.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:01 smithi081 ceph-mon[39192]: pgmap v835: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:10:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1198934144' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T19:10:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:01 smithi081 ceph-mon[39192]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T19:10:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:01 smithi081 ceph-mon[29384]: pgmap v835: 142 pgs: 142 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:10:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1198934144' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T19:10:01.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:01 smithi081 ceph-mon[29384]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T19:10:01.569 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:01.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:01.694 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:01.702 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:01.702 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:01.702 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:01.703 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:01.720 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:01.720 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:01.737 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:01.738 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:01.754 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:01.770 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:01.770 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:01.787 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:01.787 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:01.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:01.818 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:01.920 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:01.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:01.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:02.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:02.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:02.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:10:02.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T19:10:02.166 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:02.289 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:02.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:02.348 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:02.371 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:02.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:02.434 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:02.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:02 smithi104 ceph-mon[32438]: from='client.36455 v1:172.21.15.81:0/2413917169' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:02.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:02 smithi104 ceph-mon[32438]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T19:10:02.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3255082054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:02.493 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:02 smithi081 ceph-mon[29384]: from='client.36455 v1:172.21.15.81:0/2413917169' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:02.521 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:02 smithi081 ceph-mon[29384]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T19:10:02.521 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3255082054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:02.521 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:02 smithi081 ceph-mon[39192]: from='client.36455 v1:172.21.15.81:0/2413917169' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:02.521 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:02 smithi081 ceph-mon[39192]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T19:10:02.521 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3255082054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:02.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:02.557 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:02.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:02.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:02.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:02.801 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:02.845 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:02.846 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:02.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:02.925 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:02.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:03.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:03.047 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:03.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:03.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:03.261 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:10:03.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:03.294 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:03.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:03.417 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:03 smithi104 ceph-mon[32438]: pgmap v838: 158 pgs: 15 unknown, 143 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-09-21T19:10:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:03 smithi104 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:10:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:03 smithi104 ceph-mon[32438]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T19:10:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4030665772' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:03 smithi104 ceph-mon[32438]: from='client.36467 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:03.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:03.541 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[39192]: pgmap v838: 158 pgs: 15 unknown, 143 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-09-21T19:10:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[39192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:10:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[39192]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T19:10:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4030665772' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:03.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[39192]: from='client.36467 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:03.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[29384]: pgmap v838: 158 pgs: 15 unknown, 143 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-09-21T19:10:03.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[29384]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T19:10:03.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[29384]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T19:10:03.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4030665772' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:03.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:03 smithi081 ceph-mon[29384]: from='client.36467 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:03.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:03.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:10:03.665 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:03.764 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:03.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:03.788 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:03.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:03.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:03.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:03.915 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:04.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:04.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:04.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:04.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:04.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:04.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:04 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:04.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:04 smithi104 ceph-mon[32438]: from='client.36467 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:04.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:04 smithi104 ceph-mon[32438]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T19:10:04.352 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:04.352 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:04.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:04.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:04.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:04.530 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:04.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:04 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:04 smithi081 ceph-mon[39192]: from='client.36467 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:04 smithi081 ceph-mon[39192]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T19:10:04.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:04 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:04.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:04 smithi081 ceph-mon[29384]: from='client.36467 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:04.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:04 smithi081 ceph-mon[29384]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T19:10:04.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:04.653 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:04.755 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:04.775 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:04.791 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:04.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:04.898 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:04.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:05.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.023 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:05.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:05.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:10:05.124 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:05.145 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:05.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:05.257 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T19:10:05.274 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:05.318 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:05.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:05.396 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:05.438 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:05 smithi104 ceph-mon[32438]: pgmap v841: 158 pgs: 2 unknown, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:10:05.438 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/983309381' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:10:05.498 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:10:05.498 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1236,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":418,"num_osds":8,"num_up_osds":8,"osd_up_since":1726945614,"num_in_osds":8,"osd_in_since":1726945541,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":156},{"state_name":"unknown","count":2}],"num_pgs":158,"num_pools":7,"num_objects":199,"data_bytes":596507,"bytes_used":4619268096,"bytes_avail":763341017088,"bytes_total":767960285184,"unknown_pgs_ratio":0.012658228166401386,"read_bytes_sec":1279,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"btime":"2024-09-21T18:47:59:361071+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.81:8443/","prometheus":"http://172.21.15.81:9283/"}},"servicemap":{"epoch":16,"modified":"2024-09-21T19:09:37.775080+0000","services":{"rgw":{"daemons":{"summary":"","14514":{"start_epoch":12,"start_stamp":"2024-09-21T18:53:14.512185+0000","gid":14514,"addr":"172.21.15.81:0/604236037","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5129-g0ecbb435 (0ecbb435437226bc091a0c2d4549ba25bf724efe) squid (dev)","ceph_version_short":"19.3.0-5129-g0ecbb435","container_hostname":"smithi081","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi081","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487280","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"795589cd-ca7e-4c5a-ab69-8d483c7dc9a8","zone_name":"default","zonegroup_id":"a87017f7-965d-49bd-b7a2-d72f4caa6227","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T19:10:05.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:05 smithi081 ceph-mon[39192]: pgmap v841: 158 pgs: 2 unknown, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:10:05.498 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/983309381' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:10:05.499 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:05 smithi081 ceph-mon[29384]: pgmap v841: 158 pgs: 2 unknown, 156 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:10:05.499 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/983309381' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T19:10:05.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:05.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:05.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:05.643 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:05.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:05.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:10:05.824 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:05.824 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:05.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:05.892 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:05.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:05.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:06.015 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:06.117 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:06.138 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:06.199 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-21T19:10:06.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:10:06.241 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:06.261 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:06.363 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:06.384 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:06.472 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/983309381' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:10:06.473 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:06 smithi104 ceph-mon[32438]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T19:10:06.473 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2086567021' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:10:06.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:06.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:10:06.507 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:06.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/983309381' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:10:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:06 smithi081 ceph-mon[39192]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T19:10:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2086567021' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:10:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/983309381' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T19:10:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:06 smithi081 ceph-mon[29384]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T19:10:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2086567021' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T19:10:06.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:06.631 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:06.677 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:06.681 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:06.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:06.744 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:06.755 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:06.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:10:06.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:06.881 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:06.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:06.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:07.005 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:07.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:07.127 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:07.226 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:10:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:10:07.229 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:07.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:07.301 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:07.301 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:07.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:07.373 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:07.475 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:07.496 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:07 smithi104 ceph-mon[32438]: pgmap v843: 158 pgs: 158 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:10:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:07 smithi104 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:10:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:07 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:07 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[29384]: pgmap v843: 158 pgs: 158 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[29384]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[39192]: pgmap v843: 158 pgs: 158 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[39192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T19:10:07.561 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:07.562 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:07 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:07.598 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:10:07.600 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:10:07.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:07.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:07.635 DEBUG:teuthology.orchestra.run.smithi081:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T19:10:07.670 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T19:10:07.670 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T19:10:07.670 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:07.671 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:07.687 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:07.687 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:07.703 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:07.703 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:07.720 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:07.720 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:07.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:07.730 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:hints = 1 2024-09-21T19:10:07.730 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T19:10:07.730 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object prefix: benchmark_data_smithi081_181703 2024-09-21T19:10:07.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:10:07.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 0 0 0 0 0 0 - 0 2024-09-21T19:10:07.747 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:07.747 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:07.756 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:07.783 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:07.783 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:07.817 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:07.817 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:07.849 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:07.849 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:07.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:07.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:07.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:07.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:10:08.016 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:08.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:08.155 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:08.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:08.292 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:08.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:08.429 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:08.429 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:08.531 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:08.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1025762151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:08.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:08 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:10:08.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:08 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:10:08.563 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:08.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1025762151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:08.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:08 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:10:08.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:08 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:10:08.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1025762151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:08.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:08 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:10:08.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:08 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:10:08.666 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:08.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:08.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 1 15 3579 3564 222.738 222.75 0.00459299 0.00447478 2024-09-21T19:10:08.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:08.832 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:08.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:10:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:10:08.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:08.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:09.076 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:09.114 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:09.162 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:10:09.162 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149397 10.a 10:5708e5f1:::benchmark_data_smithi081_155596_object31924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.95028439799999, 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56206911699999995, 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.163 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149397 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3900000000000005e-07 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207083+0000", 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.142e-06 2024-09-21T19:10:09.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207087+0000", 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6809999999999997e-06 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207092+0000", 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4989999999999996e-06 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207456+0000", 2024-09-21T19:10:09.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036441799999999998 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207464+0000", 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2269999999999993e-06 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.765864+0000", 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.55839969 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.765987+0000", 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012294199999999999 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.766111+0000", 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124268 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.768991+0000", 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879904 2024-09-21T19:10:09.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769047+0000", 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5457000000000001e-05 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769062+0000", 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5401999999999999e-05 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769108+0000", 2024-09-21T19:10:09.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5580000000000001e-05 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769149+0000", 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1368000000000001e-05 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149401 10.a 10:5176693a:::benchmark_data_smithi081_155596_object31928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.207800+0000", 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.949564434, 2024-09-21T19:10:09.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.57158363599999995, 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149401 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207800+0000", 2024-09-21T19:10:09.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207800+0000", 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207801+0000", 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2899999999999999e-06 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207806+0000", 2024-09-21T19:10:09.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9529999999999998e-06 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207818+0000", 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1945e-05 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207825+0000", 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6839999999999999e-06 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207872+0000", 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6999000000000002e-05 2024-09-21T19:10:09.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207879+0000", 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.379e-06 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769179+0000", 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56129959100000004 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769253+0000", 2024-09-21T19:10:09.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3997000000000003e-05 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769326+0000", 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2886000000000007e-05 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774586+0000", 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052604319999999998 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774601+0000", 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4745e-05 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779292+0000", 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046908790000000002 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779342+0000", 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0544000000000003e-05 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779384+0000", 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1312e-05 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149415 10.a 10:58e095b4:::benchmark_data_smithi081_155596_object31942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.221767+0000", 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.93559771, 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.55778803399999999, 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149415 2024-09-21T19:10:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221767+0000", 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221767+0000", 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221768+0000", 2024-09-21T19:10:09.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.728e-06 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221774+0000", 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2020000000000003e-06 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221787+0000", 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3413e-05 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221795+0000", 2024-09-21T19:10:09.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9840000000000001e-06 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221956+0000", 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016143 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221966+0000", 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0108999999999999e-05 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.771078+0000", 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54911152699999999 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.771155+0000", 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7137000000000004e-05 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.771223+0000", 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8151000000000005e-05 2024-09-21T19:10:09.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774649+0000", 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034256159999999998 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774713+0000", 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3899000000000006e-05 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779438+0000", 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047251480000000002 2024-09-21T19:10:09.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779491+0000", 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3065000000000001e-05 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779555+0000", 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3625000000000002e-05 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.184 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149439 10.a 10:598e2f20:::benchmark_data_smithi081_155596_object31966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.242682+0000", 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.914682107, 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.537047512, 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.185 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149439 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242682+0000", 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242682+0000", 2024-09-21T19:10:09.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242683+0000", 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5000000000000001e-07 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242686+0000", 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9239999999999999e-06 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242710+0000", 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128e-05 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242716+0000", 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5300000000000004e-06 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242772+0000", 2024-09-21T19:10:09.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5874000000000001e-05 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242781+0000", 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1039999999999996e-06 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.772708+0000", 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.529927284 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.772783+0000", 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5451999999999995e-05 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.772849+0000", 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5692999999999996e-05 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774701+0000", 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001851669 2024-09-21T19:10:09.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774719+0000", 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8284999999999999e-05 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779643+0000", 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049241600000000003 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779693+0000", 2024-09-21T19:10:09.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9941000000000002e-05 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779730+0000", 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6517999999999998e-05 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.192 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149442 10.a 10:57628efb:::benchmark_data_smithi081_155596_object31969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.246877+0000", 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.91048689500001, 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.53906758300000002, 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149442 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.193 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246877+0000", 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246877+0000", 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246878+0000", 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9100000000000003e-07 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246881+0000", 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9359999999999999e-06 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246897+0000", 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5753e-05 2024-09-21T19:10:09.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246908+0000", 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1532000000000001e-05 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246972+0000", 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3788000000000007e-05 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246985+0000", 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.308e-05 2024-09-21T19:10:09.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774753+0000", 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52776764600000003 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774857+0000", 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010378 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774969+0000", 2024-09-21T19:10:09.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011192 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779566+0000", 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045972069999999999 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779600+0000", 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4319999999999997e-05 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785815+0000", 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006214448 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785901+0000", 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6349999999999998e-05 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785945+0000", 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4131999999999997e-05 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149455 10.a 10:5dea2684:::benchmark_data_smithi081_155596_object31982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.260978+0000", 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.89638682499999, 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52510691499999995, 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149455 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260978+0000", 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260978+0000", 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260980+0000", 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0310000000000001e-06 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260988+0000", 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6880000000000005e-06 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260999+0000", 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0530000000000001e-05 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.261007+0000", 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0239999999999997e-06 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.261093+0000", 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6580000000000001e-05 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.261103+0000", 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4380000000000001e-06 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.777104+0000", 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51600082300000005 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.777218+0000", 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113892 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.777337+0000", 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119475 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779740+0000", 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002402829 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779759+0000", 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9296999999999998e-05 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785986+0000", 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062272100000000004 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786052+0000", 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5537999999999995e-05 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786084+0000", 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.256e-05 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149476 10.a 10:54c43eb7:::benchmark_data_smithi081_155596_object32003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.279885+0000", 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.87747925399999, 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51528070100000001, 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.207 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149476 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279885+0000", 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279885+0000", 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279886+0000", 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.285e-06 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279891+0000", 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4580000000000003e-06 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279900+0000", 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7649999999999999e-06 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279907+0000", 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1690000000000003e-06 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.280422+0000", 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00051480700000000005 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.280427+0000", 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6879999999999996e-06 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779798+0000", 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49937083500000001 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779910+0000", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111414 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.780020+0000", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011003899999999999 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786130+0000", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006110959 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792039+0000", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059088609999999996 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795063+0000", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030232560000000002 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795120+0000", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7145999999999998e-05 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795166+0000", 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6019000000000001e-05 2024-09-21T19:10:09.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149487 10.a 10:5fee5d06:::benchmark_data_smithi081_155596_object32014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.288631+0000", 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.86873301200001, 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51888262600000001, 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149487 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288631+0000", 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288631+0000", 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288632+0000", 2024-09-21T19:10:09.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3900000000000003e-07 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288636+0000", 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6569999999999999e-06 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288642+0000", 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7769999999999999e-06 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288646+0000", 2024-09-21T19:10:09.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1169999999999997e-06 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288720+0000", 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3895999999999999e-05 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288725+0000", 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5419999999999996e-06 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.782607+0000", 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49388191799999998 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.782847+0000", 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023967699999999999 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.782968+0000", 2024-09-21T19:10:09.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120704 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786166+0000", 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031983300000000001 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794941+0000", 2024-09-21T19:10:09.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.008774703 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807354+0000", 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012413258999999999 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807455+0000", 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010142800000000001 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807514+0000", 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8678999999999998e-05 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149495 10.a 10:5fba34ce:::benchmark_data_smithi081_155596_object32022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.295642+0000", 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.861722238, 2024-09-21T19:10:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51213065599999996, 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149495 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295642+0000", 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295642+0000", 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295643+0000", 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.328e-06 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295648+0000", 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3159999999999998e-06 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295655+0000", 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9800000000000001e-06 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295661+0000", 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9309999999999996e-06 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295735+0000", 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4820999999999994e-05 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295746+0000", 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0433e-05 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786197+0000", 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49045062099999998 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786299+0000", 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102802 2024-09-21T19:10:09.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786409+0000", 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109434 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794946+0000", 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0085368019999999996 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794987+0000", 2024-09-21T19:10:09.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1854000000000003e-05 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807610+0000", 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012622301000000001 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807731+0000", 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120884 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807773+0000", 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2148999999999999e-05 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149497 10.a 10:5debb899:::benchmark_data_smithi081_155596_object32024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.298977+0000", 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.858387724, 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50910686000000005, 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.229 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149497 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298977+0000", 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298977+0000", 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298978+0000", 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.373e-06 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298983+0000", 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8189999999999999e-06 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298992+0000", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9209999999999998e-06 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298998+0000", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3199999999999996e-06 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.299066+0000", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7823000000000004e-05 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.299076+0000", 2024-09-21T19:10:09.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0282000000000001e-05 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.789200+0000", 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.490123859 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.789337+0000", 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013721999999999999 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.789456+0000", 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001186 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794998+0000", 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055423909999999998 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801375+0000", 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063767989999999998 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807817+0000", 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064422699999999999 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808046+0000", 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022911599999999999 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808083+0000", 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7067000000000003e-05 2024-09-21T19:10:09.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149511 10.a 10:59ccc354:::benchmark_data_smithi081_155596_object32038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.312604+0000", 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.844760196, 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49560010599999998, 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.236 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149511 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312604+0000", 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312604+0000", 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312606+0000", 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6130000000000001e-06 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312611+0000", 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9359999999999998e-06 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312623+0000", 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.259e-05 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312631+0000", 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3100000000000003e-06 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312692+0000", 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0998000000000001e-05 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312703+0000", 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1439e-05 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792059+0000", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47935586499999999 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792135+0000", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6219999999999999e-05 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792258+0000", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000122569 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801418+0000", 2024-09-21T19:10:09.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091600880000000003 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807547+0000", 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061290930000000004 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808125+0000", 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057815500000000001 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808172+0000", 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6820999999999998e-05 2024-09-21T19:10:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808204+0000", 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2409000000000001e-05 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149512 10.a 10:5cee4d35:::benchmark_data_smithi081_155596_object32039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.313008+0000", 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.844356619, 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50836416399999995, 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149512 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313008+0000", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313008+0000", 2024-09-21T19:10:09.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313009+0000", 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.215e-06 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313013+0000", 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.121e-06 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313023+0000", 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0436e-05 2024-09-21T19:10:09.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313029+0000", 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2979999999999999e-06 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313065+0000", 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6677000000000001e-05 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313071+0000", 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.152e-06 2024-09-21T19:10:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795199+0000", 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48212796699999999 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795319+0000", 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120629 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795444+0000", 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124337 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807553+0000", 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012109488999999999 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808248+0000", 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069539600000000003 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821240+0000", 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012991864000000001 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821311+0000", 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0576999999999999e-05 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821372+0000", 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1005999999999998e-05 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149517 10.a 10:5b635e0f:::benchmark_data_smithi081_155596_object32044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.319426+0000", 2024-09-21T19:10:09.248 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.837938806, 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50211374600000003, 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149517 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319426+0000", 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319426+0000", 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319427+0000", 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5349999999999999e-06 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319432+0000", 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7779999999999998e-06 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319445+0000", 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3652000000000001e-05 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319451+0000", 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9599999999999997e-06 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319518+0000", 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6574000000000003e-05 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319535+0000", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6835999999999999e-05 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.798217+0000", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47868188699999997 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.798331+0000", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114641 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.798460+0000", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012896199999999999 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807558+0000", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0090974139999999998 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808283+0000", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072537299999999997 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821438+0000", 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013154654 2024-09-21T19:10:09.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821490+0000", 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2448999999999999e-05 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821539+0000", 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9030999999999999e-05 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149518 10.a 10:51048542:::benchmark_data_smithi081_155596_object32045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.320165+0000", 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.83719980399999, 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50150713199999997, 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149518 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.255 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320165+0000", 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320165+0000", 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320166+0000", 2024-09-21T19:10:09.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0330000000000001e-06 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320172+0000", 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6839999999999999e-06 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320181+0000", 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0950000000000002e-06 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320187+0000", 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4330000000000003e-06 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320231+0000", 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3899000000000001e-05 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320236+0000", 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2859999999999999e-06 2024-09-21T19:10:09.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801454+0000", 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48121767599999998 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801604+0000", 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000150123 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801737+0000", 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133153 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807562+0000", 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058254550000000002 2024-09-21T19:10:09.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808500+0000", 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093748900000000001 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821600+0000", 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013100067999999999 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821650+0000", 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0327000000000001e-05 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821672+0000", 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1410999999999999e-05 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149533 10.a 10:506d36d3:::benchmark_data_smithi081_155596_object32060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.355055+0000", 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.80230926199999, 2024-09-21T19:10:09.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46668358700000001, 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149533 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355055+0000", 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355055+0000", 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355058+0000", 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7420000000000002e-06 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355066+0000", 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.208e-06 2024-09-21T19:10:09.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355084+0000", 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8371000000000001e-05 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355094+0000", 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8689999999999999e-06 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355167+0000", 2024-09-21T19:10:09.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3161e-05 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355180+0000", 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2319e-05 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.804086+0000", 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.44890657899999997 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.804216+0000", 2024-09-21T19:10:09.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013010100000000001 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.804346+0000", 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012922000000000001 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807567+0000", 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032212080000000001 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.814723+0000", 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0071559900000000001 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821698+0000", 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069747979999999999 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821721+0000", 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2918999999999999e-05 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821739+0000", 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8102e-05 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149657 10.0 10:00905fd5:::benchmark_data_smithi081_155596_object32184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.881834+0000", 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.275530223, 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30976963600000001, 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.270 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149657 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881834+0000", 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881834+0000", 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881836+0000", 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.84e-06 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881841+0000", 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2610000000000001e-06 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881854+0000", 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2709e-05 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881861+0000", 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.34e-06 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.881927+0000", 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5289000000000005e-05 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.882034+0000", 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107107 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.882135+0000", 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000101643 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.885018+0000", 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002883053 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.885622+0000", 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060387899999999998 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.191514+0000", 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30589223799999998 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.191569+0000", 2024-09-21T19:10:09.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4792e-05 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.191604+0000", 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4484999999999999e-05 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149663 10.0 10:0e2e1098:::benchmark_data_smithi081_155596_object32190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.887455+0000", 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.269909583, 2024-09-21T19:10:09.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31268661800000003, 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149663 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887455+0000", 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887455+0000", 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887457+0000", 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.046e-06 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887463+0000", 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9909999999999999e-06 2024-09-21T19:10:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887474+0000", 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1377e-05 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887481+0000", 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1149999999999996e-06 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887632+0000", 2024-09-21T19:10:09.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015040799999999999 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887749+0000", 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117589 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887859+0000", 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109308 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.890594+0000", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027349150000000001 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.894705+0000", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041116779999999997 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200077+0000", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30537224099999999 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200115+0000", 2024-09-21T19:10:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8031000000000002e-05 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200141+0000", 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5919e-05 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149673 10.0 10:0f1b7172:::benchmark_data_smithi081_155596_object32200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.895866+0000", 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 109.261498011, 2024-09-21T19:10:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30436275200000001, 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149673 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.895866+0000", 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.895866+0000", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.895868+0000", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6810000000000001e-06 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.895873+0000", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1189999999999996e-06 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.895886+0000", 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3101999999999999e-05 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.895894+0000", 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2799999999999998e-06 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.896080+0000", 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000186768 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.896197+0000", 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116718 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.896316+0000", 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119058 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.899760+0000", 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034436919999999999 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.900131+0000", 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037118999999999997 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200170+0000", 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30003894199999998 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200204+0000", 2024-09-21T19:10:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4143000000000002e-05 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200229+0000", 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5058999999999998e-05 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:189632 10.7 10:e3409a5e:::benchmark_data_smithi081_155596_object72159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:10:09.288 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.876272482999994, 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.310254274, 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 189632 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:10:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4799999999999998e-07 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281095+0000", 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3769999999999999e-06 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281103+0000", 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8269999999999998e-06 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281107+0000", 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6480000000000002e-06 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.292262+0000", 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011154181000000001 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.292359+0000", 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7398999999999994e-05 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.304951+0000", 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012592348 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.305035+0000", 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3821999999999998e-05 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.305160+0000", 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012451500000000001 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.311340+0000", 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061803090000000001 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.319670+0000", 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083303030000000007 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.590170+0000", 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.27049980099999998 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.590223+0000", 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.304e-05 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.591346+0000", 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011230560000000001 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:189645 10.7 10:e34fa208:::benchmark_data_smithi081_155596_object72172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:09:02.289200+0000", 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.868164809000007, 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30089073999999999, 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:09.295 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 189645 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.289200+0000", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.289200+0000", 2024-09-21T19:10:09.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.289201+0000", 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.728e-06 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.289206+0000", 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5700000000000003e-06 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.289217+0000", 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.113e-05 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.289225+0000", 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0029999999999994e-06 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.297288+0000", 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.008063377 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.297423+0000", 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013432099999999999 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.297561+0000", 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137854 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.308321+0000", 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010760485 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.308371+0000", 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0090000000000003e-05 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.588695+0000", 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.280324291 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.588753+0000", 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7847999999999999e-05 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:09.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.590090+0000", 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001337043 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:09.301 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:10:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:09.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:09.310 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":843,"stamp":"2024-09-21T19:10:05.782853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2142,"num_read_kb":1894,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":272632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":454,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4510728,"kb_used_data":13884,"kb_used_omap":18,"kb_used_meta":4495533,"kb_avail":745450488,"statfs":{"total":767960285184,"available":763341299712,"internally_reserved":0,"allocated":14217216,"data_stored":8557171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4603426800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.542649"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":678,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251134+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:10:02.251134+0000","last_peered":"2024-09-21T19:10:02.251134+0000","last_clean":"2024-09-21T19:10:02.251134+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:10:02.251134+0000","last_undegraded":"2024-09-21T19:10:02.251134+0000","last_fullsized":"2024-09-21T19:10:02.251134+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:10.748546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":837,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.241699+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:10:01.241699+0000","last_peered":"2024-09-21T19:10:01.241699+0000","last_clean":"2024-09-21T19:10:01.241699+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:10:01.241699+0000","last_undegraded":"2024-09-21T19:10:01.241699+0000","last_fullsized":"2024-09-21T19:10:01.241699+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":776,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253326+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:10:02.253326+0000","last_peered":"2024-09-21T19:10:02.253326+0000","last_clean":"2024-09-21T19:10:02.253326+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:10:02.253326+0000","last_undegraded":"2024-09-21T19:10:02.253326+0000","last_fullsized":"2024-09-21T19:10:02.253326+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.809763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":754,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.241826+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:10:01.241826+0000","last_peered":"2024-09-21T19:10:01.241826+0000","last_clean":"2024-09-21T19:10:01.241826+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:10:01.241826+0000","last_undegraded":"2024-09-21T19:10:01.241826+0000","last_fullsized":"2024-09-21T19:10:01.241826+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":765,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254309+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:10:02.254309+0000","last_peered":"2024-09-21T19:10:02.254309+0000","last_clean":"2024-09-21T19:10:02.254309+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:10:02.254309+0000","last_undegraded":"2024-09-21T19:10:02.254309+0000","last_fullsized":"2024-09-21T19:10:02.254309+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":793,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252371+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:10:02.252371+0000","last_peered":"2024-09-21T19:10:02.252371+0000","last_clean":"2024-09-21T19:10:02.252371+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:10:02.252371+0000","last_undegraded":"2024-09-21T19:10:02.252371+0000","last_fullsized":"2024-09-21T19:10:02.252371+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":889,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254121+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:10:02.254121+0000","last_peered":"2024-09-21T19:10:02.254121+0000","last_clean":"2024-09-21T19:10:02.254121+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:10:02.254121+0000","last_undegraded":"2024-09-21T19:10:02.254121+0000","last_fullsized":"2024-09-21T19:10:02.254121+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":774,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.241645+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:10:01.241645+0000","last_peered":"2024-09-21T19:10:01.241645+0000","last_clean":"2024-09-21T19:10:01.241645+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:10:01.241645+0000","last_undegraded":"2024-09-21T19:10:01.241645+0000","last_fullsized":"2024-09-21T19:10:01.241645+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":761,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242968+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:10:01.242968+0000","last_peered":"2024-09-21T19:10:01.242968+0000","last_clean":"2024-09-21T19:10:01.242968+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:10:01.242968+0000","last_undegraded":"2024-09-21T19:10:01.242968+0000","last_fullsized":"2024-09-21T19:10:01.242968+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":818,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242182+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:10:01.242182+0000","last_peered":"2024-09-21T19:10:01.242182+0000","last_clean":"2024-09-21T19:10:01.242182+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:10:01.242182+0000","last_undegraded":"2024-09-21T19:10:01.242182+0000","last_fullsized":"2024-09-21T19:10:01.242182+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":725,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.456948+0000","last_change":"2024-09-21T19:09:56.743173+0000","last_active":"2024-09-21T19:10:02.456948+0000","last_peered":"2024-09-21T19:10:02.456948+0000","last_clean":"2024-09-21T19:10:02.456948+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:10:02.456948+0000","last_undegraded":"2024-09-21T19:10:02.456948+0000","last_fullsized":"2024-09-21T19:10:02.456948+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.743123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.743123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.761375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":636,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.250495+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:10:02.250495+0000","last_peered":"2024-09-21T19:10:02.250495+0000","last_clean":"2024-09-21T19:10:02.250495+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:10:02.250495+0000","last_undegraded":"2024-09-21T19:10:02.250495+0000","last_fullsized":"2024-09-21T19:10:02.250495+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.204017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":314,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242373+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:10:01.242373+0000","last_peered":"2024-09-21T19:10:01.242373+0000","last_clean":"2024-09-21T19:10:01.242373+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:10:01.242373+0000","last_undegraded":"2024-09-21T19:10:01.242373+0000","last_fullsized":"2024-09-21T19:10:01.242373+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.643073+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":764,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242026+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:10:01.242026+0000","last_peered":"2024-09-21T19:10:01.242026+0000","last_clean":"2024-09-21T19:10:01.242026+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:10:01.242026+0000","last_undegraded":"2024-09-21T19:10:01.242026+0000","last_fullsized":"2024-09-21T19:10:01.242026+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":754,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245204+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:10:01.245204+0000","last_peered":"2024-09-21T19:10:01.245204+0000","last_clean":"2024-09-21T19:10:01.245204+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:10:01.245204+0000","last_undegraded":"2024-09-21T19:10:01.245204+0000","last_fullsized":"2024-09-21T19:10:01.245204+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":324,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252986+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:10:02.252986+0000","last_peered":"2024-09-21T19:10:02.252986+0000","last_clean":"2024-09-21T19:10:02.252986+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:10:02.252986+0000","last_undegraded":"2024-09-21T19:10:02.252986+0000","last_fullsized":"2024-09-21T19:10:02.252986+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":783,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242345+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:10:01.242345+0000","last_peered":"2024-09-21T19:10:01.242345+0000","last_clean":"2024-09-21T19:10:01.242345+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:10:01.242345+0000","last_undegraded":"2024-09-21T19:10:01.242345+0000","last_fullsized":"2024-09-21T19:10:01.242345+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":652,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252253+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:10:02.252253+0000","last_peered":"2024-09-21T19:10:02.252253+0000","last_clean":"2024-09-21T19:10:02.252253+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:10:02.252253+0000","last_undegraded":"2024-09-21T19:10:02.252253+0000","last_fullsized":"2024-09-21T19:10:02.252253+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":227,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455638+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:10:02.455638+0000","last_peered":"2024-09-21T19:10:02.455638+0000","last_clean":"2024-09-21T19:10:02.455638+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:10:02.455638+0000","last_undegraded":"2024-09-21T19:10:02.455638+0000","last_fullsized":"2024-09-21T19:10:02.455638+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:24.726173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":779,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252469+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:10:02.252469+0000","last_peered":"2024-09-21T19:10:02.252469+0000","last_clean":"2024-09-21T19:10:02.252469+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:10:02.252469+0000","last_undegraded":"2024-09-21T19:10:02.252469+0000","last_fullsized":"2024-09-21T19:10:02.252469+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:07.738704+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":762,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254858+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:10:02.254858+0000","last_peered":"2024-09-21T19:10:02.254858+0000","last_clean":"2024-09-21T19:10:02.254858+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:10:02.254858+0000","last_undegraded":"2024-09-21T19:10:02.254858+0000","last_fullsized":"2024-09-21T19:10:02.254858+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":785,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242306+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:10:01.242306+0000","last_peered":"2024-09-21T19:10:01.242306+0000","last_clean":"2024-09-21T19:10:01.242306+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:10:01.242306+0000","last_undegraded":"2024-09-21T19:10:01.242306+0000","last_fullsized":"2024-09-21T19:10:01.242306+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":832,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251768+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:10:02.251768+0000","last_peered":"2024-09-21T19:10:02.251768+0000","last_clean":"2024-09-21T19:10:02.251768+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:10:02.251768+0000","last_undegraded":"2024-09-21T19:10:02.251768+0000","last_fullsized":"2024-09-21T19:10:02.251768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":769,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242103+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:10:01.242103+0000","last_peered":"2024-09-21T19:10:01.242103+0000","last_clean":"2024-09-21T19:10:01.242103+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:10:01.242103+0000","last_undegraded":"2024-09-21T19:10:01.242103+0000","last_fullsized":"2024-09-21T19:10:01.242103+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":792,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245970+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:10:01.245970+0000","last_peered":"2024-09-21T19:10:01.245970+0000","last_clean":"2024-09-21T19:10:01.245970+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:10:01.245970+0000","last_undegraded":"2024-09-21T19:10:01.245970+0000","last_fullsized":"2024-09-21T19:10:01.245970+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.611692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":793,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253412+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:10:02.253412+0000","last_peered":"2024-09-21T19:10:02.253412+0000","last_clean":"2024-09-21T19:10:02.253412+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:10:02.253412+0000","last_undegraded":"2024-09-21T19:10:02.253412+0000","last_fullsized":"2024-09-21T19:10:02.253412+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:08.469307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":681,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251125+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:10:02.251125+0000","last_peered":"2024-09-21T19:10:02.251125+0000","last_clean":"2024-09-21T19:10:02.251125+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:10:02.251125+0000","last_undegraded":"2024-09-21T19:10:02.251125+0000","last_fullsized":"2024-09-21T19:10:02.251125+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":862,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.512214+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:10:03.512214+0000","last_peered":"2024-09-21T19:10:03.512214+0000","last_clean":"2024-09-21T19:10:03.512214+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:10:03.512214+0000","last_undegraded":"2024-09-21T19:10:03.512214+0000","last_fullsized":"2024-09-21T19:10:03.512214+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253966+0000","last_change":"2024-09-21T19:10:01.248307+0000","last_active":"2024-09-21T19:10:02.253966+0000","last_peered":"2024-09-21T19:10:02.253966+0000","last_clean":"2024-09-21T19:10:02.253966+0000","last_became_active":"2024-09-21T19:10:01.248029+0000","last_became_peered":"2024-09-21T19:10:01.248029+0000","last_unstale":"2024-09-21T19:10:02.253966+0000","last_undegraded":"2024-09-21T19:10:02.253966+0000","last_fullsized":"2024-09-21T19:10:02.253966+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:08.979050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":223,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455524+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:10:02.455524+0000","last_peered":"2024-09-21T19:10:02.455524+0000","last_clean":"2024-09-21T19:10:02.455524+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:10:02.455524+0000","last_undegraded":"2024-09-21T19:10:02.455524+0000","last_fullsized":"2024-09-21T19:10:02.455524+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:13.700777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":829,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253836+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:10:02.253836+0000","last_peered":"2024-09-21T19:10:02.253836+0000","last_clean":"2024-09-21T19:10:02.253836+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:10:02.253836+0000","last_undegraded":"2024-09-21T19:10:02.253836+0000","last_fullsized":"2024-09-21T19:10:02.253836+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:13.885784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":767,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254830+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:10:02.254830+0000","last_peered":"2024-09-21T19:10:02.254830+0000","last_clean":"2024-09-21T19:10:02.254830+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:10:02.254830+0000","last_undegraded":"2024-09-21T19:10:02.254830+0000","last_fullsized":"2024-09-21T19:10:02.254830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.830842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1010,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242396+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:10:01.242396+0000","last_peered":"2024-09-21T19:10:01.242396+0000","last_clean":"2024-09-21T19:10:01.242396+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:10:01.242396+0000","last_undegraded":"2024-09-21T19:10:01.242396+0000","last_fullsized":"2024-09-21T19:10:01.242396+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":449,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252372+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:10:02.252372+0000","last_peered":"2024-09-21T19:10:02.252372+0000","last_clean":"2024-09-21T19:10:02.252372+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:10:02.252372+0000","last_undegraded":"2024-09-21T19:10:02.252372+0000","last_fullsized":"2024-09-21T19:10:02.252372+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:22.031875+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.252711+0000","last_change":"2024-09-21T19:10:01.252308+0000","last_active":"2024-09-21T19:10:01.252711+0000","last_peered":"2024-09-21T19:10:01.252711+0000","last_clean":"2024-09-21T19:10:01.252711+0000","last_became_active":"2024-09-21T19:10:01.252017+0000","last_became_peered":"2024-09-21T19:10:01.252017+0000","last_unstale":"2024-09-21T19:10:01.252711+0000","last_undegraded":"2024-09-21T19:10:01.252711+0000","last_fullsized":"2024-09-21T19:10:01.252711+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.639057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":554,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242962+0000","last_change":"2024-09-21T19:09:53.595884+0000","last_active":"2024-09-21T19:10:01.242962+0000","last_peered":"2024-09-21T19:10:01.242962+0000","last_clean":"2024-09-21T19:10:01.242962+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:10:01.242962+0000","last_undegraded":"2024-09-21T19:10:01.242962+0000","last_fullsized":"2024-09-21T19:10:01.242962+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:15.381125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":856,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254419+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:10:02.254419+0000","last_peered":"2024-09-21T19:10:02.254419+0000","last_clean":"2024-09-21T19:10:02.254419+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:10:02.254419+0000","last_undegraded":"2024-09-21T19:10:02.254419+0000","last_fullsized":"2024-09-21T19:10:02.254419+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":781,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.255270+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:10:02.255270+0000","last_peered":"2024-09-21T19:10:02.255270+0000","last_clean":"2024-09-21T19:10:02.255270+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:10:02.255270+0000","last_undegraded":"2024-09-21T19:10:02.255270+0000","last_fullsized":"2024-09-21T19:10:02.255270+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":757,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.241753+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:10:01.241753+0000","last_peered":"2024-09-21T19:10:01.241753+0000","last_clean":"2024-09-21T19:10:01.241753+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:10:01.241753+0000","last_undegraded":"2024-09-21T19:10:01.241753+0000","last_fullsized":"2024-09-21T19:10:01.241753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:59.662855+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":774,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242101+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:10:01.242101+0000","last_peered":"2024-09-21T19:10:01.242101+0000","last_clean":"2024-09-21T19:10:01.242101+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:10:01.242101+0000","last_undegraded":"2024-09-21T19:10:01.242101+0000","last_fullsized":"2024-09-21T19:10:01.242101+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.249082+0000","last_change":"2024-09-21T19:10:01.248578+0000","last_active":"2024-09-21T19:10:01.249082+0000","last_peered":"2024-09-21T19:10:01.249082+0000","last_clean":"2024-09-21T19:10:01.249082+0000","last_became_active":"2024-09-21T19:10:01.248286+0000","last_became_peered":"2024-09-21T19:10:01.248286+0000","last_unstale":"2024-09-21T19:10:01.249082+0000","last_undegraded":"2024-09-21T19:10:01.249082+0000","last_fullsized":"2024-09-21T19:10:01.249082+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:08.972393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":539,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252524+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:10:02.252524+0000","last_peered":"2024-09-21T19:10:02.252524+0000","last_clean":"2024-09-21T19:10:02.252524+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:10:02.252524+0000","last_undegraded":"2024-09-21T19:10:02.252524+0000","last_fullsized":"2024-09-21T19:10:02.252524+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":825,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.241888+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:10:01.241888+0000","last_peered":"2024-09-21T19:10:01.241888+0000","last_clean":"2024-09-21T19:10:01.241888+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:10:01.241888+0000","last_undegraded":"2024-09-21T19:10:01.241888+0000","last_fullsized":"2024-09-21T19:10:01.241888+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":649,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251401+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:10:02.251401+0000","last_peered":"2024-09-21T19:10:02.251401+0000","last_clean":"2024-09-21T19:10:02.251401+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:10:02.251401+0000","last_undegraded":"2024-09-21T19:10:02.251401+0000","last_fullsized":"2024-09-21T19:10:02.251401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.184114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":956,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254240+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:10:02.254240+0000","last_peered":"2024-09-21T19:10:02.254240+0000","last_clean":"2024-09-21T19:10:02.254240+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:10:02.254240+0000","last_undegraded":"2024-09-21T19:10:02.254240+0000","last_fullsized":"2024-09-21T19:10:02.254240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":785,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.244950+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:10:01.244950+0000","last_peered":"2024-09-21T19:10:01.244950+0000","last_clean":"2024-09-21T19:10:01.244950+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:10:01.244950+0000","last_undegraded":"2024-09-21T19:10:01.244950+0000","last_fullsized":"2024-09-21T19:10:01.244950+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.662781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455595+0000","last_change":"2024-09-21T19:10:01.251501+0000","last_active":"2024-09-21T19:10:02.455595+0000","last_peered":"2024-09-21T19:10:02.455595+0000","last_clean":"2024-09-21T19:10:02.455595+0000","last_became_active":"2024-09-21T19:10:01.250354+0000","last_became_peered":"2024-09-21T19:10:01.250354+0000","last_unstale":"2024-09-21T19:10:02.455595+0000","last_undegraded":"2024-09-21T19:10:02.455595+0000","last_fullsized":"2024-09-21T19:10:02.455595+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.060118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":573,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.512330+0000","last_change":"2024-09-21T19:09:55.907018+0000","last_active":"2024-09-21T19:10:03.512330+0000","last_peered":"2024-09-21T19:10:03.512330+0000","last_clean":"2024-09-21T19:10:03.512330+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:10:03.512330+0000","last_undegraded":"2024-09-21T19:10:03.512330+0000","last_fullsized":"2024-09-21T19:10:03.512330+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:11.373203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":812,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252780+0000","last_change":"2024-09-21T19:09:39.038158+0000","last_active":"2024-09-21T19:10:02.252780+0000","last_peered":"2024-09-21T19:10:02.252780+0000","last_clean":"2024-09-21T19:10:02.252780+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:10:02.252780+0000","last_undegraded":"2024-09-21T19:10:02.252780+0000","last_fullsized":"2024-09-21T19:10:02.252780+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:09:39.038064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.038064+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.698592+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":331,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251854+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:10:02.251854+0000","last_peered":"2024-09-21T19:10:02.251854+0000","last_clean":"2024-09-21T19:10:02.251854+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:10:02.251854+0000","last_undegraded":"2024-09-21T19:10:02.251854+0000","last_fullsized":"2024-09-21T19:10:02.251854+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.609167+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":838,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252260+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:10:02.252260+0000","last_peered":"2024-09-21T19:10:02.252260+0000","last_clean":"2024-09-21T19:10:02.252260+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:10:02.252260+0000","last_undegraded":"2024-09-21T19:10:02.252260+0000","last_fullsized":"2024-09-21T19:10:02.252260+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":660,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.250640+0000","last_change":"2024-09-21T19:09:47.129430+0000","last_active":"2024-09-21T19:10:02.250640+0000","last_peered":"2024-09-21T19:10:02.250640+0000","last_clean":"2024-09-21T19:10:02.250640+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:10:02.250640+0000","last_undegraded":"2024-09-21T19:10:02.250640+0000","last_fullsized":"2024-09-21T19:10:02.250640+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:47.129379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:47.129379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.229738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253716+0000","last_change":"2024-09-21T19:10:01.244039+0000","last_active":"2024-09-21T19:10:02.253716+0000","last_peered":"2024-09-21T19:10:02.253716+0000","last_clean":"2024-09-21T19:10:02.253716+0000","last_became_active":"2024-09-21T19:10:01.243730+0000","last_became_peered":"2024-09-21T19:10:01.243730+0000","last_unstale":"2024-09-21T19:10:02.253716+0000","last_undegraded":"2024-09-21T19:10:02.253716+0000","last_fullsized":"2024-09-21T19:10:02.253716+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.843156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":551,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.511582+0000","last_change":"2024-09-21T19:09:54.937870+0000","last_active":"2024-09-21T19:10:03.511582+0000","last_peered":"2024-09-21T19:10:03.511582+0000","last_clean":"2024-09-21T19:10:03.511582+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:10:03.511582+0000","last_undegraded":"2024-09-21T19:10:03.511582+0000","last_fullsized":"2024-09-21T19:10:03.511582+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:57.511376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":697,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252316+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:10:02.252316+0000","last_peered":"2024-09-21T19:10:02.252316+0000","last_clean":"2024-09-21T19:10:02.252316+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:10:02.252316+0000","last_undegraded":"2024-09-21T19:10:02.252316+0000","last_fullsized":"2024-09-21T19:10:02.252316+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":800,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253615+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:10:02.253615+0000","last_peered":"2024-09-21T19:10:02.253615+0000","last_clean":"2024-09-21T19:10:02.253615+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:10:02.253615+0000","last_undegraded":"2024-09-21T19:10:02.253615+0000","last_fullsized":"2024-09-21T19:10:02.253615+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.451996+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":678,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.509928+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:10:03.509928+0000","last_peered":"2024-09-21T19:10:03.509928+0000","last_clean":"2024-09-21T19:10:03.509928+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:10:03.509928+0000","last_undegraded":"2024-09-21T19:10:03.509928+0000","last_fullsized":"2024-09-21T19:10:03.509928+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":780,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253307+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:10:02.253307+0000","last_peered":"2024-09-21T19:10:02.253307+0000","last_clean":"2024-09-21T19:10:02.253307+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:10:02.253307+0000","last_undegraded":"2024-09-21T19:10:02.253307+0000","last_fullsized":"2024-09-21T19:10:02.253307+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.144483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254481+0000","last_change":"2024-09-21T19:10:01.250359+0000","last_active":"2024-09-21T19:10:02.254481+0000","last_peered":"2024-09-21T19:10:02.254481+0000","last_clean":"2024-09-21T19:10:02.254481+0000","last_became_active":"2024-09-21T19:10:01.250079+0000","last_became_peered":"2024-09-21T19:10:01.250079+0000","last_unstale":"2024-09-21T19:10:02.254481+0000","last_undegraded":"2024-09-21T19:10:02.254481+0000","last_fullsized":"2024-09-21T19:10:02.254481+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.879067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":516,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251249+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:10:02.251249+0000","last_peered":"2024-09-21T19:10:02.251249+0000","last_clean":"2024-09-21T19:10:02.251249+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:10:02.251249+0000","last_undegraded":"2024-09-21T19:10:02.251249+0000","last_fullsized":"2024-09-21T19:10:02.251249+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1092,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253047+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:10:02.253047+0000","last_peered":"2024-09-21T19:10:02.253047+0000","last_clean":"2024-09-21T19:10:02.253047+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:10:02.253047+0000","last_undegraded":"2024-09-21T19:10:02.253047+0000","last_fullsized":"2024-09-21T19:10:02.253047+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.313979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":798,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242468+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:10:01.242468+0000","last_peered":"2024-09-21T19:10:01.242468+0000","last_clean":"2024-09-21T19:10:01.242468+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:10:01.242468+0000","last_undegraded":"2024-09-21T19:10:01.242468+0000","last_fullsized":"2024-09-21T19:10:01.242468+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":347,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253841+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:10:02.253841+0000","last_peered":"2024-09-21T19:10:02.253841+0000","last_clean":"2024-09-21T19:10:02.253841+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:10:02.253841+0000","last_undegraded":"2024-09-21T19:10:02.253841+0000","last_fullsized":"2024-09-21T19:10:02.253841+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":645,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.250717+0000","last_change":"2024-09-21T19:09:51.157757+0000","last_active":"2024-09-21T19:10:02.250717+0000","last_peered":"2024-09-21T19:10:02.250717+0000","last_clean":"2024-09-21T19:10:02.250717+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:10:02.250717+0000","last_undegraded":"2024-09-21T19:10:02.250717+0000","last_fullsized":"2024-09-21T19:10:02.250717+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.157711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.157711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:54.408363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":747,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252121+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:10:02.252121+0000","last_peered":"2024-09-21T19:10:02.252121+0000","last_clean":"2024-09-21T19:10:02.252121+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:10:02.252121+0000","last_undegraded":"2024-09-21T19:10:02.252121+0000","last_fullsized":"2024-09-21T19:10:02.252121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.255154+0000","last_change":"2024-09-21T19:10:01.259760+0000","last_active":"2024-09-21T19:10:02.255154+0000","last_peered":"2024-09-21T19:10:02.255154+0000","last_clean":"2024-09-21T19:10:02.255154+0000","last_became_active":"2024-09-21T19:10:01.259451+0000","last_became_peered":"2024-09-21T19:10:01.259451+0000","last_unstale":"2024-09-21T19:10:02.255154+0000","last_undegraded":"2024-09-21T19:10:02.255154+0000","last_fullsized":"2024-09-21T19:10:02.255154+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:17.731865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":858,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245006+0000","last_change":"2024-09-21T19:09:55.484547+0000","last_active":"2024-09-21T19:10:01.245006+0000","last_peered":"2024-09-21T19:10:01.245006+0000","last_clean":"2024-09-21T19:10:01.245006+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:10:01.245006+0000","last_undegraded":"2024-09-21T19:10:01.245006+0000","last_fullsized":"2024-09-21T19:10:01.245006+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:15.423137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":265,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253586+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:10:02.253586+0000","last_peered":"2024-09-21T19:10:02.253586+0000","last_clean":"2024-09-21T19:10:02.253586+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:10:02.253586+0000","last_undegraded":"2024-09-21T19:10:02.253586+0000","last_fullsized":"2024-09-21T19:10:02.253586+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":822,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252648+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:10:02.252648+0000","last_peered":"2024-09-21T19:10:02.252648+0000","last_clean":"2024-09-21T19:10:02.252648+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:10:02.252648+0000","last_undegraded":"2024-09-21T19:10:02.252648+0000","last_fullsized":"2024-09-21T19:10:02.252648+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":216,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455297+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:10:02.455297+0000","last_peered":"2024-09-21T19:10:02.455297+0000","last_clean":"2024-09-21T19:10:02.455297+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:10:02.455297+0000","last_undegraded":"2024-09-21T19:10:02.455297+0000","last_fullsized":"2024-09-21T19:10:02.455297+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.753652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":223,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455678+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:10:02.455678+0000","last_peered":"2024-09-21T19:10:02.455678+0000","last_clean":"2024-09-21T19:10:02.455678+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:10:02.455678+0000","last_undegraded":"2024-09-21T19:10:02.455678+0000","last_fullsized":"2024-09-21T19:10:02.455678+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.374847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":775,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245896+0000","last_change":"2024-09-21T19:09:37.588463+0000","last_active":"2024-09-21T19:10:01.245896+0000","last_peered":"2024-09-21T19:10:01.245896+0000","last_clean":"2024-09-21T19:10:01.245896+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:10:01.245896+0000","last_undegraded":"2024-09-21T19:10:01.245896+0000","last_fullsized":"2024-09-21T19:10:01.245896+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:09:37.588390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.588390+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.015172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253905+0000","last_change":"2024-09-21T19:10:01.250406+0000","last_active":"2024-09-21T19:10:02.253905+0000","last_peered":"2024-09-21T19:10:02.253905+0000","last_clean":"2024-09-21T19:10:02.253905+0000","last_became_active":"2024-09-21T19:10:01.250100+0000","last_became_peered":"2024-09-21T19:10:01.250100+0000","last_unstale":"2024-09-21T19:10:02.253905+0000","last_undegraded":"2024-09-21T19:10:02.253905+0000","last_fullsized":"2024-09-21T19:10:02.253905+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.868697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":558,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253035+0000","last_change":"2024-09-21T19:09:08.156345+0000","last_active":"2024-09-21T19:10:02.253035+0000","last_peered":"2024-09-21T19:10:02.253035+0000","last_clean":"2024-09-21T19:10:02.253035+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:10:02.253035+0000","last_undegraded":"2024-09-21T19:10:02.253035+0000","last_fullsized":"2024-09-21T19:10:02.253035+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:08.156258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:08.156258+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.996519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":759,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251219+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:10:02.251219+0000","last_peered":"2024-09-21T19:10:02.251219+0000","last_clean":"2024-09-21T19:10:02.251219+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:10:02.251219+0000","last_undegraded":"2024-09-21T19:10:02.251219+0000","last_fullsized":"2024-09-21T19:10:02.251219+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":645,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251716+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:10:02.251716+0000","last_peered":"2024-09-21T19:10:02.251716+0000","last_clean":"2024-09-21T19:10:02.251716+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:10:02.251716+0000","last_undegraded":"2024-09-21T19:10:02.251716+0000","last_fullsized":"2024-09-21T19:10:02.251716+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1797,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245036+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:10:01.245036+0000","last_peered":"2024-09-21T19:10:01.245036+0000","last_clean":"2024-09-21T19:10:01.245036+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:10:01.245036+0000","last_undegraded":"2024-09-21T19:10:01.245036+0000","last_fullsized":"2024-09-21T19:10:01.245036+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.664705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1003,"num_read_kb":998,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":726,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.458047+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:10:02.458047+0000","last_peered":"2024-09-21T19:10:02.458047+0000","last_clean":"2024-09-21T19:10:02.458047+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:10:02.458047+0000","last_undegraded":"2024-09-21T19:10:02.458047+0000","last_fullsized":"2024-09-21T19:10:02.458047+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.747406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":774,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252696+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:10:02.252696+0000","last_peered":"2024-09-21T19:10:02.252696+0000","last_clean":"2024-09-21T19:10:02.252696+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:10:02.252696+0000","last_undegraded":"2024-09-21T19:10:02.252696+0000","last_fullsized":"2024-09-21T19:10:02.252696+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251505+0000","last_change":"2024-09-21T19:10:01.260090+0000","last_active":"2024-09-21T19:10:02.251505+0000","last_peered":"2024-09-21T19:10:02.251505+0000","last_clean":"2024-09-21T19:10:02.251505+0000","last_became_active":"2024-09-21T19:10:01.259620+0000","last_became_peered":"2024-09-21T19:10:01.259620+0000","last_unstale":"2024-09-21T19:10:02.251505+0000","last_undegraded":"2024-09-21T19:10:02.251505+0000","last_fullsized":"2024-09-21T19:10:02.251505+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:27.488759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":953,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.511905+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:10:03.511905+0000","last_peered":"2024-09-21T19:10:03.511905+0000","last_clean":"2024-09-21T19:10:03.511905+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:10:03.511905+0000","last_undegraded":"2024-09-21T19:10:03.511905+0000","last_fullsized":"2024-09-21T19:10:03.511905+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.956729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":782,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251559+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:10:02.251559+0000","last_peered":"2024-09-21T19:10:02.251559+0000","last_clean":"2024-09-21T19:10:02.251559+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:10:02.251559+0000","last_undegraded":"2024-09-21T19:10:02.251559+0000","last_fullsized":"2024-09-21T19:10:02.251559+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":864,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245093+0000","last_change":"2024-09-21T19:09:46.543580+0000","last_active":"2024-09-21T19:10:01.245093+0000","last_peered":"2024-09-21T19:10:01.245093+0000","last_clean":"2024-09-21T19:10:01.245093+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:10:01.245093+0000","last_undegraded":"2024-09-21T19:10:01.245093+0000","last_fullsized":"2024-09-21T19:10:01.245093+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:46.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:46.543525+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.937610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":781,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.246107+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:10:01.246107+0000","last_peered":"2024-09-21T19:10:01.246107+0000","last_clean":"2024-09-21T19:10:01.246107+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:10:01.246107+0000","last_undegraded":"2024-09-21T19:10:01.246107+0000","last_fullsized":"2024-09-21T19:10:01.246107+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:15.623405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":317,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254774+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:10:02.254774+0000","last_peered":"2024-09-21T19:10:02.254774+0000","last_clean":"2024-09-21T19:10:02.254774+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:10:02.254774+0000","last_undegraded":"2024-09-21T19:10:02.254774+0000","last_fullsized":"2024-09-21T19:10:02.254774+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.170603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":16,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.509928+0000","last_change":"2024-09-21T19:10:01.250432+0000","last_active":"2024-09-21T19:10:03.509928+0000","last_peered":"2024-09-21T19:10:03.509928+0000","last_clean":"2024-09-21T19:10:03.509928+0000","last_became_active":"2024-09-21T19:10:01.250116+0000","last_became_peered":"2024-09-21T19:10:01.250116+0000","last_unstale":"2024-09-21T19:10:03.509928+0000","last_undegraded":"2024-09-21T19:10:03.509928+0000","last_fullsized":"2024-09-21T19:10:03.509928+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.974644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":818,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242503+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:10:01.242503+0000","last_peered":"2024-09-21T19:10:01.242503+0000","last_clean":"2024-09-21T19:10:01.242503+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:10:01.242503+0000","last_undegraded":"2024-09-21T19:10:01.242503+0000","last_fullsized":"2024-09-21T19:10:01.242503+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":786,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242048+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:10:01.242048+0000","last_peered":"2024-09-21T19:10:01.242048+0000","last_clean":"2024-09-21T19:10:01.242048+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:10:01.242048+0000","last_undegraded":"2024-09-21T19:10:01.242048+0000","last_fullsized":"2024-09-21T19:10:01.242048+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":771,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242537+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:10:01.242537+0000","last_peered":"2024-09-21T19:10:01.242537+0000","last_clean":"2024-09-21T19:10:01.242537+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:10:01.242537+0000","last_undegraded":"2024-09-21T19:10:01.242537+0000","last_fullsized":"2024-09-21T19:10:01.242537+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.639948+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":753,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252616+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:10:02.252616+0000","last_peered":"2024-09-21T19:10:02.252616+0000","last_clean":"2024-09-21T19:10:02.252616+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:10:02.252616+0000","last_undegraded":"2024-09-21T19:10:02.252616+0000","last_fullsized":"2024-09-21T19:10:02.252616+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455357+0000","last_change":"2024-09-21T19:10:01.248144+0000","last_active":"2024-09-21T19:10:02.455357+0000","last_peered":"2024-09-21T19:10:02.455357+0000","last_clean":"2024-09-21T19:10:02.455357+0000","last_became_active":"2024-09-21T19:10:01.247852+0000","last_became_peered":"2024-09-21T19:10:01.247852+0000","last_unstale":"2024-09-21T19:10:02.455357+0000","last_undegraded":"2024-09-21T19:10:02.455357+0000","last_fullsized":"2024-09-21T19:10:02.455357+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.637382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":219,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455195+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:10:02.455195+0000","last_peered":"2024-09-21T19:10:02.455195+0000","last_clean":"2024-09-21T19:10:02.455195+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:10:02.455195+0000","last_undegraded":"2024-09-21T19:10:02.455195+0000","last_fullsized":"2024-09-21T19:10:02.455195+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.739272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":867,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242612+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:10:01.242612+0000","last_peered":"2024-09-21T19:10:01.242612+0000","last_clean":"2024-09-21T19:10:01.242612+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:10:01.242612+0000","last_undegraded":"2024-09-21T19:10:01.242612+0000","last_fullsized":"2024-09-21T19:10:01.242612+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":689,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.510379+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:10:03.510379+0000","last_peered":"2024-09-21T19:10:03.510379+0000","last_clean":"2024-09-21T19:10:03.510379+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:10:03.510379+0000","last_undegraded":"2024-09-21T19:10:03.510379+0000","last_fullsized":"2024-09-21T19:10:03.510379+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":221,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.510789+0000","last_change":"2024-09-21T19:09:50.927165+0000","last_active":"2024-09-21T19:10:03.510789+0000","last_peered":"2024-09-21T19:10:03.510789+0000","last_clean":"2024-09-21T19:10:03.510789+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:10:03.510789+0000","last_undegraded":"2024-09-21T19:10:03.510789+0000","last_fullsized":"2024-09-21T19:10:03.510789+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.927103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.927103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.477371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":219,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455782+0000","last_change":"2024-09-21T19:09:31.740952+0000","last_active":"2024-09-21T19:10:02.455782+0000","last_peered":"2024-09-21T19:10:02.455782+0000","last_clean":"2024-09-21T19:10:02.455782+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:10:02.455782+0000","last_undegraded":"2024-09-21T19:10:02.455782+0000","last_fullsized":"2024-09-21T19:10:02.455782+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:31.740891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:31.740891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:40.289541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":16,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.510931+0000","last_change":"2024-09-21T19:10:01.252391+0000","last_active":"2024-09-21T19:10:03.510931+0000","last_peered":"2024-09-21T19:10:03.510931+0000","last_clean":"2024-09-21T19:10:03.510931+0000","last_became_active":"2024-09-21T19:10:01.251894+0000","last_became_peered":"2024-09-21T19:10:01.251894+0000","last_unstale":"2024-09-21T19:10:03.510931+0000","last_undegraded":"2024-09-21T19:10:03.510931+0000","last_fullsized":"2024-09-21T19:10:03.510931+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.812506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":872,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245205+0000","last_change":"2024-09-21T19:09:39.566152+0000","last_active":"2024-09-21T19:10:01.245205+0000","last_peered":"2024-09-21T19:10:01.245205+0000","last_clean":"2024-09-21T19:10:01.245205+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:10:01.245205+0000","last_undegraded":"2024-09-21T19:10:01.245205+0000","last_fullsized":"2024-09-21T19:10:01.245205+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:39.565921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.565921+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:44.360587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":793,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242216+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:10:01.242216+0000","last_peered":"2024-09-21T19:10:01.242216+0000","last_clean":"2024-09-21T19:10:01.242216+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:10:01.242216+0000","last_undegraded":"2024-09-21T19:10:01.242216+0000","last_fullsized":"2024-09-21T19:10:01.242216+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":795,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254667+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:10:02.254667+0000","last_peered":"2024-09-21T19:10:02.254667+0000","last_clean":"2024-09-21T19:10:02.254667+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:10:02.254667+0000","last_undegraded":"2024-09-21T19:10:02.254667+0000","last_fullsized":"2024-09-21T19:10:02.254667+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":273,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455805+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:10:02.455805+0000","last_peered":"2024-09-21T19:10:02.455805+0000","last_clean":"2024-09-21T19:10:02.455805+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:10:02.455805+0000","last_undegraded":"2024-09-21T19:10:02.455805+0000","last_fullsized":"2024-09-21T19:10:02.455805+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:19.687093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":766,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242293+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:10:01.242293+0000","last_peered":"2024-09-21T19:10:01.242293+0000","last_clean":"2024-09-21T19:10:01.242293+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:10:01.242293+0000","last_undegraded":"2024-09-21T19:10:01.242293+0000","last_fullsized":"2024-09-21T19:10:01.242293+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.250793+0000","last_change":"2024-09-21T19:10:01.250343+0000","last_active":"2024-09-21T19:10:01.250793+0000","last_peered":"2024-09-21T19:10:01.250793+0000","last_clean":"2024-09-21T19:10:01.250793+0000","last_became_active":"2024-09-21T19:10:01.249931+0000","last_became_peered":"2024-09-21T19:10:01.249931+0000","last_unstale":"2024-09-21T19:10:01.250793+0000","last_undegraded":"2024-09-21T19:10:01.250793+0000","last_fullsized":"2024-09-21T19:10:01.250793+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.895536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":656,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252337+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:10:02.252337+0000","last_peered":"2024-09-21T19:10:02.252337+0000","last_clean":"2024-09-21T19:10:02.252337+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:10:02.252337+0000","last_undegraded":"2024-09-21T19:10:02.252337+0000","last_fullsized":"2024-09-21T19:10:02.252337+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":757,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253211+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:10:02.253211+0000","last_peered":"2024-09-21T19:10:02.253211+0000","last_clean":"2024-09-21T19:10:02.253211+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:10:02.253211+0000","last_undegraded":"2024-09-21T19:10:02.253211+0000","last_fullsized":"2024-09-21T19:10:02.253211+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":212,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455470+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:10:02.455470+0000","last_peered":"2024-09-21T19:10:02.455470+0000","last_clean":"2024-09-21T19:10:02.455470+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:10:02.455470+0000","last_undegraded":"2024-09-21T19:10:02.455470+0000","last_fullsized":"2024-09-21T19:10:02.455470+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:10.305625+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":651,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251663+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:10:02.251663+0000","last_peered":"2024-09-21T19:10:02.251663+0000","last_clean":"2024-09-21T19:10:02.251663+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:10:02.251663+0000","last_undegraded":"2024-09-21T19:10:02.251663+0000","last_fullsized":"2024-09-21T19:10:02.251663+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.260470+0000","last_change":"2024-09-21T19:10:01.260078+0000","last_active":"2024-09-21T19:10:01.260470+0000","last_peered":"2024-09-21T19:10:01.260470+0000","last_clean":"2024-09-21T19:10:01.260470+0000","last_became_active":"2024-09-21T19:10:01.259752+0000","last_became_peered":"2024-09-21T19:10:01.259752+0000","last_unstale":"2024-09-21T19:10:01.260470+0000","last_undegraded":"2024-09-21T19:10:01.260470+0000","last_fullsized":"2024-09-21T19:10:01.260470+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.032807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":708,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.510232+0000","last_change":"2024-09-21T19:05:39.768484+0000","last_active":"2024-09-21T19:10:03.510232+0000","last_peered":"2024-09-21T19:10:03.510232+0000","last_clean":"2024-09-21T19:10:03.510232+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:10:03.510232+0000","last_undegraded":"2024-09-21T19:10:03.510232+0000","last_fullsized":"2024-09-21T19:10:03.510232+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:05:39.768388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:39.768388+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":785,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.246706+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:10:01.246706+0000","last_peered":"2024-09-21T19:10:01.246706+0000","last_clean":"2024-09-21T19:10:01.246706+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:10:01.246706+0000","last_undegraded":"2024-09-21T19:10:01.246706+0000","last_fullsized":"2024-09-21T19:10:01.246706+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.612669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":222,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455333+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:10:02.455333+0000","last_peered":"2024-09-21T19:10:02.455333+0000","last_clean":"2024-09-21T19:10:02.455333+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:10:02.455333+0000","last_undegraded":"2024-09-21T19:10:02.455333+0000","last_fullsized":"2024-09-21T19:10:02.455333+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.112472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":765,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245062+0000","last_change":"2024-09-21T19:09:35.677140+0000","last_active":"2024-09-21T19:10:01.245062+0000","last_peered":"2024-09-21T19:10:01.245062+0000","last_clean":"2024-09-21T19:10:01.245062+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:10:01.245062+0000","last_undegraded":"2024-09-21T19:10:01.245062+0000","last_fullsized":"2024-09-21T19:10:01.245062+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:35.677043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:35.677043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:50.521879+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251069+0000","last_change":"2024-09-21T19:10:01.252118+0000","last_active":"2024-09-21T19:10:02.251069+0000","last_peered":"2024-09-21T19:10:02.251069+0000","last_clean":"2024-09-21T19:10:02.251069+0000","last_became_active":"2024-09-21T19:10:01.251738+0000","last_became_peered":"2024-09-21T19:10:01.251738+0000","last_unstale":"2024-09-21T19:10:02.251069+0000","last_undegraded":"2024-09-21T19:10:02.251069+0000","last_fullsized":"2024-09-21T19:10:02.251069+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.046092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":822,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.241904+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:10:01.241904+0000","last_peered":"2024-09-21T19:10:01.241904+0000","last_clean":"2024-09-21T19:10:01.241904+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:10:01.241904+0000","last_undegraded":"2024-09-21T19:10:01.241904+0000","last_fullsized":"2024-09-21T19:10:01.241904+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":787,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245820+0000","last_change":"2024-09-21T19:09:41.535785+0000","last_active":"2024-09-21T19:10:01.245820+0000","last_peered":"2024-09-21T19:10:01.245820+0000","last_clean":"2024-09-21T19:10:01.245820+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:10:01.245820+0000","last_undegraded":"2024-09-21T19:10:01.245820+0000","last_fullsized":"2024-09-21T19:10:01.245820+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:41.535714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:41.535714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:45.775155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":212,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455181+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:10:02.455181+0000","last_peered":"2024-09-21T19:10:02.455181+0000","last_clean":"2024-09-21T19:10:02.455181+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:10:02.455181+0000","last_undegraded":"2024-09-21T19:10:02.455181+0000","last_fullsized":"2024-09-21T19:10:02.455181+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":747,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254397+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:10:02.254397+0000","last_peered":"2024-09-21T19:10:02.254397+0000","last_clean":"2024-09-21T19:10:02.254397+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:10:02.254397+0000","last_undegraded":"2024-09-21T19:10:02.254397+0000","last_fullsized":"2024-09-21T19:10:02.254397+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":14,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253535+0000","last_change":"2024-09-21T19:10:01.260124+0000","last_active":"2024-09-21T19:10:02.253535+0000","last_peered":"2024-09-21T19:10:02.253535+0000","last_clean":"2024-09-21T19:10:02.253535+0000","last_became_active":"2024-09-21T19:10:01.259826+0000","last_became_peered":"2024-09-21T19:10:01.259826+0000","last_unstale":"2024-09-21T19:10:02.253535+0000","last_undegraded":"2024-09-21T19:10:02.253535+0000","last_fullsized":"2024-09-21T19:10:02.253535+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.172175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":378,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.244868+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:10:01.244868+0000","last_peered":"2024-09-21T19:10:01.244868+0000","last_clean":"2024-09-21T19:10:01.244868+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:10:01.244868+0000","last_undegraded":"2024-09-21T19:10:01.244868+0000","last_fullsized":"2024-09-21T19:10:01.244868+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:17.636097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":801,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.246374+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:10:01.246374+0000","last_peered":"2024-09-21T19:10:01.246374+0000","last_clean":"2024-09-21T19:10:01.246374+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:10:01.246374+0000","last_undegraded":"2024-09-21T19:10:01.246374+0000","last_fullsized":"2024-09-21T19:10:01.246374+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:31.474930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":270,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253351+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:10:02.253351+0000","last_peered":"2024-09-21T19:10:02.253351+0000","last_clean":"2024-09-21T19:10:02.253351+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:10:02.253351+0000","last_undegraded":"2024-09-21T19:10:02.253351+0000","last_fullsized":"2024-09-21T19:10:02.253351+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.806824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":785,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.510023+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:10:03.510023+0000","last_peered":"2024-09-21T19:10:03.510023+0000","last_clean":"2024-09-21T19:10:03.510023+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:10:03.510023+0000","last_undegraded":"2024-09-21T19:10:03.510023+0000","last_fullsized":"2024-09-21T19:10:03.510023+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.970506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":720,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.509751+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:10:03.509751+0000","last_peered":"2024-09-21T19:10:03.509751+0000","last_clean":"2024-09-21T19:10:03.509751+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:10:03.509751+0000","last_undegraded":"2024-09-21T19:10:03.509751+0000","last_fullsized":"2024-09-21T19:10:03.509751+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":780,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.255134+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:10:02.255134+0000","last_peered":"2024-09-21T19:10:02.255134+0000","last_clean":"2024-09-21T19:10:02.255134+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:10:02.255134+0000","last_undegraded":"2024-09-21T19:10:02.255134+0000","last_fullsized":"2024-09-21T19:10:02.255134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":316,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252045+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:10:02.252045+0000","last_peered":"2024-09-21T19:10:02.252045+0000","last_clean":"2024-09-21T19:10:02.252045+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:10:02.252045+0000","last_undegraded":"2024-09-21T19:10:02.252045+0000","last_fullsized":"2024-09-21T19:10:02.252045+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":753,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245801+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:10:01.245801+0000","last_peered":"2024-09-21T19:10:01.245801+0000","last_clean":"2024-09-21T19:10:01.245801+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:10:01.245801+0000","last_undegraded":"2024-09-21T19:10:01.245801+0000","last_fullsized":"2024-09-21T19:10:01.245801+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":663,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251311+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:10:02.251311+0000","last_peered":"2024-09-21T19:10:02.251311+0000","last_clean":"2024-09-21T19:10:02.251311+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:10:02.251311+0000","last_undegraded":"2024-09-21T19:10:02.251311+0000","last_fullsized":"2024-09-21T19:10:02.251311+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.161530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":766,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245390+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:10:01.245390+0000","last_peered":"2024-09-21T19:10:01.245390+0000","last_clean":"2024-09-21T19:10:01.245390+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:10:01.245390+0000","last_undegraded":"2024-09-21T19:10:01.245390+0000","last_fullsized":"2024-09-21T19:10:01.245390+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:16.645472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":750,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254944+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:10:02.254944+0000","last_peered":"2024-09-21T19:10:02.254944+0000","last_clean":"2024-09-21T19:10:02.254944+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:10:02.254944+0000","last_undegraded":"2024-09-21T19:10:02.254944+0000","last_fullsized":"2024-09-21T19:10:02.254944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":211,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.455683+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:10:02.455683+0000","last_peered":"2024-09-21T19:10:02.455683+0000","last_clean":"2024-09-21T19:10:02.455683+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:10:02.455683+0000","last_undegraded":"2024-09-21T19:10:02.455683+0000","last_fullsized":"2024-09-21T19:10:02.455683+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":367,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.246037+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:10:01.246037+0000","last_peered":"2024-09-21T19:10:01.246037+0000","last_clean":"2024-09-21T19:10:01.246037+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:10:01.246037+0000","last_undegraded":"2024-09-21T19:10:01.246037+0000","last_fullsized":"2024-09-21T19:10:01.246037+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:29.639722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":653,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252144+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:10:02.252144+0000","last_peered":"2024-09-21T19:10:02.252144+0000","last_clean":"2024-09-21T19:10:02.252144+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:10:02.252144+0000","last_undegraded":"2024-09-21T19:10:02.252144+0000","last_fullsized":"2024-09-21T19:10:02.252144+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":994,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252742+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:10:02.252742+0000","last_peered":"2024-09-21T19:10:02.252742+0000","last_clean":"2024-09-21T19:10:02.252742+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:10:02.252742+0000","last_undegraded":"2024-09-21T19:10:02.252742+0000","last_fullsized":"2024-09-21T19:10:02.252742+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:12.049319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":776,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242880+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:10:01.242880+0000","last_peered":"2024-09-21T19:10:01.242880+0000","last_clean":"2024-09-21T19:10:01.242880+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:10:01.242880+0000","last_undegraded":"2024-09-21T19:10:01.242880+0000","last_fullsized":"2024-09-21T19:10:01.242880+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.177889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":777,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252985+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:10:02.252985+0000","last_peered":"2024-09-21T19:10:02.252985+0000","last_clean":"2024-09-21T19:10:02.252985+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:10:02.252985+0000","last_undegraded":"2024-09-21T19:10:02.252985+0000","last_fullsized":"2024-09-21T19:10:02.252985+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":777,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245246+0000","last_change":"2024-09-21T19:09:32.615127+0000","last_active":"2024-09-21T19:10:01.245246+0000","last_peered":"2024-09-21T19:10:01.245246+0000","last_clean":"2024-09-21T19:10:01.245246+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:10:01.245246+0000","last_undegraded":"2024-09-21T19:10:01.245246+0000","last_fullsized":"2024-09-21T19:10:01.245246+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:32.615041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:32.615041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.657148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":435,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252324+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:10:02.252324+0000","last_peered":"2024-09-21T19:10:02.252324+0000","last_clean":"2024-09-21T19:10:02.252324+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:10:02.252324+0000","last_undegraded":"2024-09-21T19:10:02.252324+0000","last_fullsized":"2024-09-21T19:10:02.252324+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":652,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251580+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:10:02.251580+0000","last_peered":"2024-09-21T19:10:02.251580+0000","last_clean":"2024-09-21T19:10:02.251580+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:10:02.251580+0000","last_undegraded":"2024-09-21T19:10:02.251580+0000","last_fullsized":"2024-09-21T19:10:02.251580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:18.380729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":848,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242122+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:10:01.242122+0000","last_peered":"2024-09-21T19:10:01.242122+0000","last_clean":"2024-09-21T19:10:01.242122+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:10:01.242122+0000","last_undegraded":"2024-09-21T19:10:01.242122+0000","last_fullsized":"2024-09-21T19:10:01.242122+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":767,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.242446+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:10:01.242446+0000","last_peered":"2024-09-21T19:10:01.242446+0000","last_clean":"2024-09-21T19:10:01.242446+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:10:01.242446+0000","last_undegraded":"2024-09-21T19:10:01.242446+0000","last_fullsized":"2024-09-21T19:10:01.242446+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":777,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252631+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:10:02.252631+0000","last_peered":"2024-09-21T19:10:02.252631+0000","last_clean":"2024-09-21T19:10:02.252631+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:10:02.252631+0000","last_undegraded":"2024-09-21T19:10:02.252631+0000","last_fullsized":"2024-09-21T19:10:02.252631+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:37.041685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":772,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253512+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:10:02.253512+0000","last_peered":"2024-09-21T19:10:02.253512+0000","last_clean":"2024-09-21T19:10:02.253512+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:10:02.253512+0000","last_undegraded":"2024-09-21T19:10:02.253512+0000","last_fullsized":"2024-09-21T19:10:02.253512+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":848,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252925+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:10:02.252925+0000","last_peered":"2024-09-21T19:10:02.252925+0000","last_clean":"2024-09-21T19:10:02.252925+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:10:02.252925+0000","last_undegraded":"2024-09-21T19:10:02.252925+0000","last_fullsized":"2024-09-21T19:10:02.252925+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":771,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.246290+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:10:01.246290+0000","last_peered":"2024-09-21T19:10:01.246290+0000","last_clean":"2024-09-21T19:10:01.246290+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:10:01.246290+0000","last_undegraded":"2024-09-21T19:10:01.246290+0000","last_fullsized":"2024-09-21T19:10:01.246290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:02.443813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":773,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254540+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:10:02.254540+0000","last_peered":"2024-09-21T19:10:02.254540+0000","last_clean":"2024-09-21T19:10:02.254540+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:10:02.254540+0000","last_undegraded":"2024-09-21T19:10:02.254540+0000","last_fullsized":"2024-09-21T19:10:02.254540+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":664,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.250727+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:10:02.250727+0000","last_peered":"2024-09-21T19:10:02.250727+0000","last_clean":"2024-09-21T19:10:02.250727+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:10:02.250727+0000","last_undegraded":"2024-09-21T19:10:02.250727+0000","last_fullsized":"2024-09-21T19:10:02.250727+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":797,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.253695+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:10:02.253695+0000","last_peered":"2024-09-21T19:10:02.253695+0000","last_clean":"2024-09-21T19:10:02.253695+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:10:02.253695+0000","last_undegraded":"2024-09-21T19:10:02.253695+0000","last_fullsized":"2024-09-21T19:10:02.253695+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":710,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.457628+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:10:02.457628+0000","last_peered":"2024-09-21T19:10:02.457628+0000","last_clean":"2024-09-21T19:10:02.457628+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:10:02.457628+0000","last_undegraded":"2024-09-21T19:10:02.457628+0000","last_fullsized":"2024-09-21T19:10:02.457628+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":790,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.244661+0000","last_change":"2024-09-21T19:09:51.462137+0000","last_active":"2024-09-21T19:10:01.244661+0000","last_peered":"2024-09-21T19:10:01.244661+0000","last_clean":"2024-09-21T19:10:01.244661+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:10:01.244661+0000","last_undegraded":"2024-09-21T19:10:01.244661+0000","last_fullsized":"2024-09-21T19:10:01.244661+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.462077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.604947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":772,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254645+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:10:02.254645+0000","last_peered":"2024-09-21T19:10:02.254645+0000","last_clean":"2024-09-21T19:10:02.254645+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:10:02.254645+0000","last_undegraded":"2024-09-21T19:10:02.254645+0000","last_fullsized":"2024-09-21T19:10:02.254645+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":796,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254292+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:10:02.254292+0000","last_peered":"2024-09-21T19:10:02.254292+0000","last_clean":"2024-09-21T19:10:02.254292+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:10:02.254292+0000","last_undegraded":"2024-09-21T19:10:02.254292+0000","last_fullsized":"2024-09-21T19:10:02.254292+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":647,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.250991+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:10:02.250991+0000","last_peered":"2024-09-21T19:10:02.250991+0000","last_clean":"2024-09-21T19:10:02.250991+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:10:02.250991+0000","last_undegraded":"2024-09-21T19:10:02.250991+0000","last_fullsized":"2024-09-21T19:10:02.250991+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:59.182751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":687,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.510181+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:10:03.510181+0000","last_peered":"2024-09-21T19:10:03.510181+0000","last_clean":"2024-09-21T19:10:03.510181+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:10:03.510181+0000","last_undegraded":"2024-09-21T19:10:03.510181+0000","last_fullsized":"2024-09-21T19:10:03.510181+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":765,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-21T19:10:01.245110+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:10:01.245110+0000","last_peered":"2024-09-21T19:10:01.245110+0000","last_clean":"2024-09-21T19:10:01.245110+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:10:01.245110+0000","last_undegraded":"2024-09-21T19:10:01.245110+0000","last_fullsized":"2024-09-21T19:10:01.245110+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:06.268622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":772,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.252129+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:10:02.252129+0000","last_peered":"2024-09-21T19:10:02.252129+0000","last_clean":"2024-09-21T19:10:02.252129+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:10:02.252129+0000","last_undegraded":"2024-09-21T19:10:02.252129+0000","last_fullsized":"2024-09-21T19:10:02.252129+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":788,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.254667+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:10:02.254667+0000","last_peered":"2024-09-21T19:10:02.254667+0000","last_clean":"2024-09-21T19:10:02.254667+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:10:02.254667+0000","last_undegraded":"2024-09-21T19:10:02.254667+0000","last_fullsized":"2024-09-21T19:10:02.254667+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":760,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T19:10:03.512219+0000","last_change":"2024-09-21T19:09:48.894774+0000","last_active":"2024-09-21T19:10:03.512219+0000","last_peered":"2024-09-21T19:10:03.512219+0000","last_clean":"2024-09-21T19:10:03.512219+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:10:03.512219+0000","last_undegraded":"2024-09-21T19:10:03.512219+0000","last_fullsized":"2024-09-21T19:10:03.512219+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:09:48.894708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.894708+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.651349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":668,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-21T19:10:02.251499+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:10:02.251499+0000","last_peered":"2024-09-21T19:10:02.251499+0000","last_clean":"2024-09-21T19:10:02.251499+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:10:02.251499+0000","last_undegraded":"2024-09-21T19:10:02.251499+0000","last_fullsized":"2024-09-21T19:10:02.251499+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1003,"num_read_kb":998,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266895,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":606520,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93138632,"statfs":{"total":95995035648,"available":95373959168,"internally_reserved":0,"allocated":1855488,"data_stored":1161744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464346,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290364,"kb_used_data":1508,"kb_used_omap":4,"kb_used_meta":288827,"kb_avail":93454788,"statfs":{"total":95995035648,"available":95697702912,"internally_reserved":0,"allocated":1544192,"data_stored":847880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":295759524},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619047,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271980,"kb_used_data":2072,"kb_used_omap":4,"kb_used_meta":269883,"kb_avail":93473172,"statfs":{"total":95995035648,"available":95716528128,"internally_reserved":0,"allocated":2121728,"data_stored":1431199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":276360867},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019101,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538336,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93206816,"statfs":{"total":95995035648,"available":95443779584,"internally_reserved":0,"allocated":1662976,"data_stored":910587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182626,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737312,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93007840,"statfs":{"total":95995035648,"available":95240028160,"internally_reserved":0,"allocated":2154496,"data_stored":1432517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411560,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560480,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93184672,"statfs":{"total":95995035648,"available":95421104128,"internally_reserved":0,"allocated":1781760,"data_stored":1103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678612,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93066540,"statfs":{"total":95995035648,"available":95300136960,"internally_reserved":0,"allocated":1564672,"data_stored":834967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827124,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918028,"statfs":{"total":95995035648,"available":95148060672,"internally_reserved":0,"allocated":1531904,"data_stored":834753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:10:09.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:09.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:09 smithi104 ceph-mon[32438]: from='client.36482 v1:172.21.15.81:0/1015587502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:09 smithi104 ceph-mon[32438]: pgmap v844: 158 pgs: 158 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:10:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:09 smithi104 ceph-mon[32438]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T19:10:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:09.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:09.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[29384]: from='client.36482 v1:172.21.15.81:0/1015587502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[29384]: pgmap v844: 158 pgs: 158 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[29384]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[39192]: from='client.36482 v1:172.21.15.81:0/1015587502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[39192]: pgmap v844: 158 pgs: 158 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[39192]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T19:10:09.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:09.572 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:09.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:09.712 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:09.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 2 16 6797 6781 211.879 201.062 0.0027662 0.00471022 2024-09-21T19:10:09.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:09.846 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:09.948 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:09.980 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:09.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:10.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:10.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:10.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:10.257 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:10.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:10:10.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:10.381 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:10.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.515 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:10.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:10.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:10.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:10.691 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:10.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 3 16 9406 9390 195.603 163.062 0.0028037 0.00510543 2024-09-21T19:10:10.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:10.787 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:10.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:10.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:10:10.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:10.917 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:10.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:11.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:11.039 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:11.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:10 smithi081 ceph-mon[29384]: pgmap v846: 158 pgs: 158 active+clean; 65 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 784 B/s rd, 9.9 MiB/s wr, 158 op/s 2024-09-21T19:10:11.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/188096310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:11.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:10 smithi081 ceph-mon[29384]: from='client.36503 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:11.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:10 smithi081 ceph-mon[39192]: pgmap v846: 158 pgs: 158 active+clean; 65 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 784 B/s rd, 9.9 MiB/s wr, 158 op/s 2024-09-21T19:10:11.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:10 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/188096310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:11.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:10 smithi081 ceph-mon[39192]: from='client.36503 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:10:11.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:10 smithi104 ceph-mon[32438]: pgmap v846: 158 pgs: 158 active+clean; 65 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 784 B/s rd, 9.9 MiB/s wr, 158 op/s 2024-09-21T19:10:11.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:10 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/188096310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:11.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:10 smithi104 ceph-mon[32438]: from='client.36503 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:11.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:11.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:11.281 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:10:11.281 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:10:11.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.281 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36497.0:10575 11.8 e420/414 11:172f3dac:::benchmark_data_smithi081_181703_object10574:head v 420'651, mlcod=420'648)", 2024-09-21T19:10:11.281 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:10:11.278271+0000", 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00035417800000000002, 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037198299999999999, 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278271+0000", 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:10:11.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278271+0000", 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278272+0000", 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.525e-06 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278309+0000", 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7064999999999998e-05 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278321+0000", 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1481000000000001e-05 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278327+0000", 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0850000000000002e-06 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278385+0000", 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8295999999999999e-05 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:10:11.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:11.278408+0000", 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.27e-05 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 1 2024-09-21T19:10:11.285 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:10:11.285 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:11.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:11.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:11.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:11.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:11.602 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:11.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:11.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 4 15 11725 11710 182.949 145 0.00281042 0.00545749 2024-09-21T19:10:11.735 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:11.837 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:11.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:11.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:11.963 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:11.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.013 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:12.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:11 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:11 smithi081 ceph-mon[29384]: from='client.36503 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:10:12.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:11 smithi081 ceph-mon[29384]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T19:10:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:11 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:11 smithi081 ceph-mon[39192]: from='client.36503 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:10:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:11 smithi081 ceph-mon[39192]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T19:10:12.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:12.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:12.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:11 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:11 smithi104 ceph-mon[32438]: from='client.36503 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:10:12.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:11 smithi104 ceph-mon[32438]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T19:10:12.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:12.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:10:12.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:10:12.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:12.435 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:12.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:12.566 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:12.633 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:12.666 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:12.666 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:12.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:12.703 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:12.703 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.713 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:12.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:12.714 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:12.731 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 5 16 14296 14280 178.48 160.625 0.00994522 0.00559607 2024-09-21T19:10:12.735 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:12.735 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:12.769 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:12.802 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:12.803 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:12.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:12.836 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:12.843 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:12.872 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:12.872 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:12.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:12.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[29384]: 4.1 scrub starts 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[29384]: 4.1 scrub ok 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[29384]: pgmap v848: 158 pgs: 158 active+clean; 110 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 784 B/s rd, 17 MiB/s wr, 269 op/s 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2505582305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[39192]: 4.1 scrub starts 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[39192]: 4.1 scrub ok 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[39192]: pgmap v848: 158 pgs: 158 active+clean; 110 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 784 B/s rd, 17 MiB/s wr, 269 op/s 2024-09-21T19:10:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2505582305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:13.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:12 smithi104 ceph-mon[32438]: 4.1 scrub starts 2024-09-21T19:10:13.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:12 smithi104 ceph-mon[32438]: 4.1 scrub ok 2024-09-21T19:10:13.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:12 smithi104 ceph-mon[32438]: pgmap v848: 158 pgs: 158 active+clean; 110 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 784 B/s rd, 17 MiB/s wr, 269 op/s 2024-09-21T19:10:13.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2505582305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:13.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:13.116 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:13.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:13.245 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:13.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:13.370 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:13.428 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:13.428 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:13.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:13.603 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:13.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:13.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 6 16 16603 16587 172.763 144.188 0.0104759 0.00578163 2024-09-21T19:10:13.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:13.774 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:13.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:10:13.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:13.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:13.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:14.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:14.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[39192]: 4.a scrub starts 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[39192]: 4.a scrub ok 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[29384]: 4.a scrub starts 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[29384]: 4.a scrub ok 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T19:10:14.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:13 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T19:10:14.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:13 smithi104 ceph-mon[32438]: 4.a scrub starts 2024-09-21T19:10:14.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:13 smithi104 ceph-mon[32438]: 4.a scrub ok 2024-09-21T19:10:14.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:13 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T19:10:14.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:13 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T19:10:14.075 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:14.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.220 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:14.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:14.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:14.363 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:14.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:14.494 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:14.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:10:14.553 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:14.597 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:14.622 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:14.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 7 16 18960 18944 169.117 147.312 0.00598493 0.0059054 2024-09-21T19:10:14.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:14.798 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:14.900 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:14.931 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:14.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:14.947 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:14.990 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T19:10:14.990 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T19:10:14.990 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 3.1d scrub starts 2024-09-21T19:10:14.990 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 3.1d scrub ok 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: pgmap v849: 158 pgs: 158 active+clean; 800 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 6.1f scrub starts 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 6.1f scrub ok 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[39192]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 3.1d scrub starts 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 3.1d scrub ok 2024-09-21T19:10:14.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: pgmap v849: 158 pgs: 158 active+clean; 800 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T19:10:14.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 6.1f scrub starts 2024-09-21T19:10:14.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 6.1f scrub ok 2024-09-21T19:10:14.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T19:10:14.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T19:10:14.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:14 smithi081 ceph-mon[29384]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T19:10:15.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:15.061 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:15.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 3.1d scrub starts 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 3.1d scrub ok 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: pgmap v849: 158 pgs: 158 active+clean; 800 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 6.1f scrub starts 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 6.1f scrub ok 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T19:10:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:14 smithi104 ceph-mon[32438]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T19:10:15.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:15.199 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:15.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:15.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:15.475 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:15.516 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:15.516 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:15.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.617 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:15.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:15.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:15.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 8 16 21143 21127 165.031 136.438 0.00840955 0.00605125 2024-09-21T19:10:15.748 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:15.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:15.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:15.892 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:15.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:15.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:16.023 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:16.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:16.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:16.196 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:10:16.199 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:10:16.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:10:16.253 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:16.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:16.421 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:16.523 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:16.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:16.546 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:10:16.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:16.674 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:16.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 9 16 23090 23074 160.214 121.688 0.00373275 0.00623583 2024-09-21T19:10:16.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:16.819 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:10:16.890 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[39192]: pgmap v851: 158 pgs: 158 active+clean; 898 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-21T19:10:16.891 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T19:10:16.891 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T19:10:16.891 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3245109164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:16.891 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[39192]: from='client.36518 v1:172.21.15.81:0/1773601191' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:16.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[29384]: pgmap v851: 158 pgs: 158 active+clean; 898 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-21T19:10:16.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T19:10:16.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T19:10:16.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3245109164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:16.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:16 smithi081 ceph-mon[29384]: from='client.36518 v1:172.21.15.81:0/1773601191' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:16.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:16.955 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:16.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:17.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:17.077 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:17.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:17.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:16 smithi104 ceph-mon[32438]: pgmap v851: 158 pgs: 158 active+clean; 898 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-21T19:10:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:16 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T19:10:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:16 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T19:10:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3245109164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:16 smithi104 ceph-mon[32438]: from='client.36518 v1:172.21.15.81:0/1773601191' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:10:17.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:17.324 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:17.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:17.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:17.447 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:17.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:17.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:10:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:10:17.576 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:17.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:17.714 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:17.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 10 16 24237 24221 151.361 71.6875 0.00291682 0.00659286 2024-09-21T19:10:17.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:17.838 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:17.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:17.979 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:17.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:18.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:18.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:18.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:18.244 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: 6.13 scrub starts 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: 6.13 scrub ok 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3245109164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:10:18.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:17 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: 6.13 scrub starts 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: 6.13 scrub ok 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3245109164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:10:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: 6.13 scrub starts 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: 6.13 scrub ok 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3245109164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:10:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:17 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:10:18.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:18.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:18.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:18.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:18.653 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:18.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 11 16 26054 26038 147.924 113.562 0.00598215 0.00675355 2024-09-21T19:10:18.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:18.788 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:18.875 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:10:18.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:18.920 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:18.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:19.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:19.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:19.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:19.193 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 5.4 scrub starts 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 5.4 scrub ok 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: pgmap v853: 158 pgs: 158 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 5.15 scrub starts 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 5.15 scrub ok 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T19:10:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:18 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T19:10:19.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:19.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 5.4 scrub starts 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 5.4 scrub ok 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: pgmap v853: 158 pgs: 158 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 5.15 scrub starts 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 5.15 scrub ok 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T19:10:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 5.4 scrub starts 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 5.4 scrub ok 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: pgmap v853: 158 pgs: 158 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 5.15 scrub starts 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 5.15 scrub ok 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T19:10:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:18 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T19:10:19.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:19.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:19.458 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:19.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:19.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:19.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:19.727 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:19.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 12 16 28040 28024 145.94 124.125 0.00651686 0.00684667 2024-09-21T19:10:19.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:19.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:19.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:19.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:20.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:20.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:20.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: 3.11 scrub starts 2024-09-21T19:10:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: 3.11 scrub ok 2024-09-21T19:10:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:10:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:10:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T19:10:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: 6.9 scrub starts 2024-09-21T19:10:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:19 smithi104 ceph-mon[32438]: 6.9 scrub ok 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: 3.11 scrub starts 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: 3.11 scrub ok 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: 6.9 scrub starts 2024-09-21T19:10:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[39192]: 6.9 scrub ok 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: 3.11 scrub starts 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: 3.11 scrub ok 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: 6.9 scrub starts 2024-09-21T19:10:20.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:19 smithi081 ceph-mon[29384]: 6.9 scrub ok 2024-09-21T19:10:20.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:20.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:20.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:20.493 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:20.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:20.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:20.725 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:20.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 13 16 29369 29353 141.102 83.0625 0.00171211 0.00706274 2024-09-21T19:10:20.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:20.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:20.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:20.991 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:20.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:21.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: pgmap v855: 158 pgs: 158 active+clean; 1.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: 4.14 scrub starts 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: 4.14 scrub ok 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:10:21.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: pgmap v855: 158 pgs: 158 active+clean; 1.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: 4.14 scrub starts 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: 4.14 scrub ok 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:10:21.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:20 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:10:21.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:21.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:21.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:21.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:10:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: pgmap v855: 158 pgs: 158 active+clean; 1.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: 4.14 scrub starts 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: 4.14 scrub ok 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:10:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:20 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:10:21.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:21.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:21.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:21.524 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:21.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:21.655 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:21.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 14 16 30666 30650 136.813 81.0625 0.00698683 0.00730302 2024-09-21T19:10:21.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.797 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:21.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:21.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:21.929 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:21.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:22.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:22.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:22.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:22.193 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:22.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:21 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T19:10:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:21 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T19:10:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:21 smithi104 ceph-mon[32438]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T19:10:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:21 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:10:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:21 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:10:22.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:22.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[39192]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[29384]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:10:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:21 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:10:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:22.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:22.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:22.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:22.589 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:22.691 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:22.712 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:22.732 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 15 15 32631 32616 135.883 122.875 0.00317025 0.00734987 2024-09-21T19:10:22.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:22.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:22.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:23.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:23.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:23.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:23.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:23.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:23.373 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:23.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:23.475 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:23.505 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:23 smithi104 ceph-mon[32438]: pgmap v857: 158 pgs: 158 active+clean; 1.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T19:10:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:23 smithi104 ceph-mon[32438]: 5.0 scrub starts 2024-09-21T19:10:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:23 smithi104 ceph-mon[32438]: 5.0 scrub ok 2024-09-21T19:10:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:23 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:23 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[39192]: pgmap v857: 158 pgs: 158 active+clean; 1.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[39192]: 5.0 scrub starts 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[39192]: 5.0 scrub ok 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[29384]: pgmap v857: 158 pgs: 158 active+clean; 1.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[29384]: 5.0 scrub starts 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[29384]: 5.0 scrub ok 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:23.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:23 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:23.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:23.634 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:23.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 16 16 34264 34248 133.765 102 0.00777195 0.00747021 2024-09-21T19:10:23.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:23.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:23.896 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:23.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:23.997 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.032 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:24.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:24.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:24.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:24.300 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:24.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:24.443 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:24.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: 6.15 scrub starts 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: 6.15 scrub ok 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: 5.1e scrub starts 2024-09-21T19:10:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:24 smithi104 ceph-mon[32438]: 5.1e scrub ok 2024-09-21T19:10:24.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: 6.15 scrub starts 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: 6.15 scrub ok 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: 5.1e scrub starts 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[29384]: 5.1e scrub ok 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: 6.15 scrub starts 2024-09-21T19:10:24.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: 6.15 scrub ok 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: 5.1e scrub starts 2024-09-21T19:10:24.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:24 smithi081 ceph-mon[39192]: 5.1e scrub ok 2024-09-21T19:10:24.573 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:24.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.708 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:24.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:24.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 17 16 35990 35974 132.242 107.875 0.00642313 0.00755722 2024-09-21T19:10:24.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:24.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:24.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:24.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:24.973 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2024-09-21T19:10:15.789525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":941759003,"num_objects":14560,"num_object_clones":0,"num_object_copies":43680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14560,"num_whiteouts":0,"num_read":2152,"num_read_kb":1904,"num_write":15083,"num_write_kb":919149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":272632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16020,"ondisk_log_size":16020,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7190036,"kb_used_data":2618000,"kb_used_omap":18,"kb_used_meta":4571885,"kb_avail":742771180,"statfs":{"total":767960285184,"available":760597688320,"internally_reserved":0,"allocated":2680832000,"data_stored":2675054421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":4681611248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,3,3,8,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":918224896,"num_objects":14011,"num_object_clones":0,"num_object_copies":42033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14011,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":14011,"num_write_kb":896704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.502991"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":686,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766423+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:10:10.766423+0000","last_peered":"2024-09-21T19:10:10.766423+0000","last_clean":"2024-09-21T19:10:10.766423+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:10:10.766423+0000","last_undegraded":"2024-09-21T19:10:10.766423+0000","last_fullsized":"2024-09-21T19:10:10.766423+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:10.748546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":848,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772021+0000","last_change":"2024-09-21T19:05:27.606939+0000","last_active":"2024-09-21T19:10:10.772021+0000","last_peered":"2024-09-21T19:10:10.772021+0000","last_clean":"2024-09-21T19:10:10.772021+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:10:10.772021+0000","last_undegraded":"2024-09-21T19:10:10.772021+0000","last_fullsized":"2024-09-21T19:10:10.772021+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:27.606885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.606885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770148+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:10:10.770148+0000","last_peered":"2024-09-21T19:10:10.770148+0000","last_clean":"2024-09-21T19:10:10.770148+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:10:10.770148+0000","last_undegraded":"2024-09-21T19:10:10.770148+0000","last_fullsized":"2024-09-21T19:10:10.770148+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.809763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772419+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:10:10.772419+0000","last_peered":"2024-09-21T19:10:10.772419+0000","last_clean":"2024-09-21T19:10:10.772419+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:10:10.772419+0000","last_undegraded":"2024-09-21T19:10:10.772419+0000","last_fullsized":"2024-09-21T19:10:10.772419+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":773,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767305+0000","last_change":"2024-09-21T19:04:36.586250+0000","last_active":"2024-09-21T19:10:10.767305+0000","last_peered":"2024-09-21T19:10:10.767305+0000","last_clean":"2024-09-21T19:10:10.767305+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:10:10.767305+0000","last_undegraded":"2024-09-21T19:10:10.767305+0000","last_fullsized":"2024-09-21T19:10:10.767305+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.586188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.586188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":801,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767944+0000","last_change":"2024-09-21T19:06:20.883413+0000","last_active":"2024-09-21T19:10:10.767944+0000","last_peered":"2024-09-21T19:10:10.767944+0000","last_clean":"2024-09-21T19:10:10.767944+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:10:10.767944+0000","last_undegraded":"2024-09-21T19:10:10.767944+0000","last_fullsized":"2024-09-21T19:10:10.767944+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:20.883335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:20.883335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":898,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767785+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:10:10.767785+0000","last_peered":"2024-09-21T19:10:10.767785+0000","last_clean":"2024-09-21T19:10:10.767785+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:10:10.767785+0000","last_undegraded":"2024-09-21T19:10:10.767785+0000","last_fullsized":"2024-09-21T19:10:10.767785+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772003+0000","last_change":"2024-09-21T19:05:00.675671+0000","last_active":"2024-09-21T19:10:10.772003+0000","last_peered":"2024-09-21T19:10:10.772003+0000","last_clean":"2024-09-21T19:10:10.772003+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:10:10.772003+0000","last_undegraded":"2024-09-21T19:10:10.772003+0000","last_fullsized":"2024-09-21T19:10:10.772003+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:00.675609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.675609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":771,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.774167+0000","last_change":"2024-09-21T19:06:54.704073+0000","last_active":"2024-09-21T19:10:10.774167+0000","last_peered":"2024-09-21T19:10:10.774167+0000","last_clean":"2024-09-21T19:10:10.774167+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:10:10.774167+0000","last_undegraded":"2024-09-21T19:10:10.774167+0000","last_fullsized":"2024-09-21T19:10:10.774167+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:59.658949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:59.658949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":828,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772751+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:10:10.772751+0000","last_peered":"2024-09-21T19:10:10.772751+0000","last_clean":"2024-09-21T19:10:10.772751+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:10:10.772751+0000","last_undegraded":"2024-09-21T19:10:10.772751+0000","last_fullsized":"2024-09-21T19:10:10.772751+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:05.623319+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":733,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769257+0000","last_change":"2024-09-21T19:09:56.743173+0000","last_active":"2024-09-21T19:10:10.769257+0000","last_peered":"2024-09-21T19:10:10.769257+0000","last_clean":"2024-09-21T19:10:10.769257+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:10:10.769257+0000","last_undegraded":"2024-09-21T19:10:10.769257+0000","last_fullsized":"2024-09-21T19:10:10.769257+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.743123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.743123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.761375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":644,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.764587+0000","last_change":"2024-09-21T19:04:31.864313+0000","last_active":"2024-09-21T19:10:10.764587+0000","last_peered":"2024-09-21T19:10:10.764587+0000","last_clean":"2024-09-21T19:10:10.764587+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:10:10.764587+0000","last_undegraded":"2024-09-21T19:10:10.764587+0000","last_fullsized":"2024-09-21T19:10:10.764587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:31.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.204017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":324,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773735+0000","last_change":"2024-09-21T19:06:54.700681+0000","last_active":"2024-09-21T19:10:10.773735+0000","last_peered":"2024-09-21T19:10:10.773735+0000","last_clean":"2024-09-21T19:10:10.773735+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:10:10.773735+0000","last_undegraded":"2024-09-21T19:10:10.773735+0000","last_fullsized":"2024-09-21T19:10:10.773735+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.885139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.885139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.643073+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":774,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773137+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:10:10.773137+0000","last_peered":"2024-09-21T19:10:10.773137+0000","last_clean":"2024-09-21T19:10:10.773137+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:10:10.773137+0000","last_undegraded":"2024-09-21T19:10:10.773137+0000","last_fullsized":"2024-09-21T19:10:10.773137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772492+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:10:10.772492+0000","last_peered":"2024-09-21T19:10:10.772492+0000","last_clean":"2024-09-21T19:10:10.772492+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:10:10.772492+0000","last_undegraded":"2024-09-21T19:10:10.772492+0000","last_fullsized":"2024-09-21T19:10:10.772492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:33.369870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":332,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770053+0000","last_change":"2024-09-21T19:08:21.206201+0000","last_active":"2024-09-21T19:10:10.770053+0000","last_peered":"2024-09-21T19:10:10.770053+0000","last_clean":"2024-09-21T19:10:10.770053+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:10:10.770053+0000","last_undegraded":"2024-09-21T19:10:10.770053+0000","last_fullsized":"2024-09-21T19:10:10.770053+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:21.206124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:21.206124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":793,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773829+0000","last_change":"2024-09-21T19:06:54.704853+0000","last_active":"2024-09-21T19:10:10.773829+0000","last_peered":"2024-09-21T19:10:10.773829+0000","last_clean":"2024-09-21T19:10:10.773829+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:10:10.773829+0000","last_undegraded":"2024-09-21T19:10:10.773829+0000","last_fullsized":"2024-09-21T19:10:10.773829+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:36.688019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.688019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":660,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765287+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:10:10.765287+0000","last_peered":"2024-09-21T19:10:10.765287+0000","last_clean":"2024-09-21T19:10:10.765287+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:10:10.765287+0000","last_undegraded":"2024-09-21T19:10:10.765287+0000","last_fullsized":"2024-09-21T19:10:10.765287+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":235,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773281+0000","last_change":"2024-09-21T19:06:18.612373+0000","last_active":"2024-09-21T19:10:10.773281+0000","last_peered":"2024-09-21T19:10:10.773281+0000","last_clean":"2024-09-21T19:10:10.773281+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:10:10.773281+0000","last_undegraded":"2024-09-21T19:10:10.773281+0000","last_fullsized":"2024-09-21T19:10:10.773281+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:18.612281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:18.612281+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:24.726173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":787,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771394+0000","last_change":"2024-09-21T19:09:06.107239+0000","last_active":"2024-09-21T19:10:10.771394+0000","last_peered":"2024-09-21T19:10:10.771394+0000","last_clean":"2024-09-21T19:10:10.771394+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:10:10.771394+0000","last_undegraded":"2024-09-21T19:10:10.771394+0000","last_fullsized":"2024-09-21T19:10:10.771394+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:06.107162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:06.107162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771709+0000","last_change":"2024-09-21T19:06:54.696005+0000","last_active":"2024-09-21T19:10:10.771709+0000","last_peered":"2024-09-21T19:10:10.771709+0000","last_clean":"2024-09-21T19:10:10.771709+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:10:10.771709+0000","last_undegraded":"2024-09-21T19:10:10.771709+0000","last_fullsized":"2024-09-21T19:10:10.771709+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:58.416083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:58.416083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":795,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772894+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:10:10.772894+0000","last_peered":"2024-09-21T19:10:10.772894+0000","last_clean":"2024-09-21T19:10:10.772894+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:10:10.772894+0000","last_undegraded":"2024-09-21T19:10:10.772894+0000","last_fullsized":"2024-09-21T19:10:10.772894+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.640166+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":840,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771236+0000","last_change":"2024-09-21T19:06:43.844395+0000","last_active":"2024-09-21T19:10:10.771236+0000","last_peered":"2024-09-21T19:10:10.771236+0000","last_clean":"2024-09-21T19:10:10.771236+0000","last_became_active":"2024-09-21T19:02:40.880448+0000","last_became_peered":"2024-09-21T19:02:40.880448+0000","last_unstale":"2024-09-21T19:10:10.771236+0000","last_undegraded":"2024-09-21T19:10:10.771236+0000","last_fullsized":"2024-09-21T19:10:10.771236+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773615+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:10:10.773615+0000","last_peered":"2024-09-21T19:10:10.773615+0000","last_clean":"2024-09-21T19:10:10.773615+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:10:10.773615+0000","last_undegraded":"2024-09-21T19:10:10.773615+0000","last_fullsized":"2024-09-21T19:10:10.773615+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":802,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772609+0000","last_change":"2024-09-21T19:06:54.693317+0000","last_active":"2024-09-21T19:10:10.772609+0000","last_peered":"2024-09-21T19:10:10.772609+0000","last_clean":"2024-09-21T19:10:10.772609+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:10:10.772609+0000","last_undegraded":"2024-09-21T19:10:10.772609+0000","last_fullsized":"2024-09-21T19:10:10.772609+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:03:41.162715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:03:41.162715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.611692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":802,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770246+0000","last_change":"2024-09-21T19:08:53.272098+0000","last_active":"2024-09-21T19:10:10.770246+0000","last_peered":"2024-09-21T19:10:10.770246+0000","last_clean":"2024-09-21T19:10:10.770246+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:10:10.770246+0000","last_undegraded":"2024-09-21T19:10:10.770246+0000","last_fullsized":"2024-09-21T19:10:10.770246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:08:53.272020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:53.272020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":689,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.764669+0000","last_change":"2024-09-21T19:05:45.169775+0000","last_active":"2024-09-21T19:10:10.764669+0000","last_peered":"2024-09-21T19:10:10.764669+0000","last_clean":"2024-09-21T19:10:10.764669+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:10:10.764669+0000","last_undegraded":"2024-09-21T19:10:10.764669+0000","last_fullsized":"2024-09-21T19:10:10.764669+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:05:45.169696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:45.169696+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":870,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770717+0000","last_change":"2024-09-21T19:05:46.818737+0000","last_active":"2024-09-21T19:10:10.770717+0000","last_peered":"2024-09-21T19:10:10.770717+0000","last_clean":"2024-09-21T19:10:10.770717+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:10:10.770717+0000","last_undegraded":"2024-09-21T19:10:10.770717+0000","last_fullsized":"2024-09-21T19:10:10.770717+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:05:46.818686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:46.818686+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"420'904","reported_seq":927,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.649667+0000","last_change":"2024-09-21T19:10:01.248307+0000","last_active":"2024-09-21T19:10:12.649667+0000","last_peered":"2024-09-21T19:10:12.649667+0000","last_clean":"2024-09-21T19:10:12.649667+0000","last_became_active":"2024-09-21T19:10:01.248029+0000","last_became_peered":"2024-09-21T19:10:01.248029+0000","last_unstale":"2024-09-21T19:10:12.649667+0000","last_undegraded":"2024-09-21T19:10:12.649667+0000","last_fullsized":"2024-09-21T19:10:12.649667+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":904,"log_dups_size":0,"ondisk_log_size":904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:20.749763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":57856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":231,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772799+0000","last_change":"2024-09-21T19:06:54.693265+0000","last_active":"2024-09-21T19:10:10.772799+0000","last_peered":"2024-09-21T19:10:10.772799+0000","last_clean":"2024-09-21T19:10:10.772799+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:10:10.772799+0000","last_undegraded":"2024-09-21T19:10:10.772799+0000","last_fullsized":"2024-09-21T19:10:10.772799+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:43.534605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:43.534605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:13.700777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":837,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772036+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:10:10.772036+0000","last_peered":"2024-09-21T19:10:10.772036+0000","last_clean":"2024-09-21T19:10:10.772036+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:10:10.772036+0000","last_undegraded":"2024-09-21T19:10:10.772036+0000","last_fullsized":"2024-09-21T19:10:10.772036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:13.885784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":775,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767704+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:10:10.767704+0000","last_peered":"2024-09-21T19:10:10.767704+0000","last_clean":"2024-09-21T19:10:10.767704+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:10:10.767704+0000","last_undegraded":"2024-09-21T19:10:10.767704+0000","last_fullsized":"2024-09-21T19:10:10.767704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.830842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1022,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773848+0000","last_change":"2024-09-21T19:06:54.706072+0000","last_active":"2024-09-21T19:10:10.773848+0000","last_peered":"2024-09-21T19:10:10.773848+0000","last_clean":"2024-09-21T19:10:10.773848+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:10:10.773848+0000","last_undegraded":"2024-09-21T19:10:10.773848+0000","last_fullsized":"2024-09-21T19:10:10.773848+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'10","last_scrub_stamp":"2024-09-21T19:03:53.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:53.851169+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.644910+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":457,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771310+0000","last_change":"2024-09-21T19:08:11.330281+0000","last_active":"2024-09-21T19:10:10.771310+0000","last_peered":"2024-09-21T19:10:10.771310+0000","last_clean":"2024-09-21T19:10:10.771310+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:10:10.771310+0000","last_undegraded":"2024-09-21T19:10:10.771310+0000","last_fullsized":"2024-09-21T19:10:10.771310+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:11.330212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:11.330212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:22.031875+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"420'784","reported_seq":807,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:11.910005+0000","last_change":"2024-09-21T19:10:01.252308+0000","last_active":"2024-09-21T19:10:11.910005+0000","last_peered":"2024-09-21T19:10:11.910005+0000","last_clean":"2024-09-21T19:10:11.910005+0000","last_became_active":"2024-09-21T19:10:01.252017+0000","last_became_peered":"2024-09-21T19:10:01.252017+0000","last_unstale":"2024-09-21T19:10:11.910005+0000","last_undegraded":"2024-09-21T19:10:11.910005+0000","last_fullsized":"2024-09-21T19:10:11.910005+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.868221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51380224,"num_objects":784,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":784,"num_write_kb":50176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":564,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.774146+0000","last_change":"2024-09-21T19:09:53.595884+0000","last_active":"2024-09-21T19:10:10.774146+0000","last_peered":"2024-09-21T19:10:10.774146+0000","last_clean":"2024-09-21T19:10:10.774146+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:10:10.774146+0000","last_undegraded":"2024-09-21T19:10:10.774146+0000","last_fullsized":"2024-09-21T19:10:10.774146+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:15.381125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":864,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767386+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:10:10.767386+0000","last_peered":"2024-09-21T19:10:10.767386+0000","last_clean":"2024-09-21T19:10:10.767386+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:10:10.767386+0000","last_undegraded":"2024-09-21T19:10:10.767386+0000","last_fullsized":"2024-09-21T19:10:10.767386+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":789,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772166+0000","last_change":"2024-09-21T19:06:11.910727+0000","last_active":"2024-09-21T19:10:10.772166+0000","last_peered":"2024-09-21T19:10:10.772166+0000","last_clean":"2024-09-21T19:10:10.772166+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:10:10.772166+0000","last_undegraded":"2024-09-21T19:10:10.772166+0000","last_fullsized":"2024-09-21T19:10:10.772166+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:11.910660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:11.910660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":767,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772150+0000","last_change":"2024-09-21T19:03:45.812756+0000","last_active":"2024-09-21T19:10:10.772150+0000","last_peered":"2024-09-21T19:10:10.772150+0000","last_clean":"2024-09-21T19:10:10.772150+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:10:10.772150+0000","last_undegraded":"2024-09-21T19:10:10.772150+0000","last_fullsized":"2024-09-21T19:10:10.772150+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:45.812697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:45.812697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:59.662855+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":784,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772664+0000","last_change":"2024-09-21T19:06:51.125471+0000","last_active":"2024-09-21T19:10:10.772664+0000","last_peered":"2024-09-21T19:10:10.772664+0000","last_clean":"2024-09-21T19:10:10.772664+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:10:10.772664+0000","last_undegraded":"2024-09-21T19:10:10.772664+0000","last_fullsized":"2024-09-21T19:10:10.772664+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.125413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.125413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"420'724","reported_seq":747,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:11.429316+0000","last_change":"2024-09-21T19:10:01.248578+0000","last_active":"2024-09-21T19:10:11.429316+0000","last_peered":"2024-09-21T19:10:11.429316+0000","last_clean":"2024-09-21T19:10:11.429316+0000","last_became_active":"2024-09-21T19:10:01.248286+0000","last_became_peered":"2024-09-21T19:10:01.248286+0000","last_unstale":"2024-09-21T19:10:11.429316+0000","last_undegraded":"2024-09-21T19:10:11.429316+0000","last_fullsized":"2024-09-21T19:10:11.429316+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.219498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47448064,"num_objects":724,"num_object_clones":0,"num_object_copies":2172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":724,"num_write_kb":46336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":547,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766662+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:10:10.766662+0000","last_peered":"2024-09-21T19:10:10.766662+0000","last_clean":"2024-09-21T19:10:10.766662+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:10:10.766662+0000","last_undegraded":"2024-09-21T19:10:10.766662+0000","last_fullsized":"2024-09-21T19:10:10.766662+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":835,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772290+0000","last_change":"2024-09-21T19:06:54.700420+0000","last_active":"2024-09-21T19:10:10.772290+0000","last_peered":"2024-09-21T19:10:10.772290+0000","last_clean":"2024-09-21T19:10:10.772290+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:10:10.772290+0000","last_undegraded":"2024-09-21T19:10:10.772290+0000","last_fullsized":"2024-09-21T19:10:10.772290+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:04:58.633852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.633852+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":657,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766824+0000","last_change":"2024-09-21T19:04:25.838960+0000","last_active":"2024-09-21T19:10:10.766824+0000","last_peered":"2024-09-21T19:10:10.766824+0000","last_clean":"2024-09-21T19:10:10.766824+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:10:10.766824+0000","last_undegraded":"2024-09-21T19:10:10.766824+0000","last_fullsized":"2024-09-21T19:10:10.766824+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:04:25.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:25.838891+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.184114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":966,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767223+0000","last_change":"2024-09-21T19:04:26.469205+0000","last_active":"2024-09-21T19:10:10.767223+0000","last_peered":"2024-09-21T19:10:10.767223+0000","last_clean":"2024-09-21T19:10:10.767223+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:10:10.767223+0000","last_undegraded":"2024-09-21T19:10:10.767223+0000","last_fullsized":"2024-09-21T19:10:10.767223+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:04:26.469154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:04:26.469154+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":795,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772990+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:10:10.772990+0000","last_peered":"2024-09-21T19:10:10.772990+0000","last_clean":"2024-09-21T19:10:10.772990+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:10:10.772990+0000","last_undegraded":"2024-09-21T19:10:10.772990+0000","last_fullsized":"2024-09-21T19:10:10.772990+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.662781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"420'909","reported_seq":932,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.939974+0000","last_change":"2024-09-21T19:10:01.251501+0000","last_active":"2024-09-21T19:10:12.939974+0000","last_peered":"2024-09-21T19:10:12.939974+0000","last_clean":"2024-09-21T19:10:12.939974+0000","last_became_active":"2024-09-21T19:10:01.250354+0000","last_became_peered":"2024-09-21T19:10:01.250354+0000","last_unstale":"2024-09-21T19:10:12.939974+0000","last_undegraded":"2024-09-21T19:10:12.939974+0000","last_fullsized":"2024-09-21T19:10:12.939974+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:27.378734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59572224,"num_objects":909,"num_object_clones":0,"num_object_copies":2727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":909,"num_write_kb":58176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":579,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771178+0000","last_change":"2024-09-21T19:09:55.907018+0000","last_active":"2024-09-21T19:10:10.771178+0000","last_peered":"2024-09-21T19:10:10.771178+0000","last_clean":"2024-09-21T19:10:10.771178+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:10:10.771178+0000","last_undegraded":"2024-09-21T19:10:10.771178+0000","last_fullsized":"2024-09-21T19:10:10.771178+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:11.373203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":820,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770424+0000","last_change":"2024-09-21T19:09:39.038158+0000","last_active":"2024-09-21T19:10:10.770424+0000","last_peered":"2024-09-21T19:10:10.770424+0000","last_clean":"2024-09-21T19:10:10.770424+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:10:10.770424+0000","last_undegraded":"2024-09-21T19:10:10.770424+0000","last_fullsized":"2024-09-21T19:10:10.770424+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:09:39.038064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.038064+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.698592+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":339,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771332+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:10:10.771332+0000","last_peered":"2024-09-21T19:10:10.771332+0000","last_clean":"2024-09-21T19:10:10.771332+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:10:10.771332+0000","last_undegraded":"2024-09-21T19:10:10.771332+0000","last_fullsized":"2024-09-21T19:10:10.771332+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.609167+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":848,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765236+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:10:10.765236+0000","last_peered":"2024-09-21T19:10:10.765236+0000","last_clean":"2024-09-21T19:10:10.765236+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:10:10.765236+0000","last_undegraded":"2024-09-21T19:10:10.765236+0000","last_fullsized":"2024-09-21T19:10:10.765236+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":668,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765589+0000","last_change":"2024-09-21T19:09:47.129430+0000","last_active":"2024-09-21T19:10:10.765589+0000","last_peered":"2024-09-21T19:10:10.765589+0000","last_clean":"2024-09-21T19:10:10.765589+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:10:10.765589+0000","last_undegraded":"2024-09-21T19:10:10.765589+0000","last_fullsized":"2024-09-21T19:10:10.765589+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:47.129379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:47.129379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.229738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"420'918","reported_seq":941,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.663141+0000","last_change":"2024-09-21T19:10:01.244039+0000","last_active":"2024-09-21T19:10:12.663141+0000","last_peered":"2024-09-21T19:10:12.663141+0000","last_clean":"2024-09-21T19:10:12.663141+0000","last_became_active":"2024-09-21T19:10:01.243730+0000","last_became_peered":"2024-09-21T19:10:01.243730+0000","last_unstale":"2024-09-21T19:10:12.663141+0000","last_undegraded":"2024-09-21T19:10:12.663141+0000","last_fullsized":"2024-09-21T19:10:12.663141+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.216097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60162048,"num_objects":918,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918,"num_write_kb":58752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":557,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771105+0000","last_change":"2024-09-21T19:09:54.937870+0000","last_active":"2024-09-21T19:10:10.771105+0000","last_peered":"2024-09-21T19:10:10.771105+0000","last_clean":"2024-09-21T19:10:10.771105+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:10:10.771105+0000","last_undegraded":"2024-09-21T19:10:10.771105+0000","last_fullsized":"2024-09-21T19:10:10.771105+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:57.511376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":705,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765382+0000","last_change":"2024-09-21T19:06:54.700287+0000","last_active":"2024-09-21T19:10:10.765382+0000","last_peered":"2024-09-21T19:10:10.765382+0000","last_clean":"2024-09-21T19:10:10.765382+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:10:10.765382+0000","last_undegraded":"2024-09-21T19:10:10.765382+0000","last_fullsized":"2024-09-21T19:10:10.765382+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:05:32.978706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.978706+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771402+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:10:10.771402+0000","last_peered":"2024-09-21T19:10:10.771402+0000","last_clean":"2024-09-21T19:10:10.771402+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:10:10.771402+0000","last_undegraded":"2024-09-21T19:10:10.771402+0000","last_fullsized":"2024-09-21T19:10:10.771402+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.451996+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":684,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769341+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:10:10.769341+0000","last_peered":"2024-09-21T19:10:10.769341+0000","last_clean":"2024-09-21T19:10:10.769341+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:10:10.769341+0000","last_undegraded":"2024-09-21T19:10:10.769341+0000","last_fullsized":"2024-09-21T19:10:10.769341+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:08.779433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":788,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770186+0000","last_change":"2024-09-21T19:07:31.239386+0000","last_active":"2024-09-21T19:10:10.770186+0000","last_peered":"2024-09-21T19:10:10.770186+0000","last_clean":"2024-09-21T19:10:10.770186+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:10:10.770186+0000","last_undegraded":"2024-09-21T19:10:10.770186+0000","last_fullsized":"2024-09-21T19:10:10.770186+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:07:31.239303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:07:31.239303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.144483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"420'882","reported_seq":905,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.984909+0000","last_change":"2024-09-21T19:10:01.250359+0000","last_active":"2024-09-21T19:10:12.984909+0000","last_peered":"2024-09-21T19:10:12.984909+0000","last_clean":"2024-09-21T19:10:12.984909+0000","last_became_active":"2024-09-21T19:10:01.250079+0000","last_became_peered":"2024-09-21T19:10:01.250079+0000","last_unstale":"2024-09-21T19:10:12.984909+0000","last_undegraded":"2024-09-21T19:10:12.984909+0000","last_fullsized":"2024-09-21T19:10:12.984909+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.322647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":2646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":56448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":524,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.764555+0000","last_change":"2024-09-21T19:06:54.702251+0000","last_active":"2024-09-21T19:10:10.764555+0000","last_peered":"2024-09-21T19:10:10.764555+0000","last_clean":"2024-09-21T19:10:10.764555+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:10:10.764555+0000","last_undegraded":"2024-09-21T19:10:10.764555+0000","last_fullsized":"2024-09-21T19:10:10.764555+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:04:34.855882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:04:34.855882+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.190338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1102,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771205+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:10:10.771205+0000","last_peered":"2024-09-21T19:10:10.771205+0000","last_clean":"2024-09-21T19:10:10.771205+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:10:10.771205+0000","last_undegraded":"2024-09-21T19:10:10.771205+0000","last_fullsized":"2024-09-21T19:10:10.771205+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.313979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773458+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:10:10.773458+0000","last_peered":"2024-09-21T19:10:10.773458+0000","last_clean":"2024-09-21T19:10:10.773458+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:10:10.773458+0000","last_undegraded":"2024-09-21T19:10:10.773458+0000","last_fullsized":"2024-09-21T19:10:10.773458+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":355,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770657+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:10:10.770657+0000","last_peered":"2024-09-21T19:10:10.770657+0000","last_clean":"2024-09-21T19:10:10.770657+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:10:10.770657+0000","last_undegraded":"2024-09-21T19:10:10.770657+0000","last_fullsized":"2024-09-21T19:10:10.770657+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:10.030025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":653,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.764687+0000","last_change":"2024-09-21T19:09:51.157757+0000","last_active":"2024-09-21T19:10:10.764687+0000","last_peered":"2024-09-21T19:10:10.764687+0000","last_clean":"2024-09-21T19:10:10.764687+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:10:10.764687+0000","last_undegraded":"2024-09-21T19:10:10.764687+0000","last_fullsized":"2024-09-21T19:10:10.764687+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.157711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.157711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:54.408363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":755,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767702+0000","last_change":"2024-09-21T19:05:32.772075+0000","last_active":"2024-09-21T19:10:10.767702+0000","last_peered":"2024-09-21T19:10:10.767702+0000","last_clean":"2024-09-21T19:10:10.767702+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:10:10.767702+0000","last_undegraded":"2024-09-21T19:10:10.767702+0000","last_fullsized":"2024-09-21T19:10:10.767702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:32.772015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:32.772015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"420'919","reported_seq":942,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.976989+0000","last_change":"2024-09-21T19:10:01.259760+0000","last_active":"2024-09-21T19:10:12.976989+0000","last_peered":"2024-09-21T19:10:12.976989+0000","last_clean":"2024-09-21T19:10:12.976989+0000","last_became_active":"2024-09-21T19:10:01.259451+0000","last_became_peered":"2024-09-21T19:10:01.259451+0000","last_unstale":"2024-09-21T19:10:12.976989+0000","last_undegraded":"2024-09-21T19:10:12.976989+0000","last_fullsized":"2024-09-21T19:10:12.976989+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:00.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60227584,"num_objects":919,"num_object_clones":0,"num_object_copies":2757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":919,"num_write_kb":58816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":868,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772302+0000","last_change":"2024-09-21T19:09:55.484547+0000","last_active":"2024-09-21T19:10:10.772302+0000","last_peered":"2024-09-21T19:10:10.772302+0000","last_clean":"2024-09-21T19:10:10.772302+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:10:10.772302+0000","last_undegraded":"2024-09-21T19:10:10.772302+0000","last_fullsized":"2024-09-21T19:10:10.772302+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:15.423137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":273,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771840+0000","last_change":"2024-09-21T19:08:22.212329+0000","last_active":"2024-09-21T19:10:10.771840+0000","last_peered":"2024-09-21T19:10:10.771840+0000","last_clean":"2024-09-21T19:10:10.771840+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:10:10.771840+0000","last_undegraded":"2024-09-21T19:10:10.771840+0000","last_fullsized":"2024-09-21T19:10:10.771840+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:08:22.212236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:08:22.212236+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":830,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769675+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:10:10.769675+0000","last_peered":"2024-09-21T19:10:10.769675+0000","last_clean":"2024-09-21T19:10:10.769675+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:10:10.769675+0000","last_undegraded":"2024-09-21T19:10:10.769675+0000","last_fullsized":"2024-09-21T19:10:10.769675+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.148679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":224,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770563+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:10:10.770563+0000","last_peered":"2024-09-21T19:10:10.770563+0000","last_clean":"2024-09-21T19:10:10.770563+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:10:10.770563+0000","last_undegraded":"2024-09-21T19:10:10.770563+0000","last_fullsized":"2024-09-21T19:10:10.770563+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.753652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":231,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773198+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:10:10.773198+0000","last_peered":"2024-09-21T19:10:10.773198+0000","last_clean":"2024-09-21T19:10:10.773198+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:10:10.773198+0000","last_undegraded":"2024-09-21T19:10:10.773198+0000","last_fullsized":"2024-09-21T19:10:10.773198+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:27.374847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773246+0000","last_change":"2024-09-21T19:09:37.588463+0000","last_active":"2024-09-21T19:10:10.773246+0000","last_peered":"2024-09-21T19:10:10.773246+0000","last_clean":"2024-09-21T19:10:10.773246+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:10:10.773246+0000","last_undegraded":"2024-09-21T19:10:10.773246+0000","last_fullsized":"2024-09-21T19:10:10.773246+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:09:37.588390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.588390+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.015172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"420'968","reported_seq":991,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.985264+0000","last_change":"2024-09-21T19:10:01.250406+0000","last_active":"2024-09-21T19:10:12.985264+0000","last_peered":"2024-09-21T19:10:12.985264+0000","last_clean":"2024-09-21T19:10:12.985264+0000","last_became_active":"2024-09-21T19:10:01.250100+0000","last_became_peered":"2024-09-21T19:10:01.250100+0000","last_unstale":"2024-09-21T19:10:12.985264+0000","last_undegraded":"2024-09-21T19:10:12.985264+0000","last_fullsized":"2024-09-21T19:10:12.985264+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:11.071756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438848,"num_objects":968,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":61952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":566,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770928+0000","last_change":"2024-09-21T19:09:08.156345+0000","last_active":"2024-09-21T19:10:10.770928+0000","last_peered":"2024-09-21T19:10:10.770928+0000","last_clean":"2024-09-21T19:10:10.770928+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:10:10.770928+0000","last_undegraded":"2024-09-21T19:10:10.770928+0000","last_fullsized":"2024-09-21T19:10:10.770928+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:08.156258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:08.156258+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.996519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":767,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.764767+0000","last_change":"2024-09-21T19:06:54.706540+0000","last_active":"2024-09-21T19:10:10.764767+0000","last_peered":"2024-09-21T19:10:10.764767+0000","last_clean":"2024-09-21T19:10:10.764767+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:10:10.764767+0000","last_undegraded":"2024-09-21T19:10:10.764767+0000","last_fullsized":"2024-09-21T19:10:10.764767+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:05:33.974373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.974373+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":653,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765364+0000","last_change":"2024-09-21T19:04:36.892132+0000","last_active":"2024-09-21T19:10:10.765364+0000","last_peered":"2024-09-21T19:10:10.765364+0000","last_clean":"2024-09-21T19:10:10.765364+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:10:10.765364+0000","last_undegraded":"2024-09-21T19:10:10.765364+0000","last_fullsized":"2024-09-21T19:10:10.765364+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:36.892075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:36.892075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1817,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.839602+0000","last_change":"2024-09-21T19:04:00.138122+0000","last_active":"2024-09-21T19:10:10.839602+0000","last_peered":"2024-09-21T19:10:10.839602+0000","last_clean":"2024-09-21T19:10:10.839602+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:10:10.839602+0000","last_undegraded":"2024-09-21T19:10:10.839602+0000","last_fullsized":"2024-09-21T19:10:10.839602+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:04:00.138043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:04:00.138043+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:23.664705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1013,"num_read_kb":1008,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":734,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770692+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:10:10.770692+0000","last_peered":"2024-09-21T19:10:10.770692+0000","last_clean":"2024-09-21T19:10:10.770692+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:10:10.770692+0000","last_undegraded":"2024-09-21T19:10:10.770692+0000","last_fullsized":"2024-09-21T19:10:10.770692+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.747406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":782,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770763+0000","last_change":"2024-09-21T19:08:23.185177+0000","last_active":"2024-09-21T19:10:10.770763+0000","last_peered":"2024-09-21T19:10:10.770763+0000","last_clean":"2024-09-21T19:10:10.770763+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:10:10.770763+0000","last_undegraded":"2024-09-21T19:10:10.770763+0000","last_fullsized":"2024-09-21T19:10:10.770763+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:23.185105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:23.185105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"420'947","reported_seq":970,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.990810+0000","last_change":"2024-09-21T19:10:01.260090+0000","last_active":"2024-09-21T19:10:12.990810+0000","last_peered":"2024-09-21T19:10:12.990810+0000","last_clean":"2024-09-21T19:10:12.990810+0000","last_became_active":"2024-09-21T19:10:01.259620+0000","last_became_peered":"2024-09-21T19:10:01.259620+0000","last_unstale":"2024-09-21T19:10:12.990810+0000","last_undegraded":"2024-09-21T19:10:12.990810+0000","last_fullsized":"2024-09-21T19:10:12.990810+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.395244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62062592,"num_objects":947,"num_object_clones":0,"num_object_copies":2841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":60608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":961,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771317+0000","last_change":"2024-09-21T19:04:28.815376+0000","last_active":"2024-09-21T19:10:10.771317+0000","last_peered":"2024-09-21T19:10:10.771317+0000","last_clean":"2024-09-21T19:10:10.771317+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:10:10.771317+0000","last_undegraded":"2024-09-21T19:10:10.771317+0000","last_fullsized":"2024-09-21T19:10:10.771317+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:04:28.815291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:28.815291+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.956729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":790,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766890+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:10:10.766890+0000","last_peered":"2024-09-21T19:10:10.766890+0000","last_clean":"2024-09-21T19:10:10.766890+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:10:10.766890+0000","last_undegraded":"2024-09-21T19:10:10.766890+0000","last_fullsized":"2024-09-21T19:10:10.766890+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":874,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773086+0000","last_change":"2024-09-21T19:09:46.543580+0000","last_active":"2024-09-21T19:10:10.773086+0000","last_peered":"2024-09-21T19:10:10.773086+0000","last_clean":"2024-09-21T19:10:10.773086+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:10:10.773086+0000","last_undegraded":"2024-09-21T19:10:10.773086+0000","last_fullsized":"2024-09-21T19:10:10.773086+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:46.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:46.543525+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.937610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772756+0000","last_change":"2024-09-21T19:06:54.692231+0000","last_active":"2024-09-21T19:10:10.772756+0000","last_peered":"2024-09-21T19:10:10.772756+0000","last_clean":"2024-09-21T19:10:10.772756+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:10:10.772756+0000","last_undegraded":"2024-09-21T19:10:10.772756+0000","last_fullsized":"2024-09-21T19:10:10.772756+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:59.160041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:59.160041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:15.623405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":325,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771612+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:10:10.771612+0000","last_peered":"2024-09-21T19:10:10.771612+0000","last_clean":"2024-09-21T19:10:10.771612+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:10:10.771612+0000","last_undegraded":"2024-09-21T19:10:10.771612+0000","last_fullsized":"2024-09-21T19:10:10.771612+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:16.170603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"420'1136","reported_seq":1159,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:14.368602+0000","last_change":"2024-09-21T19:10:01.250432+0000","last_active":"2024-09-21T19:10:14.368602+0000","last_peered":"2024-09-21T19:10:14.368602+0000","last_clean":"2024-09-21T19:10:14.368602+0000","last_became_active":"2024-09-21T19:10:01.250116+0000","last_became_peered":"2024-09-21T19:10:01.250116+0000","last_unstale":"2024-09-21T19:10:14.368602+0000","last_undegraded":"2024-09-21T19:10:14.368602+0000","last_fullsized":"2024-09-21T19:10:14.368602+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":1136,"log_dups_size":0,"ondisk_log_size":1136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:24.804001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74448896,"num_objects":1136,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1136,"num_write_kb":72704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":828,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773925+0000","last_change":"2024-09-21T19:05:37.660913+0000","last_active":"2024-09-21T19:10:10.773925+0000","last_peered":"2024-09-21T19:10:10.773925+0000","last_clean":"2024-09-21T19:10:10.773925+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:10:10.773925+0000","last_undegraded":"2024-09-21T19:10:10.773925+0000","last_fullsized":"2024-09-21T19:10:10.773925+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:37.660843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:37.660843+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772609+0000","last_change":"2024-09-21T19:05:42.891865+0000","last_active":"2024-09-21T19:10:10.772609+0000","last_peered":"2024-09-21T19:10:10.772609+0000","last_clean":"2024-09-21T19:10:10.772609+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:10:10.772609+0000","last_undegraded":"2024-09-21T19:10:10.772609+0000","last_fullsized":"2024-09-21T19:10:10.772609+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:30.559507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.559507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":781,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.774086+0000","last_change":"2024-09-21T19:06:54.702958+0000","last_active":"2024-09-21T19:10:10.774086+0000","last_peered":"2024-09-21T19:10:10.774086+0000","last_clean":"2024-09-21T19:10:10.774086+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:10:10.774086+0000","last_undegraded":"2024-09-21T19:10:10.774086+0000","last_fullsized":"2024-09-21T19:10:10.774086+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.634061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.634061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.639948+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":761,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766771+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:10:10.766771+0000","last_peered":"2024-09-21T19:10:10.766771+0000","last_clean":"2024-09-21T19:10:10.766771+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:10:10.766771+0000","last_undegraded":"2024-09-21T19:10:10.766771+0000","last_fullsized":"2024-09-21T19:10:10.766771+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.884689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"420'870","reported_seq":893,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.948688+0000","last_change":"2024-09-21T19:10:01.248144+0000","last_active":"2024-09-21T19:10:12.948688+0000","last_peered":"2024-09-21T19:10:12.948688+0000","last_clean":"2024-09-21T19:10:12.948688+0000","last_became_active":"2024-09-21T19:10:01.247852+0000","last_became_peered":"2024-09-21T19:10:01.247852+0000","last_unstale":"2024-09-21T19:10:12.948688+0000","last_undegraded":"2024-09-21T19:10:12.948688+0000","last_fullsized":"2024-09-21T19:10:12.948688+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.027503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57016320,"num_objects":870,"num_object_clones":0,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":55680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":227,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769891+0000","last_change":"2024-09-21T19:06:54.690303+0000","last_active":"2024-09-21T19:10:10.769891+0000","last_peered":"2024-09-21T19:10:10.769891+0000","last_clean":"2024-09-21T19:10:10.769891+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:10:10.769891+0000","last_undegraded":"2024-09-21T19:10:10.769891+0000","last_fullsized":"2024-09-21T19:10:10.769891+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:01.593588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.593588+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:11.739272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":877,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.774045+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:10:10.774045+0000","last_peered":"2024-09-21T19:10:10.774045+0000","last_clean":"2024-09-21T19:10:10.774045+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:10:10.774045+0000","last_undegraded":"2024-09-21T19:10:10.774045+0000","last_fullsized":"2024-09-21T19:10:10.774045+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":695,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769793+0000","last_change":"2024-09-21T19:05:49.829190+0000","last_active":"2024-09-21T19:10:10.769793+0000","last_peered":"2024-09-21T19:10:10.769793+0000","last_clean":"2024-09-21T19:10:10.769793+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:10:10.769793+0000","last_undegraded":"2024-09-21T19:10:10.769793+0000","last_fullsized":"2024-09-21T19:10:10.769793+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:49.829125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:49.829125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":228,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770341+0000","last_change":"2024-09-21T19:09:50.927165+0000","last_active":"2024-09-21T19:10:10.770341+0000","last_peered":"2024-09-21T19:10:10.770341+0000","last_clean":"2024-09-21T19:10:10.770341+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:10:10.770341+0000","last_undegraded":"2024-09-21T19:10:10.770341+0000","last_fullsized":"2024-09-21T19:10:10.770341+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.927103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.927103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.477371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":228,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772957+0000","last_change":"2024-09-21T19:09:31.740952+0000","last_active":"2024-09-21T19:10:10.772957+0000","last_peered":"2024-09-21T19:10:10.772957+0000","last_clean":"2024-09-21T19:10:10.772957+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:10:10.772957+0000","last_undegraded":"2024-09-21T19:10:10.772957+0000","last_fullsized":"2024-09-21T19:10:10.772957+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:31.740891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:31.740891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:40.289541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"420'1111","reported_seq":1134,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:14.357125+0000","last_change":"2024-09-21T19:10:01.252391+0000","last_active":"2024-09-21T19:10:14.357125+0000","last_peered":"2024-09-21T19:10:14.357125+0000","last_clean":"2024-09-21T19:10:14.357125+0000","last_became_active":"2024-09-21T19:10:01.251894+0000","last_became_peered":"2024-09-21T19:10:01.251894+0000","last_unstale":"2024-09-21T19:10:14.357125+0000","last_undegraded":"2024-09-21T19:10:14.357125+0000","last_fullsized":"2024-09-21T19:10:14.357125+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:17.954196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72810496,"num_objects":1111,"num_object_clones":0,"num_object_copies":3333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1111,"num_write_kb":71104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":882,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772293+0000","last_change":"2024-09-21T19:09:39.566152+0000","last_active":"2024-09-21T19:10:10.772293+0000","last_peered":"2024-09-21T19:10:10.772293+0000","last_clean":"2024-09-21T19:10:10.772293+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:10:10.772293+0000","last_undegraded":"2024-09-21T19:10:10.772293+0000","last_fullsized":"2024-09-21T19:10:10.772293+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:39.565921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:39.565921+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:44.360587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":803,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773622+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:10:10.773622+0000","last_peered":"2024-09-21T19:10:10.773622+0000","last_clean":"2024-09-21T19:10:10.773622+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:10:10.773622+0000","last_undegraded":"2024-09-21T19:10:10.773622+0000","last_fullsized":"2024-09-21T19:10:10.773622+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":803,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767786+0000","last_change":"2024-09-21T19:06:09.897972+0000","last_active":"2024-09-21T19:10:10.767786+0000","last_peered":"2024-09-21T19:10:10.767786+0000","last_clean":"2024-09-21T19:10:10.767786+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:10:10.767786+0000","last_undegraded":"2024-09-21T19:10:10.767786+0000","last_fullsized":"2024-09-21T19:10:10.767786+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:09.897921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.897921+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":284,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.662589+0000","last_change":"2024-09-21T19:06:54.696268+0000","last_active":"2024-09-21T19:10:12.662589+0000","last_peered":"2024-09-21T19:10:12.662589+0000","last_clean":"2024-09-21T19:10:12.662589+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:10:12.662589+0000","last_undegraded":"2024-09-21T19:10:12.662589+0000","last_fullsized":"2024-09-21T19:10:12.662589+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'9","last_scrub_stamp":"2024-09-21T19:05:41.118956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:41.118956+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:19.687093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":776,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773770+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:10:10.773770+0000","last_peered":"2024-09-21T19:10:10.773770+0000","last_clean":"2024-09-21T19:10:10.773770+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:10:10.773770+0000","last_undegraded":"2024-09-21T19:10:10.773770+0000","last_fullsized":"2024-09-21T19:10:10.773770+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"420'748","reported_seq":771,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:11.909253+0000","last_change":"2024-09-21T19:10:01.250343+0000","last_active":"2024-09-21T19:10:11.909253+0000","last_peered":"2024-09-21T19:10:11.909253+0000","last_clean":"2024-09-21T19:10:11.909253+0000","last_became_active":"2024-09-21T19:10:01.249931+0000","last_became_peered":"2024-09-21T19:10:01.249931+0000","last_unstale":"2024-09-21T19:10:11.909253+0000","last_undegraded":"2024-09-21T19:10:11.909253+0000","last_fullsized":"2024-09-21T19:10:11.909253+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:14.985381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49020928,"num_objects":748,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":748,"num_write_kb":47872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":664,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765484+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:10:10.765484+0000","last_peered":"2024-09-21T19:10:10.765484+0000","last_clean":"2024-09-21T19:10:10.765484+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:10:10.765484+0000","last_undegraded":"2024-09-21T19:10:10.765484+0000","last_fullsized":"2024-09-21T19:10:10.765484+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":765,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767318+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:10:10.767318+0000","last_peered":"2024-09-21T19:10:10.767318+0000","last_clean":"2024-09-21T19:10:10.767318+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:10:10.767318+0000","last_undegraded":"2024-09-21T19:10:10.767318+0000","last_fullsized":"2024-09-21T19:10:10.767318+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:05.893237+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":220,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772707+0000","last_change":"2024-09-21T19:09:07.710551+0000","last_active":"2024-09-21T19:10:10.772707+0000","last_peered":"2024-09-21T19:10:10.772707+0000","last_clean":"2024-09-21T19:10:10.772707+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:10:10.772707+0000","last_undegraded":"2024-09-21T19:10:10.772707+0000","last_fullsized":"2024-09-21T19:10:10.772707+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:07.710482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:07.710482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":659,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765262+0000","last_change":"2024-09-21T19:05:42.902679+0000","last_active":"2024-09-21T19:10:10.765262+0000","last_peered":"2024-09-21T19:10:10.765262+0000","last_clean":"2024-09-21T19:10:10.765262+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:10:10.765262+0000","last_undegraded":"2024-09-21T19:10:10.765262+0000","last_fullsized":"2024-09-21T19:10:10.765262+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:01.901272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:01.901272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"420'745","reported_seq":768,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:11.903740+0000","last_change":"2024-09-21T19:10:01.260078+0000","last_active":"2024-09-21T19:10:11.903740+0000","last_peered":"2024-09-21T19:10:11.903740+0000","last_clean":"2024-09-21T19:10:11.903740+0000","last_became_active":"2024-09-21T19:10:01.259752+0000","last_became_peered":"2024-09-21T19:10:01.259752+0000","last_unstale":"2024-09-21T19:10:11.903740+0000","last_undegraded":"2024-09-21T19:10:11.903740+0000","last_fullsized":"2024-09-21T19:10:11.903740+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:07.839116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":722,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.835830+0000","last_change":"2024-09-21T19:10:12.835830+0000","last_active":"2024-09-21T19:10:12.835830+0000","last_peered":"2024-09-21T19:10:12.835830+0000","last_clean":"2024-09-21T19:10:12.835830+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:10:12.835830+0000","last_undegraded":"2024-09-21T19:10:12.835830+0000","last_fullsized":"2024-09-21T19:10:12.835830+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:10:12.835752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:12.835752+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:18.327395+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":795,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773451+0000","last_change":"2024-09-21T19:06:54.692286+0000","last_active":"2024-09-21T19:10:10.773451+0000","last_peered":"2024-09-21T19:10:10.773451+0000","last_clean":"2024-09-21T19:10:10.773451+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:10:10.773451+0000","last_undegraded":"2024-09-21T19:10:10.773451+0000","last_fullsized":"2024-09-21T19:10:10.773451+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:29.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:29.259554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.612669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":231,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772795+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:10:10.772795+0000","last_peered":"2024-09-21T19:10:10.772795+0000","last_clean":"2024-09-21T19:10:10.772795+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:10:10.772795+0000","last_undegraded":"2024-09-21T19:10:10.772795+0000","last_fullsized":"2024-09-21T19:10:10.772795+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":775,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773008+0000","last_change":"2024-09-21T19:09:35.677140+0000","last_active":"2024-09-21T19:10:10.773008+0000","last_peered":"2024-09-21T19:10:10.773008+0000","last_clean":"2024-09-21T19:10:10.773008+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:10:10.773008+0000","last_undegraded":"2024-09-21T19:10:10.773008+0000","last_fullsized":"2024-09-21T19:10:10.773008+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:35.677043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:35.677043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:50.521879+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"420'921","reported_seq":944,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.818917+0000","last_change":"2024-09-21T19:10:01.252118+0000","last_active":"2024-09-21T19:10:12.818917+0000","last_peered":"2024-09-21T19:10:12.818917+0000","last_clean":"2024-09-21T19:10:12.818917+0000","last_became_active":"2024-09-21T19:10:01.251738+0000","last_became_peered":"2024-09-21T19:10:01.251738+0000","last_unstale":"2024-09-21T19:10:12.818917+0000","last_undegraded":"2024-09-21T19:10:12.818917+0000","last_fullsized":"2024-09-21T19:10:12.818917+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:10.078205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":921,"num_write_kb":58944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":832,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772455+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:10:10.772455+0000","last_peered":"2024-09-21T19:10:10.772455+0000","last_clean":"2024-09-21T19:10:10.772455+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:10:10.772455+0000","last_undegraded":"2024-09-21T19:10:10.772455+0000","last_fullsized":"2024-09-21T19:10:10.772455+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":797,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772951+0000","last_change":"2024-09-21T19:09:41.535785+0000","last_active":"2024-09-21T19:10:10.772951+0000","last_peered":"2024-09-21T19:10:10.772951+0000","last_clean":"2024-09-21T19:10:10.772951+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:10:10.772951+0000","last_undegraded":"2024-09-21T19:10:10.772951+0000","last_fullsized":"2024-09-21T19:10:10.772951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:41.535714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:41.535714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:45.775155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":220,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769804+0000","last_change":"2024-09-21T19:08:54.760564+0000","last_active":"2024-09-21T19:10:10.769804+0000","last_peered":"2024-09-21T19:10:10.769804+0000","last_clean":"2024-09-21T19:10:10.769804+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:10:10.769804+0000","last_undegraded":"2024-09-21T19:10:10.769804+0000","last_fullsized":"2024-09-21T19:10:10.769804+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:54.760520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:54.760520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.759186+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":755,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767492+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:10:10.767492+0000","last_peered":"2024-09-21T19:10:10.767492+0000","last_clean":"2024-09-21T19:10:10.767492+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:10:10.767492+0000","last_undegraded":"2024-09-21T19:10:10.767492+0000","last_fullsized":"2024-09-21T19:10:10.767492+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"420'875","reported_seq":898,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:12.679490+0000","last_change":"2024-09-21T19:10:01.260124+0000","last_active":"2024-09-21T19:10:12.679490+0000","last_peered":"2024-09-21T19:10:12.679490+0000","last_clean":"2024-09-21T19:10:12.679490+0000","last_became_active":"2024-09-21T19:10:01.259826+0000","last_became_peered":"2024-09-21T19:10:01.259826+0000","last_unstale":"2024-09-21T19:10:12.679490+0000","last_undegraded":"2024-09-21T19:10:12.679490+0000","last_fullsized":"2024-09-21T19:10:12.679490+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.671546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":56000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":388,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772722+0000","last_change":"2024-09-21T19:06:54.689710+0000","last_active":"2024-09-21T19:10:10.772722+0000","last_peered":"2024-09-21T19:10:10.772722+0000","last_clean":"2024-09-21T19:10:10.772722+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:10:10.772722+0000","last_undegraded":"2024-09-21T19:10:10.772722+0000","last_fullsized":"2024-09-21T19:10:10.772722+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:03:56.136566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:03:56.136566+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:17.636097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":811,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773222+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:10:10.773222+0000","last_peered":"2024-09-21T19:10:10.773222+0000","last_clean":"2024-09-21T19:10:10.773222+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:10:10.773222+0000","last_undegraded":"2024-09-21T19:10:10.773222+0000","last_fullsized":"2024-09-21T19:10:10.773222+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:31.474930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":280,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766692+0000","last_change":"2024-09-21T19:05:37.842820+0000","last_active":"2024-09-21T19:10:10.766692+0000","last_peered":"2024-09-21T19:10:10.766692+0000","last_clean":"2024-09-21T19:10:10.766692+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:10:10.766692+0000","last_undegraded":"2024-09-21T19:10:10.766692+0000","last_fullsized":"2024-09-21T19:10:10.766692+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'10","last_scrub_stamp":"2024-09-21T19:02:46.144365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.144365+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.806824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769427+0000","last_change":"2024-09-21T19:06:54.701859+0000","last_active":"2024-09-21T19:10:10.769427+0000","last_peered":"2024-09-21T19:10:10.769427+0000","last_clean":"2024-09-21T19:10:10.769427+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:10:10.769427+0000","last_undegraded":"2024-09-21T19:10:10.769427+0000","last_fullsized":"2024-09-21T19:10:10.769427+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:24.782657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:24.782657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:58.970506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":726,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770594+0000","last_change":"2024-09-21T19:06:54.695951+0000","last_active":"2024-09-21T19:10:10.770594+0000","last_peered":"2024-09-21T19:10:10.770594+0000","last_clean":"2024-09-21T19:10:10.770594+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:10:10.770594+0000","last_undegraded":"2024-09-21T19:10:10.770594+0000","last_fullsized":"2024-09-21T19:10:10.770594+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:27.767763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:27.767763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:04.844585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":789,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772026+0000","last_change":"2024-09-21T19:04:31.523836+0000","last_active":"2024-09-21T19:10:10.772026+0000","last_peered":"2024-09-21T19:10:10.772026+0000","last_clean":"2024-09-21T19:10:10.772026+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:10:10.772026+0000","last_undegraded":"2024-09-21T19:10:10.772026+0000","last_fullsized":"2024-09-21T19:10:10.772026+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:04:31.523766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:31.523766+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":324,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767622+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:10:10.767622+0000","last_peered":"2024-09-21T19:10:10.767622+0000","last_clean":"2024-09-21T19:10:10.767622+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:10:10.767622+0000","last_undegraded":"2024-09-21T19:10:10.767622+0000","last_fullsized":"2024-09-21T19:10:10.767622+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.823920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773157+0000","last_change":"2024-09-21T19:08:59.673024+0000","last_active":"2024-09-21T19:10:10.773157+0000","last_peered":"2024-09-21T19:10:10.773157+0000","last_clean":"2024-09-21T19:10:10.773157+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:10:10.773157+0000","last_undegraded":"2024-09-21T19:10:10.773157+0000","last_fullsized":"2024-09-21T19:10:10.773157+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:59.672905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:59.672905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:14.714397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":671,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766725+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:10:10.766725+0000","last_peered":"2024-09-21T19:10:10.766725+0000","last_clean":"2024-09-21T19:10:10.766725+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:10:10.766725+0000","last_undegraded":"2024-09-21T19:10:10.766725+0000","last_fullsized":"2024-09-21T19:10:10.766725+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:09.161530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":776,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772510+0000","last_change":"2024-09-21T19:03:51.172211+0000","last_active":"2024-09-21T19:10:10.772510+0000","last_peered":"2024-09-21T19:10:10.772510+0000","last_clean":"2024-09-21T19:10:10.772510+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:10:10.772510+0000","last_undegraded":"2024-09-21T19:10:10.772510+0000","last_fullsized":"2024-09-21T19:10:10.772510+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:51.171874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:51.171874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:16.645472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":759,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771793+0000","last_change":"2024-09-21T19:05:48.782953+0000","last_active":"2024-09-21T19:10:10.771793+0000","last_peered":"2024-09-21T19:10:10.771793+0000","last_clean":"2024-09-21T19:10:10.771793+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:10:10.771793+0000","last_undegraded":"2024-09-21T19:10:10.771793+0000","last_fullsized":"2024-09-21T19:10:10.771793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:48.782897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:48.782897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":219,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773373+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:10:10.773373+0000","last_peered":"2024-09-21T19:10:10.773373+0000","last_clean":"2024-09-21T19:10:10.773373+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:10:10.773373+0000","last_undegraded":"2024-09-21T19:10:10.773373+0000","last_fullsized":"2024-09-21T19:10:10.773373+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:21.314979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":377,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773308+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:10:10.773308+0000","last_peered":"2024-09-21T19:10:10.773308+0000","last_clean":"2024-09-21T19:10:10.773308+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:10:10.773308+0000","last_undegraded":"2024-09-21T19:10:10.773308+0000","last_fullsized":"2024-09-21T19:10:10.773308+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:29.639722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":661,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765994+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:10:10.765994+0000","last_peered":"2024-09-21T19:10:10.765994+0000","last_clean":"2024-09-21T19:10:10.765994+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:10:10.765994+0000","last_undegraded":"2024-09-21T19:10:10.765994+0000","last_fullsized":"2024-09-21T19:10:10.765994+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1004,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766904+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:10:10.766904+0000","last_peered":"2024-09-21T19:10:10.766904+0000","last_clean":"2024-09-21T19:10:10.766904+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:10:10.766904+0000","last_undegraded":"2024-09-21T19:10:10.766904+0000","last_fullsized":"2024-09-21T19:10:10.766904+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:12.049319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":787,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.774041+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:10:10.774041+0000","last_peered":"2024-09-21T19:10:10.774041+0000","last_clean":"2024-09-21T19:10:10.774041+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:10:10.774041+0000","last_undegraded":"2024-09-21T19:10:10.774041+0000","last_fullsized":"2024-09-21T19:10:10.774041+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:03.177889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767092+0000","last_change":"2024-09-21T19:04:30.535034+0000","last_active":"2024-09-21T19:10:10.767092+0000","last_peered":"2024-09-21T19:10:10.767092+0000","last_clean":"2024-09-21T19:10:10.767092+0000","last_became_active":"2024-09-21T18:53:09.717621+0000","last_became_peered":"2024-09-21T18:53:09.717621+0000","last_unstale":"2024-09-21T19:10:10.767092+0000","last_undegraded":"2024-09-21T19:10:10.767092+0000","last_fullsized":"2024-09-21T19:10:10.767092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:04:30.534895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:30.534895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":787,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773313+0000","last_change":"2024-09-21T19:09:32.615127+0000","last_active":"2024-09-21T19:10:10.773313+0000","last_peered":"2024-09-21T19:10:10.773313+0000","last_clean":"2024-09-21T19:10:10.773313+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:10:10.773313+0000","last_undegraded":"2024-09-21T19:10:10.773313+0000","last_fullsized":"2024-09-21T19:10:10.773313+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:32.615041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:32.615041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:43.657148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":445,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771265+0000","last_change":"2024-09-21T19:08:13.250734+0000","last_active":"2024-09-21T19:10:10.771265+0000","last_peered":"2024-09-21T19:10:10.771265+0000","last_clean":"2024-09-21T19:10:10.771265+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:10:10.771265+0000","last_undegraded":"2024-09-21T19:10:10.771265+0000","last_fullsized":"2024-09-21T19:10:10.771265+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:08:13.250645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:13.250645+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":660,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767070+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:10:10.767070+0000","last_peered":"2024-09-21T19:10:10.767070+0000","last_clean":"2024-09-21T19:10:10.767070+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:10:10.767070+0000","last_undegraded":"2024-09-21T19:10:10.767070+0000","last_fullsized":"2024-09-21T19:10:10.767070+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:18.380729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":858,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773227+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:10:10.773227+0000","last_peered":"2024-09-21T19:10:10.773227+0000","last_clean":"2024-09-21T19:10:10.773227+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:10:10.773227+0000","last_undegraded":"2024-09-21T19:10:10.773227+0000","last_fullsized":"2024-09-21T19:10:10.773227+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":777,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773785+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:10:10.773785+0000","last_peered":"2024-09-21T19:10:10.773785+0000","last_clean":"2024-09-21T19:10:10.773785+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:10:10.773785+0000","last_undegraded":"2024-09-21T19:10:10.773785+0000","last_fullsized":"2024-09-21T19:10:10.773785+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:08.642054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770418+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:10:10.770418+0000","last_peered":"2024-09-21T19:10:10.770418+0000","last_clean":"2024-09-21T19:10:10.770418+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:10:10.770418+0000","last_undegraded":"2024-09-21T19:10:10.770418+0000","last_fullsized":"2024-09-21T19:10:10.770418+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:37.041685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766852+0000","last_change":"2024-09-21T19:06:54.699100+0000","last_active":"2024-09-21T19:10:10.766852+0000","last_peered":"2024-09-21T19:10:10.766852+0000","last_clean":"2024-09-21T19:10:10.766852+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:10:10.766852+0000","last_undegraded":"2024-09-21T19:10:10.766852+0000","last_fullsized":"2024-09-21T19:10:10.766852+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:58.663546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:58.663546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":856,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770520+0000","last_change":"2024-09-21T19:07:26.387952+0000","last_active":"2024-09-21T19:10:10.770520+0000","last_peered":"2024-09-21T19:10:10.770520+0000","last_clean":"2024-09-21T19:10:10.770520+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:10:10.770520+0000","last_undegraded":"2024-09-21T19:10:10.770520+0000","last_fullsized":"2024-09-21T19:10:10.770520+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:07:26.387733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:07:26.387733+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:06.007266+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":781,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.773347+0000","last_change":"2024-09-21T19:08:58.660566+0000","last_active":"2024-09-21T19:10:10.773347+0000","last_peered":"2024-09-21T19:10:10.773347+0000","last_clean":"2024-09-21T19:10:10.773347+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:10:10.773347+0000","last_undegraded":"2024-09-21T19:10:10.773347+0000","last_fullsized":"2024-09-21T19:10:10.773347+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.660500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.660500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.521068+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":781,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767577+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:10:10.767577+0000","last_peered":"2024-09-21T19:10:10.767577+0000","last_clean":"2024-09-21T19:10:10.767577+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:10:10.767577+0000","last_undegraded":"2024-09-21T19:10:10.767577+0000","last_fullsized":"2024-09-21T19:10:10.767577+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":672,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.765678+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:10:10.765678+0000","last_peered":"2024-09-21T19:10:10.765678+0000","last_clean":"2024-09-21T19:10:10.765678+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:10:10.765678+0000","last_undegraded":"2024-09-21T19:10:10.765678+0000","last_fullsized":"2024-09-21T19:10:10.765678+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":805,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767628+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:10:10.767628+0000","last_peered":"2024-09-21T19:10:10.767628+0000","last_clean":"2024-09-21T19:10:10.767628+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:10:10.767628+0000","last_undegraded":"2024-09-21T19:10:10.767628+0000","last_fullsized":"2024-09-21T19:10:10.767628+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":718,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.769977+0000","last_change":"2024-09-21T19:06:54.691961+0000","last_active":"2024-09-21T19:10:10.769977+0000","last_peered":"2024-09-21T19:10:10.769977+0000","last_clean":"2024-09-21T19:10:10.769977+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:10:10.769977+0000","last_undegraded":"2024-09-21T19:10:10.769977+0000","last_fullsized":"2024-09-21T19:10:10.769977+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.444704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.444704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:02.766402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":800,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771947+0000","last_change":"2024-09-21T19:09:51.462137+0000","last_active":"2024-09-21T19:10:10.771947+0000","last_peered":"2024-09-21T19:10:10.771947+0000","last_clean":"2024-09-21T19:10:10.771947+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:10:10.771947+0000","last_undegraded":"2024-09-21T19:10:10.771947+0000","last_fullsized":"2024-09-21T19:10:10.771947+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.462077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:01.604947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767730+0000","last_change":"2024-09-21T19:04:23.459146+0000","last_active":"2024-09-21T19:10:10.767730+0000","last_peered":"2024-09-21T19:10:10.767730+0000","last_clean":"2024-09-21T19:10:10.767730+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:10:10.767730+0000","last_undegraded":"2024-09-21T19:10:10.767730+0000","last_fullsized":"2024-09-21T19:10:10.767730+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:23.459083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:04:23.459083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":804,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767297+0000","last_change":"2024-09-21T19:06:21.924894+0000","last_active":"2024-09-21T19:10:10.767297+0000","last_peered":"2024-09-21T19:10:10.767297+0000","last_clean":"2024-09-21T19:10:10.767297+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:10:10.767297+0000","last_undegraded":"2024-09-21T19:10:10.767297+0000","last_fullsized":"2024-09-21T19:10:10.767297+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:06:21.924735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:21.924735+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":655,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766302+0000","last_change":"2024-09-21T19:06:54.702062+0000","last_active":"2024-09-21T19:10:10.766302+0000","last_peered":"2024-09-21T19:10:10.766302+0000","last_clean":"2024-09-21T19:10:10.766302+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:10:10.766302+0000","last_undegraded":"2024-09-21T19:10:10.766302+0000","last_fullsized":"2024-09-21T19:10:10.766302+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:55.627499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:03:55.627499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:59.182751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":693,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770804+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:10:10.770804+0000","last_peered":"2024-09-21T19:10:10.770804+0000","last_clean":"2024-09-21T19:10:10.770804+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:10:10.770804+0000","last_undegraded":"2024-09-21T19:10:10.770804+0000","last_fullsized":"2024-09-21T19:10:10.770804+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:05.847567+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":775,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.772400+0000","last_change":"2024-09-21T19:08:57.705386+0000","last_active":"2024-09-21T19:10:10.772400+0000","last_peered":"2024-09-21T19:10:10.772400+0000","last_clean":"2024-09-21T19:10:10.772400+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:10:10.772400+0000","last_undegraded":"2024-09-21T19:10:10.772400+0000","last_fullsized":"2024-09-21T19:10:10.772400+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:57.705327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:08:57.705327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:07.569040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.771080+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:10:10.771080+0000","last_peered":"2024-09-21T19:10:10.771080+0000","last_clean":"2024-09-21T19:10:10.771080+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:10:10.771080+0000","last_undegraded":"2024-09-21T19:10:10.771080+0000","last_fullsized":"2024-09-21T19:10:10.771080+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:25.261004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":797,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.767539+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:10:10.767539+0000","last_peered":"2024-09-21T19:10:10.767539+0000","last_clean":"2024-09-21T19:10:10.767539+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:10:10.767539+0000","last_undegraded":"2024-09-21T19:10:10.767539+0000","last_fullsized":"2024-09-21T19:10:10.767539+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":766,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.770860+0000","last_change":"2024-09-21T19:09:48.894774+0000","last_active":"2024-09-21T19:10:10.770860+0000","last_peered":"2024-09-21T19:10:10.770860+0000","last_clean":"2024-09-21T19:10:10.770860+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:10:10.770860+0000","last_undegraded":"2024-09-21T19:10:10.770860+0000","last_fullsized":"2024-09-21T19:10:10.770860+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:09:48.894708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.894708+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:10:56.651349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":676,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-21T19:10:10.766969+0000","last_change":"2024-09-21T19:05:42.904512+0000","last_active":"2024-09-21T19:10:10.766969+0000","last_peered":"2024-09-21T19:10:10.766969+0000","last_clean":"2024-09-21T19:10:10.766969+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:10:10.766969+0000","last_undegraded":"2024-09-21T19:10:10.766969+0000","last_fullsized":"2024-09-21T19:10:10.766969+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:26.929117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:26.929117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":941162496,"num_objects":14361,"num_object_clones":0,"num_object_copies":43083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14361,"num_write_kb":919104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2885746688,"data_stored":2885746688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14361,"ondisk_log_size":14361,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1013,"num_read_kb":1008,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266897,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879716,"kb_used_data":275152,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92865436,"statfs":{"total":95995035648,"available":95094206464,"internally_reserved":0,"allocated":281755648,"data_stored":281036015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464348,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874456,"kb_used_data":535624,"kb_used_omap":4,"kb_used_meta":338811,"kb_avail":92870696,"statfs":{"total":95995035648,"available":95099592704,"internally_reserved":0,"allocated":548478976,"data_stored":547767106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":346943140},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619049,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":498136,"kb_used_data":232392,"kb_used_omap":4,"kb_used_meta":265723,"kb_avail":93247016,"statfs":{"total":95995035648,"available":95484944384,"internally_reserved":0,"allocated":237969408,"data_stored":237257992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":272101027},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019103,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695572,"kb_used_data":150724,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93049580,"statfs":{"total":95995035648,"available":95282769920,"internally_reserved":0,"allocated":154341376,"data_stored":153627339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182628,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1148920,"kb_used_data":409704,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92596232,"statfs":{"total":95995035648,"available":94818541568,"internally_reserved":0,"allocated":419536896,"data_stored":418794030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411562,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894860,"kb_used_data":336124,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92850292,"statfs":{"total":95995035648,"available":95078699008,"internally_reserved":0,"allocated":344190976,"data_stored":343491853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106884,"kb_used_data":412084,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92638268,"statfs":{"total":95995035648,"available":94861586432,"internally_reserved":0,"allocated":421974016,"data_stored":421218422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738612,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1091492,"kb_used_data":266196,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92653660,"statfs":{"total":95995035648,"available":94877347840,"internally_reserved":0,"allocated":272584704,"data_stored":271861664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295895040,"data_stored":295895040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":468975616,"data_stored":468975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363462656,"data_stored":363462656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":426311680,"data_stored":426311680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":184811520,"data_stored":184811520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":252051456,"data_stored":252051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":591986688,"data_stored":591986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":302252032,"data_stored":302252032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:10:24.979 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T19:10:24.979 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:10:24.980 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T19:10:24.980 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T19:10:24.980 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T19:10:24.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T19:10:24.988 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:24.997 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:24.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:25.003 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:25.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:25.142 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:25.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:25.284 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:25.360 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:10:25.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.409 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:25.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: pgmap v859: 158 pgs: 158 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T19:10:25.475 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: pgmap v859: 158 pgs: 158 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T19:10:25.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:25 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T19:10:25.487 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T19:10:25.487 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T19:10:25.488 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: pgmap v859: 158 pgs: 158 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-21T19:10:25.488 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:10:25.488 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:10:25.488 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T19:10:25.488 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:25 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T19:10:25.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:25.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:25.655 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:25.656 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:25.685 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:25.687 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:25.687 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:25.719 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:25.719 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:25.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 18 16 37628 37612 130.582 102.375 0.00583727 0.00765207 2024-09-21T19:10:25.755 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:25.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:25.788 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:25.788 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:25.814 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:25.820 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:25.820 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:25.853 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:25.854 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:25.887 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:25.887 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:25.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:25.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:25.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:26.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:26.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:26.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:26.234 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:26.333 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:10:26.333 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T19:10:26.333 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:10:26.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:26.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:10:26.376 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:26.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:38873 11.6 11:6ce25306:::benchmark_data_smithi081_181703_object38872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:26.486896+0000", 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.013703537999999999, 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013736196000000001, 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:26.505 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 38873 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.486896+0000", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.486896+0000", 2024-09-21T19:10:26.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.486899+0000", 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1579999999999999e-06 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.486924+0000", 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5058999999999998e-05 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:26.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.486945+0000", 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1200999999999999e-05 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.486953+0000", 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6689999999999995e-06 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.487029+0000", 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5407999999999994e-05 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.487282+0000", 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025274199999999997 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.487429+0000", 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014723700000000001 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.490038+0000", 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026091759999999999 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.509 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.497019+0000", 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0069810189999999998 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:38884 11.1 e426/414 11:81f60ab3:::benchmark_data_smithi081_181703_object38883:head v 426'2438, mlcod=426'2436)", 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:26.493154+0000", 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0074457999999999998, 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0075651650000000004, 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:26.510 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493154+0000", 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493154+0000", 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:26.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493156+0000", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.536e-06 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493385+0000", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022977699999999999 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493396+0000", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0479000000000001e-05 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493403+0000", 2024-09-21T19:10:26.512 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9670000000000004e-06 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493710+0000", 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030725799999999998 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.493730+0000", 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9851e-05 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:38891 11.6 11:6c799a84:::benchmark_data_smithi081_181703_object38890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:26.499492+0000", 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0011081019999999999, 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012820189999999999, 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 38891 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:26.514 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.499492+0000", 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.499492+0000", 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.499610+0000", 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118268 2024-09-21T19:10:26.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.499637+0000", 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6962999999999999e-05 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.499654+0000", 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6823000000000001e-05 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.499740+0000", 2024-09-21T19:10:26.516 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5791000000000006e-05 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.500264+0000", 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000524651 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.500398+0000", 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013342200000000001 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:26.500542+0000", 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014447200000000001 2024-09-21T19:10:26.517 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.518 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:26.518 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.518 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:26.518 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:26.518 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 3 2024-09-21T19:10:26.518 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:26.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 4.17 scrub starts 2024-09-21T19:10:26.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 4.17 scrub ok 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T19:10:26.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2441545384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.520 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:26.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 4.17 scrub starts 2024-09-21T19:10:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 4.17 scrub ok 2024-09-21T19:10:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T19:10:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T19:10:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T19:10:26.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2441545384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 4.17 scrub starts 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 4.17 scrub ok 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T19:10:26.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:10:26.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:10:26.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T19:10:26.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T19:10:26.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2441545384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:26.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:26.649 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:26.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 19 16 39262 39246 129.083 102.125 0.00798292 0.00774206 2024-09-21T19:10:26.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:26.795 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:26.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:26.922 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:26.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:27.024 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.062 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:27.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:27.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:27.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:27.341 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:27.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: pgmap v861: 158 pgs: 158 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: 6.1b scrub starts 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: 6.1b scrub ok 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:10:27.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:10:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: pgmap v861: 158 pgs: 158 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: 6.1b scrub starts 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: 6.1b scrub ok 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:10:27.464 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:27 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:10:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:27.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:27.616 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:27.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:27.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:10:27.732164+0000 min lat: 0.00122502 max lat: 0.458913 avg lat: 0.00777269 2024-09-21T19:10:27.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:10:27.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 20 16 41164 41148 128.573 118.875 0.00902398 0.00777269 2024-09-21T19:10:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:27.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: pgmap v861: 158 pgs: 158 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-21T19:10:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: 6.1b scrub starts 2024-09-21T19:10:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: 6.1b scrub ok 2024-09-21T19:10:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T19:10:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T19:10:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:10:27.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:27 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:10:27.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:27.879 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:27.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:27.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:28.025 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:28.127 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:28.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:28.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:28.318 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:28.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:28.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:28.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:10:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:28.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:28.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T19:10:28.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T19:10:28.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T19:10:28.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T19:10:28.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T19:10:28.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T19:10:28.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T19:10:28.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T19:10:28.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:10:28.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:28 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:10:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:28.626 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T19:10:28.626 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T19:10:28.627 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T19:10:28.627 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T19:10:28.627 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T19:10:28.627 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T19:10:28.627 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:10:28.627 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:10:28.707 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:28.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 21 16 42966 42950 127.813 112.625 0.00480309 0.00781885 2024-09-21T19:10:28.739 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:28.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:28.916 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:28.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:29.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:10:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:10:29.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:29.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:29.189 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:29.254 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:29.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:29.294 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[29384]: 6.11 scrub starts 2024-09-21T19:10:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:29.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:29.464 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[29384]: 6.11 scrub ok 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[29384]: pgmap v862: 158 pgs: 158 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[39192]: 6.11 scrub starts 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[39192]: 6.11 scrub ok 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[39192]: pgmap v862: 158 pgs: 158 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-21T19:10:29.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:29.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:29.601 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:29.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:29.733 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:29.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 22 16 44514 44498 126.4 96.75 0.0124961 0.00790649 2024-09-21T19:10:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:29.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:29 smithi104 ceph-mon[32438]: 6.11 scrub starts 2024-09-21T19:10:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:29 smithi104 ceph-mon[32438]: 6.11 scrub ok 2024-09-21T19:10:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:29 smithi104 ceph-mon[32438]: pgmap v862: 158 pgs: 158 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-21T19:10:29.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:29.757 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:10:29.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:29.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:10:29.870 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:29.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:29.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:30.002 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:30.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:30.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.272 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:30.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:30.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:30.408 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:30.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: 2.0 scrub starts 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: 2.0 scrub ok 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: 6.6 scrub starts 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: 6.6 scrub ok 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2956566445' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2956566445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T19:10:30.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[29384]: pgmap v864: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 779 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: 2.0 scrub starts 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: 2.0 scrub ok 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: 6.6 scrub starts 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: 6.6 scrub ok 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2956566445' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2956566445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T19:10:30.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:30 smithi081 ceph-mon[39192]: pgmap v864: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 779 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T19:10:30.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:30.680 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:30.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 23 16 45979 45963 124.885 91.5625 0.00811234 0.00800146 2024-09-21T19:10:30.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: 2.0 scrub starts 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: 2.0 scrub ok 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: 6.6 scrub starts 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: 6.6 scrub ok 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2956566445' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2956566445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T19:10:30.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:30 smithi104 ceph-mon[32438]: pgmap v864: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 779 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T19:10:30.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.816 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:30.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:30.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:30.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:30.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:31.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.084 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:31.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:31.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:31.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:31.352 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:31.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:31.475 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:31.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:31 smithi081 ceph-mon[29384]: 3.d scrub starts 2024-09-21T19:10:31.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:31 smithi081 ceph-mon[29384]: 3.d scrub ok 2024-09-21T19:10:31.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:31 smithi081 ceph-mon[39192]: 3.d scrub starts 2024-09-21T19:10:31.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:31 smithi081 ceph-mon[39192]: 3.d scrub ok 2024-09-21T19:10:31.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:31.597 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:31.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:31.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 24 16 47450 47434 123.512 91.9375 0.012954 0.00809068 2024-09-21T19:10:31.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:31.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:31 smithi104 ceph-mon[32438]: 3.d scrub starts 2024-09-21T19:10:31.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:31 smithi104 ceph-mon[32438]: 3.d scrub ok 2024-09-21T19:10:31.839 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:31.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:31.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:31.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:32.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:32.074 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:10:32.074 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:10:32.074 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T19:10:32.074 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T19:10:32.075 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T19:10:32.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-21T19:10:32.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:32.136 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:32.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:32.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:32.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:32.423 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:32.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:32.565 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:32.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:32 smithi081 ceph-mon[39192]: pgmap v865: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-21T19:10:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:32.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:32 smithi081 ceph-mon[29384]: pgmap v865: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-21T19:10:32.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:32.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:32.733 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 25 16 49035 49019 122.534 99.0625 0.0109955 0.00815536 2024-09-21T19:10:32.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:32 smithi104 ceph-mon[32438]: pgmap v865: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-21T19:10:32.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:32.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:32.986 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:32.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:33.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:33.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:33.142 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:33.142 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:10:33.142 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:10:33.142 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:10:33.143 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:33.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:33.246 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:33.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:33.379 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:33.481 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.508 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:33.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:33.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.651 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:33.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:33.734 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 26 16 50661 50645 121.73 101.625 0.00470243 0.00820992 2024-09-21T19:10:33.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.791 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:33.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:33.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:33.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:33.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:34.023 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:34.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:34.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:10:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.105 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:34.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:34.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:34.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.386 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:34.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:34.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:34.514 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:34.616 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:34.645 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:34.734 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 27 16 52115 52099 120.587 90.875 0.00298065 0.00828323 2024-09-21T19:10:34.747 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:34.783 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:34.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:34.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:10:34.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:34.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:34.926 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:34 smithi081 ceph-mon[29384]: pgmap v866: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-21T19:10:34.927 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:34 smithi081 ceph-mon[39192]: pgmap v866: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-21T19:10:34.932 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:34.932 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "" 2024-09-21T19:10:34.932 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:35.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:35.057 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:35.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:35.189 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:34 smithi104 ceph-mon[32438]: pgmap v866: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-21T19:10:35.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:35.327 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:35.327 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:35.327 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:35.328 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:35.328 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:35.328 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:35.328 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:35.328 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:35.347 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:35.347 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:35.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:35.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:35.416 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:35.417 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:35.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:35.449 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:35.449 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:35.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:35.597 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:35.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:35.734 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 28 16 53547 53531 119.476 89.5 0.00581396 0.00836495 2024-09-21T19:10:35.737 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:35.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:35.872 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:35.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:35.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:36.006 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:36.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:35 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1672269333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:36.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:35 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1672269333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:36.108 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:36.152 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:36.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:36.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:35 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1672269333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:36.303 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:36.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.448 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:36.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:36.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:36.582 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:36.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:36.734 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 29 16 55166 55150 118.845 101.188 0.00575657 0.00840915 2024-09-21T19:10:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:36.775 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:36.815 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:36.866 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:36.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:36.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:37.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:37.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:36 smithi081 ceph-mon[39192]: pgmap v867: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-21T19:10:37.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:36 smithi081 ceph-mon[29384]: pgmap v867: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-21T19:10:37.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:37.141 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:37.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:37.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:36 smithi104 ceph-mon[32438]: pgmap v867: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-21T19:10:37.283 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:37.347 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:10:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:10:37.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:37.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:37.553 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:37.655 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:37.690 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:37.734 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 30 16 56589 56573 117.848 88.9375 0.0167683 0.00848041 2024-09-21T19:10:37.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.821 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:37.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:10:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:10:37.922 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:37.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-21T19:10:37.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-21T19:10:37.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:37.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:37.947 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-21T19:10:37.947 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-21T19:10:37.948 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:37.948 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:37.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:37.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:38.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:38.079 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:38.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:38.219 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-21T19:10:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-21T19:10:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:38.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:38.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:38.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:38.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:38.487 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:38.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:38.622 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:38.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:10:38.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:38.735 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 31 16 58070 58054 117.029 92.5625 0.00622097 0.00853898 2024-09-21T19:10:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:38.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:10:38.864 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:38.883 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:10:38.890 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:38.890 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:10:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:38.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:38.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:39.063 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: pgmap v868: 158 pgs: 158 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-21T19:10:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-21T19:10:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T19:10:39.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4281819594' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: from='client.36542 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: from='client.36542 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[39192]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: pgmap v868: 158 pgs: 158 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4281819594' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: from='client.36542 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: from='client.36542 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:39.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:38 smithi081 ceph-mon[29384]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T19:10:39.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:39.196 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: pgmap v868: 158 pgs: 158 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4281819594' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: from='client.36542 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: from='client.36542 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:10:39.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:38 smithi104 ceph-mon[32438]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T19:10:39.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.328 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:39.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:39.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:39.468 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:39.569 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:39.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:39.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:39.735 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 32 16 59456 59440 116.079 86.625 0.0072526 0.0086099 2024-09-21T19:10:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.741 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:39.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:39.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:39.865 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:39.870 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:39.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:39.922 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:39.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:40.012 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:40.013 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:40.016 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:40.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:40.150 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:40.252 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:40.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:40.285 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:40.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:40.421 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:40.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:40.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:40.660 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:40.661 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:40.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:40.698 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:40.698 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:40.731 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:40.731 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:40.735 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 33 16 60776 60760 115.061 82.5 0.0127986 0.00868495 2024-09-21T19:10:40.765 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:40.765 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:40.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:40.799 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:40.800 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:40.822 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:40.833 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:40.834 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:40.866 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:40.866 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:40.903 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:40.903 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:40.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:40.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:40.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:41.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:41.116 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:41.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:40 smithi104 ceph-mon[32438]: pgmap v871: 158 pgs: 1 peering, 157 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T19:10:41.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/687375758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:41.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:40 smithi081 ceph-mon[29384]: pgmap v871: 158 pgs: 1 peering, 157 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T19:10:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/687375758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:41.304 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:40 smithi081 ceph-mon[39192]: pgmap v871: 158 pgs: 1 peering, 157 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T19:10:41.304 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/687375758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:41.323 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:10:41.324 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:10:41.324 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T19:10:41.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T19:10:41.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.417 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:41.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:61706 11.f e429/414 11:fb39be97:::benchmark_data_smithi081_181703_object61705:head v 429'3923, mlcod=429'3920)", 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.471345+0000", 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0044755639999999996, 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045044990000000003, 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.484 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471345+0000", 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471345+0000", 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471347+0000", 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2050000000000001e-06 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471385+0000", 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8547999999999998e-05 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471405+0000", 2024-09-21T19:10:41.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9826999999999998e-05 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471415+0000", 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5060000000000001e-06 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471489+0000", 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4340999999999999e-05 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.487 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471511+0000", 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2064000000000001e-05 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:61707 11.f e429/414)", 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.471446+0000", 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0043745549999999996, 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044718980000000002, 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471446+0000", 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471446+0000", 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471446+0000", 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2399999999999998e-07 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471484+0000", 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8034000000000002e-05 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471490+0000", 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6010000000000002e-06 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.490 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.471495+0000", 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6959999999999996e-06 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:61712 11.0 11:05f2da15:::benchmark_data_smithi081_181703_object61711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.472749+0000", 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0030709470000000001, 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003218701, 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.491 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61712 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472749+0000", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472749+0000", 2024-09-21T19:10:41.492 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472751+0000", 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7880000000000001e-06 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472791+0000", 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9875e-05 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472801+0000", 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0202e-05 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472808+0000", 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6139999999999997e-06 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472865+0000", 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7581999999999999e-05 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.494 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.472995+0000", 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013002900000000001 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,7", 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.473137+0000", 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014113600000000001 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:61708 11.b e429/414 11:d4817c9e:::benchmark_data_smithi081_181703_object61707:head v 429'3839, mlcod=429'3837)", 2024-09-21T19:10:41.495 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.474645+0000", 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001174792, 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013912740000000001, 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474645+0000", 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474645+0000", 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474648+0000", 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.001e-06 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474679+0000", 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1659000000000003e-05 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474693+0000", 2024-09-21T19:10:41.497 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4065e-05 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474704+0000", 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0849999999999999e-05 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474971+0000", 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000266746 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:41.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474989+0000", 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8419000000000001e-05 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:61705 11.6 e429/414 11:61d5a9a1:::benchmark_data_smithi081_181703_object61704:head v 429'3890, mlcod=429'3886)", 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.474664+0000", 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001156081, 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001430176, 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474664+0000", 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474664+0000", 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474666+0000", 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.499e-06 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.500 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474691+0000", 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5748000000000001e-05 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474702+0000", 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1022999999999999e-05 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474709+0000", 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.956e-06 2024-09-21T19:10:41.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474785+0000", 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5476999999999996e-05 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474806+0000", 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1109999999999999e-05 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.502 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:61713 11.f e429/414)", 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.474740+0000", 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001080475, 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001402983, 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474740+0000", 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.503 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474740+0000", 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474741+0000", 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1800000000000005e-07 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474772+0000", 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1563999999999998e-05 2024-09-21T19:10:41.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474778+0000", 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1410000000000002e-06 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474783+0000", 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9640000000000002e-06 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:61714 11.0 11:0563a1ee:::benchmark_data_smithi081_181703_object61713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:10:41.474870+0000", 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00094995400000000003, 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013249469999999999, 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 61714 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:41.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474870+0000", 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474870+0000", 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474872+0000", 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.268e-06 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.507 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474892+0000", 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0632000000000002e-05 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474902+0000", 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7869999999999996e-06 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.474908+0000", 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2210000000000002e-06 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:41.508 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:10:41.475317+0000", 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000408668 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 7 2024-09-21T19:10:41.509 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:41.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:41.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:41.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:41.718 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:41.736 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 34 16 62017 62001 113.956 77.5625 0.0100568 0.00877007 2024-09-21T19:10:41.811 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:10:41.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:41.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:41.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:42.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:42.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:42.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:42.292 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:42.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:42.428 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:42.530 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:42.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:42.696 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:42.736 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 35 16 63329 63313 113.043 82 0.00732462 0.00884102 2024-09-21T19:10:42.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:42.839 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:42.941 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:42.978 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:42.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:43.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:10:43.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:42 smithi081 ceph-mon[29384]: pgmap v872: 158 pgs: 1 peering, 157 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T19:10:43.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:42 smithi081 ceph-mon[39192]: pgmap v872: 158 pgs: 1 peering, 157 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T19:10:43.072 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:43.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:43.106 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:43.202 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:42 smithi104 ceph-mon[32438]: pgmap v872: 158 pgs: 1 peering, 157 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T19:10:43.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.258 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:43.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:43.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:43.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:43.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:43.537 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:43.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:43.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:43.718 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:43.736 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 36 16 64481 64465 111.903 72 0.0266706 0.00892954 2024-09-21T19:10:43.781 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:43.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:43.842 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:43.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:43.973 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:43.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:44.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:44.123 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 429, 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T19:10:38.863570+0000", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T19:06:54.357382+0000", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T19:05:41.856710+0000", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5806080, 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "nodeep-scrub", 2024-09-21T19:10:44.124 INFO:teuthology.orchestra.run.smithi081.stdout: "noscrub", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 20, 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 11, 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T19:10:44.125 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.126 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 11, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 11, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 11, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 11, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 11, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.127 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "170", 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "168", 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.128 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.129 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.130 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.1800000667572021, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.1800000667572021, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-21T19:10:44.131 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.132 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.133 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T19:10:44.134 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.135 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.136 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6500000953674316, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6500000953674316, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T19:10:44.137 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.138 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:10:44.139 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.140 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.141 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.142 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:10:44.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T19:10:44.144 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.145 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.146 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.147 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.25, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.25, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.25, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.25, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T19:10:44.150 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.151 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.152 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.153 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.154 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5, 2024-09-21T19:10:44.155 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.5, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.5, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:10:44.156 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:10:44.157 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:10:44.158 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.159 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.160 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:10:44.161 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 11, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_4", 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T19:09:59.705202+0000", 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:10:44.162 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 16, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 16, 2024-09-21T19:10:44.163 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 16, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 16, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 16, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "418", 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:10:44.164 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:10:44.165 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:10:44.166 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:10:44.167 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:10:44.168 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.169 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.170 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T19:10:44.171 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T19:10:44.172 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.173 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.174 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.175 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.177 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.177 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T19:10:44.177 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T19:10:44.178 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.179 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.180 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.181 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.182 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.183 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 428, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:10:44.184 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.185 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.187 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:10:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.191 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 0.16912841796875, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 37, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 326, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 348, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 348, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 327, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:10:44.192 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.194 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/3649492645", 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/3649492645", 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/3649492645", 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/3649492645", 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T19:10:44.195 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 45, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 238, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 261, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 239, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.196 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.197 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.198 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/2795996229", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/2795996229", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/2795996229", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/2795996229", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:10:44.200 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:10:44.201 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.202 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:09:27.199421+0000", 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:10:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:07.797276+0000", 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.205 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:19.932106+0000", 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:09:55.632813+0000", 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:10:44.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:18.847048+0000", 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.207 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:24.337182+0000", 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 326 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:09:37.226663+0000", 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 238 2024-09-21T19:10:44.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:22.667125+0000", 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.209 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.6", 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:10:44.210 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.14", 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.1d", 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.212 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T19:10:44.212 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T19:10:44.212 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.212 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:44.213 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.213 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.9", 2024-09-21T19:10:44.213 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 7, 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 3 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.c", 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.1f", 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 3 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "6.2", 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:10:44.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 3, 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 0 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [], 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:10:44.216 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T19:10:44.217 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T19:10:44.218 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T19:10:44.219 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T19:10:44.219 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:44.219 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:44.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:44.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.387 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:44.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:44.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:10:44.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:44.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:44.638 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:44.737 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 37 16 65477 65461 110.556 62.25 0.00744666 0.0090396 2024-09-21T19:10:44.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:44.785 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:44.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:44.915 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:44.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:45.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:45.038 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:45.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:45.178 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:44 smithi104 ceph-mon[32438]: pgmap v873: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1258680728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:10:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3710696397' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:44 smithi104 ceph-mon[32438]: from='client.26401 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:45.280 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[39192]: pgmap v873: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1258680728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:10:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3710696397' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:45.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[39192]: from='client.26401 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[29384]: pgmap v873: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1258680728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:10:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3710696397' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:45.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:44 smithi081 ceph-mon[29384]: from='client.26401 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:10:45.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:45.450 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:45.540 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:45.540 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:10:45.540 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:10:45.540 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:10:45.540 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:45.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:45.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:45.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:45.737 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 38 16 66642 66626 109.563 72.8125 0.0103958 0.00912276 2024-09-21T19:10:45.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:45.858 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:45.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:45.994 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:45.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:46.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:46.128 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:46.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:46.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:45 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:45 smithi104 ceph-mon[32438]: from='client.26401 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:10:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:45 smithi104 ceph-mon[32438]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T19:10:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:45 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T19:10:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:45 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T19:10:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[39192]: from='client.26401 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:10:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[39192]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T19:10:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T19:10:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T19:10:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:10:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[29384]: from='client.26401 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:10:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[29384]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T19:10:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T19:10:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:45 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T19:10:46.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:46.387 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:46.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:46.534 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:10:46.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:46.666 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:46.738 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 39 16 68003 67987 108.934 85.0625 0.0150068 0.0091741 2024-09-21T19:10:46.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:46.803 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:46.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:46.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:46.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:47.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:47.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:47.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[29384]: pgmap v875: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 645 B/s rd, 81 MiB/s wr, 1.30k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[29384]: 5.17 scrub starts 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[29384]: 5.17 scrub ok 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[39192]: pgmap v875: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 645 B/s rd, 81 MiB/s wr, 1.30k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[39192]: 5.17 scrub starts 2024-09-21T19:10:47.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:46 smithi081 ceph-mon[39192]: 5.17 scrub ok 2024-09-21T19:10:47.229 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:46 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T19:10:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:46 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T19:10:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:46 smithi104 ceph-mon[32438]: pgmap v875: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 645 B/s rd, 81 MiB/s wr, 1.30k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:46 smithi104 ceph-mon[32438]: 5.17 scrub starts 2024-09-21T19:10:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:46 smithi104 ceph-mon[32438]: 5.17 scrub ok 2024-09-21T19:10:47.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.370 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:47.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:47.471 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.505 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:47.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:47.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:10:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:10:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:10:47.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:47.645 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:47.738 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:10:47.736742+0000 min lat: 0.00122502 max lat: 0.458913 avg lat: 0.00922565 2024-09-21T19:10:47.738 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:10:47.738 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 40 16 69359 69343 108.33 84.75 0.0106566 0.00922565 2024-09-21T19:10:47.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:47.774 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:47.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:47.806 INFO:tasks.thrashosds.thrasher:j is {'epoch': 429, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T19:10:38.863570+0000', 'last_up_change': '2024-09-21T19:06:54.357382+0000', 'last_in_change': '2024-09-21T19:05:41.856710+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '168', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T19:09:59.705202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '418', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 428, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.16912841796875, 'last_clean_begin': 37, 'last_clean_end': 326, 'up_from': 348, 'up_thru': 348, 'down_at': 327, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 3649492645}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 3649492645}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 3649492645}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 3649492645}]}, 'public_addr': '172.21.15.104:6804/3649492645', 'cluster_addr': '172.21.15.104:6805/3649492645', 'heartbeat_back_addr': '172.21.15.104:6807/3649492645', 'heartbeat_front_addr': '172.21.15.104:6806/3649492645', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 45, 'last_clean_end': 238, 'up_from': 261, 'up_thru': 414, 'down_at': 239, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 2795996229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 2795996229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 2795996229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 2795996229}]}, 'public_addr': '172.21.15.104:6808/2795996229', 'cluster_addr': '172.21.15.104:6809/2795996229', 'heartbeat_back_addr': '172.21.15.104:6811/2795996229', 'heartbeat_front_addr': '172.21.15.104:6810/2795996229', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:09:27.199421+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:07.797276+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:19.932106+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:09:55.632813+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:18.847048+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:24.337182+0000', 'dead_epoch': 326}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:09:37.226663+0000', 'dead_epoch': 238}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:22.667125+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '5.9', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.c', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1f', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.2', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T19:10:47.806 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1d 2024-09-21T19:10:47.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd rm-pg-upmap-items 4.1d 2024-09-21T19:10:47.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:47.917 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:47.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:48.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:48.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:48.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:47 smithi081 ceph-mon[39192]: 3.13 scrub starts 2024-09-21T19:10:48.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:47 smithi081 ceph-mon[39192]: 3.13 scrub ok 2024-09-21T19:10:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:47 smithi081 ceph-mon[29384]: 3.13 scrub starts 2024-09-21T19:10:48.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:47 smithi081 ceph-mon[29384]: 3.13 scrub ok 2024-09-21T19:10:48.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.194 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:48.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:48.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:47 smithi104 ceph-mon[32438]: 3.13 scrub starts 2024-09-21T19:10:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:47 smithi104 ceph-mon[32438]: 3.13 scrub ok 2024-09-21T19:10:48.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:48.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:10:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.342 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:48.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:48.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.508 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:48.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:48.609 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:48.636 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:48.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:48.739 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 41 16 70550 70534 107.501 74.4375 0.0102765 0.00929656 2024-09-21T19:10:48.789 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:48.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:48.904 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:48.930 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:48.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:10:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:10:48.950 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:10:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:48.951 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:48.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:49.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:49.078 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:48 smithi081 ceph-mon[29384]: pgmap v876: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:49.078 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:48 smithi081 ceph-mon[39192]: pgmap v876: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:49.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:49.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.221 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:49.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:49.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:48 smithi104 ceph-mon[32438]: pgmap v876: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:49.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:49.355 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:49.459 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:49.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:49.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:49.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:49.750 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:49.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 42 16 71776 71760 106.726 76.625 0.00381616 0.00936329 2024-09-21T19:10:49.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:49.880 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:49.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:49.962 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:49.963 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:49 smithi081 ceph-mon[39192]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T19:10:49.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:49.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:49 smithi081 ceph-mon[29384]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T19:10:49.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.021 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:50.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:50.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:50.149 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:50.236 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235882 8.21 8:87f52933:::benchmark_data_smithi081_105538_object103650:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 536.55506913900001, 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0591238139999999, 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.237 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235882 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676577+0000", 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676578+0000", 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4799999999999997e-07 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676581+0000", 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0920000000000002e-06 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676588+0000", 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3090000000000001e-06 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676594+0000", 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1589999999999998e-06 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.795011+0000", 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.118417017 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.795019+0000", 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0039999999999999e-06 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.735295+0000", 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.94027655099999996 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.735701+0000", 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040593399999999999 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.241 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235888 8.a 8:559d0eda:::benchmark_data_smithi081_105538_object103656:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 536.55503565000004, 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.075440073, 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:10:50.242 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235888 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676611+0000", 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676614+0000", 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3970000000000001e-06 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676619+0000", 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2859999999999999e-06 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676624+0000", 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.04e-06 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679419+0000", 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027943460000000001 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679424+0000", 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.215e-06 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876061+0000", 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19663730400000001 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876071+0000", 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6500000000000008e-06 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.751965+0000", 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87589449500000005 2024-09-21T19:10:50.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752051+0000", 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5783000000000002e-05 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:235893 8.a 8:56f22e24:::benchmark_data_smithi081_105538_object103661:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 536.55501483499995, 2024-09-21T19:10:50.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1078101929999999, 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 235893 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676632+0000", 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2e-07 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676634+0000", 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9439999999999999e-06 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676640+0000", 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6949999999999997e-06 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.676644+0000", 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6739999999999996e-06 2024-09-21T19:10:50.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:50.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679477+0000", 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028324510000000002 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting_for_map not empty", 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.679482+0000", 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7180000000000004e-06 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876097+0000", 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19661515199999999 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876105+0000", 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.524e-06 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752411+0000", 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.87630529599999996 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for missing object", 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.752544+0000", 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013372500000000001 2024-09-21T19:10:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784273+0000", 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031728734000000001 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:54.784442+0000", 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016886000000000001 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237696 8.1a 8:5de8ae1c:::benchmark_data_smithi081_105538_object105464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 533.53878128099996, 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2294143280000001, 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237696 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692865+0000", 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692868+0000", 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3070000000000001e-06 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692875+0000", 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9779999999999999e-06 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692894+0000", 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9734000000000001e-05 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692906+0000", 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1218e-05 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693046+0000", 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140288 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693056+0000", 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0366999999999999e-05 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.918656+0000", 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2255999339999999 2024-09-21T19:10:50.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.918766+0000", 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000109454 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.919037+0000", 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027156700000000001 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921749+0000", 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027122779999999998 2024-09-21T19:10:50.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921789+0000", 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9567000000000003e-05 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.921942+0000", 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000152612 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922003+0000", 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1637999999999995e-05 2024-09-21T19:10:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922094+0000", 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0926000000000003e-05 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922144+0000", 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0142000000000002e-05 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922280+0000", 2024-09-21T19:10:50.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013531799999999999 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237743 8.1a 8:5f844437:::benchmark_data_smithi081_105538_object105511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 533.538717217, 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2389039639999999, 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.262 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237743 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692929+0000", 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692930+0000", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4099999999999998e-07 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692933+0000", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.371e-06 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692940+0000", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4679999999999997e-06 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692945+0000", 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5600000000000001e-06 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693080+0000", 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000134232 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693085+0000", 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.57e-06 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922497+0000", 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.229411636 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922649+0000", 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015161100000000001 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.922904+0000", 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025589599999999998 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931394+0000", 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084900319999999998 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931456+0000", 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1847000000000006e-05 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931520+0000", 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3540999999999994e-05 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931559+0000", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9418000000000001e-05 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931652+0000", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3154999999999998e-05 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931710+0000", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7414999999999998e-05 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931833+0000", 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123471 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237758 8.1a 8:5b1e9ef3:::benchmark_data_smithi081_105538_object105526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 533.53869408699995, 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2525915910000001, 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:10:50.269 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237758 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692952+0000", 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692953+0000", 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7999999999999996e-07 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692955+0000", 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3410000000000001e-06 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692961+0000", 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2209999999999996e-06 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.692965+0000", 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7990000000000001e-06 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693105+0000", 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013992799999999999 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.693111+0000", 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924108+0000", 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.230996792 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924214+0000", 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106278 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,5,7", 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.924401+0000", 2024-09-21T19:10:50.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018696000000000001 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931404+0000", 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0070028060000000003 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931875+0000", 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00047151400000000003 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.931917+0000", 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1340000000000001e-05 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.932034+0000", 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011719500000000001 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945070+0000", 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013035573 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945131+0000", 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1702e-05 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.945544+0000", 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041286200000000001 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220084 9.0 9:0e4fd42a:::benchmark_data_smithi081_131274_object92229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.793318+0000", 2024-09-21T19:10:50.276 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.43832837299999, 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.122923267, 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220084 2024-09-21T19:10:50.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:10:50.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:49 smithi104 ceph-mon[32438]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T19:10:50.278 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793318+0000", 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793318+0000", 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793320+0000", 2024-09-21T19:10:50.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9300000000000002e-06 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793327+0000", 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5740000000000001e-06 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793341+0000", 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.398e-05 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793349+0000", 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.225e-06 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793432+0000", 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2929000000000005e-05 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793440+0000", 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5669999999999995e-06 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913658+0000", 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1202174709999999 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913729+0000", 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1544000000000005e-05 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913801+0000", 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1413000000000002e-05 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916062+0000", 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022611839999999999 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916133+0000", 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0709000000000004e-05 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916149+0000", 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6439e-05 2024-09-21T19:10:50.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916194+0000", 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4857000000000001e-05 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916241+0000", 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7444999999999997e-05 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220086 9.6 9:64ade3dc:::benchmark_data_smithi081_131274_object92231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.795252+0000", 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.436394248, 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.12467045, 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220086 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795252+0000", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795252+0000", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795254+0000", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.734e-06 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795259+0000", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2950000000000001e-06 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795272+0000", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2602000000000001e-05 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795279+0000", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1219999999999997e-06 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795338+0000", 2024-09-21T19:10:50.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8900000000000002e-05 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.795347+0000", 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8899999999999996e-06 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913320+0000", 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1179732060000001 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913417+0000", 2024-09-21T19:10:50.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7256000000000005e-05 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.913550+0000", 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013275799999999999 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.915789+0000", 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002238982 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919819+0000", 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040299159999999997 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919867+0000", 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7783e-05 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919897+0000", 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0576e-05 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919923+0000", 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5429999999999999e-05 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220106 9.0 9:05263109:::benchmark_data_smithi081_131274_object92251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.814872+0000", 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.41677450600002, 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1053909749999999, 2024-09-21T19:10:50.291 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220106 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814872+0000", 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814872+0000", 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814874+0000", 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0949999999999998e-06 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814880+0000", 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4210000000000003e-06 2024-09-21T19:10:50.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814896+0000", 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6657999999999999e-05 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814904+0000", 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6850000000000001e-06 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814964+0000", 2024-09-21T19:10:50.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9774000000000001e-05 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.814972+0000", 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0860000000000002e-06 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916287+0000", 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.101315327 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916386+0000", 2024-09-21T19:10:50.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9087999999999997e-05 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916506+0000", 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120146 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920079+0000", 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003572464 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920168+0000", 2024-09-21T19:10:50.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9536999999999993e-05 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920183+0000", 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4664999999999999e-05 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920224+0000", 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1270999999999998e-05 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920263+0000", 2024-09-21T19:10:50.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8757999999999997e-05 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220112 9.0 9:03ea8174:::benchmark_data_smithi081_131274_object92257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.816780+0000", 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.41486670400002, 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1074493400000001, 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.298 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220112 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816780+0000", 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816780+0000", 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816781+0000", 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4750000000000001e-06 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816786+0000", 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0239999999999997e-06 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816799+0000", 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2174000000000001e-05 2024-09-21T19:10:50.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816806+0000", 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7260000000000002e-06 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816860+0000", 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4076000000000002e-05 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.816866+0000", 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2240000000000003e-06 2024-09-21T19:10:50.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918333+0000", 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.101467937 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918437+0000", 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103133 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918573+0000", 2024-09-21T19:10:50.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013647799999999999 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920339+0000", 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001766297 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922149+0000", 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018096519999999999 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924130+0000", 2024-09-21T19:10:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019811120000000002 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924185+0000", 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4701999999999999e-05 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924229+0000", 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4329999999999997e-05 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220117 9.6 9:6f3a98df:::benchmark_data_smithi081_131274_object92262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.822735+0000", 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.40891130300002, 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0975038029999999, 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220117 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822735+0000", 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822735+0000", 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822737+0000", 2024-09-21T19:10:50.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.871e-06 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822743+0000", 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9680000000000003e-06 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822756+0000", 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2984e-05 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822763+0000", 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9199999999999998e-06 2024-09-21T19:10:50.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822822+0000", 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9385000000000001e-05 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.822830+0000", 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5909999999999998e-06 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.915826+0000", 2024-09-21T19:10:50.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.092996222 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.915926+0000", 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9735999999999995e-05 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.916035+0000", 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000108657 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.919964+0000", 2024-09-21T19:10:50.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039297070000000002 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920037+0000", 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2786000000000004e-05 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920150+0000", 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112744 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920201+0000", 2024-09-21T19:10:50.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0902000000000002e-05 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920239+0000", 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8330000000000001e-05 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220122 9.0 9:0c6321ee:::benchmark_data_smithi081_131274_object92267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.825986+0000", 2024-09-21T19:10:50.311 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.40566035799998, 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.100532968, 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220122 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825986+0000", 2024-09-21T19:10:50.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825986+0000", 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825988+0000", 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7069999999999999e-06 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.825993+0000", 2024-09-21T19:10:50.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4500000000000003e-06 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826006+0000", 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2561999999999999e-05 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826013+0000", 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0600000000000002e-06 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826061+0000", 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8186999999999998e-05 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.826067+0000", 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.434e-06 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920392+0000", 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.094324946 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920507+0000", 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114973 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920619+0000", 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112753 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924297+0000", 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036775319999999998 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926375+0000", 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020784319999999998 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926420+0000", 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4790999999999999e-05 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926476+0000", 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5685e-05 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926519+0000", 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3455999999999999e-05 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.318 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220136 9.6 9:62065d13:::benchmark_data_smithi081_131274_object92281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.319 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.837800+0000", 2024-09-21T19:10:50.319 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.39384696799999, 2024-09-21T19:10:50.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.084514252, 2024-09-21T19:10:50.319 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220136 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837800+0000", 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837800+0000", 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837801+0000", 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5319999999999999e-06 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837806+0000", 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7949999999999998e-06 2024-09-21T19:10:50.321 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837817+0000", 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1245e-05 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.837824+0000", 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7499999999999997e-06 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.838416+0000", 2024-09-21T19:10:50.322 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000591809 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.838425+0000", 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9409999999999996e-06 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918115+0000", 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.079690332 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.323 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918208+0000", 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2885000000000003e-05 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.918278+0000", 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0358999999999995e-05 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922124+0000", 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038462309999999999 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.324 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922174+0000", 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9570999999999997e-05 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922234+0000", 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9478999999999998e-05 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922273+0000", 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9539999999999998e-05 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.325 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922314+0000", 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0782999999999999e-05 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220141 9.6 9:67bf545e:::benchmark_data_smithi081_131274_object92286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.841044+0000", 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.39060298200002, 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0901363319999999, 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.326 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220141 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841044+0000", 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.327 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841044+0000", 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841045+0000", 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.607e-06 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841050+0000", 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.792e-06 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.328 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841060+0000", 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0179e-05 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841068+0000", 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7190000000000001e-06 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841120+0000", 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1857999999999997e-05 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.329 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.841128+0000", 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0949999999999996e-06 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920269+0000", 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0791413379999999 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920385+0000", 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011542799999999999 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.330 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.920500+0000", 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115887 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924337+0000", 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003836665 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931030+0000", 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066925429999999996 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.331 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931084+0000", 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3983000000000001e-05 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931141+0000", 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7275999999999998e-05 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.332 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931180+0000", 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8961999999999997e-05 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220162 9.6 9:6dc944fb:::benchmark_data_smithi081_131274_object92307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.858423+0000", 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.37322356800001, 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0732035870000001, 2024-09-21T19:10:50.333 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220162 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858423+0000", 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.334 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858423+0000", 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858425+0000", 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0219999999999999e-06 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858431+0000", 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.575e-06 2024-09-21T19:10:50.335 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858442+0000", 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1494999999999999e-05 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858449+0000", 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4089999999999997e-06 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858503+0000", 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3995000000000001e-05 2024-09-21T19:10:50.336 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.858509+0000", 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3750000000000002e-06 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922344+0000", 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0638351479999999 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922455+0000", 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111475 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.337 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.922580+0000", 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012425500000000001 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.930959+0000", 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0083796870000000002 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931314+0000", 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035444600000000001 2024-09-21T19:10:50.338 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931328+0000", 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4002000000000001e-05 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931589+0000", 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026142500000000003 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931627+0000", 2024-09-21T19:10:50.339 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7277999999999998e-05 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220172 9.6 9:63b6decf:::benchmark_data_smithi081_131274_object92317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.864693+0000", 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.36695324499999, 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0673737990000001, 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.340 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220172 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864693+0000", 2024-09-21T19:10:50.341 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.342 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864693+0000", 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864695+0000", 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.257e-06 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864699+0000", 2024-09-21T19:10:50.343 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4610000000000001e-06 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864706+0000", 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.374e-06 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864716+0000", 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9059999999999997e-06 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.344 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864769+0000", 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3052e-05 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.864775+0000", 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7189999999999998e-06 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924367+0000", 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0595922900000001 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.345 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924469+0000", 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010186100000000001 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.924573+0000", 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010414800000000001 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931014+0000", 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064408640000000001 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931637+0000", 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062255100000000005 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931969+0000", 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00033202399999999999 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932028+0000", 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8981000000000003e-05 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.349 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:50.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:50.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932067+0000", 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9310999999999997e-05 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220174 9.6 9:6fbae686:::benchmark_data_smithi081_131274_object92319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.870009+0000", 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.36163805799998, 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.065893177, 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.350 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220174 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870009+0000", 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.351 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870009+0000", 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870010+0000", 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.232e-06 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870014+0000", 2024-09-21T19:10:50.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1300000000000003e-06 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870022+0000", 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5169999999999999e-06 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870028+0000", 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1150000000000002e-06 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870081+0000", 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3016000000000002e-05 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.870086+0000", 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4049999999999999e-06 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926686+0000", 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.056600395 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926781+0000", 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5039999999999998e-05 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.926889+0000", 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010791899999999999 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931641+0000", 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047520599999999998 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.931778+0000", 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013652599999999999 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935798+0000", 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040200610000000001 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935853+0000", 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4672000000000001e-05 2024-09-21T19:10:50.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935902+0000", 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9088999999999997e-05 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220193 9.6 9:68b62631:::benchmark_data_smithi081_131274_object92338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.883802+0000", 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.34784498300002, 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.052301138, 2024-09-21T19:10:50.357 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220193 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883802+0000", 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.358 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883802+0000", 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883803+0000", 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.556e-06 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883808+0000", 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.746e-06 2024-09-21T19:10:50.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883820+0000", 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2238999999999999e-05 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883828+0000", 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.413e-06 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.360 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883882+0000", 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4839000000000001e-05 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.883888+0000", 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6110000000000001e-06 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.928928+0000", 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.045039823 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.361 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.929033+0000", 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010493300000000001 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.929098+0000", 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5024999999999995e-05 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932076+0000", 2024-09-21T19:10:50.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029780090000000002 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.932105+0000", 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.932e-05 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.935964+0000", 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038585859999999998 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.936061+0000", 2024-09-21T19:10:50.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7151000000000005e-05 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.936103+0000", 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1887000000000001e-05 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:220203 9.6 9:6d933243:::benchmark_data_smithi081_131274_object92348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:26.894230+0000", 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 323.33741644700001, 2024-09-21T19:10:50.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.048930103, 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 220203 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894230+0000", 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894230+0000", 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894231+0000", 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.158e-06 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894235+0000", 2024-09-21T19:10:50.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1540000000000004e-06 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894245+0000", 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7559999999999994e-06 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894251+0000", 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9039999999999997e-06 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894295+0000", 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4288e-05 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.894302+0000", 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6420000000000001e-06 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.933209+0000", 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0389072500000001 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.368 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.933303+0000", 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3661999999999999e-05 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.933574+0000", 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027142600000000001 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.939924+0000", 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063491440000000001 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943005+0000", 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030819480000000002 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943077+0000", 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1402999999999994e-05 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943122+0000", 2024-09-21T19:10:50.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4982999999999999e-05 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:27.943160+0000", 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8384999999999999e-05 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:243954 9.0 9:05c846c6:::benchmark_data_smithi081_131274_object116099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.906685+0000", 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 302.32496134000002, 2024-09-21T19:10:50.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.046879836, 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243954 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906685+0000", 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906685+0000", 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906686+0000", 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8999999999999996e-07 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906688+0000", 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5679999999999998e-06 2024-09-21T19:10:50.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906695+0000", 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5409999999999997e-06 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906700+0000", 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5929999999999999e-06 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906752+0000", 2024-09-21T19:10:50.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2562999999999998e-05 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906867+0000", 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011485900000000001 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,6", 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.906974+0000", 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106817 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.910354+0000", 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003380699 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.911367+0000", 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001013 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.913393+0000", 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020258680000000001 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.921236+0000", 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078429820000000001 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.921249+0000", 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3012e-05 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.953475+0000", 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0322257130000001 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:10:50.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.953565+0000", 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0031000000000006e-05 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:10:50.378 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:10:50.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:50.477 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:50.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.615 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:50.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:50.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:50.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 43 16 72908 72892 105.889 70.75 0.0686622 0.00943477 2024-09-21T19:10:50.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:50.867 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:50.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:50.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:50.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:10:50.949 INFO:teuthology.orchestra.run.smithi081.stderr:clear 4.1d pg_upmap_items mapping 2024-09-21T19:10:50.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:51.007 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:51.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.135 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:51.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:51.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:50 smithi104 ceph-mon[32438]: pgmap v878: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:50 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:10:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:50 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:10:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3331533874' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} : dispatch 2024-09-21T19:10:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:50 smithi104 ceph-mon[32438]: from='client.36563 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} : dispatch 2024-09-21T19:10:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1851404005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[39192]: pgmap v878: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3331533874' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} : dispatch 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[39192]: from='client.36563 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} : dispatch 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1851404005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[29384]: pgmap v878: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3331533874' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} : dispatch 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[29384]: from='client.36563 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"} : dispatch 2024-09-21T19:10:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1851404005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:10:51.365 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:51.399 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:51.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:51.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:51.649 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:10:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:10:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:10:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:10:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:10:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:10:51.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:51.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 44 16 74106 74090 105.185 74.875 0.0177599 0.00950181 2024-09-21T19:10:51.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:51.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:51.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:51.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:51.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:52.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:52.081 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:52.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:52.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:52.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:51 smithi104 ceph-mon[32438]: from='client.36563 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]': finished 2024-09-21T19:10:52.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:51 smithi104 ceph-mon[32438]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T19:10:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:51 smithi081 ceph-mon[29384]: from='client.36563 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]': finished 2024-09-21T19:10:52.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:51 smithi081 ceph-mon[29384]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T19:10:52.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:51 smithi081 ceph-mon[39192]: from='client.36563 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1d"}]': finished 2024-09-21T19:10:52.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:51 smithi081 ceph-mon[39192]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T19:10:52.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:52.344 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:52.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:52.471 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:52.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:52.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:52.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:52.744 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:52.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 45 16 75268 75252 104.461 72.625 0.0177895 0.0095677 2024-09-21T19:10:52.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:52.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:52.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.023 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:53.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:53.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:53.158 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: pgmap v880: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 743 B/s rd, 58 MiB/s wr, 930 op/s 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: 6.2 scrub starts 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: 6.2 scrub ok 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T19:10:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:53.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:53.293 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: pgmap v880: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 743 B/s rd, 58 MiB/s wr, 930 op/s 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: 6.2 scrub starts 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: 6.2 scrub ok 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: pgmap v880: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 743 B/s rd, 58 MiB/s wr, 930 op/s 2024-09-21T19:10:53.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: 6.2 scrub starts 2024-09-21T19:10:53.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: 6.2 scrub ok 2024-09-21T19:10:53.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T19:10:53.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:10:53.395 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:53.427 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:53.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:53.565 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:53.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:53.692 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:53.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 46 16 76500 76484 103.864 77 0.00811872 0.00962313 2024-09-21T19:10:53.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:53.822 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:53.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:53.954 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:53.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:54.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.095 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:54.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:54.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:54.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:53 smithi104 ceph-mon[32438]: 4.10 scrub starts 2024-09-21T19:10:54.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:53 smithi104 ceph-mon[32438]: 4.10 scrub ok 2024-09-21T19:10:54.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:53 smithi081 ceph-mon[29384]: 4.10 scrub starts 2024-09-21T19:10:54.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:53 smithi081 ceph-mon[29384]: 4.10 scrub ok 2024-09-21T19:10:54.262 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:53 smithi081 ceph-mon[39192]: 4.10 scrub starts 2024-09-21T19:10:54.262 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:53 smithi081 ceph-mon[39192]: 4.10 scrub ok 2024-09-21T19:10:54.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:54.360 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:54.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:54.497 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:54.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:54.635 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:54.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:54.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 47 16 77715 77699 103.27 75.9375 0.0178559 0.00967853 2024-09-21T19:10:54.773 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:54.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:54.910 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:54.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:55.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:55.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:55.170 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:55.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:54 smithi104 ceph-mon[32438]: pgmap v882: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T19:10:55.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:55.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:54 smithi081 ceph-mon[29384]: pgmap v882: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T19:10:55.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:55.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:54 smithi081 ceph-mon[39192]: pgmap v882: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T19:10:55.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:55.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:55.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:55.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:55.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:55.717 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:55.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 48 16 78924 78908 102.693 75.5625 0.0102055 0.00973344 2024-09-21T19:10:55.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:55.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:55.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:55.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:55.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:56.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:56.126 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:56.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:55 smithi104 ceph-mon[32438]: 3.1e scrub starts 2024-09-21T19:10:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:55 smithi104 ceph-mon[32438]: 3.1e scrub ok 2024-09-21T19:10:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:55 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:10:56.262 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:55 smithi081 ceph-mon[39192]: 3.1e scrub starts 2024-09-21T19:10:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:55 smithi081 ceph-mon[39192]: 3.1e scrub ok 2024-09-21T19:10:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:55 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:10:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:55 smithi081 ceph-mon[29384]: 3.1e scrub starts 2024-09-21T19:10:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:55 smithi081 ceph-mon[29384]: 3.1e scrub ok 2024-09-21T19:10:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:55 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:10:56.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.395 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:56.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:56.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:56.522 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:56.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:56.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:56.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 49 16 80060 80044 102.046 71 0.00471249 0.00979422 2024-09-21T19:10:56.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:56.799 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:56.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:56.928 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:56.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:57.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:57.053 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:57.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:10:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.187 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:57.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:10:57.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:56 smithi081 ceph-mon[29384]: pgmap v883: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 850 B/s rd, 62 MiB/s wr, 990 op/s; 36 B/s, 0 objects/s recovering 2024-09-21T19:10:57.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:56 smithi081 ceph-mon[29384]: 3.b scrub starts 2024-09-21T19:10:57.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:56 smithi081 ceph-mon[29384]: 3.b scrub ok 2024-09-21T19:10:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:56 smithi081 ceph-mon[39192]: pgmap v883: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 850 B/s rd, 62 MiB/s wr, 990 op/s; 36 B/s, 0 objects/s recovering 2024-09-21T19:10:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:56 smithi081 ceph-mon[39192]: 3.b scrub starts 2024-09-21T19:10:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:56 smithi081 ceph-mon[39192]: 3.b scrub ok 2024-09-21T19:10:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:56 smithi104 ceph-mon[32438]: pgmap v883: 158 pgs: 158 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 850 B/s rd, 62 MiB/s wr, 990 op/s; 36 B/s, 0 objects/s recovering 2024-09-21T19:10:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:56 smithi104 ceph-mon[32438]: 3.b scrub starts 2024-09-21T19:10:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:56 smithi104 ceph-mon[32438]: 3.b scrub ok 2024-09-21T19:10:57.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.324 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:57.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:57.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:57.453 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:57.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:57.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:10:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:10:57.593 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:57.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:57.728 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:57.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 50 16 81193 81177 101.422 70.8125 0.00826663 0.00985471 2024-09-21T19:10:57.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:57.855 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:57.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:57.987 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:57.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:58.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:58.125 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:58.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:58.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: 3.7 scrub starts 2024-09-21T19:10:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: 3.7 scrub ok 2024-09-21T19:10:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:10:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:10:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T19:10:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:57 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T19:10:58.260 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: 3.7 scrub starts 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: 3.7 scrub ok 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: 3.7 scrub starts 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: 3.7 scrub ok 2024-09-21T19:10:58.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:10:58.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:10:58.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:10:58.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T19:10:58.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:57 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T19:10:58.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:58.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:58.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:10:58.540 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:10:58.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:58.680 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:58.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 51 16 82372 82356 100.878 73.6875 0.00722656 0.00990838 2024-09-21T19:10:58.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:10:58.818 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:10:58.920 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:10:58.957 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:58.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:10:59.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:10:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:10:59.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:59.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:59.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.225 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:59.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:59.326 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:10:59.357 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:10:59.459 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:59.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:59 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:10:59.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:59 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:10:59.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:10:59 smithi081 ceph-mon[29384]: pgmap v884: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 27 B/s, 0 objects/s recovering 2024-09-21T19:10:59.482 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:59 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:10:59.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:59 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:10:59.483 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:10:59 smithi081 ceph-mon[39192]: pgmap v884: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 27 B/s, 0 objects/s recovering 2024-09-21T19:10:59.486 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:10:59.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:59 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:10:59.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:59 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:10:59.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:10:59 smithi104 ceph-mon[32438]: pgmap v884: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 27 B/s, 0 objects/s recovering 2024-09-21T19:10:59.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:10:59.626 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:10:59.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:10:59.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 52 16 83640 83624 100.462 79.25 0.0145631 0.00994923 2024-09-21T19:10:59.766 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:10:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:10:59.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:10:59.899 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:10:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:10:59.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:00.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:00.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:00.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:00.181 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:00.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:00.315 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:00.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: 6.8 scrub starts 2024-09-21T19:11:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: 6.8 scrub ok 2024-09-21T19:11:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T19:11:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T19:11:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T19:11:00.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T19:11:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:00.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:00.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: 6.8 scrub starts 2024-09-21T19:11:00.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: 6.8 scrub ok 2024-09-21T19:11:00.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T19:11:00.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T19:11:00.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T19:11:00.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:00 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T19:11:00.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:00.443 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:00.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: 6.8 scrub starts 2024-09-21T19:11:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: 6.8 scrub ok 2024-09-21T19:11:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T19:11:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T19:11:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T19:11:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:00 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T19:11:00.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:00.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:00.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:11:00.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:00.713 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:00.736 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:00.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 53 16 84874 84858 100.019 77.125 0.0104918 0.00999315 2024-09-21T19:11:00.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:00.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:00.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:00.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.028 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:01.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:01.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:01.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:01.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:01.320 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:01.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:01.487 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.488 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:01.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:01.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: pgmap v885: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 24 B/s, 0 objects/s recovering 2024-09-21T19:11:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: 5.9 scrub starts 2024-09-21T19:11:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: 5.9 scrub ok 2024-09-21T19:11:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: 6.1 scrub starts 2024-09-21T19:11:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: 6.1 scrub ok 2024-09-21T19:11:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T19:11:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:01 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: pgmap v885: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 24 B/s, 0 objects/s recovering 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: 5.9 scrub starts 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: 5.9 scrub ok 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: 6.1 scrub starts 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: 6.1 scrub ok 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T19:11:01.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: pgmap v885: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 24 B/s, 0 objects/s recovering 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: 5.9 scrub starts 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: 5.9 scrub ok 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: 6.1 scrub starts 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: 6.1 scrub ok 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T19:11:01.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:01 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T19:11:01.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:01.619 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:01.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:01.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 54 16 85952 85936 99.4147 67.375 0.013206 0.0100535 2024-09-21T19:11:01.760 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:01.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:01.897 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:01.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:01.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:02.040 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:02.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:11:02.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:02.142 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:11:02.166 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:02.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:02.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:02 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:11:02.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:02 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:11:02.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1736952753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:02.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:02 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:11:02.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:02 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:11:02.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1736952753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:02.298 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:02.399 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:02.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:02.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:02 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:11:02.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:02 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:11:02.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1736952753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:02.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:02.568 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:02.663 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:02.663 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:02.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:02.708 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:02.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 55 16 87101 87085 98.913 71.8125 0.00794612 0.0101051 2024-09-21T19:11:02.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:02.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:02.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:02.973 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:02.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:03.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:03.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:03.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: pgmap v886: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 21 B/s, 0 objects/s recovering 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1736952753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: 3.17 scrub starts 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: 3.17 scrub ok 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T19:11:03.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: pgmap v886: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 21 B/s, 0 objects/s recovering 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1736952753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: 3.17 scrub starts 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: 3.17 scrub ok 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T19:11:03.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:03 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T19:11:03.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:11:03.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:03.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:03.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:03.502 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:03.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: pgmap v886: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 21 B/s, 0 objects/s recovering 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1736952753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: 3.17 scrub starts 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: 3.17 scrub ok 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T19:11:03.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:03 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T19:11:03.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:03.629 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:03.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:03.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 56 16 88302 88286 98.487 75.0625 0.0147406 0.0101485 2024-09-21T19:11:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.769 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:03.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:03.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:03.881 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:03.941 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:03.945 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:03.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:04.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:04.091 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:04.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:04.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[39192]: 3.16 scrub starts 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[39192]: 3.16 scrub ok 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[39192]: 6.18 scrub starts 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[39192]: 6.18 scrub ok 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[29384]: 3.16 scrub starts 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[29384]: 3.16 scrub ok 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[29384]: 6.18 scrub starts 2024-09-21T19:11:04.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:04 smithi081 ceph-mon[29384]: 6.18 scrub ok 2024-09-21T19:11:04.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:04.481 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:04.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:04 smithi104 ceph-mon[32438]: 3.16 scrub starts 2024-09-21T19:11:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:04 smithi104 ceph-mon[32438]: 3.16 scrub ok 2024-09-21T19:11:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:04 smithi104 ceph-mon[32438]: 6.18 scrub starts 2024-09-21T19:11:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:04 smithi104 ceph-mon[32438]: 6.18 scrub ok 2024-09-21T19:11:04.511 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:04.535 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:11:04.535 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:11:04.535 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T19:11:04.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T19:11:04.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:04.748 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:04.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 57 16 89402 89386 97.9654 68.75 0.0134086 0.0102028 2024-09-21T19:11:04.775 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:04.872 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:04.873 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:04.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:04.902 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:04.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:04.911 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:04.911 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:04.956 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:05.000 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:05.001 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:05.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:05.040 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:11:05.040 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:11:05.040 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:11:05.040 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:11:05.040 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:11:05.043 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:05.043 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:05.061 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:05.085 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:05.085 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:05.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:05.118 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:05.153 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:05.153 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:05.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:05.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:05.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:05.350 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:05.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:05.480 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:05 smithi104 ceph-mon[32438]: pgmap v888: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-21T19:11:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:05 smithi104 ceph-mon[32438]: 3.6 scrub starts 2024-09-21T19:11:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:05 smithi104 ceph-mon[32438]: 3.6 scrub ok 2024-09-21T19:11:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:05 smithi104 ceph-mon[32438]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T19:11:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1682336853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:05.523 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[29384]: pgmap v888: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[29384]: 3.6 scrub starts 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[29384]: 3.6 scrub ok 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[29384]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1682336853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[39192]: pgmap v888: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[39192]: 3.6 scrub starts 2024-09-21T19:11:05.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[39192]: 3.6 scrub ok 2024-09-21T19:11:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[39192]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T19:11:05.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1682336853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:05.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.620 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:05.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:05.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:05.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 58 16 90501 90485 97.4607 68.6875 0.00820266 0.0102556 2024-09-21T19:11:05.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:11:05.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:05.881 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:05.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:05.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:06.008 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:06.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:06.135 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:06.239 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.267 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:06.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:06.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:06.406 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:06 smithi104 ceph-mon[32438]: 5.1 scrub starts 2024-09-21T19:11:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:06 smithi104 ceph-mon[32438]: 5.1 scrub ok 2024-09-21T19:11:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:06 smithi104 ceph-mon[32438]: 5.5 scrub starts 2024-09-21T19:11:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:06 smithi104 ceph-mon[32438]: 5.5 scrub ok 2024-09-21T19:11:06.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:06.541 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[29384]: 5.1 scrub starts 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[29384]: 5.1 scrub ok 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[29384]: 5.5 scrub starts 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[29384]: 5.5 scrub ok 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[39192]: 5.1 scrub starts 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[39192]: 5.1 scrub ok 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[39192]: 5.5 scrub starts 2024-09-21T19:11:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:06 smithi081 ceph-mon[39192]: 5.5 scrub ok 2024-09-21T19:11:06.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:06.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:11:06.668 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:06.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 59 16 91646 91630 97.0219 71.5625 0.0244956 0.0103013 2024-09-21T19:11:06.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.803 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:06.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:06.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:06.948 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:06.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:07.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:07.088 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:07.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:07.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:07.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:07.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:07.391 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:07.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:07.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: pgmap v890: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-21T19:11:07.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T19:11:07.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: pgmap v890: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T19:11:07.477 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:07.478 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:07.478 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:11:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:11:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:11:07.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: pgmap v890: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-21T19:11:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T19:11:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T19:11:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T19:11:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T19:11:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:07 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:07.553 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:07.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:07.655 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:07.699 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:07.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:07.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:11:07.756620+0000 min lat: 0.00122502 max lat: 0.458913 avg lat: 0.0103579 2024-09-21T19:11:07.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:11:07.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 60 16 92695 92679 96.4975 65.5625 0.0184061 0.0103579 2024-09-21T19:11:07.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:07.819 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:07.853 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:07.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:07.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:07.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:08.004 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:08.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:08.156 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 436, 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T19:11:07.167439+0000", 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T19:06:54.357382+0000", 2024-09-21T19:11:08.174 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T19:05:41.856710+0000", 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5799936, 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 20, 2024-09-21T19:11:08.175 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 11, 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T19:11:08.176 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.177 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 11, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 11, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 11, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 11, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 11, 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.178 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "170", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "168", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.179 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:11:08.180 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:11:08.180 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.180 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.180 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.180 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.180 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.182 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.183 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.1800000667572021, 2024-09-21T19:11:08.184 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.1800000667572021, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T19:11:08.185 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T19:11:08.186 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.187 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.188 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.189 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6500000953674316, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6500000953674316, 2024-09-21T19:11:08.190 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.191 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.192 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T19:11:08.193 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.194 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.195 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:11:08.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.198 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.200 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.201 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:11:08.202 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.25, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.25, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.25, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.25, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.204 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.205 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.206 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.207 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.208 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5, 2024-09-21T19:11:08.209 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.5, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.5, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T19:11:08.210 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.211 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.212 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:11:08.213 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.214 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.215 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:11:08.216 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:11:08.217 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 11, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_4", 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T19:09:59.705202+0000", 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:11:08.218 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 16, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 16, 2024-09-21T19:11:08.219 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 16, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 16, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 16, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "418", 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:11:08.220 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:11:08.221 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:11:08.222 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:11:08.223 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T19:11:08.224 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T19:11:08.225 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.226 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.227 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.228 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T19:11:08.228 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T19:11:08.228 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T19:11:08.228 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T19:11:08.228 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:11:08.229 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:11:08.231 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T19:11:08.232 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T19:11:08.235 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.236 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.237 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 432, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T19:11:08.238 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.239 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:11:08.240 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.242 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T19:11:08.243 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.244 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.245 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 0.16912841796875, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 37, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 326, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 348, 2024-09-21T19:11:08.246 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 348, 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 327, 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.247 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.248 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:11:08.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/3649492645", 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/3649492645", 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/3649492645", 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/3649492645", 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.250 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 45, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 238, 2024-09-21T19:11:08.251 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 261, 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 239, 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.252 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:11:08.253 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/2795996229", 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/2795996229", 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/2795996229", 2024-09-21T19:11:08.254 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/2795996229", 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:11:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 414, 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:11:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.257 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T19:11:08.258 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.259 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:50.384083+0000", 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:07.797276+0000", 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:19.932106+0000", 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:09:55.632813+0000", 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:11:08.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:18.847048+0000", 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.262 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:24.337182+0000", 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 326 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:06.571191+0000", 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 238 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.263 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:10:22.667125+0000", 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.6", 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.14", 2024-09-21T19:11:08.265 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.9", 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 7, 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 3 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.c", 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.1f", 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.267 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 3 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "6.2", 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 3, 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 0 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [], 2024-09-21T19:11:08.268 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:11:08.269 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T19:11:08.270 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.271 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:11:08.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.313 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:08.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: 11.7 scrub starts 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: 11.2 scrub starts 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T19:11:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/8961409' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: 11.7 scrub starts 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: 11.2 scrub starts 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T19:11:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/8961409' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:11:08.414 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:08.444 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:08.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: 11.7 scrub starts 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: 11.2 scrub starts 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T19:11:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/8961409' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:11:08.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:08.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:08.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.730 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:08.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:08.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 61 16 93612 93596 95.8553 57.3125 0.0401811 0.0104263 2024-09-21T19:11:08.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:08.875 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:08.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:08.931 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:11:08.931 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:11:08.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:11:08.931 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:11:08.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.931 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.35258.0:235880 8.e e235/172 8:7f5b8aa9:::benchmark_data_smithi081_105538_object103648:head v 235'14796, mlcod=235'14793)", 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:53.660995+0000", 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 555.25007211299999, 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.215059784, 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.660995+0000", 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.660995+0000", 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.660996+0000", 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2780000000000001e-06 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661004+0000", 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.537e-06 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661014+0000", 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8509999999999994e-06 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.661021+0000", 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1049999999999997e-06 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.673549+0000", 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012528169 2024-09-21T19:11:08.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.673572+0000", 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2654e-05 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:53.876055+0000", 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20248319000000001 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.35258.0:237698 8.1d 8:bb593120:::benchmark_data_smithi081_105538_object105466:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:01:56.696353+0000", 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 552.21471467900005, 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2170390849999999, 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.35258", 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 237698 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696353+0000", 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696353+0000", 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696354+0000", 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7489999999999999e-06 2024-09-21T19:11:08.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696359+0000", 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9300000000000002e-06 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696369+0000", 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5850000000000006e-06 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696378+0000", 2024-09-21T19:11:08.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8780000000000004e-06 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696533+0000", 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015554699999999999 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:56.696542+0000", 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5779999999999999e-06 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907858+0000", 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2113164430000001 2024-09-21T19:11:08.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.907991+0000", 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000132287 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,1,3,5,7", 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.908107+0000", 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011627400000000001 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.909917+0000", 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001810617 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.909964+0000", 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6118999999999997e-05 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.911547+0000", 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001583751 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912015+0000", 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000468159 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.912881+0000", 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000865985 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913148+0000", 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026665800000000002 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913220+0000", 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1758e-05 2024-09-21T19:11:08.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:01:57.913392+0000", 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000171767 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244006 9.6 9:6e3a6723:::benchmark_data_smithi081_131274_object116151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 320.96322289, 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0195242950000001, 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244006 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947847+0000", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3649999999999998e-06 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947854+0000", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5519999999999999e-06 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947872+0000", 2024-09-21T19:11:08.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999000000000002e-05 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947880+0000", 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4200000000000001e-06 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948077+0000", 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019761400000000001 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948118+0000", 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0926000000000001e-05 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959355+0000", 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.011236561 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959470+0000", 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115578 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959574+0000", 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010335700000000001 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962650+0000", 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030760499999999999 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962661+0000", 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1022000000000001e-05 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.948 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967081+0000", 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044202369999999996 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967163+0000", 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1880999999999998e-05 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967369+0000", 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020573299999999999 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.949 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244020 9.6 9:649892a9:::benchmark_data_smithi081_131274_object116165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 320.95470920600002, 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0117105959999999, 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:11:08.950 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244020 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956361+0000", 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2320000000000001e-06 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956369+0000", 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7789999999999996e-06 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956389+0000", 2024-09-21T19:11:08.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9944000000000001e-05 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956398+0000", 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1600000000000004e-06 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956553+0000", 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015503399999999999 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956577+0000", 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3506000000000002e-05 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964359+0000", 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0077825030000001 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964473+0000", 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113613 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964573+0000", 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9993000000000003e-05 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967381+0000", 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028085699999999998 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967707+0000", 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000325555 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967927+0000", 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021985700000000001 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967974+0000", 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.761e-05 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.968069+0000", 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4240000000000006e-05 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:31267 10.9 10:9783b47f:::benchmark_data_smithi081_155596_object31266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:26.607735+0000", 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 282.30333251399998, 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20489924400000001, 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.957 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 31267 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607735+0000", 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607735+0000", 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607737+0000", 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3290000000000001e-06 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607761+0000", 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4369999999999999e-05 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607773+0000", 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1241e-05 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607780+0000", 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7300000000000005e-06 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.607945+0000", 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016431700000000001 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.608072+0000", 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012773 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 7", 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.608196+0000", 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012397600000000001 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609756+0000", 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001559653 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812319+0000", 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20256344600000001 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812379+0000", 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9110000000000002e-05 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812634+0000", 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025534199999999998 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.962 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:31274 10.9 10:924e9fbb:::benchmark_data_smithi081_155596_object31273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:26.609958+0000", 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 282.301109215, 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.20294334, 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.963 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 31274 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609958+0000", 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609958+0000", 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609960+0000", 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.34e-06 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609983+0000", 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2238999999999998e-05 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.609994+0000", 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0981e-05 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610000+0000", 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.815e-06 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610407+0000", 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00040638000000000002 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610538+0000", 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013086900000000001 2024-09-21T19:11:08.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 7", 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.610658+0000", 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012067599999999999 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.810099+0000", 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.19944103199999999 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812689+0000", 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025899830000000001 2024-09-21T19:11:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812831+0000", 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014130699999999999 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:26.812901+0000", 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0717999999999996e-05 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.968 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33317 10.5 10:a1fd2c68:::benchmark_data_smithi081_155596_object33316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.080876+0000", 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.83019164900003, 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24413483999999999, 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33317 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080876+0000", 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080876+0000", 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080877+0000", 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5179999999999999e-06 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080901+0000", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4229000000000001e-05 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080915+0000", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4166e-05 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.080935+0000", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9604000000000001e-05 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081048+0000", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112958 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081171+0000", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123039 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081439+0000", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000268244 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084329+0000", 2024-09-21T19:11:08.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028894839999999999 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084352+0000", 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3459999999999999e-05 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324916+0000", 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24056408900000001 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324971+0000", 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4605999999999999e-05 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325010+0000", 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9443000000000002e-05 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33320 10.5 10:a8d5ea65:::benchmark_data_smithi081_155596_object33319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.081030+0000", 2024-09-21T19:11:08.974 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.83003723299998, 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.244095749, 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33320 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081030+0000", 2024-09-21T19:11:08.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081030+0000", 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081031+0000", 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3239999999999999e-06 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081054+0000", 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2931000000000001e-05 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081064+0000", 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9839999999999996e-06 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.081071+0000", 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7710000000000001e-06 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.082782+0000", 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017113300000000001 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.082905+0000", 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012306900000000001 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083057+0000", 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015153500000000001 2024-09-21T19:11:08.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084421+0000", 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001363813 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086857+0000", 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002436485 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325056+0000", 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23819896199999999 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325096+0000", 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9486000000000001e-05 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325126+0000", 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0059000000000001e-05 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.980 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33324 10.3 10:c24b9b15:::benchmark_data_smithi081_155596_object33323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.083022+0000", 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.82804547299997, 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.241109666, 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.981 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33324 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083022+0000", 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083022+0000", 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083023+0000", 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6109999999999999e-06 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083045+0000", 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1503000000000001e-05 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083055+0000", 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0385e-05 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083062+0000", 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7009999999999999e-06 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083114+0000", 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2135999999999997e-05 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083224+0000", 2024-09-21T19:11:08.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010973599999999999 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3", 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083351+0000", 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012762400000000001 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086269+0000", 2024-09-21T19:11:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029178239999999999 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.323973+0000", 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23770374 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324045+0000", 2024-09-21T19:11:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2119999999999994e-05 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324131+0000", 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6285999999999993e-05 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33326 10.3 10:c4df0ca1:::benchmark_data_smithi081_155596_object33325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.083993+0000", 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.82707427000003, 2024-09-21T19:11:08.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24031893300000001, 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33326 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083993+0000", 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083993+0000", 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.083994+0000", 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4980000000000001e-06 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084018+0000", 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3116999999999999e-05 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084030+0000", 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2183e-05 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084038+0000", 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7980000000000006e-06 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084782+0000", 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00074450700000000005 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084920+0000", 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000137849 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3", 2024-09-21T19:11:08.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085101+0000", 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00018122099999999999 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086382+0000", 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001280757 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324184+0000", 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.237802455 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324237+0000", 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2840000000000002e-05 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.324312+0000", 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4708000000000004e-05 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.994 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33329 10.5 10:a18a2d61:::benchmark_data_smithi081_155596_object33328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.084952+0000", 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.82611556500001, 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.24032500900000001, 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33329 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.995 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084952+0000", 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084952+0000", 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084953+0000", 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.296e-06 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084979+0000", 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6511000000000001e-05 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084991+0000", 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1089e-05 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.084998+0000", 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4120000000000004e-06 2024-09-21T19:11:08.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085089+0000", 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1058999999999997e-05 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085203+0000", 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114437 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:08.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.085332+0000", 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012876900000000001 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086575+0000", 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001242504 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.086939+0000", 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036457500000000001 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:08.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325201+0000", 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23826132699999999 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325246+0000", 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5204000000000003e-05 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325277+0000", 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0825999999999999e-05 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33342 10.5 10:a1946825:::benchmark_data_smithi081_155596_object33341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.092684+0000", 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.81838334899999, 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23319526600000001, 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.001 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33342 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092684+0000", 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092684+0000", 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092685+0000", 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1990000000000001e-06 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092704+0000", 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9352999999999999e-05 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092714+0000", 2024-09-21T19:11:09.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7180000000000008e-06 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092719+0000", 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.955e-06 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092761+0000", 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2271000000000002e-05 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092882+0000", 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012021 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.093025+0000", 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000143141 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094974+0000", 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019494709999999999 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094998+0000", 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3421000000000002e-05 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325787+0000", 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23078942699999999 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325841+0000", 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3797e-05 2024-09-21T19:11:09.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325879+0000", 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8303000000000002e-05 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:33344 10.5 10:af6f1865:::benchmark_data_smithi081_155596_object33343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:06:28.092888+0000", 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 280.81817933999997, 2024-09-21T19:11:09.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.233110239, 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 33344 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092888+0000", 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092888+0000", 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092889+0000", 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.04e-06 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092914+0000", 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5072999999999999e-05 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092923+0000", 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8769999999999999e-06 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.092931+0000", 2024-09-21T19:11:09.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.909e-06 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094112+0000", 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011809800000000001 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094190+0000", 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8523999999999996e-05 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:09.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.094270+0000", 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9764000000000004e-05 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.095095+0000", 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082455899999999999 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.095392+0000", 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029711899999999999 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325932+0000", 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.23053996800000001 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325965+0000", 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3707000000000001e-05 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:06:28.325998+0000", 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2719000000000003e-05 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189643 10.3 10:cbf6bec4:::benchmark_data_smithi081_155596_object72170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.622112661, 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.28335258899999999, 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189643 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288956+0000", 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.674e-06 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288961+0000", 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0039999999999999e-06 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288973+0000", 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2011000000000001e-05 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288980+0000", 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1729999999999998e-06 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289033+0000", 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2995999999999999e-05 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289155+0000", 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121433 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289279+0000", 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012419300000000001 2024-09-21T19:11:09.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.292626+0000", 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033469849999999998 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.295941+0000", 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033146389999999999 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572053+0000", 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27611242699999999 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572259+0000", 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020604900000000001 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572307+0000", 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8004999999999999e-05 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.020 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189647 10.5 10:a120596a:::benchmark_data_smithi081_155596_object72174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.617483298, 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29500822199999999, 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189647 2024-09-21T19:11:09.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293586+0000", 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8470000000000001e-06 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293592+0000", 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5810000000000002e-06 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293603+0000", 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0728e-05 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293610+0000", 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2250000000000003e-06 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293666+0000", 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5977999999999998e-05 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293807+0000", 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014065599999999999 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293923+0000", 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011639800000000001 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.296808+0000", 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028842630000000002 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.297628+0000", 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082086500000000005 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588498+0000", 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29086982900000002 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588546+0000", 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7862999999999999e-05 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588592+0000", 2024-09-21T19:11:09.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5989000000000003e-05 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189666 10.5 10:afed0067:::benchmark_data_smithi081_155596_object72193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.597516242, 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27515819499999999, 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.027 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189666 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:11:09.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313553+0000", 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0530000000000001e-06 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313559+0000", 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3489999999999997e-06 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313573+0000", 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.324e-05 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313581+0000", 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5320000000000006e-06 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313644+0000", 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3293000000000005e-05 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313760+0000", 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011513799999999999 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313872+0000", 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011291 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.316982+0000", 2024-09-21T19:11:09.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031098860000000001 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.317682+0000", 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000699711 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588631+0000", 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27094870199999999 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588676+0000", 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5027000000000001e-05 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588709+0000", 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3353999999999999e-05 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189670 10.3 10:c9bbc12c:::benchmark_data_smithi081_155596_object72197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318154+0000", 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.592912992, 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26539114400000002, 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189670 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318154+0000", 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318154+0000", 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318156+0000", 2024-09-21T19:11:09.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9599999999999999e-06 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318162+0000", 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8619999999999999e-06 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318177+0000", 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5451000000000002e-05 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318185+0000", 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2309999999999999e-06 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318245+0000", 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9849000000000002e-05 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318368+0000", 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123416 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318444+0000", 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6353000000000006e-05 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321387+0000", 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002943153 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321453+0000", 2024-09-21T19:11:09.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5414999999999996e-05 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583333+0000", 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.261880057 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583423+0000", 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0205999999999997e-05 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583545+0000", 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012219100000000001 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189672 10.5 10:a358f405:::benchmark_data_smithi081_155596_object72199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.592455028, 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27021128500000002, 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.040 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189672 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318613+0000", 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1829999999999999e-06 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318618+0000", 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3819999999999997e-06 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:11:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318630+0000", 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2054e-05 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318637+0000", 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0690000000000004e-06 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318711+0000", 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4373000000000001e-05 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318818+0000", 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010637999999999999 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318920+0000", 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010200300000000001 2024-09-21T19:11:09.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.324913+0000", 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005993651 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.324929+0000", 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5424999999999999e-05 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588745+0000", 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26381601999999998 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588790+0000", 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5621000000000002e-05 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588823+0000", 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3124000000000003e-05 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.047 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189673 10.5 10:a287fbc8:::benchmark_data_smithi081_155596_object72200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.592300497, 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27468967399999999, 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.048 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189673 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.049 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318768+0000", 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318773+0000", 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6940000000000003e-06 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318783+0000", 2024-09-21T19:11:09.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0121e-05 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318790+0000", 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1210000000000001e-06 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321882+0000", 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030923249999999999 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321979+0000", 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6842000000000006e-05 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.322109+0000", 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012934000000000001 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.325016+0000", 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002907251 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.325126+0000", 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110315 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593359+0000", 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26823260999999998 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593411+0000", 2024-09-21T19:11:09.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2719000000000001e-05 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593456+0000", 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4946e-05 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189677 10.3 10:c4e3b9d2:::benchmark_data_smithi081_155596_object72204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:09.054 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.331628+0000", 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 126.579438787, 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.25206215199999998, 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189677 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.055 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331628+0000", 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331628+0000", 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331630+0000", 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7850000000000001e-06 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331635+0000", 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7609999999999998e-06 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331645+0000", 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0312000000000001e-05 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331651+0000", 2024-09-21T19:11:09.057 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6119999999999998e-06 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331699+0000", 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7834000000000003e-05 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331778+0000", 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9080999999999997e-05 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:11:09.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.331852+0000", 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4070000000000001e-05 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.334467+0000", 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002614894 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.334579+0000", 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112322 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.059 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583607+0000", 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.249028095 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583656+0000", 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8356999999999998e-05 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.583691+0000", 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5029e-05 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.060 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.061 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:09.061 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:09.061 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:11:09.080 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:09.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:09.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:11:09.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:09.414 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:09.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:09.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T19:11:09.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: pgmap v892: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 3.1c scrub starts 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 3.1c scrub ok 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 6.0 scrub starts 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 6.0 scrub ok 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 3.10 scrub starts 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 3.10 scrub ok 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[29384]: 11.7 scrub ok 2024-09-21T19:11:09.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: pgmap v892: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 3.1c scrub starts 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 3.1c scrub ok 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 6.0 scrub starts 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 6.0 scrub ok 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 3.10 scrub starts 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 3.10 scrub ok 2024-09-21T19:11:09.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:09 smithi081 ceph-mon[39192]: 11.7 scrub ok 2024-09-21T19:11:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.576 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:09.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:09.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.710 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:09.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: pgmap v892: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 3.1c scrub starts 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 3.1c scrub ok 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 6.0 scrub starts 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 6.0 scrub ok 2024-09-21T19:11:09.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 3.10 scrub starts 2024-09-21T19:11:09.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 3.10 scrub ok 2024-09-21T19:11:09.758 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:09 smithi104 ceph-mon[32438]: 11.7 scrub ok 2024-09-21T19:11:09.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 62 15 94503 94488 95.2086 55.75 0.0191135 0.0104981 2024-09-21T19:11:09.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.841 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:09.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:09.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:09.979 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:09.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:10.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.114 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:10.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.248 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:10.349 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:10.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.501 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[29384]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[29384]: pgmap v894: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1426279793' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[29384]: from='client.36590 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:10.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[39192]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T19:11:10.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[39192]: pgmap v894: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-21T19:11:10.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1426279793' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:10.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:10 smithi081 ceph-mon[39192]: from='client.36590 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:10.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:10.625 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:10.727 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:10.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:10 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:10.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:10 smithi104 ceph-mon[32438]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T19:11:10.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:10 smithi104 ceph-mon[32438]: pgmap v894: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-21T19:11:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:10 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1426279793' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:10.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:10 smithi104 ceph-mon[32438]: from='client.36590 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:10.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 63 16 95364 95348 94.5507 53.75 0.00381465 0.0105449 2024-09-21T19:11:10.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:10.973 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:10.995 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:10.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:11.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.141 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:11.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:11:11.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:11.262 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:11.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:11.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:11.535 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: 6.12 scrub starts 2024-09-21T19:11:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: 6.12 scrub ok 2024-09-21T19:11:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: 11.2 scrub ok 2024-09-21T19:11:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T19:11:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T19:11:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: from='client.36590 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:11:11.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[29384]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T19:11:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: 6.12 scrub starts 2024-09-21T19:11:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: 6.12 scrub ok 2024-09-21T19:11:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: 11.2 scrub ok 2024-09-21T19:11:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T19:11:11.568 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T19:11:11.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:11.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: from='client.36590 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:11:11.569 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:11 smithi081 ceph-mon[39192]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T19:11:11.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:11.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: 6.12 scrub starts 2024-09-21T19:11:11.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: 6.12 scrub ok 2024-09-21T19:11:11.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: 11.2 scrub ok 2024-09-21T19:11:11.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T19:11:11.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T19:11:11.657 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:11.658 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: from='client.36590 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:11:11.658 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:11 smithi104 ceph-mon[32438]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T19:11:11.672 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:11.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 64 16 96136 96120 93.8274 48.25 0.00965756 0.0106521 2024-09-21T19:11:11.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:11.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:11.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:11.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:12.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:12.076 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:12.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:12.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:12.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:12.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.490 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:12.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:12.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:12 smithi081 ceph-mon[39192]: pgmap v896: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 960 op/s 2024-09-21T19:11:12.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:12 smithi081 ceph-mon[29384]: pgmap v896: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 960 op/s 2024-09-21T19:11:12.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:12.715 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:12.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:12.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:12 smithi104 ceph-mon[32438]: pgmap v896: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 960 op/s 2024-09-21T19:11:12.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 65 16 97248 97232 93.4532 69.5 0.0119955 0.0106958 2024-09-21T19:11:12.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:12.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:12.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:13.019 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:13.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:13.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:13.291 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:13.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.427 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:13.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:13.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.563 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:13.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:13.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:13.693 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:13.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 66 16 98320 98304 93.0524 67 0.0124349 0.0107421 2024-09-21T19:11:13.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.829 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:13.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:13.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:13.964 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:13.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:14.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:14.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:14.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:14.222 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:14.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.351 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:14.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:14.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:14.486 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:14.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:14.610 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:14.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:14.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:14.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 67 16 99307 99291 92.5843 61.6875 0.00265486 0.0107814 2024-09-21T19:11:14.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:14.857 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:14.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:14.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:15.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.131 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:15.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:15.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:14 smithi104 ceph-mon[32438]: pgmap v897: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 770 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-21T19:11:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:14 smithi104 ceph-mon[32438]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T19:11:15.269 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:14 smithi081 ceph-mon[29384]: pgmap v897: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 770 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-21T19:11:15.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:14 smithi081 ceph-mon[29384]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T19:11:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:14 smithi081 ceph-mon[39192]: pgmap v897: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 770 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-21T19:11:15.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:14 smithi081 ceph-mon[39192]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T19:11:15.372 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:15.401 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:15.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:15.523 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:15.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:15.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:15.656 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:15.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:15.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 68 15 100206 100191 92.0501 56.25 0.00818269 0.0108591 2024-09-21T19:11:15.796 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:15.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:15.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:15.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:16.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:16.068 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:16.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:16.203 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:16.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:16.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:16.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.450 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:16.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:16.551 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.576 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:16.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:16.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:16.703 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:16.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 69 16 101011 100995 91.4444 50.25 0.0167469 0.0109311 2024-09-21T19:11:16.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:16.841 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:16.943 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:16.968 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:16.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:16.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:11:17.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:17.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:16 smithi081 ceph-mon[39192]: pgmap v899: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 849 B/s rd, 48 MiB/s wr, 773 op/s 2024-09-21T19:11:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:16 smithi081 ceph-mon[29384]: pgmap v899: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 849 B/s rd, 48 MiB/s wr, 773 op/s 2024-09-21T19:11:17.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:16 smithi104 ceph-mon[32438]: pgmap v899: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 849 B/s rd, 48 MiB/s wr, 773 op/s 2024-09-21T19:11:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:17.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:17.392 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:17.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:17.533 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:17.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:11:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:11:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:11:17.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.704 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:17.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:17.715 INFO:tasks.thrashosds.thrasher:j is {'epoch': 436, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T19:11:07.167439+0000', 'last_up_change': '2024-09-21T19:06:54.357382+0000', 'last_in_change': '2024-09-21T19:05:41.856710+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '168', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T19:09:59.705202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '418', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.16912841796875, 'last_clean_begin': 37, 'last_clean_end': 326, 'up_from': 348, 'up_thru': 348, 'down_at': 327, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 3649492645}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 3649492645}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 3649492645}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 3649492645}]}, 'public_addr': '172.21.15.104:6804/3649492645', 'cluster_addr': '172.21.15.104:6805/3649492645', 'heartbeat_back_addr': '172.21.15.104:6807/3649492645', 'heartbeat_front_addr': '172.21.15.104:6806/3649492645', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 45, 'last_clean_end': 238, 'up_from': 261, 'up_thru': 414, 'down_at': 239, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 2795996229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 2795996229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 2795996229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 2795996229}]}, 'public_addr': '172.21.15.104:6808/2795996229', 'cluster_addr': '172.21.15.104:6809/2795996229', 'heartbeat_back_addr': '172.21.15.104:6811/2795996229', 'heartbeat_front_addr': '172.21.15.104:6810/2795996229', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:50.384083+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:07.797276+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:19.932106+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:09:55.632813+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:18.847048+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:24.337182+0000', 'dead_epoch': 326}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:06.571191+0000', 'dead_epoch': 238}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:10:22.667125+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.9', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.c', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1f', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.2', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T19:11:17.715 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T19:11:17.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 70 16 102108 102092 91.1153 68.5625 0.0113286 0.0109703 2024-09-21T19:11:17.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:17.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:17.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:17.971 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:17.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:18.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:18.122 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:18.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:18.234 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:18.239 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:18.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:18.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:18.397 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:18.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:18.533 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:18.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:18.660 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:18.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 71 15 103155 103140 90.7545 65.5 0.0140013 0.0110143 2024-09-21T19:11:18.762 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:18.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:18.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:18.919 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:18.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:103318 11.b 11:d857a216:::benchmark_data_smithi081_181703_object103317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:18.915297+0000", 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0018163109999999999, 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018495880000000001, 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:18.921 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 103318 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915297+0000", 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915297+0000", 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915298+0000", 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9350000000000001e-06 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915324+0000", 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5296999999999999e-05 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915340+0000", 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5953000000000001e-05 2024-09-21T19:11:18.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915347+0000", 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7559999999999999e-06 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916012+0000", 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00066426799999999998 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916158+0000", 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000146453 2024-09-21T19:11:18.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,1", 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916326+0000", 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016790800000000001 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop_reply(client.36497.0:103315 11.9 e439/414 ondisk, result = 0)", 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:18.915386+0000", 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.001726446, 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001828442, 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:18.925 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915386+0000", 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915386+0000", 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.926 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915387+0000", 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0299999999999997e-07 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915391+0000", 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5820000000000002e-06 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915397+0000", 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5899999999999996e-06 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915402+0000", 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9679999999999997e-06 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915651+0000", 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024825599999999998 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915658+0000", 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4159999999999998e-06 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:103319 11.b 11.c12af0fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:18.915446+0000", 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0016663870000000001, 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001818648, 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 103319 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:18.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915446+0000", 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915446+0000", 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915447+0000", 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.06e-07 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915471+0000", 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3465999999999999e-05 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915483+0000", 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2798e-05 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.915489+0000", 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9220000000000002e-06 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.931 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:103323 11.8 11:1a37e4ab:::benchmark_data_smithi081_181703_object103322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:18.916143+0000", 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00097005400000000003, 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011788079999999999, 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 103323 2024-09-21T19:11:18.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916143+0000", 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916143+0000", 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916144+0000", 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5090000000000001e-06 2024-09-21T19:11:18.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916200+0000", 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5661000000000001e-05 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916212+0000", 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2339e-05 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916220+0000", 2024-09-21T19:11:18.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0099999999999995e-06 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916665+0000", 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000444237 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916812+0000", 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014698099999999999 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916941+0000", 2024-09-21T19:11:18.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012981900000000001 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:103320 11.1 e439/414)", 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:18.916299+0000", 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00081363599999999996, 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010783800000000001, 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916299+0000", 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916299+0000", 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916300+0000", 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8800000000000002e-07 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916328+0000", 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7866000000000001e-05 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916337+0000", 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6319999999999997e-06 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.916343+0000", 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2910000000000004e-06 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:18.938 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:103315 11.9 11:991336ee:::benchmark_data_smithi081_181703_object103314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:18.909436+0000", 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0076769029999999997, 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0079855929999999992, 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 103315 2024-09-21T19:11:18.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.909436+0000", 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.909436+0000", 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:11:18.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.909438+0000", 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.829e-06 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.909483+0000", 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4724000000000001e-05 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.909497+0000", 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4544e-05 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:11:18.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.909505+0000", 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6020000000000002e-06 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.911807+0000", 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002302681 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.911948+0000", 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014019999999999999 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:11:18.942 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,6", 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:18.912083+0000", 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013552900000000001 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 6 2024-09-21T19:11:18.943 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:11:19.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:18 smithi104 ceph-mon[32438]: pgmap v900: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 846 op/s 2024-09-21T19:11:19.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:11:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:11:19.021 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:19.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:19.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:18 smithi081 ceph-mon[29384]: pgmap v900: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 846 op/s 2024-09-21T19:11:19.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:18 smithi081 ceph-mon[39192]: pgmap v900: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 846 op/s 2024-09-21T19:11:19.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:19.181 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:19.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:19.309 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:19.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:19.438 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:19.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:19.577 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:19.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:19.709 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:19.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 72 16 104300 104284 90.4871 71.5 0.00624452 0.0110464 2024-09-21T19:11:19.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:19.838 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:19.923 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:19.923 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:19.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:19.966 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:19.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:19.971 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:19.971 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:20.008 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:20.008 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:20.008 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3916967371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:20.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:20.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3916967371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:20.040 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:20.041 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:20.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:20.069 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:20.069 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:20.104 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:20.106 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:20.106 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:20.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:20.141 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:20.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:11:20.175 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:20.175 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:20.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:20.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:20.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3916967371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:20.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:20.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:11:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:20.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:20.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:20.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:20.680 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:20.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 73 16 105319 105303 90.1198 63.6875 0.00562442 0.0110913 2024-09-21T19:11:20.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:20.916 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:20.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:21.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:21.102 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:21.123 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:21.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:20 smithi081 ceph-mon[29384]: pgmap v901: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T19:11:21.155 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:20 smithi081 ceph-mon[39192]: pgmap v901: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T19:11:21.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:21.228 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:20 smithi104 ceph-mon[32438]: pgmap v901: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T19:11:21.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:21.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:21.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:21.529 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:21.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:21.671 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:21.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 74 16 106412 106396 89.8251 68.3125 0.0101728 0.0111284 2024-09-21T19:11:21.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.814 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:21.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:21.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:21.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:21.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:22.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:22.090 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:22.135 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:22.135 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:22.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:21 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:22.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:21 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:22.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:21 smithi081 ceph-mon[29384]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T19:11:22.136 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:21 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:22.136 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:21 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:22.136 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:21 smithi081 ceph-mon[39192]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T19:11:22.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:21 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:21 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:21 smithi104 ceph-mon[32438]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T19:11:22.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:22.364 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:22.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:22.505 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:22.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:22.633 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:22.717 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:11:22.717 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:11:22.717 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T19:11:22.717 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T19:11:22.717 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:11:22.734 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:22.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 75 16 107522 107506 89.5524 69.375 0.0139447 0.0111621 2024-09-21T19:11:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:22.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:22.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:22.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:22 smithi104 ceph-mon[32438]: pgmap v903: 158 pgs: 158 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T19:11:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1170973148' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:23.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:22 smithi104 ceph-mon[32438]: from='client.26443 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:23.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:11:23.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[29384]: pgmap v903: 158 pgs: 158 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T19:11:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1170973148' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[29384]: from='client.26443 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[39192]: pgmap v903: 158 pgs: 158 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T19:11:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1170973148' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:23.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:22 smithi081 ceph-mon[39192]: from='client.26443 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:23.076 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:23.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:11:23.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:23.203 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:23.273 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:23.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:23.330 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:11:23.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:23.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:23.463 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:23.564 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:23.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:23.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:23.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:23.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 76 16 108542 108526 89.213 63.75 0.00767656 0.0112047 2024-09-21T19:11:23.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:23.861 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:23.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:23.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:24.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:23 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:24.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:23 smithi104 ceph-mon[32438]: from='client.26443 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:24.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:23 smithi104 ceph-mon[32438]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T19:11:24.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:24.141 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:24.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:24.273 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:23 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:23 smithi081 ceph-mon[39192]: from='client.26443 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:23 smithi081 ceph-mon[39192]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T19:11:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:23 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:23 smithi081 ceph-mon[29384]: from='client.26443 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:23 smithi081 ceph-mon[29384]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T19:11:24.375 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:24.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:24.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:24.533 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:24.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:24.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:24.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:24.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 77 16 109763 109747 89.0454 76.3125 0.0311869 0.0112247 2024-09-21T19:11:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.836 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:24.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:24.912 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:24.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:24.971 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:24 smithi081 ceph-mon[29384]: pgmap v905: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T19:11:24.971 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:24 smithi081 ceph-mon[39192]: pgmap v905: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T19:11:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:24.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:24.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:25.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:25.123 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:25.131 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:11:25.134 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:11:25.226 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:25.256 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:25.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:24 smithi104 ceph-mon[32438]: pgmap v905: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T19:11:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:25.358 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:25.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:25.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:25.519 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:25.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:25.657 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:25.759 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:25.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 78 16 110848 110832 88.7732 67.8125 0.00614283 0.0112605 2024-09-21T19:11:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.785 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:25.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:25.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:25.917 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:25.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:26.019 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:26.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:26.152 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:26.186 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:26.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:25 smithi104 ceph-mon[32438]: from='client.16731 v1:172.21.15.81:0/3573444156' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:26.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:26.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:25 smithi081 ceph-mon[39192]: from='client.16731 v1:172.21.15.81:0/3573444156' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:26.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:25 smithi081 ceph-mon[29384]: from='client.16731 v1:172.21.15.81:0/3573444156' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:26.323 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:26.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:11:26.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:26.491 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:26.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.652 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:26.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:26.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:26.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 79 16 111937 111921 88.5106 68.0625 0.0169155 0.0112924 2024-09-21T19:11:26.777 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:26.831 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:26.832 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:26.871 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:26.871 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:26.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:26.902 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:26.902 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:26.908 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:26.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:26.943 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:26.943 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:26.978 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:26.978 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[29384]: pgmap v906: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/720270864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[39192]: pgmap v906: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:26.979 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/720270864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:27.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:27.011 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:27.011 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.050 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:27.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:27.051 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:27.084 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:27.084 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:27.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:27.186 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:27.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:26 smithi104 ceph-mon[32438]: pgmap v906: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-21T19:11:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:26 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:26 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:27.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/720270864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:27.282 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:27.282 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:27.283 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:11:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:11:27.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:27.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:27.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:27.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:27.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:27.605 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:27.654 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":905,"stamp":"2024-09-21T19:11:23.818295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7010982427,"num_objects":107169,"num_object_clones":0,"num_object_copies":321507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107169,"num_whiteouts":0,"num_read":2222,"num_read_kb":1974,"num_write":107692,"num_write_kb":6846125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":272852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81929,"ondisk_log_size":81929,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25472160,"kb_used_data":20433680,"kb_used_omap":18,"kb_used_meta":5038317,"kb_avail":724489056,"statfs":{"total":767960285184,"available":741876793344,"internally_reserved":0,"allocated":20924088320,"data_stored":20917464355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":5159237616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":628097024,"num_objects":9584,"num_object_clones":0,"num_object_copies":28752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9584,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9584,"num_write_kb":613376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002428"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":727,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914892+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:11:20.914892+0000","last_peered":"2024-09-21T19:11:20.914892+0000","last_clean":"2024-09-21T19:11:20.914892+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:11:20.914892+0000","last_undegraded":"2024-09-21T19:11:20.914892+0000","last_fullsized":"2024-09-21T19:11:20.914892+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.472742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":896,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.910743+0000","last_change":"2024-09-21T19:10:17.698193+0000","last_active":"2024-09-21T19:11:20.910743+0000","last_peered":"2024-09-21T19:11:20.910743+0000","last_clean":"2024-09-21T19:11:20.910743+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:11:20.910743+0000","last_undegraded":"2024-09-21T19:11:20.910743+0000","last_fullsized":"2024-09-21T19:11:20.910743+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:10:17.698093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.698093+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.038064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":825,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909065+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:11:20.909065+0000","last_peered":"2024-09-21T19:11:20.909065+0000","last_clean":"2024-09-21T19:11:20.909065+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:11:20.909065+0000","last_undegraded":"2024-09-21T19:11:20.909065+0000","last_fullsized":"2024-09-21T19:11:20.909065+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:44.208018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":805,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911090+0000","last_change":"2024-09-21T19:06:54.703995+0000","last_active":"2024-09-21T19:11:20.911090+0000","last_peered":"2024-09-21T19:11:20.911090+0000","last_clean":"2024-09-21T19:11:20.911090+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:11:20.911090+0000","last_undegraded":"2024-09-21T19:11:20.911090+0000","last_fullsized":"2024-09-21T19:11:20.911090+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:40.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:40.634655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:30.836849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":822,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916109+0000","last_change":"2024-09-21T19:10:25.846651+0000","last_active":"2024-09-21T19:11:20.916109+0000","last_peered":"2024-09-21T19:11:20.916109+0000","last_clean":"2024-09-21T19:11:20.916109+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:11:20.916109+0000","last_undegraded":"2024-09-21T19:11:20.916109+0000","last_fullsized":"2024-09-21T19:11:20.916109+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:25.846276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.846276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:44.275130+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":850,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914190+0000","last_change":"2024-09-21T19:10:54.811144+0000","last_active":"2024-09-21T19:11:20.914190+0000","last_peered":"2024-09-21T19:11:20.914190+0000","last_clean":"2024-09-21T19:11:20.914190+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:11:20.914190+0000","last_undegraded":"2024-09-21T19:11:20.914190+0000","last_fullsized":"2024-09-21T19:11:20.914190+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:54.811103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:54.811103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.471846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":938,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914794+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:11:20.914794+0000","last_peered":"2024-09-21T19:11:20.914794+0000","last_clean":"2024-09-21T19:11:20.914794+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:11:20.914794+0000","last_undegraded":"2024-09-21T19:11:20.914794+0000","last_fullsized":"2024-09-21T19:11:20.914794+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":833,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.910643+0000","last_change":"2024-09-21T19:10:16.657183+0000","last_active":"2024-09-21T19:11:20.910643+0000","last_peered":"2024-09-21T19:11:20.910643+0000","last_clean":"2024-09-21T19:11:20.910643+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:11:20.910643+0000","last_undegraded":"2024-09-21T19:11:20.910643+0000","last_fullsized":"2024-09-21T19:11:20.910643+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.657131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.657131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":820,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911908+0000","last_change":"2024-09-21T19:10:13.614795+0000","last_active":"2024-09-21T19:11:20.911908+0000","last_peered":"2024-09-21T19:11:20.911908+0000","last_clean":"2024-09-21T19:11:20.911908+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:11:20.911908+0000","last_undegraded":"2024-09-21T19:11:20.911908+0000","last_fullsized":"2024-09-21T19:11:20.911908+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.614720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.614720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:27.974740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":869,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911288+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:11:20.911288+0000","last_peered":"2024-09-21T19:11:20.911288+0000","last_clean":"2024-09-21T19:11:20.911288+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:11:20.911288+0000","last_undegraded":"2024-09-21T19:11:20.911288+0000","last_fullsized":"2024-09-21T19:11:20.911288+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.138431+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":781,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.903052+0000","last_change":"2024-09-21T19:11:00.940146+0000","last_active":"2024-09-21T19:11:20.903052+0000","last_peered":"2024-09-21T19:11:20.903052+0000","last_clean":"2024-09-21T19:11:20.903052+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:11:20.903052+0000","last_undegraded":"2024-09-21T19:11:20.903052+0000","last_fullsized":"2024-09-21T19:11:20.903052+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.940070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.940070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.146532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":693,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915207+0000","last_change":"2024-09-21T19:11:03.440073+0000","last_active":"2024-09-21T19:11:20.915207+0000","last_peered":"2024-09-21T19:11:20.915207+0000","last_clean":"2024-09-21T19:11:20.915207+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:11:20.915207+0000","last_undegraded":"2024-09-21T19:11:20.915207+0000","last_fullsized":"2024-09-21T19:11:20.915207+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.439988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.439988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:07.256937+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":373,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911681+0000","last_change":"2024-09-21T19:11:08.134154+0000","last_active":"2024-09-21T19:11:20.911681+0000","last_peered":"2024-09-21T19:11:20.911681+0000","last_clean":"2024-09-21T19:11:20.911681+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:11:20.911681+0000","last_undegraded":"2024-09-21T19:11:20.911681+0000","last_fullsized":"2024-09-21T19:11:20.911681+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.134097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.134097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.482578+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":815,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911184+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:11:20.911184+0000","last_peered":"2024-09-21T19:11:20.911184+0000","last_clean":"2024-09-21T19:11:20.911184+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:11:20.911184+0000","last_undegraded":"2024-09-21T19:11:20.911184+0000","last_fullsized":"2024-09-21T19:11:20.911184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.851204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":805,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907303+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:11:20.907303+0000","last_peered":"2024-09-21T19:11:20.907303+0000","last_clean":"2024-09-21T19:11:20.907303+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:11:20.907303+0000","last_undegraded":"2024-09-21T19:11:20.907303+0000","last_fullsized":"2024-09-21T19:11:20.907303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.552609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":381,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.908892+0000","last_change":"2024-09-21T19:10:13.100266+0000","last_active":"2024-09-21T19:11:20.908892+0000","last_peered":"2024-09-21T19:11:20.908892+0000","last_clean":"2024-09-21T19:11:20.908892+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:11:20.908892+0000","last_undegraded":"2024-09-21T19:11:20.908892+0000","last_fullsized":"2024-09-21T19:11:20.908892+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.100191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.100191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:34.286351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":842,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911785+0000","last_change":"2024-09-21T19:10:26.819197+0000","last_active":"2024-09-21T19:11:20.911785+0000","last_peered":"2024-09-21T19:11:20.911785+0000","last_clean":"2024-09-21T19:11:20.911785+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:11:20.911785+0000","last_undegraded":"2024-09-21T19:11:20.911785+0000","last_fullsized":"2024-09-21T19:11:20.911785+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:26.819128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:26.819128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:47.207572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":701,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.917477+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:11:20.917477+0000","last_peered":"2024-09-21T19:11:20.917477+0000","last_clean":"2024-09-21T19:11:20.917477+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:11:20.917477+0000","last_undegraded":"2024-09-21T19:11:20.917477+0000","last_fullsized":"2024-09-21T19:11:20.917477+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:40.466668+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":284,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.905569+0000","last_change":"2024-09-21T19:10:25.436206+0000","last_active":"2024-09-21T19:11:20.905569+0000","last_peered":"2024-09-21T19:11:20.905569+0000","last_clean":"2024-09-21T19:11:20.905569+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:11:20.905569+0000","last_undegraded":"2024-09-21T19:11:20.905569+0000","last_fullsized":"2024-09-21T19:11:20.905569+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:10:25.436158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.436158+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.770755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":836,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909444+0000","last_change":"2024-09-21T19:10:25.109965+0000","last_active":"2024-09-21T19:11:20.909444+0000","last_peered":"2024-09-21T19:11:20.909444+0000","last_clean":"2024-09-21T19:11:20.909444+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:11:20.909444+0000","last_undegraded":"2024-09-21T19:11:20.909444+0000","last_fullsized":"2024-09-21T19:11:20.909444+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:25.109893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.109893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.095758+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":818,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915422+0000","last_change":"2024-09-21T19:10:13.916666+0000","last_active":"2024-09-21T19:11:20.915422+0000","last_peered":"2024-09-21T19:11:20.915422+0000","last_clean":"2024-09-21T19:11:20.915422+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:11:20.915422+0000","last_undegraded":"2024-09-21T19:11:20.915422+0000","last_fullsized":"2024-09-21T19:11:20.915422+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.916604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.916604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":836,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911427+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:11:20.911427+0000","last_peered":"2024-09-21T19:11:20.911427+0000","last_clean":"2024-09-21T19:11:20.911427+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:11:20.911427+0000","last_undegraded":"2024-09-21T19:11:20.911427+0000","last_fullsized":"2024-09-21T19:11:20.911427+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.111619+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":900,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915714+0000","last_change":"2024-09-21T19:10:52.029412+0000","last_active":"2024-09-21T19:11:20.915714+0000","last_peered":"2024-09-21T19:11:20.915714+0000","last_clean":"2024-09-21T19:11:20.915714+0000","last_became_active":"2024-09-21T19:10:52.013122+0000","last_became_peered":"2024-09-21T19:10:52.013122+0000","last_unstale":"2024-09-21T19:11:20.915714+0000","last_undegraded":"2024-09-21T19:11:20.915714+0000","last_fullsized":"2024-09-21T19:11:20.915714+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":820,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911581+0000","last_change":"2024-09-21T19:06:13.785197+0000","last_active":"2024-09-21T19:11:20.911581+0000","last_peered":"2024-09-21T19:11:20.911581+0000","last_clean":"2024-09-21T19:11:20.911581+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:11:20.911581+0000","last_undegraded":"2024-09-21T19:11:20.911581+0000","last_fullsized":"2024-09-21T19:11:20.911581+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:13.785148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:13.785148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:31.792346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":851,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.908095+0000","last_change":"2024-09-21T19:10:13.592189+0000","last_active":"2024-09-21T19:11:20.908095+0000","last_peered":"2024-09-21T19:11:20.908095+0000","last_clean":"2024-09-21T19:11:20.908095+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:11:20.908095+0000","last_undegraded":"2024-09-21T19:11:20.908095+0000","last_fullsized":"2024-09-21T19:11:20.908095+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:13.592116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.592116+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":850,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909140+0000","last_change":"2024-09-21T19:10:27.115013+0000","last_active":"2024-09-21T19:11:20.909140+0000","last_peered":"2024-09-21T19:11:20.909140+0000","last_clean":"2024-09-21T19:11:20.909140+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:11:20.909140+0000","last_undegraded":"2024-09-21T19:11:20.909140+0000","last_fullsized":"2024-09-21T19:11:20.909140+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:27.114925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.114925+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:50.719198+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":737,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916043+0000","last_change":"2024-09-21T19:10:27.501434+0000","last_active":"2024-09-21T19:11:20.916043+0000","last_peered":"2024-09-21T19:11:20.916043+0000","last_clean":"2024-09-21T19:11:20.916043+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:11:20.916043+0000","last_undegraded":"2024-09-21T19:11:20.916043+0000","last_fullsized":"2024-09-21T19:11:20.916043+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:27.500959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.500959+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:39.504634+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":930,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786413+0000","last_change":"2024-09-21T19:10:15.802776+0000","last_active":"2024-09-21T19:11:14.786413+0000","last_peered":"2024-09-21T19:11:14.786413+0000","last_clean":"2024-09-21T19:11:14.786413+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:11:14.786413+0000","last_undegraded":"2024-09-21T19:11:14.786413+0000","last_fullsized":"2024-09-21T19:11:14.786413+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:10:15.802724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:15.802724+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:27.023101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"440'6604","reported_seq":6667,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:22.804752+0000","last_change":"2024-09-21T19:10:01.248307+0000","last_active":"2024-09-21T19:11:22.804752+0000","last_peered":"2024-09-21T19:11:22.804752+0000","last_clean":"2024-09-21T19:11:22.804752+0000","last_became_active":"2024-09-21T19:10:01.248029+0000","last_became_peered":"2024-09-21T19:10:01.248029+0000","last_unstale":"2024-09-21T19:11:22.804752+0000","last_undegraded":"2024-09-21T19:11:22.804752+0000","last_fullsized":"2024-09-21T19:11:22.804752+0000","mapping_epoch":414,"log_start":"423'1700","ondisk_log_start":"423'1700","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:21.027444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432799744,"num_objects":6604,"num_object_clones":0,"num_object_copies":19812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":422656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":279,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906786+0000","last_change":"2024-09-21T19:10:21.515835+0000","last_active":"2024-09-21T19:11:20.906786+0000","last_peered":"2024-09-21T19:11:20.906786+0000","last_clean":"2024-09-21T19:11:20.906786+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:11:20.906786+0000","last_undegraded":"2024-09-21T19:11:20.906786+0000","last_fullsized":"2024-09-21T19:11:20.906786+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:21.515713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.515713+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.289543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":878,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909912+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:11:20.909912+0000","last_peered":"2024-09-21T19:11:20.909912+0000","last_clean":"2024-09-21T19:11:20.909912+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:11:20.909912+0000","last_undegraded":"2024-09-21T19:11:20.909912+0000","last_fullsized":"2024-09-21T19:11:20.909912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.082123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":816,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916295+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:11:20.916295+0000","last_peered":"2024-09-21T19:11:20.916295+0000","last_clean":"2024-09-21T19:11:20.916295+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:11:20.916295+0000","last_undegraded":"2024-09-21T19:11:20.916295+0000","last_fullsized":"2024-09-21T19:11:20.916295+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1085,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911829+0000","last_change":"2024-09-21T19:11:10.165309+0000","last_active":"2024-09-21T19:11:20.911829+0000","last_peered":"2024-09-21T19:11:20.911829+0000","last_clean":"2024-09-21T19:11:20.911829+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:11:20.911829+0000","last_undegraded":"2024-09-21T19:11:20.911829+0000","last_fullsized":"2024-09-21T19:11:20.911829+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:11:10.165236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:10.165236+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.702978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":505,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909385+0000","last_change":"2024-09-21T19:10:23.103828+0000","last_active":"2024-09-21T19:11:20.909385+0000","last_peered":"2024-09-21T19:11:20.909385+0000","last_clean":"2024-09-21T19:11:20.909385+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:11:20.909385+0000","last_undegraded":"2024-09-21T19:11:20.909385+0000","last_fullsized":"2024-09-21T19:11:20.909385+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:23.103763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.103763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:34.402051+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"440'6735","reported_seq":6798,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:21.912333+0000","last_change":"2024-09-21T19:10:01.252308+0000","last_active":"2024-09-21T19:11:21.912333+0000","last_peered":"2024-09-21T19:11:21.912333+0000","last_clean":"2024-09-21T19:11:21.912333+0000","last_became_active":"2024-09-21T19:10:01.252017+0000","last_became_peered":"2024-09-21T19:10:01.252017+0000","last_unstale":"2024-09-21T19:11:21.912333+0000","last_undegraded":"2024-09-21T19:11:21.912333+0000","last_fullsized":"2024-09-21T19:11:21.912333+0000","mapping_epoch":414,"log_start":"423'1800","ondisk_log_start":"423'1800","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.828286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441384960,"num_objects":6735,"num_object_clones":0,"num_object_copies":20205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6735,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":605,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911840+0000","last_change":"2024-09-21T19:09:53.595884+0000","last_active":"2024-09-21T19:11:20.911840+0000","last_peered":"2024-09-21T19:11:20.911840+0000","last_clean":"2024-09-21T19:11:20.911840+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:11:20.911840+0000","last_undegraded":"2024-09-21T19:11:20.911840+0000","last_fullsized":"2024-09-21T19:11:20.911840+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":905,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915518+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:11:20.915518+0000","last_peered":"2024-09-21T19:11:20.915518+0000","last_clean":"2024-09-21T19:11:20.915518+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:11:20.915518+0000","last_undegraded":"2024-09-21T19:11:20.915518+0000","last_fullsized":"2024-09-21T19:11:20.915518+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.668914+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":838,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915902+0000","last_change":"2024-09-21T19:10:56.831121+0000","last_active":"2024-09-21T19:11:20.915902+0000","last_peered":"2024-09-21T19:11:20.915902+0000","last_clean":"2024-09-21T19:11:20.915902+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:11:20.915902+0000","last_undegraded":"2024-09-21T19:11:20.915902+0000","last_fullsized":"2024-09-21T19:11:20.915902+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:56.831042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:56.831042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:00.224419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":815,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.910645+0000","last_change":"2024-09-21T19:11:05.056988+0000","last_active":"2024-09-21T19:11:20.910645+0000","last_peered":"2024-09-21T19:11:20.910645+0000","last_clean":"2024-09-21T19:11:20.910645+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:11:20.910645+0000","last_undegraded":"2024-09-21T19:11:20.910645+0000","last_fullsized":"2024-09-21T19:11:20.910645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.056931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.056931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:22.842104+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":832,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911198+0000","last_change":"2024-09-21T19:10:51.864691+0000","last_active":"2024-09-21T19:11:20.911198+0000","last_peered":"2024-09-21T19:11:20.911198+0000","last_clean":"2024-09-21T19:11:20.911198+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:11:20.911198+0000","last_undegraded":"2024-09-21T19:11:20.911198+0000","last_fullsized":"2024-09-21T19:11:20.911198+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:51.864627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.864627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:01.755943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"440'6588","reported_seq":6651,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:21.454556+0000","last_change":"2024-09-21T19:10:01.248578+0000","last_active":"2024-09-21T19:11:21.454556+0000","last_peered":"2024-09-21T19:11:21.454556+0000","last_clean":"2024-09-21T19:11:21.454556+0000","last_became_active":"2024-09-21T19:10:01.248286+0000","last_became_peered":"2024-09-21T19:10:01.248286+0000","last_unstale":"2024-09-21T19:11:21.454556+0000","last_undegraded":"2024-09-21T19:11:21.454556+0000","last_fullsized":"2024-09-21T19:11:21.454556+0000","mapping_epoch":414,"log_start":"424'1900","ondisk_log_start":"424'1900","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431751168,"num_objects":6588,"num_object_clones":0,"num_object_copies":19764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6588,"num_write_kb":421632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":588,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.919500+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:11:20.919500+0000","last_peered":"2024-09-21T19:11:20.919500+0000","last_clean":"2024-09-21T19:11:20.919500+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:11:20.919500+0000","last_undegraded":"2024-09-21T19:11:20.919500+0000","last_fullsized":"2024-09-21T19:11:20.919500+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:36.692032+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":883,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.910871+0000","last_change":"2024-09-21T19:10:11.634360+0000","last_active":"2024-09-21T19:11:20.910871+0000","last_peered":"2024-09-21T19:11:20.910871+0000","last_clean":"2024-09-21T19:11:20.910871+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:11:20.910871+0000","last_undegraded":"2024-09-21T19:11:20.910871+0000","last_fullsized":"2024-09-21T19:11:20.910871+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:10:11.634280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:11.634280+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:25.695788+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":706,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916773+0000","last_change":"2024-09-21T19:11:04.465550+0000","last_active":"2024-09-21T19:11:20.916773+0000","last_peered":"2024-09-21T19:11:20.916773+0000","last_clean":"2024-09-21T19:11:20.916773+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:11:20.916773+0000","last_undegraded":"2024-09-21T19:11:20.916773+0000","last_fullsized":"2024-09-21T19:11:20.916773+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:04.464221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:04.464221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:31.185556+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":1029,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.913365+0000","last_change":"2024-09-21T19:10:21.865360+0000","last_active":"2024-09-21T19:11:20.913365+0000","last_peered":"2024-09-21T19:11:20.913365+0000","last_clean":"2024-09-21T19:11:20.913365+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:11:20.913365+0000","last_undegraded":"2024-09-21T19:11:20.913365+0000","last_fullsized":"2024-09-21T19:11:20.913365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:10:21.865158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.865158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:42.703841+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":835,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907716+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:11:20.907716+0000","last_peered":"2024-09-21T19:11:20.907716+0000","last_clean":"2024-09-21T19:11:20.907716+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:11:20.907716+0000","last_undegraded":"2024-09-21T19:11:20.907716+0000","last_fullsized":"2024-09-21T19:11:20.907716+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.789154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"440'6664","reported_seq":6726,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:23.040712+0000","last_change":"2024-09-21T19:10:01.251501+0000","last_active":"2024-09-21T19:11:23.040712+0000","last_peered":"2024-09-21T19:11:23.040712+0000","last_clean":"2024-09-21T19:11:23.040712+0000","last_became_active":"2024-09-21T19:10:01.250354+0000","last_became_peered":"2024-09-21T19:10:01.250354+0000","last_unstale":"2024-09-21T19:11:23.040712+0000","last_undegraded":"2024-09-21T19:11:23.040712+0000","last_fullsized":"2024-09-21T19:11:23.040712+0000","mapping_epoch":414,"log_start":"419'400","ondisk_log_start":"419'400","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:27.378734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436731904,"num_objects":6664,"num_object_clones":0,"num_object_copies":19992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6664,"num_write_kb":426496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":617,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786455+0000","last_change":"2024-09-21T19:09:55.907018+0000","last_active":"2024-09-21T19:11:14.786455+0000","last_peered":"2024-09-21T19:11:14.786455+0000","last_clean":"2024-09-21T19:11:14.786455+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:11:14.786455+0000","last_undegraded":"2024-09-21T19:11:14.786455+0000","last_fullsized":"2024-09-21T19:11:14.786455+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":868,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909944+0000","last_change":"2024-09-21T19:10:57.167898+0000","last_active":"2024-09-21T19:11:20.909944+0000","last_peered":"2024-09-21T19:11:20.909944+0000","last_clean":"2024-09-21T19:11:20.909944+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:11:20.909944+0000","last_undegraded":"2024-09-21T19:11:20.909944+0000","last_fullsized":"2024-09-21T19:11:20.909944+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:57.167826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.167826+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:01.720799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":379,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915807+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:11:20.915807+0000","last_peered":"2024-09-21T19:11:20.915807+0000","last_clean":"2024-09-21T19:11:20.915807+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:11:20.915807+0000","last_undegraded":"2024-09-21T19:11:20.915807+0000","last_fullsized":"2024-09-21T19:11:20.915807+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":903,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.917421+0000","last_change":"2024-09-21T19:06:54.700645+0000","last_active":"2024-09-21T19:11:20.917421+0000","last_peered":"2024-09-21T19:11:20.917421+0000","last_clean":"2024-09-21T19:11:20.917421+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:11:20.917421+0000","last_undegraded":"2024-09-21T19:11:20.917421+0000","last_fullsized":"2024-09-21T19:11:20.917421+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'12","last_scrub_stamp":"2024-09-21T19:05:44.193752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:44.193752+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.399811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":716,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915337+0000","last_change":"2024-09-21T19:11:08.473248+0000","last_active":"2024-09-21T19:11:20.915337+0000","last_peered":"2024-09-21T19:11:20.915337+0000","last_clean":"2024-09-21T19:11:20.915337+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:11:20.915337+0000","last_undegraded":"2024-09-21T19:11:20.915337+0000","last_fullsized":"2024-09-21T19:11:20.915337+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.473184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.473184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:38.292717+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"440'6850","reported_seq":6913,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:22.763191+0000","last_change":"2024-09-21T19:10:01.244039+0000","last_active":"2024-09-21T19:11:22.763191+0000","last_peered":"2024-09-21T19:11:22.763191+0000","last_clean":"2024-09-21T19:11:22.763191+0000","last_became_active":"2024-09-21T19:10:01.243730+0000","last_became_peered":"2024-09-21T19:10:01.243730+0000","last_unstale":"2024-09-21T19:11:22.763191+0000","last_undegraded":"2024-09-21T19:11:22.763191+0000","last_fullsized":"2024-09-21T19:11:22.763191+0000","mapping_epoch":414,"log_start":"424'2000","ondisk_log_start":"424'2000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.216097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448921600,"num_objects":6850,"num_object_clones":0,"num_object_copies":20550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6850,"num_write_kb":438400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":595,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786357+0000","last_change":"2024-09-21T19:09:54.937870+0000","last_active":"2024-09-21T19:11:14.786357+0000","last_peered":"2024-09-21T19:11:14.786357+0000","last_clean":"2024-09-21T19:11:14.786357+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:11:14.786357+0000","last_undegraded":"2024-09-21T19:11:14.786357+0000","last_fullsized":"2024-09-21T19:11:14.786357+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":754,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.917562+0000","last_change":"2024-09-21T19:10:25.488039+0000","last_active":"2024-09-21T19:11:20.917562+0000","last_peered":"2024-09-21T19:11:20.917562+0000","last_clean":"2024-09-21T19:11:20.917562+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:11:20.917562+0000","last_undegraded":"2024-09-21T19:11:20.917562+0000","last_fullsized":"2024-09-21T19:11:20.917562+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:10:25.487878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.487878+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:46.383223+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":848,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909422+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:11:20.909422+0000","last_peered":"2024-09-21T19:11:20.909422+0000","last_clean":"2024-09-21T19:11:20.909422+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:11:20.909422+0000","last_undegraded":"2024-09-21T19:11:20.909422+0000","last_fullsized":"2024-09-21T19:11:20.909422+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.041720+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":723,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.785727+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:11:14.785727+0000","last_peered":"2024-09-21T19:11:14.785727+0000","last_clean":"2024-09-21T19:11:14.785727+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:11:14.785727+0000","last_undegraded":"2024-09-21T19:11:14.785727+0000","last_fullsized":"2024-09-21T19:11:14.785727+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":836,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.908987+0000","last_change":"2024-09-21T19:11:00.111371+0000","last_active":"2024-09-21T19:11:20.908987+0000","last_peered":"2024-09-21T19:11:20.908987+0000","last_clean":"2024-09-21T19:11:20.908987+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:11:20.908987+0000","last_undegraded":"2024-09-21T19:11:20.908987+0000","last_fullsized":"2024-09-21T19:11:20.908987+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.111305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.111305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.568023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"440'6758","reported_seq":6821,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:23.078603+0000","last_change":"2024-09-21T19:10:01.250359+0000","last_active":"2024-09-21T19:11:23.078603+0000","last_peered":"2024-09-21T19:11:23.078603+0000","last_clean":"2024-09-21T19:11:23.078603+0000","last_became_active":"2024-09-21T19:10:01.250079+0000","last_became_peered":"2024-09-21T19:10:01.250079+0000","last_unstale":"2024-09-21T19:11:23.078603+0000","last_undegraded":"2024-09-21T19:11:23.078603+0000","last_fullsized":"2024-09-21T19:11:23.078603+0000","mapping_epoch":414,"log_start":"426'2400","ondisk_log_start":"426'2400","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442892288,"num_objects":6758,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":572,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915621+0000","last_change":"2024-09-21T19:11:07.477321+0000","last_active":"2024-09-21T19:11:20.915621+0000","last_peered":"2024-09-21T19:11:20.915621+0000","last_clean":"2024-09-21T19:11:20.915621+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:11:20.915621+0000","last_undegraded":"2024-09-21T19:11:20.915621+0000","last_fullsized":"2024-09-21T19:11:20.915621+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:07.476444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.476444+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:35.128950+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1156,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909232+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:11:20.909232+0000","last_peered":"2024-09-21T19:11:20.909232+0000","last_clean":"2024-09-21T19:11:20.909232+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:11:20.909232+0000","last_undegraded":"2024-09-21T19:11:20.909232+0000","last_fullsized":"2024-09-21T19:11:20.909232+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.035088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":848,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911358+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:11:20.911358+0000","last_peered":"2024-09-21T19:11:20.911358+0000","last_clean":"2024-09-21T19:11:20.911358+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:11:20.911358+0000","last_undegraded":"2024-09-21T19:11:20.911358+0000","last_fullsized":"2024-09-21T19:11:20.911358+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.906766+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":395,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.917026+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:11:20.917026+0000","last_peered":"2024-09-21T19:11:20.917026+0000","last_clean":"2024-09-21T19:11:20.917026+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:11:20.917026+0000","last_undegraded":"2024-09-21T19:11:20.917026+0000","last_fullsized":"2024-09-21T19:11:20.917026+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.115247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":702,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914499+0000","last_change":"2024-09-21T19:11:05.417598+0000","last_active":"2024-09-21T19:11:20.914499+0000","last_peered":"2024-09-21T19:11:20.914499+0000","last_clean":"2024-09-21T19:11:20.914499+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:11:20.914499+0000","last_undegraded":"2024-09-21T19:11:20.914499+0000","last_fullsized":"2024-09-21T19:11:20.914499+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.417365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.417365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.055191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":804,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914268+0000","last_change":"2024-09-21T19:10:28.717418+0000","last_active":"2024-09-21T19:11:20.914268+0000","last_peered":"2024-09-21T19:11:20.914268+0000","last_clean":"2024-09-21T19:11:20.914268+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:11:20.914268+0000","last_undegraded":"2024-09-21T19:11:20.914268+0000","last_fullsized":"2024-09-21T19:11:20.914268+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:28.717310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.717310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.163329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"440'6702","reported_seq":6765,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:23.106867+0000","last_change":"2024-09-21T19:10:01.259760+0000","last_active":"2024-09-21T19:11:23.106867+0000","last_peered":"2024-09-21T19:11:23.106867+0000","last_clean":"2024-09-21T19:11:23.106867+0000","last_became_active":"2024-09-21T19:10:01.259451+0000","last_became_peered":"2024-09-21T19:10:01.259451+0000","last_unstale":"2024-09-21T19:11:23.106867+0000","last_undegraded":"2024-09-21T19:11:23.106867+0000","last_fullsized":"2024-09-21T19:11:23.106867+0000","mapping_epoch":414,"log_start":"426'2300","ondisk_log_start":"426'2300","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439222272,"num_objects":6702,"num_object_clones":0,"num_object_copies":20106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6702,"num_write_kb":428928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":909,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907095+0000","last_change":"2024-09-21T19:09:55.484547+0000","last_active":"2024-09-21T19:11:20.907095+0000","last_peered":"2024-09-21T19:11:20.907095+0000","last_clean":"2024-09-21T19:11:20.907095+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:11:20.907095+0000","last_undegraded":"2024-09-21T19:11:20.907095+0000","last_fullsized":"2024-09-21T19:11:20.907095+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":321,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909636+0000","last_change":"2024-09-21T19:10:20.080922+0000","last_active":"2024-09-21T19:11:20.909636+0000","last_peered":"2024-09-21T19:11:20.909636+0000","last_clean":"2024-09-21T19:11:20.909636+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:11:20.909636+0000","last_undegraded":"2024-09-21T19:11:20.909636+0000","last_fullsized":"2024-09-21T19:11:20.909636+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:10:20.080874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:20.080874+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:45.892100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":871,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909402+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:11:20.909402+0000","last_peered":"2024-09-21T19:11:20.909402+0000","last_clean":"2024-09-21T19:11:20.909402+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:11:20.909402+0000","last_undegraded":"2024-09-21T19:11:20.909402+0000","last_fullsized":"2024-09-21T19:11:20.909402+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:35.138769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":264,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.904798+0000","last_change":"2024-09-21T19:06:53.904748+0000","last_active":"2024-09-21T19:11:20.904798+0000","last_peered":"2024-09-21T19:11:20.904798+0000","last_clean":"2024-09-21T19:11:20.904798+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:11:20.904798+0000","last_undegraded":"2024-09-21T19:11:20.904798+0000","last_fullsized":"2024-09-21T19:11:20.904798+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:06:53.902363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.902363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:30.368816+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":271,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.905979+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:11:20.905979+0000","last_peered":"2024-09-21T19:11:20.905979+0000","last_clean":"2024-09-21T19:11:20.905979+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:11:20.905979+0000","last_undegraded":"2024-09-21T19:11:20.905979+0000","last_fullsized":"2024-09-21T19:11:20.905979+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:43.358042+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":833,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907722+0000","last_change":"2024-09-21T19:10:45.720231+0000","last_active":"2024-09-21T19:11:20.907722+0000","last_peered":"2024-09-21T19:11:20.907722+0000","last_clean":"2024-09-21T19:11:20.907722+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:11:20.907722+0000","last_undegraded":"2024-09-21T19:11:20.907722+0000","last_fullsized":"2024-09-21T19:11:20.907722+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:45.720186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.720186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.576025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"440'6785","reported_seq":6847,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:23.088149+0000","last_change":"2024-09-21T19:10:01.250406+0000","last_active":"2024-09-21T19:11:23.088149+0000","last_peered":"2024-09-21T19:11:23.088149+0000","last_clean":"2024-09-21T19:11:23.088149+0000","last_became_active":"2024-09-21T19:10:01.250100+0000","last_became_peered":"2024-09-21T19:10:01.250100+0000","last_unstale":"2024-09-21T19:11:23.088149+0000","last_undegraded":"2024-09-21T19:11:23.088149+0000","last_fullsized":"2024-09-21T19:11:23.088149+0000","mapping_epoch":414,"log_start":"426'2400","ondisk_log_start":"426'2400","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444661760,"num_objects":6785,"num_object_clones":0,"num_object_copies":20355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6785,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":615,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909671+0000","last_change":"2024-09-21T19:10:28.147523+0000","last_active":"2024-09-21T19:11:20.909671+0000","last_peered":"2024-09-21T19:11:20.909671+0000","last_clean":"2024-09-21T19:11:20.909671+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:11:20.909671+0000","last_undegraded":"2024-09-21T19:11:20.909671+0000","last_fullsized":"2024-09-21T19:11:20.909671+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:28.147274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.147274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:39.517401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":816,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916584+0000","last_change":"2024-09-21T19:10:26.510494+0000","last_active":"2024-09-21T19:11:20.916584+0000","last_peered":"2024-09-21T19:11:20.916584+0000","last_clean":"2024-09-21T19:11:20.916584+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:11:20.916584+0000","last_undegraded":"2024-09-21T19:11:20.916584+0000","last_fullsized":"2024-09-21T19:11:20.916584+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:10:26.510232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:26.510232+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.440211+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":702,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916524+0000","last_change":"2024-09-21T19:10:18.502592+0000","last_active":"2024-09-21T19:11:20.916524+0000","last_peered":"2024-09-21T19:11:20.916524+0000","last_clean":"2024-09-21T19:11:20.916524+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:11:20.916524+0000","last_undegraded":"2024-09-21T19:11:20.916524+0000","last_fullsized":"2024-09-21T19:11:20.916524+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:18.502526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:18.502526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.162083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1936,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.923926+0000","last_change":"2024-09-21T19:10:28.526838+0000","last_active":"2024-09-21T19:11:20.923926+0000","last_peered":"2024-09-21T19:11:20.923926+0000","last_clean":"2024-09-21T19:11:20.923926+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:11:20.923926+0000","last_undegraded":"2024-09-21T19:11:20.923926+0000","last_fullsized":"2024-09-21T19:11:20.923926+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:10:28.526790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.526790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:50.333975+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1083,"num_read_kb":1078,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":775,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.903762+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:11:20.903762+0000","last_peered":"2024-09-21T19:11:20.903762+0000","last_clean":"2024-09-21T19:11:20.903762+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:11:20.903762+0000","last_undegraded":"2024-09-21T19:11:20.903762+0000","last_fullsized":"2024-09-21T19:11:20.903762+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.851635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909027+0000","last_change":"2024-09-21T19:10:17.073561+0000","last_active":"2024-09-21T19:11:20.909027+0000","last_peered":"2024-09-21T19:11:20.909027+0000","last_clean":"2024-09-21T19:11:20.909027+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:11:20.909027+0000","last_undegraded":"2024-09-21T19:11:20.909027+0000","last_fullsized":"2024-09-21T19:11:20.909027+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.073521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.073521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:30.597379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"440'6754","reported_seq":6817,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:23.094361+0000","last_change":"2024-09-21T19:10:01.260090+0000","last_active":"2024-09-21T19:11:23.094361+0000","last_peered":"2024-09-21T19:11:23.094361+0000","last_clean":"2024-09-21T19:11:23.094361+0000","last_became_active":"2024-09-21T19:10:01.259620+0000","last_became_peered":"2024-09-21T19:10:01.259620+0000","last_unstale":"2024-09-21T19:11:23.094361+0000","last_undegraded":"2024-09-21T19:11:23.094361+0000","last_fullsized":"2024-09-21T19:11:23.094361+0000","mapping_epoch":414,"log_start":"426'2400","ondisk_log_start":"426'2400","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442630144,"num_objects":6754,"num_object_clones":0,"num_object_copies":20262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":432256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":1021,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786298+0000","last_change":"2024-09-21T19:11:02.540038+0000","last_active":"2024-09-21T19:11:14.786298+0000","last_peered":"2024-09-21T19:11:14.786298+0000","last_clean":"2024-09-21T19:11:14.786298+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:11:14.786298+0000","last_undegraded":"2024-09-21T19:11:14.786298+0000","last_fullsized":"2024-09-21T19:11:14.786298+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:02.539974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.539974+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.492636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915289+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:11:20.915289+0000","last_peered":"2024-09-21T19:11:20.915289+0000","last_clean":"2024-09-21T19:11:20.915289+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:11:20.915289+0000","last_undegraded":"2024-09-21T19:11:20.915289+0000","last_fullsized":"2024-09-21T19:11:20.915289+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:34.702481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":915,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906903+0000","last_change":"2024-09-21T19:09:46.543580+0000","last_active":"2024-09-21T19:11:20.906903+0000","last_peered":"2024-09-21T19:11:20.906903+0000","last_clean":"2024-09-21T19:11:20.906903+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:11:20.906903+0000","last_undegraded":"2024-09-21T19:11:20.906903+0000","last_fullsized":"2024-09-21T19:11:20.906903+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:46.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:46.543525+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:17.736334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":839,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.908234+0000","last_change":"2024-09-21T19:10:17.553053+0000","last_active":"2024-09-21T19:11:20.908234+0000","last_peered":"2024-09-21T19:11:20.908234+0000","last_clean":"2024-09-21T19:11:20.908234+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:11:20.908234+0000","last_undegraded":"2024-09-21T19:11:20.908234+0000","last_fullsized":"2024-09-21T19:11:20.908234+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.553007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.553007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:47.373270+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":365,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915346+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:11:20.915346+0000","last_peered":"2024-09-21T19:11:20.915346+0000","last_clean":"2024-09-21T19:11:20.915346+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:11:20.915346+0000","last_undegraded":"2024-09-21T19:11:20.915346+0000","last_fullsized":"2024-09-21T19:11:20.915346+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"439'6529","reported_seq":6590,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:19.430001+0000","last_change":"2024-09-21T19:10:01.250432+0000","last_active":"2024-09-21T19:11:19.430001+0000","last_peered":"2024-09-21T19:11:19.430001+0000","last_clean":"2024-09-21T19:11:19.430001+0000","last_became_active":"2024-09-21T19:10:01.250116+0000","last_became_peered":"2024-09-21T19:10:01.250116+0000","last_unstale":"2024-09-21T19:11:19.430001+0000","last_undegraded":"2024-09-21T19:11:19.430001+0000","last_fullsized":"2024-09-21T19:11:19.430001+0000","mapping_epoch":414,"log_start":"421'1200","ondisk_log_start":"421'1200","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:24.804001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427884544,"num_objects":6529,"num_object_clones":0,"num_object_copies":19587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6529,"num_write_kb":417856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":877,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911914+0000","last_change":"2024-09-21T19:10:23.782154+0000","last_active":"2024-09-21T19:11:20.911914+0000","last_peered":"2024-09-21T19:11:20.911914+0000","last_clean":"2024-09-21T19:11:20.911914+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:11:20.911914+0000","last_undegraded":"2024-09-21T19:11:20.911914+0000","last_fullsized":"2024-09-21T19:11:20.911914+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:10:23.782107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.782107+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:24.541190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":845,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911041+0000","last_change":"2024-09-21T19:10:19.729765+0000","last_active":"2024-09-21T19:11:20.911041+0000","last_peered":"2024-09-21T19:11:20.911041+0000","last_clean":"2024-09-21T19:11:20.911041+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:11:20.911041+0000","last_undegraded":"2024-09-21T19:11:20.911041+0000","last_fullsized":"2024-09-21T19:11:20.911041+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:19.729685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.729685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:20.999386+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":829,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911911+0000","last_change":"2024-09-21T19:10:58.999039+0000","last_active":"2024-09-21T19:11:20.911911+0000","last_peered":"2024-09-21T19:11:20.911911+0000","last_clean":"2024-09-21T19:11:20.911911+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:11:20.911911+0000","last_undegraded":"2024-09-21T19:11:20.911911+0000","last_fullsized":"2024-09-21T19:11:20.911911+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.998847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.998847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:59.614433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":802,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.919608+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:11:20.919608+0000","last_peered":"2024-09-21T19:11:20.919608+0000","last_clean":"2024-09-21T19:11:20.919608+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:11:20.919608+0000","last_undegraded":"2024-09-21T19:11:20.919608+0000","last_fullsized":"2024-09-21T19:11:20.919608+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.968888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"440'6617","reported_seq":6746,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:23.070097+0000","last_change":"2024-09-21T19:11:09.898977+0000","last_active":"2024-09-21T19:11:23.070097+0000","last_peered":"2024-09-21T19:11:23.070097+0000","last_clean":"2024-09-21T19:11:23.070097+0000","last_became_active":"2024-09-21T19:10:01.247852+0000","last_became_peered":"2024-09-21T19:10:01.247852+0000","last_unstale":"2024-09-21T19:11:23.070097+0000","last_undegraded":"2024-09-21T19:11:23.070097+0000","last_fullsized":"2024-09-21T19:11:23.070097+0000","mapping_epoch":414,"log_start":"419'300","ondisk_log_start":"419'300","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'5812","last_scrub_stamp":"2024-09-21T19:11:09.898694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.898694+0000","objects_scrubbed":5735,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:34.977808+0000","scrub_duration":2494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433651712,"num_objects":6617,"num_object_clones":0,"num_object_copies":19851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":275,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906894+0000","last_change":"2024-09-21T19:10:14.570018+0000","last_active":"2024-09-21T19:11:20.906894+0000","last_peered":"2024-09-21T19:11:20.906894+0000","last_clean":"2024-09-21T19:11:20.906894+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:11:20.906894+0000","last_undegraded":"2024-09-21T19:11:20.906894+0000","last_fullsized":"2024-09-21T19:11:20.906894+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:14.569931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:14.569931+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":918,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911814+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:11:20.911814+0000","last_peered":"2024-09-21T19:11:20.911814+0000","last_clean":"2024-09-21T19:11:20.911814+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:11:20.911814+0000","last_undegraded":"2024-09-21T19:11:20.911814+0000","last_fullsized":"2024-09-21T19:11:20.911814+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:41.851179+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":741,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786171+0000","last_change":"2024-09-21T19:10:17.814004+0000","last_active":"2024-09-21T19:11:14.786171+0000","last_peered":"2024-09-21T19:11:14.786171+0000","last_clean":"2024-09-21T19:11:14.786171+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:11:14.786171+0000","last_undegraded":"2024-09-21T19:11:14.786171+0000","last_fullsized":"2024-09-21T19:11:14.786171+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.813917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.813917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:46.777123+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":266,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786100+0000","last_change":"2024-09-21T19:09:50.927165+0000","last_active":"2024-09-21T19:11:14.786100+0000","last_peered":"2024-09-21T19:11:14.786100+0000","last_clean":"2024-09-21T19:11:14.786100+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:11:14.786100+0000","last_undegraded":"2024-09-21T19:11:14.786100+0000","last_fullsized":"2024-09-21T19:11:14.786100+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.927103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.927103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:19.477371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":276,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.905409+0000","last_change":"2024-09-21T19:10:45.402838+0000","last_active":"2024-09-21T19:11:20.905409+0000","last_peered":"2024-09-21T19:11:20.905409+0000","last_clean":"2024-09-21T19:11:20.905409+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:11:20.905409+0000","last_undegraded":"2024-09-21T19:11:20.905409+0000","last_fullsized":"2024-09-21T19:11:20.905409+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:45.402769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.402769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:57.226877+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"439'6519","reported_seq":6580,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:19.395293+0000","last_change":"2024-09-21T19:10:01.252391+0000","last_active":"2024-09-21T19:11:19.395293+0000","last_peered":"2024-09-21T19:11:19.395293+0000","last_clean":"2024-09-21T19:11:19.395293+0000","last_became_active":"2024-09-21T19:10:01.251894+0000","last_became_peered":"2024-09-21T19:10:01.251894+0000","last_unstale":"2024-09-21T19:11:19.395293+0000","last_undegraded":"2024-09-21T19:11:19.395293+0000","last_fullsized":"2024-09-21T19:11:19.395293+0000","mapping_epoch":414,"log_start":"421'1200","ondisk_log_start":"421'1200","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427229184,"num_objects":6519,"num_object_clones":0,"num_object_copies":19557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":417216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":930,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906948+0000","last_change":"2024-09-21T19:10:51.731370+0000","last_active":"2024-09-21T19:11:20.906948+0000","last_peered":"2024-09-21T19:11:20.906948+0000","last_clean":"2024-09-21T19:11:20.906948+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:11:20.906948+0000","last_undegraded":"2024-09-21T19:11:20.906948+0000","last_fullsized":"2024-09-21T19:11:20.906948+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:51.731299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.731299+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:56.746842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":844,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911588+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:11:20.911588+0000","last_peered":"2024-09-21T19:11:20.911588+0000","last_clean":"2024-09-21T19:11:20.911588+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:11:20.911588+0000","last_undegraded":"2024-09-21T19:11:20.911588+0000","last_fullsized":"2024-09-21T19:11:20.911588+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:40.839786+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":852,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.921411+0000","last_change":"2024-09-21T19:10:55.909242+0000","last_active":"2024-09-21T19:11:20.921411+0000","last_peered":"2024-09-21T19:11:20.921411+0000","last_clean":"2024-09-21T19:11:20.921411+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:11:20.921411+0000","last_undegraded":"2024-09-21T19:11:20.921411+0000","last_fullsized":"2024-09-21T19:11:20.921411+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:55.909172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:55.909172+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.738817+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":346,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.905205+0000","last_change":"2024-09-21T19:10:24.479930+0000","last_active":"2024-09-21T19:11:20.905205+0000","last_peered":"2024-09-21T19:11:20.905205+0000","last_clean":"2024-09-21T19:11:20.905205+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:11:20.905205+0000","last_undegraded":"2024-09-21T19:11:20.905205+0000","last_fullsized":"2024-09-21T19:11:20.905205+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:10:24.479847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:24.479847+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:31.063463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":817,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911655+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:11:20.911655+0000","last_peered":"2024-09-21T19:11:20.911655+0000","last_clean":"2024-09-21T19:11:20.911655+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:11:20.911655+0000","last_undegraded":"2024-09-21T19:11:20.911655+0000","last_fullsized":"2024-09-21T19:11:20.911655+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:42.815657+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"440'6559","reported_seq":6622,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:21.911665+0000","last_change":"2024-09-21T19:10:01.250343+0000","last_active":"2024-09-21T19:11:21.911665+0000","last_peered":"2024-09-21T19:11:21.911665+0000","last_clean":"2024-09-21T19:11:21.911665+0000","last_became_active":"2024-09-21T19:10:01.249931+0000","last_became_peered":"2024-09-21T19:10:01.249931+0000","last_unstale":"2024-09-21T19:11:21.911665+0000","last_undegraded":"2024-09-21T19:11:21.911665+0000","last_fullsized":"2024-09-21T19:11:21.911665+0000","mapping_epoch":414,"log_start":"423'1700","ondisk_log_start":"423'1700","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.706001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429850624,"num_objects":6559,"num_object_clones":0,"num_object_copies":19677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":419776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":704,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.917650+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:11:20.917650+0000","last_peered":"2024-09-21T19:11:20.917650+0000","last_clean":"2024-09-21T19:11:20.917650+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:11:20.917650+0000","last_undegraded":"2024-09-21T19:11:20.917650+0000","last_fullsized":"2024-09-21T19:11:20.917650+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.499806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":805,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.920127+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:11:20.920127+0000","last_peered":"2024-09-21T19:11:20.920127+0000","last_clean":"2024-09-21T19:11:20.920127+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:11:20.920127+0000","last_undegraded":"2024-09-21T19:11:20.920127+0000","last_fullsized":"2024-09-21T19:11:20.920127+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.953609+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":268,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906710+0000","last_change":"2024-09-21T19:10:27.397361+0000","last_active":"2024-09-21T19:11:20.906710+0000","last_peered":"2024-09-21T19:11:20.906710+0000","last_clean":"2024-09-21T19:11:20.906710+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:11:20.906710+0000","last_undegraded":"2024-09-21T19:11:20.906710+0000","last_fullsized":"2024-09-21T19:11:20.906710+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.397188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.397188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:57.228642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":708,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916016+0000","last_change":"2024-09-21T19:10:19.485820+0000","last_active":"2024-09-21T19:11:20.916016+0000","last_peered":"2024-09-21T19:11:20.916016+0000","last_clean":"2024-09-21T19:11:20.916016+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:11:20.916016+0000","last_undegraded":"2024-09-21T19:11:20.916016+0000","last_fullsized":"2024-09-21T19:11:20.916016+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:19.485588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.485588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:28.923857+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"440'6631","reported_seq":6694,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:21.936149+0000","last_change":"2024-09-21T19:10:01.260078+0000","last_active":"2024-09-21T19:11:21.936149+0000","last_peered":"2024-09-21T19:11:21.936149+0000","last_clean":"2024-09-21T19:11:21.936149+0000","last_became_active":"2024-09-21T19:10:01.259752+0000","last_became_peered":"2024-09-21T19:10:01.259752+0000","last_unstale":"2024-09-21T19:11:21.936149+0000","last_undegraded":"2024-09-21T19:11:21.936149+0000","last_fullsized":"2024-09-21T19:11:21.936149+0000","mapping_epoch":414,"log_start":"423'1700","ondisk_log_start":"423'1700","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.779638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434569216,"num_objects":6631,"num_object_clones":0,"num_object_copies":19893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":424384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":760,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786040+0000","last_change":"2024-09-21T19:10:12.835830+0000","last_active":"2024-09-21T19:11:14.786040+0000","last_peered":"2024-09-21T19:11:14.786040+0000","last_clean":"2024-09-21T19:11:14.786040+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:11:14.786040+0000","last_undegraded":"2024-09-21T19:11:14.786040+0000","last_fullsized":"2024-09-21T19:11:14.786040+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:10:12.835752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:12.835752+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:18.327395+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":844,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.908175+0000","last_change":"2024-09-21T19:10:29.539254+0000","last_active":"2024-09-21T19:11:20.908175+0000","last_peered":"2024-09-21T19:11:20.908175+0000","last_clean":"2024-09-21T19:11:20.908175+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:11:20.908175+0000","last_undegraded":"2024-09-21T19:11:20.908175+0000","last_fullsized":"2024-09-21T19:11:20.908175+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:29.539210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:29.539210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.989413+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":271,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.904556+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:11:20.904556+0000","last_peered":"2024-09-21T19:11:20.904556+0000","last_clean":"2024-09-21T19:11:20.904556+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:11:20.904556+0000","last_undegraded":"2024-09-21T19:11:20.904556+0000","last_fullsized":"2024-09-21T19:11:20.904556+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.430448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":824,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906837+0000","last_change":"2024-09-21T19:10:58.720745+0000","last_active":"2024-09-21T19:11:20.906837+0000","last_peered":"2024-09-21T19:11:20.906837+0000","last_clean":"2024-09-21T19:11:20.906837+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:11:20.906837+0000","last_undegraded":"2024-09-21T19:11:20.906837+0000","last_fullsized":"2024-09-21T19:11:20.906837+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.720634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.720634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.021358+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"440'6827","reported_seq":6889,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:22.893047+0000","last_change":"2024-09-21T19:10:01.252118+0000","last_active":"2024-09-21T19:11:22.893047+0000","last_peered":"2024-09-21T19:11:22.893047+0000","last_clean":"2024-09-21T19:11:22.893047+0000","last_became_active":"2024-09-21T19:10:01.251738+0000","last_became_peered":"2024-09-21T19:10:01.251738+0000","last_unstale":"2024-09-21T19:11:22.893047+0000","last_undegraded":"2024-09-21T19:11:22.893047+0000","last_fullsized":"2024-09-21T19:11:22.893047+0000","mapping_epoch":414,"log_start":"421'1300","ondisk_log_start":"421'1300","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5527,"log_dups_size":0,"ondisk_log_size":5527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.479895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447414272,"num_objects":6827,"num_object_clones":0,"num_object_copies":20481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":873,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.910969+0000","last_change":"2024-09-21T19:06:54.699608+0000","last_active":"2024-09-21T19:11:20.910969+0000","last_peered":"2024-09-21T19:11:20.910969+0000","last_clean":"2024-09-21T19:11:20.910969+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:11:20.910969+0000","last_undegraded":"2024-09-21T19:11:20.910969+0000","last_fullsized":"2024-09-21T19:11:20.910969+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:05:42.677420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.677420+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:29.883408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":846,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909538+0000","last_change":"2024-09-21T19:10:57.731366+0000","last_active":"2024-09-21T19:11:20.909538+0000","last_peered":"2024-09-21T19:11:20.909538+0000","last_clean":"2024-09-21T19:11:20.909538+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:11:20.909538+0000","last_undegraded":"2024-09-21T19:11:20.909538+0000","last_fullsized":"2024-09-21T19:11:20.909538+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:57.731319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.731319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.882041+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":268,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906886+0000","last_change":"2024-09-21T19:11:06.454799+0000","last_active":"2024-09-21T19:11:20.906886+0000","last_peered":"2024-09-21T19:11:20.906886+0000","last_clean":"2024-09-21T19:11:20.906886+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:11:20.906886+0000","last_undegraded":"2024-09-21T19:11:20.906886+0000","last_fullsized":"2024-09-21T19:11:20.906886+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:06.454727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.454727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.197782+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":796,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.920966+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:11:20.920966+0000","last_peered":"2024-09-21T19:11:20.920966+0000","last_clean":"2024-09-21T19:11:20.920966+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:11:20.920966+0000","last_undegraded":"2024-09-21T19:11:20.920966+0000","last_fullsized":"2024-09-21T19:11:20.920966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"440'6848","reported_seq":6979,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:22.787563+0000","last_change":"2024-09-21T19:11:09.208009+0000","last_active":"2024-09-21T19:11:22.787563+0000","last_peered":"2024-09-21T19:11:22.787563+0000","last_clean":"2024-09-21T19:11:22.787563+0000","last_became_active":"2024-09-21T19:10:01.259826+0000","last_became_peered":"2024-09-21T19:10:01.259826+0000","last_unstale":"2024-09-21T19:11:22.787563+0000","last_undegraded":"2024-09-21T19:11:22.787563+0000","last_fullsized":"2024-09-21T19:11:22.787563+0000","mapping_epoch":414,"log_start":"424'2000","ondisk_log_start":"424'2000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'5961","last_scrub_stamp":"2024-09-21T19:11:09.207825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.207825+0000","objects_scrubbed":5889,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.058679+0000","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448790528,"num_objects":6848,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6848,"num_write_kb":438272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":436,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906692+0000","last_change":"2024-09-21T19:10:20.596047+0000","last_active":"2024-09-21T19:11:20.906692+0000","last_peered":"2024-09-21T19:11:20.906692+0000","last_clean":"2024-09-21T19:11:20.906692+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:11:20.906692+0000","last_undegraded":"2024-09-21T19:11:20.906692+0000","last_fullsized":"2024-09-21T19:11:20.906692+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:10:20.595995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:20.595995+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:26.129621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":852,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.908163+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:11:20.908163+0000","last_peered":"2024-09-21T19:11:20.908163+0000","last_clean":"2024-09-21T19:11:20.908163+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:11:20.908163+0000","last_undegraded":"2024-09-21T19:11:20.908163+0000","last_fullsized":"2024-09-21T19:11:20.908163+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:31.541007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":342,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914077+0000","last_change":"2024-09-21T19:10:59.827898+0000","last_active":"2024-09-21T19:11:20.914077+0000","last_peered":"2024-09-21T19:11:20.914077+0000","last_clean":"2024-09-21T19:11:20.914077+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:11:20.914077+0000","last_undegraded":"2024-09-21T19:11:20.914077+0000","last_fullsized":"2024-09-21T19:11:20.914077+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:10:59.827833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.827833+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.990995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":837,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.785828+0000","last_change":"2024-09-21T19:10:59.672382+0000","last_active":"2024-09-21T19:11:14.785828+0000","last_peered":"2024-09-21T19:11:14.785828+0000","last_clean":"2024-09-21T19:11:14.785828+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:11:14.785828+0000","last_undegraded":"2024-09-21T19:11:14.785828+0000","last_fullsized":"2024-09-21T19:11:14.785828+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:59.672320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.672320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:59.920295+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":772,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.785629+0000","last_change":"2024-09-21T19:11:08.539733+0000","last_active":"2024-09-21T19:11:14.785629+0000","last_peered":"2024-09-21T19:11:14.785629+0000","last_clean":"2024-09-21T19:11:14.785629+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:11:14.785629+0000","last_undegraded":"2024-09-21T19:11:14.785629+0000","last_fullsized":"2024-09-21T19:11:14.785629+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.539689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.539689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:35.735538+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":837,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915725+0000","last_change":"2024-09-21T19:10:24.873768+0000","last_active":"2024-09-21T19:11:20.915725+0000","last_peered":"2024-09-21T19:11:20.915725+0000","last_clean":"2024-09-21T19:11:20.915725+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:11:20.915725+0000","last_undegraded":"2024-09-21T19:11:20.915725+0000","last_fullsized":"2024-09-21T19:11:20.915725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:10:24.873701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:24.873701+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.273198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":365,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.913937+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:11:20.913937+0000","last_peered":"2024-09-21T19:11:20.913937+0000","last_clean":"2024-09-21T19:11:20.913937+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:11:20.913937+0000","last_undegraded":"2024-09-21T19:11:20.913937+0000","last_fullsized":"2024-09-21T19:11:20.913937+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.027866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":812,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907635+0000","last_change":"2024-09-21T19:10:22.627798+0000","last_active":"2024-09-21T19:11:20.907635+0000","last_peered":"2024-09-21T19:11:20.907635+0000","last_clean":"2024-09-21T19:11:20.907635+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:11:20.907635+0000","last_undegraded":"2024-09-21T19:11:20.907635+0000","last_fullsized":"2024-09-21T19:11:20.907635+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:22.627740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:22.627740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:30.537034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":712,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916682+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:11:20.916682+0000","last_peered":"2024-09-21T19:11:20.916682+0000","last_clean":"2024-09-21T19:11:20.916682+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:11:20.916682+0000","last_undegraded":"2024-09-21T19:11:20.916682+0000","last_fullsized":"2024-09-21T19:11:20.916682+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.467919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":824,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907190+0000","last_change":"2024-09-21T19:10:18.539646+0000","last_active":"2024-09-21T19:11:20.907190+0000","last_peered":"2024-09-21T19:11:20.907190+0000","last_clean":"2024-09-21T19:11:20.907190+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:11:20.907190+0000","last_undegraded":"2024-09-21T19:11:20.907190+0000","last_fullsized":"2024-09-21T19:11:20.907190+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:18.539586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:18.539586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":807,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915513+0000","last_change":"2024-09-21T19:10:46.660901+0000","last_active":"2024-09-21T19:11:20.915513+0000","last_peered":"2024-09-21T19:11:20.915513+0000","last_clean":"2024-09-21T19:11:20.915513+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:11:20.915513+0000","last_undegraded":"2024-09-21T19:11:20.915513+0000","last_fullsized":"2024-09-21T19:11:20.915513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.660838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.660838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:07.972413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":260,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.905648+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:11:20.905648+0000","last_peered":"2024-09-21T19:11:20.905648+0000","last_clean":"2024-09-21T19:11:20.905648+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:11:20.905648+0000","last_undegraded":"2024-09-21T19:11:20.905648+0000","last_fullsized":"2024-09-21T19:11:20.905648+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:40.410888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":418,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907861+0000","last_change":"2024-09-21T19:06:54.692561+0000","last_active":"2024-09-21T19:11:20.907861+0000","last_peered":"2024-09-21T19:11:20.907861+0000","last_clean":"2024-09-21T19:11:20.907861+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:11:20.907861+0000","last_undegraded":"2024-09-21T19:11:20.907861+0000","last_fullsized":"2024-09-21T19:11:20.907861+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:05:00.125117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:00.125117+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:30.574749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":701,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.919341+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:11:20.919341+0000","last_peered":"2024-09-21T19:11:20.919341+0000","last_clean":"2024-09-21T19:11:20.919341+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:11:20.919341+0000","last_undegraded":"2024-09-21T19:11:20.919341+0000","last_fullsized":"2024-09-21T19:11:20.919341+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:37.462446+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1058,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.919736+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:11:20.919736+0000","last_peered":"2024-09-21T19:11:20.919736+0000","last_clean":"2024-09-21T19:11:20.919736+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:11:20.919736+0000","last_undegraded":"2024-09-21T19:11:20.919736+0000","last_fullsized":"2024-09-21T19:11:20.919736+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":827,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911817+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:11:20.911817+0000","last_peered":"2024-09-21T19:11:20.911817+0000","last_clean":"2024-09-21T19:11:20.911817+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:11:20.911817+0000","last_undegraded":"2024-09-21T19:11:20.911817+0000","last_fullsized":"2024-09-21T19:11:20.911817+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":852,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.919940+0000","last_change":"2024-09-21T19:10:39.003954+0000","last_active":"2024-09-21T19:11:20.919940+0000","last_peered":"2024-09-21T19:11:20.919940+0000","last_clean":"2024-09-21T19:11:20.919940+0000","last_became_active":"2024-09-21T19:10:38.923267+0000","last_became_peered":"2024-09-21T19:10:38.923267+0000","last_unstale":"2024-09-21T19:11:20.919940+0000","last_undegraded":"2024-09-21T19:11:20.919940+0000","last_fullsized":"2024-09-21T19:11:20.919940+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:19.876033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.876033+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:26.310986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":835,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907992+0000","last_change":"2024-09-21T19:10:46.686915+0000","last_active":"2024-09-21T19:11:20.907992+0000","last_peered":"2024-09-21T19:11:20.907992+0000","last_clean":"2024-09-21T19:11:20.907992+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:11:20.907992+0000","last_undegraded":"2024-09-21T19:11:20.907992+0000","last_fullsized":"2024-09-21T19:11:20.907992+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.686843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.686843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.453957+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":508,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909396+0000","last_change":"2024-09-21T19:10:18.103484+0000","last_active":"2024-09-21T19:11:20.909396+0000","last_peered":"2024-09-21T19:11:20.909396+0000","last_clean":"2024-09-21T19:11:20.909396+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:11:20.909396+0000","last_undegraded":"2024-09-21T19:11:20.909396+0000","last_fullsized":"2024-09-21T19:11:20.909396+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:10:18.103406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:18.103406+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:30.133249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":701,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916952+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:11:20.916952+0000","last_peered":"2024-09-21T19:11:20.916952+0000","last_clean":"2024-09-21T19:11:20.916952+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:11:20.916952+0000","last_undegraded":"2024-09-21T19:11:20.916952+0000","last_fullsized":"2024-09-21T19:11:20.916952+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.496959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":898,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911219+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:11:20.911219+0000","last_peered":"2024-09-21T19:11:20.911219+0000","last_clean":"2024-09-21T19:11:20.911219+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:11:20.911219+0000","last_undegraded":"2024-09-21T19:11:20.911219+0000","last_fullsized":"2024-09-21T19:11:20.911219+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:37.861523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":817,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.911719+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:11:20.911719+0000","last_peered":"2024-09-21T19:11:20.911719+0000","last_clean":"2024-09-21T19:11:20.911719+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:11:20.911719+0000","last_undegraded":"2024-09-21T19:11:20.911719+0000","last_fullsized":"2024-09-21T19:11:20.911719+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.166063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":825,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909782+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:11:20.909782+0000","last_peered":"2024-09-21T19:11:20.909782+0000","last_clean":"2024-09-21T19:11:20.909782+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:11:20.909782+0000","last_undegraded":"2024-09-21T19:11:20.909782+0000","last_fullsized":"2024-09-21T19:11:20.909782+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:39.125799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":828,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914253+0000","last_change":"2024-09-21T19:10:27.770820+0000","last_active":"2024-09-21T19:11:20.914253+0000","last_peered":"2024-09-21T19:11:20.914253+0000","last_clean":"2024-09-21T19:11:20.914253+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:11:20.914253+0000","last_undegraded":"2024-09-21T19:11:20.914253+0000","last_fullsized":"2024-09-21T19:11:20.914253+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.770751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.770751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:45.709466+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":904,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.920557+0000","last_change":"2024-09-21T19:11:06.137061+0000","last_active":"2024-09-21T19:11:20.920557+0000","last_peered":"2024-09-21T19:11:20.920557+0000","last_clean":"2024-09-21T19:11:20.920557+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:11:20.920557+0000","last_undegraded":"2024-09-21T19:11:20.920557+0000","last_fullsized":"2024-09-21T19:11:20.920557+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:06.136973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.136973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.021915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":829,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907945+0000","last_change":"2024-09-21T19:11:07.755091+0000","last_active":"2024-09-21T19:11:20.907945+0000","last_peered":"2024-09-21T19:11:20.907945+0000","last_clean":"2024-09-21T19:11:20.907945+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:11:20.907945+0000","last_undegraded":"2024-09-21T19:11:20.907945+0000","last_fullsized":"2024-09-21T19:11:20.907945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:07.755026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.755026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.561489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":821,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.921210+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:11:20.921210+0000","last_peered":"2024-09-21T19:11:20.921210+0000","last_clean":"2024-09-21T19:11:20.921210+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:11:20.921210+0000","last_undegraded":"2024-09-21T19:11:20.921210+0000","last_fullsized":"2024-09-21T19:11:20.921210+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":712,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915417+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:11:20.915417+0000","last_peered":"2024-09-21T19:11:20.915417+0000","last_clean":"2024-09-21T19:11:20.915417+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:11:20.915417+0000","last_undegraded":"2024-09-21T19:11:20.915417+0000","last_fullsized":"2024-09-21T19:11:20.915417+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.411163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":845,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914442+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:11:20.914442+0000","last_peered":"2024-09-21T19:11:20.914442+0000","last_clean":"2024-09-21T19:11:20.914442+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:11:20.914442+0000","last_undegraded":"2024-09-21T19:11:20.914442+0000","last_fullsized":"2024-09-21T19:11:20.914442+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:35.695151+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":766,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.903273+0000","last_change":"2024-09-21T19:11:03.004430+0000","last_active":"2024-09-21T19:11:20.903273+0000","last_peered":"2024-09-21T19:11:20.903273+0000","last_clean":"2024-09-21T19:11:20.903273+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:11:20.903273+0000","last_undegraded":"2024-09-21T19:11:20.903273+0000","last_fullsized":"2024-09-21T19:11:20.903273+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.004386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.004386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.746086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":848,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.906565+0000","last_change":"2024-09-21T19:11:01.724461+0000","last_active":"2024-09-21T19:11:20.906565+0000","last_peered":"2024-09-21T19:11:20.906565+0000","last_clean":"2024-09-21T19:11:20.906565+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:11:20.906565+0000","last_undegraded":"2024-09-21T19:11:20.906565+0000","last_fullsized":"2024-09-21T19:11:20.906565+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:01.724417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:01.724417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:02.723695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":828,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.921335+0000","last_change":"2024-09-21T19:10:16.861750+0000","last_active":"2024-09-21T19:11:20.921335+0000","last_peered":"2024-09-21T19:11:20.921335+0000","last_clean":"2024-09-21T19:11:20.921335+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:11:20.921335+0000","last_undegraded":"2024-09-21T19:11:20.921335+0000","last_fullsized":"2024-09-21T19:11:20.921335+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.861671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.861671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.883789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":852,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915418+0000","last_change":"2024-09-21T19:10:52.788414+0000","last_active":"2024-09-21T19:11:20.915418+0000","last_peered":"2024-09-21T19:11:20.915418+0000","last_clean":"2024-09-21T19:11:20.915418+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:11:20.915418+0000","last_undegraded":"2024-09-21T19:11:20.915418+0000","last_fullsized":"2024-09-21T19:11:20.915418+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:10:52.788342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:52.788342+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:04.702000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":704,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.914762+0000","last_change":"2024-09-21T19:11:02.421016+0000","last_active":"2024-09-21T19:11:20.914762+0000","last_peered":"2024-09-21T19:11:20.914762+0000","last_clean":"2024-09-21T19:11:20.914762+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:11:20.914762+0000","last_undegraded":"2024-09-21T19:11:20.914762+0000","last_fullsized":"2024-09-21T19:11:20.914762+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:02.420940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.420940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.532139+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":731,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.785762+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:11:14.785762+0000","last_peered":"2024-09-21T19:11:14.785762+0000","last_clean":"2024-09-21T19:11:14.785762+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:11:14.785762+0000","last_undegraded":"2024-09-21T19:11:14.785762+0000","last_fullsized":"2024-09-21T19:11:14.785762+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":823,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.907180+0000","last_change":"2024-09-21T19:11:09.805596+0000","last_active":"2024-09-21T19:11:20.907180+0000","last_peered":"2024-09-21T19:11:20.907180+0000","last_clean":"2024-09-21T19:11:20.907180+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:11:20.907180+0000","last_undegraded":"2024-09-21T19:11:20.907180+0000","last_fullsized":"2024-09-21T19:11:20.907180+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:09.805527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.805527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:24.581512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":821,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.909239+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:11:20.909239+0000","last_peered":"2024-09-21T19:11:20.909239+0000","last_clean":"2024-09-21T19:11:20.909239+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:11:20.909239+0000","last_undegraded":"2024-09-21T19:11:20.909239+0000","last_fullsized":"2024-09-21T19:11:20.909239+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.129647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":837,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.915660+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:11:20.915660+0000","last_peered":"2024-09-21T19:11:20.915660+0000","last_clean":"2024-09-21T19:11:20.915660+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:11:20.915660+0000","last_undegraded":"2024-09-21T19:11:20.915660+0000","last_fullsized":"2024-09-21T19:11:20.915660+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":813,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-21T19:11:14.786396+0000","last_change":"2024-09-21T19:11:00.629261+0000","last_active":"2024-09-21T19:11:14.786396+0000","last_peered":"2024-09-21T19:11:14.786396+0000","last_clean":"2024-09-21T19:11:14.786396+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:11:14.786396+0000","last_undegraded":"2024-09-21T19:11:14.786396+0000","last_fullsized":"2024-09-21T19:11:14.786396+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:00.629191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.629191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.169952+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":724,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T19:11:20.916863+0000","last_change":"2024-09-21T19:10:23.500631+0000","last_active":"2024-09-21T19:11:20.916863+0000","last_peered":"2024-09-21T19:11:20.916863+0000","last_clean":"2024-09-21T19:11:20.916863+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:11:20.916863+0000","last_undegraded":"2024-09-21T19:11:20.916863+0000","last_fullsized":"2024-09-21T19:11:20.916863+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:23.500562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.500562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:42.786621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7010385920,"num_objects":106970,"num_object_clones":0,"num_object_copies":320910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106970,"num_write_kb":6846080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21006450688,"data_stored":21006450688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80270,"ondisk_log_size":80270,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":5446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1083,"num_read_kb":1078,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266911,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3187092,"kb_used_data":2542272,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":90558060,"statfs":{"total":95995035648,"available":92731453440,"internally_reserved":0,"allocated":2603286528,"data_stored":2602461103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":660273610},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464361,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3698868,"kb_used_data":3293412,"kb_used_omap":4,"kb_used_meta":405435,"kb_avail":90046284,"statfs":{"total":95995035648,"available":92207394816,"internally_reserved":0,"allocated":3372453888,"data_stored":3371649464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":415166116},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619063,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043532,"kb_used_data":1729208,"kb_used_omap":4,"kb_used_meta":314299,"kb_avail":91701620,"statfs":{"total":95995035648,"available":93902458880,"internally_reserved":0,"allocated":1770708992,"data_stored":1769891784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":321842851},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019117,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2278492,"kb_used_data":1701704,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91466660,"statfs":{"total":95995035648,"available":93661859840,"internally_reserved":0,"allocated":1742544896,"data_stored":1741711779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182642,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3824616,"kb_used_data":3009048,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":89920536,"statfs":{"total":95995035648,"available":92078628864,"internally_reserved":0,"allocated":3081265152,"data_stored":3080416494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411576,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3197372,"kb_used_data":2574764,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90547780,"statfs":{"total":95995035648,"available":92720926720,"internally_reserved":0,"allocated":2636558336,"data_stored":2635753421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4227128,"kb_used_data":3433764,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":89518024,"statfs":{"total":95995035648,"available":91666456576,"internally_reserved":0,"allocated":3516174336,"data_stored":3515312950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738626,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3015060,"kb_used_data":2149508,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90730092,"statfs":{"total":95995035648,"available":92907614208,"internally_reserved":0,"allocated":2201096192,"data_stored":2200267360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2208038912,"data_stored":2208038912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3544055808,"data_stored":3544055808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2651258880,"data_stored":2651258880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3079995392,"data_stored":3079995392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1745551360,"data_stored":1745551360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1773666304,"data_stored":1773666304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3396927488,"data_stored":3396927488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2606956544,"data_stored":2606956544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:11:27.661 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-21T19:11:27.707 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:27.740 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:27.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-21T19:11:27.760672+0000 min lat: 0.00122502 max lat: 0.53624 avg lat: 0.0113204 2024-09-21T19:11:27.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T19:11:27.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 80 16 113085 113069 88.3011 71.75 0.0158427 0.0113204 2024-09-21T19:11:27.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:27.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:27.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:27.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:28.020 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:28.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:28.130 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:11:28.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:28.256 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:27 smithi104 ceph-mon[32438]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T19:11:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:27 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:28.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:27 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:28.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-mon[39192]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T19:11:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-mon[29384]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T19:11:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:27 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:28.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.433 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:28.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:28.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:28.567 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.578 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:28.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:28.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:28.707 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:28.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 81 16 114308 114292 88.1546 76.4375 0.00465988 0.0113395 2024-09-21T19:11:28.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:28.848 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:28.899 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:28.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:28.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:11:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:11:28.980 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:28.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:29.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:29.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:29.150 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:29.150 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:11:29.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.150 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:114747 11.e 11:767c5098:::benchmark_data_smithi081_181703_object114746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:29.142967+0000", 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0043134870000000004, 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043450069999999997, 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:29.151 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 114747 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.142967+0000", 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.142967+0000", 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.142969+0000", 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5710000000000001e-06 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.143022+0000", 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2768e-05 2024-09-21T19:11:29.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.143031+0000", 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1160000000000005e-06 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.143036+0000", 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8389999999999997e-06 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.143089+0000", 2024-09-21T19:11:29.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3236000000000003e-05 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.143219+0000", 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013050299999999999 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.143357+0000", 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013730400000000001 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.146488+0000", 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031315190000000001 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:29.146677+0000", 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018859000000000001 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 1 2024-09-21T19:11:29.157 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:29.184 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:28 smithi081 ceph-mon[39192]: pgmap v908: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-09-21T19:11:29.184 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:28 smithi081 ceph-mon[39192]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T19:11:29.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:28 smithi081 ceph-mon[29384]: pgmap v908: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-09-21T19:11:29.185 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:28 smithi081 ceph-mon[29384]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T19:11:29.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:29.246 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:28 smithi104 ceph-mon[32438]: pgmap v908: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-09-21T19:11:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:28 smithi104 ceph-mon[32438]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T19:11:29.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:29.373 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:29.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.521 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:29.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:29.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:29.654 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:29.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:29.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 82 16 115427 115411 87.9316 69.9375 0.0118411 0.0113676 2024-09-21T19:11:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.795 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:29.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:29.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:29.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:29.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:11:30.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:30.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:29 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/657961699' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:11:30.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:29 smithi104 ceph-mon[32438]: from='client.26461 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:11:30.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:30.074 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:30.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.219 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:30.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:29 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/657961699' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:11:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:29 smithi081 ceph-mon[29384]: from='client.26461 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:11:30.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:30.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:29 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/657961699' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:11:30.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:29 smithi081 ceph-mon[39192]: from='client.26461 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:11:30.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:30.361 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:30.404 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:30.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:30.495 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:30.597 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.635 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:30.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:30.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:30.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 83 16 116657 116641 87.7984 76.875 0.0143865 0.0113847 2024-09-21T19:11:30.764 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:30.813 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:11:30.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:30.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:11:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:30.902 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:30.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:30.967 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:30.967 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:11:30.967 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:11:30.967 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:11:30.967 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:31.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:31.051 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:31.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[29384]: pgmap v910: 158 pgs: 158 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 787 B/s rd, 60 MiB/s wr, 958 op/s 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[29384]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[29384]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[39192]: pgmap v910: 158 pgs: 158 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 787 B/s rd, 60 MiB/s wr, 958 op/s 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:31.227 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[39192]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:11:31.228 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:30 smithi081 ceph-mon[39192]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T19:11:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.232 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:31.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:30 smithi104 ceph-mon[32438]: pgmap v910: 158 pgs: 158 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 787 B/s rd, 60 MiB/s wr, 958 op/s 2024-09-21T19:11:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:30 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:30 smithi104 ceph-mon[32438]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:11:31.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:30 smithi104 ceph-mon[32438]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T19:11:31.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:31.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:31.494 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:31.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:31.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.757 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:31.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:31.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 84 16 117710 117694 87.5366 65.8125 0.0200619 0.0114192 2024-09-21T19:11:31.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:11:31.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:31.897 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:31.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:31.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:32.009 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: 4.9 scrub starts 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: 4.9 scrub ok 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: 6.9 scrub starts 2024-09-21T19:11:32.010 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:31 smithi104 ceph-mon[32438]: 6.9 scrub ok 2024-09-21T19:11:32.042 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:32.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.177 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:32.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:32.213 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:32.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:32.299 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: 4.9 scrub starts 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: 4.9 scrub ok 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: 6.9 scrub starts 2024-09-21T19:11:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[39192]: 6.9 scrub ok 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: 4.9 scrub starts 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: 4.9 scrub ok 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: 6.9 scrub starts 2024-09-21T19:11:32.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:31 smithi081 ceph-mon[29384]: 6.9 scrub ok 2024-09-21T19:11:32.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:32.434 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:32.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:32.567 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:32.662 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:11:32.662 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:11:32.662 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T19:11:32.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:11:32.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.691 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:32.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:32.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 85 16 118759 118743 87.2781 65.5625 0.0112787 0.0114527 2024-09-21T19:11:32.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:32.813 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:32.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:32.944 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:32.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:33.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: 5.c scrub starts 2024-09-21T19:11:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: 5.c scrub ok 2024-09-21T19:11:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: pgmap v912: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 985 op/s 2024-09-21T19:11:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: 5.3 scrub starts 2024-09-21T19:11:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: 5.3 scrub ok 2024-09-21T19:11:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T19:11:33.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:32 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T19:11:33.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:33.085 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:33.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:33.223 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: 5.c scrub starts 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: 5.c scrub ok 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: pgmap v912: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 985 op/s 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: 5.3 scrub starts 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: 5.3 scrub ok 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: 5.c scrub starts 2024-09-21T19:11:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: 5.c scrub ok 2024-09-21T19:11:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: pgmap v912: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 985 op/s 2024-09-21T19:11:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: 5.3 scrub starts 2024-09-21T19:11:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: 5.3 scrub ok 2024-09-21T19:11:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T19:11:33.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:32 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T19:11:33.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:33.348 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:33.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.490 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:33.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:33.591 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.620 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:33.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:33.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:33.744 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:33.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 86 16 119520 119504 86.8164 47.5625 0.00729911 0.0115132 2024-09-21T19:11:33.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:33.880 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:33.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:33.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:34.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235896 8.22 8:45adf4b4:::benchmark_data_smithi081_105538_object103664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 579.19421790299998, 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.039943074, 2024-09-21T19:11:34.069 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235896 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675140+0000", 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675143+0000", 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.729e-06 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675150+0000", 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6820000000000003e-06 2024-09-21T19:11:34.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675168+0000", 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7875000000000001e-05 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675178+0000", 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5610000000000004e-06 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731748+0000", 2024-09-21T19:11:34.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056570691999999999 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731754+0000", 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.976e-06 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.700981+0000", 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96922663099999995 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701110+0000", 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012864500000000001 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.701320+0000", 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021059200000000001 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702216+0000", 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089577400000000001 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.702245+0000", 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8608000000000001e-05 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.714979+0000", 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012734354999999999 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715038+0000", 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8588999999999998e-05 2024-09-21T19:11:34.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.715083+0000", 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5364999999999997e-05 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:235898 8.2 8:4337194e:::benchmark_data_smithi081_105538_object103666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 579.19416452500002, 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0181600850000001, 2024-09-21T19:11:34.076 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 235898 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675193+0000", 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675194+0000", 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1699999999999997e-07 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675197+0000", 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8559999999999998e-06 2024-09-21T19:11:34.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675203+0000", 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2369999999999997e-06 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.675208+0000", 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9989999999999999e-06 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731592+0000", 2024-09-21T19:11:34.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.056383961000000003 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:53.731604+0000", 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2278e-05 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685684+0000", 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.95407945800000005 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.685861+0000", 2024-09-21T19:11:34.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017756400000000001 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.686082+0000", 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022110200000000001 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.687697+0000", 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016145230000000001 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.690382+0000", 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026848850000000001 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693261+0000", 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879291 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693314+0000", 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2845e-05 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:54.693353+0000", 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9369000000000002e-05 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237679 8.1b 8:deadaa1e:::benchmark_data_smithi081_105538_object105447:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 577.18471224500001, 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0964987129999999, 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.083 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237679 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684646+0000", 2024-09-21T19:11:34.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684647+0000", 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.153e-06 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684651+0000", 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3370000000000001e-06 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684660+0000", 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6470000000000004e-06 2024-09-21T19:11:34.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684667+0000", 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3089999999999998e-06 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684783+0000", 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011565600000000001 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684791+0000", 2024-09-21T19:11:34.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7809999999999994e-06 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691384+0000", 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0065922899999999 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691392+0000", 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.636e-06 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781063+0000", 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0896703569999999 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781144+0000", 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1546999999999997e-05 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237686 8.1b 8:df3678b5:::benchmark_data_smithi081_105538_object105454:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 577.18467318800003, 2024-09-21T19:11:34.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.173105171, 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237686 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684685+0000", 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1299999999999999e-07 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684688+0000", 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5830000000000001e-06 2024-09-21T19:11:34.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684693+0000", 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5910000000000003e-06 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684699+0000", 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2900000000000002e-06 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684822+0000", 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012324499999999999 2024-09-21T19:11:34.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684830+0000", 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6750000000000002e-06 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691420+0000", 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006590262 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691426+0000", 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4559999999999997e-06 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781412+0000", 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.089985687 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for missing object", 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781478+0000", 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6272000000000005e-05 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857711+0000", 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.076232366999999995 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857790+0000", 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9029999999999994e-05 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237687 8.1b 8:dc29a777:::benchmark_data_smithi081_105538_object105455:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 577.18465285000002, 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1710659049999999, 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.094 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237687 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:11:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684705+0000", 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4899999999999995e-07 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684708+0000", 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1339999999999998e-06 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684713+0000", 2024-09-21T19:11:34.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9749999999999998e-06 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684734+0000", 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1206e-05 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684861+0000", 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012699799999999999 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.684870+0000", 2024-09-21T19:11:34.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9770000000000006e-06 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691453+0000", 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.006583418 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691459+0000", 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2580000000000001e-06 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781524+0000", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0900641010000001 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781643+0000", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119735 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.781902+0000", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025915999999999998 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784476+0000", 2024-09-21T19:11:34.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025738229999999998 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784488+0000", 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1401e-05 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784513+0000", 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5462000000000001e-05 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855646+0000", 2024-09-21T19:11:34.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071132476 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855686+0000", 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0157999999999997e-05 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855771+0000", 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5074000000000006e-05 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237708 8.1b 8:dbdbeb85:::benchmark_data_smithi081_105538_object105476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 577.16496525399998, 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.151669386, 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237708 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704393+0000", 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704394+0000", 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.855e-06 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704399+0000", 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9699999999999998e-06 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704413+0000", 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3560000000000001e-05 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704421+0000", 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6399999999999997e-06 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704473+0000", 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.27e-05 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.704483+0000", 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4520000000000003e-06 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691484+0000", 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98700145800000005 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691491+0000", 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782825+0000", 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0913340810000001 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.782935+0000", 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110647 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.783135+0000", 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020000399999999999 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784491+0000", 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013554319999999999 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784543+0000", 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2609999999999999e-05 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784566+0000", 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2945000000000001e-05 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855871+0000", 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071304949000000006 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.855929+0000", 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7924999999999999e-05 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.856062+0000", 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013279700000000001 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.35258.0:237711 8.1b 8:dd96cf97:::benchmark_data_smithi081_105538_object105479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 577.16276566700003, 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1506975869999998, 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.109 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.35258", 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/2691508306", 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 237711 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706592+0000", 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706594+0000", 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9099999999999999e-06 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706599+0000", 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0259999999999998e-06 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706618+0000", 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8499e-05 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706633+0000", 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5931999999999999e-05 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706751+0000", 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117395 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:55.706762+0000", 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1316e-05 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691770+0000", 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.985007619 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:56.691776+0000", 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5490000000000003e-06 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784627+0000", 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0928504649999999 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784699+0000", 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2539000000000005e-05 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2,3", 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.784817+0000", 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000118029 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785805+0000", 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098732000000000008 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785853+0000", 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8554000000000003e-05 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.785945+0000", 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1835000000000004e-05 2024-09-21T19:11:34.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857195+0000", 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.071249589000000002 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857226+0000", 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1705999999999998e-05 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.857290+0000", 2024-09-21T19:11:34.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3304000000000002e-05 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2026 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94695648315648 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 575.59696786100005, 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46135494399999999, 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:11:34.117 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2026 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272390+0000", 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272393+0000", 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4999999999999999e-06 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272403+0000", 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9019999999999994e-06 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272421+0000", 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7499999999999998e-05 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272431+0000", 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0607e-05 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272504+0000", 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2151999999999996e-05 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272515+0000", 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1241e-05 2024-09-21T19:11:34.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702573+0000", 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.430058259 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.702717+0000", 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143931 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.733745+0000", 2024-09-21T19:11:34.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.031027852000000002 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2027 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 575.596900162, 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.464594917, 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.122 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2027 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272458+0000", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4000000000000001e-07 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272462+0000", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4199999999999999e-06 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272469+0000", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272474+0000", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4249999999999997e-06 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272539+0000", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5066000000000003e-05 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.272549+0000", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8709999999999992e-06 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704278+0000", 2024-09-21T19:11:34.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.43172896500000002 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704401+0000", 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012265999999999999 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.737053+0000", 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.032651612000000003 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.14514.0:2036 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94695651780864 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 575.31554022199998, 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15105808700000001, 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.14514", 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/604236037", 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 2036 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.127 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553818+0000", 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0399999999999996e-07 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553821+0000", 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6769999999999999e-06 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553827+0000", 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8000000000000004e-06 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553831+0000", 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6550000000000003e-06 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553873+0000", 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1241e-05 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.553879+0000", 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5429999999999999e-06 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704657+0000", 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.150777833 2024-09-21T19:11:34.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704730+0000", 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3398000000000004e-05 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:01:57.704876+0000", 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014533600000000001 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.15816.0:76067 9.c e280/254 9:3b8d2f8b:::benchmark_data_smithi081_131274_object76066:head v 280'4754, mlcod=280'4748)", 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 489.52397221799998, 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.23228328300000001, 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345386+0000", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.345387+0000", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5460000000000001e-06 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570553+0000", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.22516550099999999 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570563+0000", 2024-09-21T19:11:34.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0163000000000001e-05 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570569+0000", 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7660000000000004e-06 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570644+0000", 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5710999999999994e-05 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.570678+0000", 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3608000000000001e-05 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577637+0000", 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069593909999999997 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:03:23.577669+0000", 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1597000000000003e-05 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.135 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:29887 10.1 10:85f2786f:::benchmark_data_smithi081_155596_object29886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 307.04525229799998, 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12143838799999999, 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 29887 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.136 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824105+0000", 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824107+0000", 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.767e-06 2024-09-21T19:11:34.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.824151+0000", 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4229e-05 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933322+0000", 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.10917025 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.933345+0000", 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3669999999999999e-05 2024-09-21T19:11:34.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934070+0000", 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072502099999999996 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934269+0000", 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019867600000000001 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.934418+0000", 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014940099999999999 2024-09-21T19:11:34.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.940623+0000", 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062042729999999997 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.940648+0000", 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5347999999999999e-05 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945384+0000", 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047363209999999999 2024-09-21T19:11:34.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945492+0000", 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000107899 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:25.945544+0000", 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1533000000000003e-05 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.141 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31276 10.e 10:78850574:::benchmark_data_smithi081_155596_object31275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 306.25803597599997, 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20212575599999999, 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31276 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611322+0000", 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611324+0000", 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9740000000000001e-06 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611355+0000", 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1634999999999998e-05 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611367+0000", 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1613999999999999e-05 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611374+0000", 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8029999999999999e-06 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611557+0000", 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018305700000000001 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611683+0000", 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012569999999999999 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.611833+0000", 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015060799999999999 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.613723+0000", 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00188988 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614412+0000", 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068917700000000002 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813312+0000", 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.198899348 2024-09-21T19:11:34.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813403+0000", 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1162000000000006e-05 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813448+0000", 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4798e-05 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31282 10.e 10:7508a8e6:::benchmark_data_smithi081_155596_object31281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 306.25500161600002, 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19924375599999999, 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31282 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614356+0000", 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614359+0000", 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4099999999999998e-06 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614395+0000", 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6782999999999997e-05 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614409+0000", 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3772e-05 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614417+0000", 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9030000000000004e-06 2024-09-21T19:11:34.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614476+0000", 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8980000000000001e-05 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614640+0000", 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016440500000000001 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 2,6", 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.614800+0000", 2024-09-21T19:11:34.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015944200000000001 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.615994+0000", 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001193784 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.616143+0000", 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000149013 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813485+0000", 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.197342778 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813565+0000", 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9160999999999996e-05 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:26.813600+0000", 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5324999999999998e-05 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:31755 10.1 10:87c30842:::benchmark_data_smithi081_155596_object31754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 305.859074663, 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.163083229, 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 31755 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.154 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010283+0000", 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010285+0000", 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8959999999999999e-06 2024-09-21T19:11:34.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010383+0000", 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7940000000000001e-05 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010393+0000", 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0414999999999999e-05 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010409+0000", 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5746000000000001e-05 2024-09-21T19:11:34.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010487+0000", 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8318000000000005e-05 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010631+0000", 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000143502 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,7", 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.010769+0000", 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013836 2024-09-21T19:11:34.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.015045+0000", 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042753640000000002 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.015059+0000", 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4182000000000001e-05 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173282+0000", 2024-09-21T19:11:34.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.15822359799999999 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173328+0000", 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5524e-05 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:06:27.173366+0000", 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8383999999999997e-05 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:65428 11.e 11:7d7e10f4:::benchmark_data_smithi081_181703_object65427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:10:44.550281+0000", 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.319077166, 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13865825200000001, 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 65428 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.550281+0000", 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.550281+0000", 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.550282+0000", 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.063e-06 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.550354+0000", 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2575000000000002e-05 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.550364+0000", 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0134e-05 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.550371+0000", 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6490000000000002e-06 2024-09-21T19:11:34.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.552669+0000", 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022975830000000002 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.552803+0000", 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013404399999999999 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.552939+0000", 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013650700000000001 2024-09-21T19:11:34.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.554839+0000", 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001900234 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.557695+0000", 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002855865 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.688869+0000", 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13117352900000001 2024-09-21T19:11:34.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.688908+0000", 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9422999999999998e-05 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:10:44.688939+0000", 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0645999999999998e-05 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.165 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:95293 11.e 11:70e6721f:::benchmark_data_smithi081_181703_object95292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:10.437545+0000", 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.431812395000001, 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.136412901, 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95293 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.437545+0000", 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.437545+0000", 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.437547+0000", 2024-09-21T19:11:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.703e-06 2024-09-21T19:11:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.437613+0000", 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6078e-05 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.437626+0000", 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.287e-05 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.437634+0000", 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6290000000000001e-06 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441808+0000", 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041741970000000002 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441903+0000", 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4765000000000005e-05 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441990+0000", 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7637000000000001e-05 2024-09-21T19:11:34.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.445257+0000", 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032670070000000002 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.448069+0000", 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028114030000000001 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.573871+0000", 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12580192300000001 2024-09-21T19:11:34.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.573914+0000", 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3223999999999998e-05 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.573958+0000", 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4465000000000002e-05 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.173 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:95296 11.e 11:7322c2ac:::benchmark_data_smithi081_181703_object95295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:10.439302+0000", 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.430055880000001, 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13479693800000001, 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95296 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.439302+0000", 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.439302+0000", 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.439303+0000", 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.178e-06 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.439364+0000", 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0482999999999997e-05 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.439372+0000", 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6279999999999994e-06 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.439379+0000", 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2289999999999997e-06 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.443315+0000", 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039360510000000003 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.443449+0000", 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133807 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.443602+0000", 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015318800000000001 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.447868+0000", 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042651130000000001 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.448074+0000", 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020688600000000001 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.574012+0000", 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12593794899999999 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.574071+0000", 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9069e-05 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.574099+0000", 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7356999999999999e-05 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:95297 11.e 11:743734dd:::benchmark_data_smithi081_181703_object95296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:11:34.179 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:10.441749+0000", 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.427608521, 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13243450900000001, 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95297 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441749+0000", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441749+0000", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441751+0000", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.567e-06 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441817+0000", 2024-09-21T19:11:34.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5755999999999999e-05 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441827+0000", 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0679000000000001e-05 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.441833+0000", 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3639999999999998e-06 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.445298+0000", 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034650829999999999 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.445381+0000", 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3399999999999994e-05 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.445541+0000", 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000159483 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.448079+0000", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025384449999999999 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.448350+0000", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027120600000000002 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.574129+0000", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.12577912899999999 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.574159+0000", 2024-09-21T19:11:34.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9598e-05 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.574184+0000", 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4799000000000001e-05 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:100774 11.e 11:71b3684c:::benchmark_data_smithi081_181703_object100773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:16.217238+0000", 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.652119458000001, 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.35419971500000003, 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:34.185 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100774 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217238+0000", 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217238+0000", 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217240+0000", 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.962e-06 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217278+0000", 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7490999999999997e-05 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217291+0000", 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3107e-05 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217298+0000", 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2819999999999999e-06 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217373+0000", 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4538999999999999e-05 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217512+0000", 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013977999999999999 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217651+0000", 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013883400000000001 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.220716+0000", 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030648030000000001 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.223253+0000", 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002537157 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.571363+0000", 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.34810986199999999 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.571399+0000", 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6293e-05 2024-09-21T19:11:34.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.571438+0000", 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8605000000000001e-05 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:34.191 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:34.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:34.229 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:33 smithi104 ceph-mon[32438]: 11.c scrub starts 2024-09-21T19:11:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:33 smithi104 ceph-mon[32438]: 6.1a scrub starts 2024-09-21T19:11:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:33 smithi104 ceph-mon[32438]: 6.1a scrub ok 2024-09-21T19:11:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:33 smithi081 ceph-mon[39192]: 11.c scrub starts 2024-09-21T19:11:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:33 smithi081 ceph-mon[39192]: 6.1a scrub starts 2024-09-21T19:11:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:33 smithi081 ceph-mon[39192]: 6.1a scrub ok 2024-09-21T19:11:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:33 smithi081 ceph-mon[29384]: 11.c scrub starts 2024-09-21T19:11:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:33 smithi081 ceph-mon[29384]: 6.1a scrub starts 2024-09-21T19:11:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:33 smithi081 ceph-mon[29384]: 6.1a scrub ok 2024-09-21T19:11:34.331 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:34.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:34.490 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:34.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:34.630 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:34.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:34.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 87 16 120476 120460 86.5053 59.75 0.0279474 0.0115553 2024-09-21T19:11:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:34.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:34.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:34.998 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:35.026 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:35.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:35.157 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: pgmap v913: 158 pgs: 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T19:11:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T19:11:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T19:11:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: 5.1c scrub starts 2024-09-21T19:11:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: 5.1c scrub ok 2024-09-21T19:11:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T19:11:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:34 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T19:11:35.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:35.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: pgmap v913: 158 pgs: 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T19:11:35.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T19:11:35.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T19:11:35.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: 5.1c scrub starts 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: 5.1c scrub ok 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: pgmap v913: 158 pgs: 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: 5.1c scrub starts 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: 5.1c scrub ok 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T19:11:35.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:34 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T19:11:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:35.340 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:35.343 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:35.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:35.561 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:35.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:35.707 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:35.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 88 16 121506 121490 86.2538 64.375 0.00762453 0.0115888 2024-09-21T19:11:35.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:11:35.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:35.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:35.984 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:35.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:36.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:36.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:36.152 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:36.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:36.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:35 smithi104 ceph-mon[32438]: 4.15 scrub starts 2024-09-21T19:11:36.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:35 smithi104 ceph-mon[32438]: 4.15 scrub ok 2024-09-21T19:11:36.266 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:35 smithi081 ceph-mon[29384]: 4.15 scrub starts 2024-09-21T19:11:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:35 smithi081 ceph-mon[29384]: 4.15 scrub ok 2024-09-21T19:11:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:35 smithi081 ceph-mon[39192]: 4.15 scrub starts 2024-09-21T19:11:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:35 smithi081 ceph-mon[39192]: 4.15 scrub ok 2024-09-21T19:11:36.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:36.400 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:36.502 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:36.529 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:36.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.669 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:36.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:122549 11.f 11:f4f108b7:::benchmark_data_smithi081_181703_object122548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:36.730130+0000", 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.019601229000000001, 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01963434, 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:36.753 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122549 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730130+0000", 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730130+0000", 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730131+0000", 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1200000000000001e-06 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730207+0000", 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5631999999999997e-05 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730219+0000", 2024-09-21T19:11:36.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1738e-05 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730225+0000", 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4370000000000004e-06 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730276+0000", 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1093e-05 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730412+0000", 2024-09-21T19:11:36.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013561500000000001 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.730545+0000", 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013355100000000001 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.736725+0000", 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061793880000000001 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.743154+0000", 2024-09-21T19:11:36.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064293589999999999 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:122550 11.f 11:f1d2b95a:::benchmark_data_smithi081_181703_object122549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:36.731525+0000", 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.018206028999999999, 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018327203, 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:36.758 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122550 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731525+0000", 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731525+0000", 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.759 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731527+0000", 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4130000000000001e-06 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731608+0000", 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.14e-05 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731622+0000", 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4031e-05 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731629+0000", 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0940000000000002e-06 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.733764+0000", 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021344850000000002 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.733886+0000", 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012194200000000001 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.734017+0000", 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013168699999999999 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740008+0000", 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005990441 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.743195+0000", 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031868809999999999 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:36.762 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:122551 11.f 11:fa6a1cac:::benchmark_data_smithi081_181703_object122550:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:36.731651+0000", 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.018080687000000002, 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018247279000000002, 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122551 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.763 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731651+0000", 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731651+0000", 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731652+0000", 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4099999999999997e-07 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731748+0000", 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6038000000000004e-05 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731756+0000", 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5809999999999997e-06 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 89 16 122584 122568 86.0417 67.375 0.00877349 0.0116174 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.731763+0000", 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5749999999999997e-06 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:36.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.736752+0000", 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049891960000000004 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.736868+0000", 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011551 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.736997+0000", 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012966400000000001 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.743116+0000", 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061192099999999999 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.743218+0000", 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010135400000000001 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:122562 11.f 11:f9555282:::benchmark_data_smithi081_181703_object122561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T19:11:36.768 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:36.740461+0000", 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0092705719999999995, 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0094845239999999994, 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122562 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740461+0000", 2024-09-21T19:11:36.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740461+0000", 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740463+0000", 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7460000000000001e-06 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:36.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740492+0000", 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9638e-05 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740507+0000", 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4304999999999999e-05 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740514+0000", 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8609999999999998e-06 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:36.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740571+0000", 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6996000000000001e-05 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740722+0000", 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015025600000000001 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.740881+0000", 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015918599999999999 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.743271+0000", 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023898750000000001 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.746231+0000", 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002960318 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36497.0:122567 11.3 e444/414 11:c298f50f:::benchmark_data_smithi081_181703_object122566:head v 444'7665, mlcod=444'7663)", 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:36.748086+0000", 2024-09-21T19:11:36.773 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0016457500000000001, 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001898521, 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748086+0000", 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748086+0000", 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748087+0000", 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3200000000000001e-06 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748108+0000", 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1466000000000001e-05 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748130+0000", 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1838999999999999e-05 2024-09-21T19:11:36.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748142+0000", 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1420000000000001e-05 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748206+0000", 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4688000000000001e-05 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.748229+0000", 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2540000000000001e-05 2024-09-21T19:11:36.776 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:122569 11.5 11:adba3fb6:::benchmark_data_smithi081_181703_object122568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:36.749127+0000", 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.00060429600000000004, 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088558599999999995, 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:36.777 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 122569 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749127+0000", 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749127+0000", 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:36.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749129+0000", 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4360000000000001e-06 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749243+0000", 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114566 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749256+0000", 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3183999999999999e-05 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749263+0000", 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1049999999999997e-06 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749325+0000", 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1397999999999997e-05 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749459+0000", 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013386499999999999 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:36.749620+0000", 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016102299999999999 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 6 2024-09-21T19:11:36.781 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:36.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:36.812 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:36.913 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:36.944 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:36.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:37.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:37.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:37.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:37.222 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: 11.1 scrub starts 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: pgmap v914: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.22k op/s 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: 11.c scrub ok 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: 3.2 scrub starts 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: 3.2 scrub ok 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: 6.10 scrub starts 2024-09-21T19:11:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:36 smithi104 ceph-mon[32438]: 6.10 scrub ok 2024-09-21T19:11:37.297 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: 11.1 scrub starts 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: pgmap v914: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.22k op/s 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: 11.c scrub ok 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: 3.2 scrub starts 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: 3.2 scrub ok 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: 6.10 scrub starts 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[29384]: 6.10 scrub ok 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: 11.1 scrub starts 2024-09-21T19:11:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: pgmap v914: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.22k op/s 2024-09-21T19:11:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: 11.c scrub ok 2024-09-21T19:11:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: 3.2 scrub starts 2024-09-21T19:11:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: 3.2 scrub ok 2024-09-21T19:11:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: 6.10 scrub starts 2024-09-21T19:11:37.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:36 smithi081 ceph-mon[39192]: 6.10 scrub ok 2024-09-21T19:11:37.317 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:11:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:11:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:11:37.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:37.372 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:37.474 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:37.504 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:37.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:37.642 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:37.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:11:37.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total time run: 90.0078 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total writes made: 123618 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Write size: 65536 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object size: 65536 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Bandwidth (MB/sec): 85.8384 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Bandwidth: 31.2016 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max bandwidth (MB/sec): 222.75 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min bandwidth (MB/sec): 47.5625 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average IOPS: 1373 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev IOPS: 499.225 2024-09-21T19:11:37.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max IOPS: 3564 2024-09-21T19:11:37.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min IOPS: 761 2024-09-21T19:11:37.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average Latency(s): 0.0116458 2024-09-21T19:11:37.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Latency(s): 0.0129865 2024-09-21T19:11:37.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max latency(s): 0.53624 2024-09-21T19:11:37.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min latency(s): 0.00122502 2024-09-21T19:11:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.766 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:37.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:37.771 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T19:11:37.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:37.897 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:37.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:37.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:38.004 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:38 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T19:11:38.004 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:38 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T19:11:38.004 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:38.005 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:38 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:11:38.005 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:38 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:11:38.049 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:38.060 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:38.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.189 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:38.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:38.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:38.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:11:38.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:38 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:11:38.326 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:38.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.467 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:38.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:38.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:38.619 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:11:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:11:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:11:38.620 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:38.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:38.735 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:38.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:38.881 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:38.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:38.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:39.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:11:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:11:39.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: 3.f scrub starts 2024-09-21T19:11:39.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: 3.f scrub ok 2024-09-21T19:11:39.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: pgmap v915: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 933 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-21T19:11:39.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: 4.11 scrub starts 2024-09-21T19:11:39.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: 4.11 scrub ok 2024-09-21T19:11:39.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T19:11:39.018 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T19:11:39.021 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: 3.f scrub starts 2024-09-21T19:11:39.021 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: 3.f scrub ok 2024-09-21T19:11:39.021 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: pgmap v915: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 933 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-21T19:11:39.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: 4.11 scrub starts 2024-09-21T19:11:39.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: 4.11 scrub ok 2024-09-21T19:11:39.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T19:11:39.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T19:11:39.022 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/4189585704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:39.037 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.040 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:39.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:39.087 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:39.087 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:39.097 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:39.113 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:11:39.116 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:11:39.128 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:39.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:39.172 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:39.172 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:39.190 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:39.207 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:39.207 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:39.241 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:39.242 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:39.272 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:39.272 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:39.273 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/4189585704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:39.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:39.304 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:39.304 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:39.330 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:39.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:39.469 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:39.501 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: 3.f scrub starts 2024-09-21T19:11:39.501 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: 3.f scrub ok 2024-09-21T19:11:39.501 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: pgmap v915: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 933 B/s rd, 61 MiB/s wr, 979 op/s 2024-09-21T19:11:39.501 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: 4.11 scrub starts 2024-09-21T19:11:39.501 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: 4.11 scrub ok 2024-09-21T19:11:39.502 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T19:11:39.502 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T19:11:39.502 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/4189585704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:39.571 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:39.598 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:39.700 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:39.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:39.831 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:39.867 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:39.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:39.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:11:39.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:40.040 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:40.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:40.186 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: 11.1 scrub ok 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: from='client.36617 v1:172.21.15.81:0/3250457582' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: 4.1b scrub starts 2024-09-21T19:11:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:40 smithi104 ceph-mon[32438]: 4.1b scrub ok 2024-09-21T19:11:40.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: 11.1 scrub ok 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: from='client.36617 v1:172.21.15.81:0/3250457582' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: 4.1b scrub starts 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[29384]: 4.1b scrub ok 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: 11.1 scrub ok 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T19:11:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T19:11:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: from='client.36617 v1:172.21.15.81:0/3250457582' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: 4.1b scrub starts 2024-09-21T19:11:40.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:40 smithi081 ceph-mon[39192]: 4.1b scrub ok 2024-09-21T19:11:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.331 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:40.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:40.341 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:40.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:40.473 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:40.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:40.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:40.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:40.745 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:40.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:40.875 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:40.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:40.977 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:41.010 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:41.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.153 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:41.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:41.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[29384]: pgmap v916: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.05k op/s 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[29384]: 3.12 scrub starts 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[29384]: 3.12 scrub ok 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[39192]: pgmap v916: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.05k op/s 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T19:11:41.272 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T19:11:41.273 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[39192]: 3.12 scrub starts 2024-09-21T19:11:41.273 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:41 smithi081 ceph-mon[39192]: 3.12 scrub ok 2024-09-21T19:11:41.282 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:41.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:41 smithi104 ceph-mon[32438]: pgmap v916: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.05k op/s 2024-09-21T19:11:41.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:41 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T19:11:41.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:41 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T19:11:41.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:41 smithi104 ceph-mon[32438]: 3.12 scrub starts 2024-09-21T19:11:41.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:41 smithi104 ceph-mon[32438]: 3.12 scrub ok 2024-09-21T19:11:41.384 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:41.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:41.544 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:41.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:41.668 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:41.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:41.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:41.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:41.942 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:41.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:42.041 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:11:42.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:42.078 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:42.079 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:42.080 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:42.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.215 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:42.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:42.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 5.1a scrub starts 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 5.1a scrub ok 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2281688437' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: from='client.26473 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 4.b scrub starts 2024-09-21T19:11:42.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:42 smithi104 ceph-mon[32438]: 4.b scrub ok 2024-09-21T19:11:42.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 5.1a scrub starts 2024-09-21T19:11:42.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 5.1a scrub ok 2024-09-21T19:11:42.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T19:11:42.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T19:11:42.295 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2281688437' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: from='client.26473 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 4.b scrub starts 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[39192]: 4.b scrub ok 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 5.1a scrub starts 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 5.1a scrub ok 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T19:11:42.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T19:11:42.297 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T19:11:42.297 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2281688437' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:42.297 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: from='client.26473 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:42.297 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 4.b scrub starts 2024-09-21T19:11:42.297 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:42 smithi081 ceph-mon[29384]: 4.b scrub ok 2024-09-21T19:11:42.315 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":915,"stamp":"2024-09-21T19:11:37.826983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7870093851,"num_objects":120278,"num_object_clones":0,"num_object_copies":360834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120278,"num_whiteouts":0,"num_read":2236,"num_read_kb":1988,"num_write":120801,"num_write_kb":7685101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":272852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82038,"ondisk_log_size":82038,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28052532,"kb_used_data":22959740,"kb_used_omap":18,"kb_used_meta":5092461,"kb_avail":721908684,"statfs":{"total":767960285184,"available":739234492416,"internally_reserved":0,"allocated":23510773760,"data_stored":23505853673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":5214681072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,6,7,14,9,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":633339904,"num_objects":9664,"num_object_clones":0,"num_object_copies":28992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":9664,"num_write_kb":618496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.876444"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":735,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830922+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:11:30.830922+0000","last_peered":"2024-09-21T19:11:30.830922+0000","last_clean":"2024-09-21T19:11:30.830922+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:11:30.830922+0000","last_undegraded":"2024-09-21T19:11:30.830922+0000","last_fullsized":"2024-09-21T19:11:30.830922+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.472742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":904,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.808341+0000","last_change":"2024-09-21T19:10:17.698193+0000","last_active":"2024-09-21T19:11:30.808341+0000","last_peered":"2024-09-21T19:11:30.808341+0000","last_clean":"2024-09-21T19:11:30.808341+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:11:30.808341+0000","last_undegraded":"2024-09-21T19:11:30.808341+0000","last_fullsized":"2024-09-21T19:11:30.808341+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:10:17.698093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.698093+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.038064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":833,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810562+0000","last_change":"2024-09-21T19:09:37.069611+0000","last_active":"2024-09-21T19:11:30.810562+0000","last_peered":"2024-09-21T19:11:30.810562+0000","last_clean":"2024-09-21T19:11:30.810562+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:11:30.810562+0000","last_undegraded":"2024-09-21T19:11:30.810562+0000","last_fullsized":"2024-09-21T19:11:30.810562+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:37.069548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:37.069548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:44.208018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":821,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.079408+0000","last_change":"2024-09-21T19:11:33.079408+0000","last_active":"2024-09-21T19:11:33.079408+0000","last_peered":"2024-09-21T19:11:33.079408+0000","last_clean":"2024-09-21T19:11:33.079408+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:11:33.079408+0000","last_undegraded":"2024-09-21T19:11:33.079408+0000","last_fullsized":"2024-09-21T19:11:33.079408+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:33.079348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:33.079348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:00.185027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":830,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816979+0000","last_change":"2024-09-21T19:10:25.846651+0000","last_active":"2024-09-21T19:11:30.816979+0000","last_peered":"2024-09-21T19:11:30.816979+0000","last_clean":"2024-09-21T19:11:30.816979+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:11:30.816979+0000","last_undegraded":"2024-09-21T19:11:30.816979+0000","last_fullsized":"2024-09-21T19:11:30.816979+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:25.846276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.846276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:44.275130+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":858,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816226+0000","last_change":"2024-09-21T19:10:54.811144+0000","last_active":"2024-09-21T19:11:30.816226+0000","last_peered":"2024-09-21T19:11:30.816226+0000","last_clean":"2024-09-21T19:11:30.816226+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:11:30.816226+0000","last_undegraded":"2024-09-21T19:11:30.816226+0000","last_fullsized":"2024-09-21T19:11:30.816226+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:54.811103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:54.811103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.471846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":946,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.817057+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:11:30.817057+0000","last_peered":"2024-09-21T19:11:30.817057+0000","last_clean":"2024-09-21T19:11:30.817057+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:11:30.817057+0000","last_undegraded":"2024-09-21T19:11:30.817057+0000","last_fullsized":"2024-09-21T19:11:30.817057+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":841,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.808252+0000","last_change":"2024-09-21T19:10:16.657183+0000","last_active":"2024-09-21T19:11:30.808252+0000","last_peered":"2024-09-21T19:11:30.808252+0000","last_clean":"2024-09-21T19:11:30.808252+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:11:30.808252+0000","last_undegraded":"2024-09-21T19:11:30.808252+0000","last_fullsized":"2024-09-21T19:11:30.808252+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.657131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.657131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":828,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811660+0000","last_change":"2024-09-21T19:10:13.614795+0000","last_active":"2024-09-21T19:11:30.811660+0000","last_peered":"2024-09-21T19:11:30.811660+0000","last_clean":"2024-09-21T19:11:30.811660+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:11:30.811660+0000","last_undegraded":"2024-09-21T19:11:30.811660+0000","last_fullsized":"2024-09-21T19:11:30.811660+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.614720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.614720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":877,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.808890+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:11:30.808890+0000","last_peered":"2024-09-21T19:11:30.808890+0000","last_clean":"2024-09-21T19:11:30.808890+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:11:30.808890+0000","last_undegraded":"2024-09-21T19:11:30.808890+0000","last_fullsized":"2024-09-21T19:11:30.808890+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.138431+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":789,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.819413+0000","last_change":"2024-09-21T19:11:00.940146+0000","last_active":"2024-09-21T19:11:30.819413+0000","last_peered":"2024-09-21T19:11:30.819413+0000","last_clean":"2024-09-21T19:11:30.819413+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:11:30.819413+0000","last_undegraded":"2024-09-21T19:11:30.819413+0000","last_fullsized":"2024-09-21T19:11:30.819413+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.940070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.940070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.146532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":701,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829270+0000","last_change":"2024-09-21T19:11:03.440073+0000","last_active":"2024-09-21T19:11:30.829270+0000","last_peered":"2024-09-21T19:11:30.829270+0000","last_clean":"2024-09-21T19:11:30.829270+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:11:30.829270+0000","last_undegraded":"2024-09-21T19:11:30.829270+0000","last_fullsized":"2024-09-21T19:11:30.829270+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.439988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.439988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:07.256937+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":381,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810494+0000","last_change":"2024-09-21T19:11:08.134154+0000","last_active":"2024-09-21T19:11:30.810494+0000","last_peered":"2024-09-21T19:11:30.810494+0000","last_clean":"2024-09-21T19:11:30.810494+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:11:30.810494+0000","last_undegraded":"2024-09-21T19:11:30.810494+0000","last_fullsized":"2024-09-21T19:11:30.810494+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.134097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.134097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.482578+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":823,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.808614+0000","last_change":"2024-09-21T19:06:12.793192+0000","last_active":"2024-09-21T19:11:30.808614+0000","last_peered":"2024-09-21T19:11:30.808614+0000","last_clean":"2024-09-21T19:11:30.808614+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:11:30.808614+0000","last_undegraded":"2024-09-21T19:11:30.808614+0000","last_fullsized":"2024-09-21T19:11:30.808614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:06:12.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:12.793136+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.851204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":813,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812292+0000","last_change":"2024-09-21T19:09:05.634088+0000","last_active":"2024-09-21T19:11:30.812292+0000","last_peered":"2024-09-21T19:11:30.812292+0000","last_clean":"2024-09-21T19:11:30.812292+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:11:30.812292+0000","last_undegraded":"2024-09-21T19:11:30.812292+0000","last_fullsized":"2024-09-21T19:11:30.812292+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:05.634016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:05.634016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.552609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":389,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810387+0000","last_change":"2024-09-21T19:10:13.100266+0000","last_active":"2024-09-21T19:11:30.810387+0000","last_peered":"2024-09-21T19:11:30.810387+0000","last_clean":"2024-09-21T19:11:30.810387+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:11:30.810387+0000","last_undegraded":"2024-09-21T19:11:30.810387+0000","last_fullsized":"2024-09-21T19:11:30.810387+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.100191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.100191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:34.286351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":850,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810417+0000","last_change":"2024-09-21T19:10:26.819197+0000","last_active":"2024-09-21T19:11:30.810417+0000","last_peered":"2024-09-21T19:11:30.810417+0000","last_clean":"2024-09-21T19:11:30.810417+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:11:30.810417+0000","last_undegraded":"2024-09-21T19:11:30.810417+0000","last_fullsized":"2024-09-21T19:11:30.810417+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:26.819128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:26.819128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:47.207572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":709,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829670+0000","last_change":"2024-09-21T19:06:49.921851+0000","last_active":"2024-09-21T19:11:30.829670+0000","last_peered":"2024-09-21T19:11:30.829670+0000","last_clean":"2024-09-21T19:11:30.829670+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:11:30.829670+0000","last_undegraded":"2024-09-21T19:11:30.829670+0000","last_fullsized":"2024-09-21T19:11:30.829670+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:49.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:49.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:40.466668+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":292,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818480+0000","last_change":"2024-09-21T19:10:25.436206+0000","last_active":"2024-09-21T19:11:30.818480+0000","last_peered":"2024-09-21T19:11:30.818480+0000","last_clean":"2024-09-21T19:11:30.818480+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:11:30.818480+0000","last_undegraded":"2024-09-21T19:11:30.818480+0000","last_fullsized":"2024-09-21T19:11:30.818480+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:10:25.436158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.436158+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":852,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:31.048129+0000","last_change":"2024-09-21T19:11:31.048129+0000","last_active":"2024-09-21T19:11:31.048129+0000","last_peered":"2024-09-21T19:11:31.048129+0000","last_clean":"2024-09-21T19:11:31.048129+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:11:31.048129+0000","last_undegraded":"2024-09-21T19:11:31.048129+0000","last_fullsized":"2024-09-21T19:11:31.048129+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.048064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.048064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:39.917247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":826,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816133+0000","last_change":"2024-09-21T19:10:13.916666+0000","last_active":"2024-09-21T19:11:30.816133+0000","last_peered":"2024-09-21T19:11:30.816133+0000","last_clean":"2024-09-21T19:11:30.816133+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:11:30.816133+0000","last_undegraded":"2024-09-21T19:11:30.816133+0000","last_fullsized":"2024-09-21T19:11:30.816133+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.916604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.916604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":844,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809022+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:11:30.809022+0000","last_peered":"2024-09-21T19:11:30.809022+0000","last_clean":"2024-09-21T19:11:30.809022+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:11:30.809022+0000","last_undegraded":"2024-09-21T19:11:30.809022+0000","last_fullsized":"2024-09-21T19:11:30.809022+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.111619+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":908,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.817009+0000","last_change":"2024-09-21T19:10:52.029412+0000","last_active":"2024-09-21T19:11:30.817009+0000","last_peered":"2024-09-21T19:11:30.817009+0000","last_clean":"2024-09-21T19:11:30.817009+0000","last_became_active":"2024-09-21T19:10:52.013122+0000","last_became_peered":"2024-09-21T19:10:52.013122+0000","last_unstale":"2024-09-21T19:11:30.817009+0000","last_undegraded":"2024-09-21T19:11:30.817009+0000","last_fullsized":"2024-09-21T19:11:30.817009+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:06:43.844337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:43.844337+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:34.072955+0000","last_change":"2024-09-21T19:11:34.072955+0000","last_active":"2024-09-21T19:11:34.072955+0000","last_peered":"2024-09-21T19:11:34.072955+0000","last_clean":"2024-09-21T19:11:34.072955+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:11:34.072955+0000","last_undegraded":"2024-09-21T19:11:34.072955+0000","last_fullsized":"2024-09-21T19:11:34.072955+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:34.072904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.072904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.755481+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":859,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.813829+0000","last_change":"2024-09-21T19:10:13.592189+0000","last_active":"2024-09-21T19:11:30.813829+0000","last_peered":"2024-09-21T19:11:30.813829+0000","last_clean":"2024-09-21T19:11:30.813829+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:11:30.813829+0000","last_undegraded":"2024-09-21T19:11:30.813829+0000","last_fullsized":"2024-09-21T19:11:30.813829+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:13.592116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.592116+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.755272+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":858,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810677+0000","last_change":"2024-09-21T19:10:27.115013+0000","last_active":"2024-09-21T19:11:30.810677+0000","last_peered":"2024-09-21T19:11:30.810677+0000","last_clean":"2024-09-21T19:11:30.810677+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:11:30.810677+0000","last_undegraded":"2024-09-21T19:11:30.810677+0000","last_fullsized":"2024-09-21T19:11:30.810677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:27.114925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.114925+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:50.719198+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":745,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830653+0000","last_change":"2024-09-21T19:10:27.501434+0000","last_active":"2024-09-21T19:11:30.830653+0000","last_peered":"2024-09-21T19:11:30.830653+0000","last_clean":"2024-09-21T19:11:30.830653+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:11:30.830653+0000","last_undegraded":"2024-09-21T19:11:30.830653+0000","last_fullsized":"2024-09-21T19:11:30.830653+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:27.500959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.500959+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:39.504634+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":943,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821465+0000","last_change":"2024-09-21T19:10:15.802776+0000","last_active":"2024-09-21T19:11:30.821465+0000","last_peered":"2024-09-21T19:11:30.821465+0000","last_clean":"2024-09-21T19:11:30.821465+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:11:30.821465+0000","last_undegraded":"2024-09-21T19:11:30.821465+0000","last_fullsized":"2024-09-21T19:11:30.821465+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:10:15.802724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:15.802724+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"444'7320","reported_seq":7391,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:32.788074+0000","last_change":"2024-09-21T19:10:01.248307+0000","last_active":"2024-09-21T19:11:32.788074+0000","last_peered":"2024-09-21T19:11:32.788074+0000","last_clean":"2024-09-21T19:11:32.788074+0000","last_became_active":"2024-09-21T19:10:01.248029+0000","last_became_peered":"2024-09-21T19:10:01.248029+0000","last_unstale":"2024-09-21T19:11:32.788074+0000","last_undegraded":"2024-09-21T19:11:32.788074+0000","last_fullsized":"2024-09-21T19:11:32.788074+0000","mapping_epoch":414,"log_start":"426'2400","ondisk_log_start":"426'2400","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:21.027444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479723520,"num_objects":7320,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7320,"num_write_kb":468480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":287,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820556+0000","last_change":"2024-09-21T19:10:21.515835+0000","last_active":"2024-09-21T19:11:30.820556+0000","last_peered":"2024-09-21T19:11:30.820556+0000","last_clean":"2024-09-21T19:11:30.820556+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:11:30.820556+0000","last_undegraded":"2024-09-21T19:11:30.820556+0000","last_fullsized":"2024-09-21T19:11:30.820556+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:21.515713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.515713+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.289543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":886,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820578+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:11:30.820578+0000","last_peered":"2024-09-21T19:11:30.820578+0000","last_clean":"2024-09-21T19:11:30.820578+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:11:30.820578+0000","last_undegraded":"2024-09-21T19:11:30.820578+0000","last_fullsized":"2024-09-21T19:11:30.820578+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.082123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":824,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821179+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:11:30.821179+0000","last_peered":"2024-09-21T19:11:30.821179+0000","last_clean":"2024-09-21T19:11:30.821179+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:11:30.821179+0000","last_undegraded":"2024-09-21T19:11:30.821179+0000","last_fullsized":"2024-09-21T19:11:30.821179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1096,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810570+0000","last_change":"2024-09-21T19:11:10.165309+0000","last_active":"2024-09-21T19:11:30.810570+0000","last_peered":"2024-09-21T19:11:30.810570+0000","last_clean":"2024-09-21T19:11:30.810570+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:11:30.810570+0000","last_undegraded":"2024-09-21T19:11:30.810570+0000","last_fullsized":"2024-09-21T19:11:30.810570+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:11:10.165236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:10.165236+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.702978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":513,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811990+0000","last_change":"2024-09-21T19:10:23.103828+0000","last_active":"2024-09-21T19:11:30.811990+0000","last_peered":"2024-09-21T19:11:30.811990+0000","last_clean":"2024-09-21T19:11:30.811990+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:11:30.811990+0000","last_undegraded":"2024-09-21T19:11:30.811990+0000","last_fullsized":"2024-09-21T19:11:30.811990+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:23.103763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.103763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:34.402051+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"444'7824","reported_seq":7895,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:36.905201+0000","last_change":"2024-09-21T19:10:01.252308+0000","last_active":"2024-09-21T19:11:36.905201+0000","last_peered":"2024-09-21T19:11:36.905201+0000","last_clean":"2024-09-21T19:11:36.905201+0000","last_became_active":"2024-09-21T19:10:01.252017+0000","last_became_peered":"2024-09-21T19:10:01.252017+0000","last_unstale":"2024-09-21T19:11:36.905201+0000","last_undegraded":"2024-09-21T19:11:36.905201+0000","last_fullsized":"2024-09-21T19:11:36.905201+0000","mapping_epoch":414,"log_start":"427'2900","ondisk_log_start":"427'2900","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.828286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512753664,"num_objects":7824,"num_object_clones":0,"num_object_copies":23472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":500736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":613,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811573+0000","last_change":"2024-09-21T19:09:53.595884+0000","last_active":"2024-09-21T19:11:30.811573+0000","last_peered":"2024-09-21T19:11:30.811573+0000","last_clean":"2024-09-21T19:11:30.811573+0000","last_became_active":"2024-09-21T19:05:42.888177+0000","last_became_peered":"2024-09-21T19:05:42.888177+0000","last_unstale":"2024-09-21T19:11:30.811573+0000","last_undegraded":"2024-09-21T19:11:30.811573+0000","last_fullsized":"2024-09-21T19:11:30.811573+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":913,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820901+0000","last_change":"2024-09-21T19:05:36.820913+0000","last_active":"2024-09-21T19:11:30.820901+0000","last_peered":"2024-09-21T19:11:30.820901+0000","last_clean":"2024-09-21T19:11:30.820901+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:11:30.820901+0000","last_undegraded":"2024-09-21T19:11:30.820901+0000","last_fullsized":"2024-09-21T19:11:30.820901+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:05:36.820837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:36.820837+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.668914+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":846,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816674+0000","last_change":"2024-09-21T19:10:56.831121+0000","last_active":"2024-09-21T19:11:30.816674+0000","last_peered":"2024-09-21T19:11:30.816674+0000","last_clean":"2024-09-21T19:11:30.816674+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:11:30.816674+0000","last_undegraded":"2024-09-21T19:11:30.816674+0000","last_fullsized":"2024-09-21T19:11:30.816674+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:56.831042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:56.831042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:00.224419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":823,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809148+0000","last_change":"2024-09-21T19:11:05.056988+0000","last_active":"2024-09-21T19:11:30.809148+0000","last_peered":"2024-09-21T19:11:30.809148+0000","last_clean":"2024-09-21T19:11:30.809148+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:11:30.809148+0000","last_undegraded":"2024-09-21T19:11:30.809148+0000","last_fullsized":"2024-09-21T19:11:30.809148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.056931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.056931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:22.842104+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":840,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.808800+0000","last_change":"2024-09-21T19:10:51.864691+0000","last_active":"2024-09-21T19:11:30.808800+0000","last_peered":"2024-09-21T19:11:30.808800+0000","last_clean":"2024-09-21T19:11:30.808800+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:11:30.808800+0000","last_undegraded":"2024-09-21T19:11:30.808800+0000","last_fullsized":"2024-09-21T19:11:30.808800+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:51.864627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.864627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:01.755943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"444'7556","reported_seq":7627,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:36.425638+0000","last_change":"2024-09-21T19:10:01.248578+0000","last_active":"2024-09-21T19:11:36.425638+0000","last_peered":"2024-09-21T19:11:36.425638+0000","last_clean":"2024-09-21T19:11:36.425638+0000","last_became_active":"2024-09-21T19:10:01.248286+0000","last_became_peered":"2024-09-21T19:10:01.248286+0000","last_unstale":"2024-09-21T19:11:36.425638+0000","last_undegraded":"2024-09-21T19:11:36.425638+0000","last_fullsized":"2024-09-21T19:11:36.425638+0000","mapping_epoch":414,"log_start":"427'2800","ondisk_log_start":"427'2800","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.205302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495190016,"num_objects":7556,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7556,"num_write_kb":483584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":596,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816004+0000","last_change":"2024-09-21T19:06:54.695601+0000","last_active":"2024-09-21T19:11:30.816004+0000","last_peered":"2024-09-21T19:11:30.816004+0000","last_clean":"2024-09-21T19:11:30.816004+0000","last_became_active":"2024-09-21T19:06:54.691706+0000","last_became_peered":"2024-09-21T19:06:54.691706+0000","last_unstale":"2024-09-21T19:11:30.816004+0000","last_undegraded":"2024-09-21T19:11:30.816004+0000","last_fullsized":"2024-09-21T19:11:30.816004+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:05:28.684905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:05:28.684905+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:36.692032+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":891,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809233+0000","last_change":"2024-09-21T19:10:11.634360+0000","last_active":"2024-09-21T19:11:30.809233+0000","last_peered":"2024-09-21T19:11:30.809233+0000","last_clean":"2024-09-21T19:11:30.809233+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:11:30.809233+0000","last_undegraded":"2024-09-21T19:11:30.809233+0000","last_fullsized":"2024-09-21T19:11:30.809233+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:10:11.634280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:11.634280+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":714,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.831032+0000","last_change":"2024-09-21T19:11:04.465550+0000","last_active":"2024-09-21T19:11:30.831032+0000","last_peered":"2024-09-21T19:11:30.831032+0000","last_clean":"2024-09-21T19:11:30.831032+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:11:30.831032+0000","last_undegraded":"2024-09-21T19:11:30.831032+0000","last_fullsized":"2024-09-21T19:11:30.831032+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:04.464221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:04.464221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:31.185556+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":1039,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821122+0000","last_change":"2024-09-21T19:10:21.865360+0000","last_active":"2024-09-21T19:11:30.821122+0000","last_peered":"2024-09-21T19:11:30.821122+0000","last_clean":"2024-09-21T19:11:30.821122+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:11:30.821122+0000","last_undegraded":"2024-09-21T19:11:30.821122+0000","last_fullsized":"2024-09-21T19:11:30.821122+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:10:21.865158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.865158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:42.703841+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":843,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812870+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:11:30.812870+0000","last_peered":"2024-09-21T19:11:30.812870+0000","last_clean":"2024-09-21T19:11:30.812870+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:11:30.812870+0000","last_undegraded":"2024-09-21T19:11:30.812870+0000","last_fullsized":"2024-09-21T19:11:30.812870+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.789154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"444'7380","reported_seq":7450,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.076066+0000","last_change":"2024-09-21T19:10:01.251501+0000","last_active":"2024-09-21T19:11:33.076066+0000","last_peered":"2024-09-21T19:11:33.076066+0000","last_clean":"2024-09-21T19:11:33.076066+0000","last_became_active":"2024-09-21T19:10:01.250354+0000","last_became_peered":"2024-09-21T19:10:01.250354+0000","last_unstale":"2024-09-21T19:11:33.076066+0000","last_undegraded":"2024-09-21T19:11:33.076066+0000","last_fullsized":"2024-09-21T19:11:33.076066+0000","mapping_epoch":414,"log_start":"420'1100","ondisk_log_start":"420'1100","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.606410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483655680,"num_objects":7380,"num_object_clones":0,"num_object_copies":22140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7380,"num_write_kb":472320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":628,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.822088+0000","last_change":"2024-09-21T19:09:55.907018+0000","last_active":"2024-09-21T19:11:30.822088+0000","last_peered":"2024-09-21T19:11:30.822088+0000","last_clean":"2024-09-21T19:11:30.822088+0000","last_became_active":"2024-09-21T19:02:40.879160+0000","last_became_peered":"2024-09-21T19:02:40.879160+0000","last_unstale":"2024-09-21T19:11:30.822088+0000","last_undegraded":"2024-09-21T19:11:30.822088+0000","last_fullsized":"2024-09-21T19:11:30.822088+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:22.726643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":876,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811049+0000","last_change":"2024-09-21T19:10:57.167898+0000","last_active":"2024-09-21T19:11:30.811049+0000","last_peered":"2024-09-21T19:11:30.811049+0000","last_clean":"2024-09-21T19:11:30.811049+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:11:30.811049+0000","last_undegraded":"2024-09-21T19:11:30.811049+0000","last_fullsized":"2024-09-21T19:11:30.811049+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:57.167826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.167826+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:01.720799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":388,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818377+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:11:30.818377+0000","last_peered":"2024-09-21T19:11:30.818377+0000","last_clean":"2024-09-21T19:11:30.818377+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:11:30.818377+0000","last_undegraded":"2024-09-21T19:11:30.818377+0000","last_fullsized":"2024-09-21T19:11:30.818377+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":921,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:32.614829+0000","last_change":"2024-09-21T19:11:32.614783+0000","last_active":"2024-09-21T19:11:32.614829+0000","last_peered":"2024-09-21T19:11:32.614829+0000","last_clean":"2024-09-21T19:11:32.614829+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:11:32.614829+0000","last_undegraded":"2024-09-21T19:11:32.614829+0000","last_fullsized":"2024-09-21T19:11:32.614829+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'14","last_scrub_stamp":"2024-09-21T19:11:32.614707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:32.614707+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:45.115750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":724,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829425+0000","last_change":"2024-09-21T19:11:08.473248+0000","last_active":"2024-09-21T19:11:30.829425+0000","last_peered":"2024-09-21T19:11:30.829425+0000","last_clean":"2024-09-21T19:11:30.829425+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:11:30.829425+0000","last_undegraded":"2024-09-21T19:11:30.829425+0000","last_fullsized":"2024-09-21T19:11:30.829425+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.473184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.473184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:38.292717+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"444'7543","reported_seq":7614,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:32.770997+0000","last_change":"2024-09-21T19:10:01.244039+0000","last_active":"2024-09-21T19:11:32.770997+0000","last_peered":"2024-09-21T19:11:32.770997+0000","last_clean":"2024-09-21T19:11:32.770997+0000","last_became_active":"2024-09-21T19:10:01.243730+0000","last_became_peered":"2024-09-21T19:10:01.243730+0000","last_unstale":"2024-09-21T19:11:32.770997+0000","last_undegraded":"2024-09-21T19:11:32.770997+0000","last_fullsized":"2024-09-21T19:11:32.770997+0000","mapping_epoch":414,"log_start":"426'2700","ondisk_log_start":"426'2700","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494338048,"num_objects":7543,"num_object_clones":0,"num_object_copies":22629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":482752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":606,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821502+0000","last_change":"2024-09-21T19:09:54.937870+0000","last_active":"2024-09-21T19:11:30.821502+0000","last_peered":"2024-09-21T19:11:30.821502+0000","last_clean":"2024-09-21T19:11:30.821502+0000","last_became_active":"2024-09-21T19:02:40.877183+0000","last_became_peered":"2024-09-21T19:02:40.877183+0000","last_unstale":"2024-09-21T19:11:30.821502+0000","last_undegraded":"2024-09-21T19:11:30.821502+0000","last_fullsized":"2024-09-21T19:11:30.821502+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.733013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":762,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829718+0000","last_change":"2024-09-21T19:10:25.488039+0000","last_active":"2024-09-21T19:11:30.829718+0000","last_peered":"2024-09-21T19:11:30.829718+0000","last_clean":"2024-09-21T19:11:30.829718+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:11:30.829718+0000","last_undegraded":"2024-09-21T19:11:30.829718+0000","last_fullsized":"2024-09-21T19:11:30.829718+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:10:25.487878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.487878+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:46.383223+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":856,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.814226+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:11:30.814226+0000","last_peered":"2024-09-21T19:11:30.814226+0000","last_clean":"2024-09-21T19:11:30.814226+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:11:30.814226+0000","last_undegraded":"2024-09-21T19:11:30.814226+0000","last_fullsized":"2024-09-21T19:11:30.814226+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.041720+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":733,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821197+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:11:30.821197+0000","last_peered":"2024-09-21T19:11:30.821197+0000","last_clean":"2024-09-21T19:11:30.821197+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:11:30.821197+0000","last_undegraded":"2024-09-21T19:11:30.821197+0000","last_fullsized":"2024-09-21T19:11:30.821197+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:17.702612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":844,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810485+0000","last_change":"2024-09-21T19:11:00.111371+0000","last_active":"2024-09-21T19:11:30.810485+0000","last_peered":"2024-09-21T19:11:30.810485+0000","last_clean":"2024-09-21T19:11:30.810485+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:11:30.810485+0000","last_undegraded":"2024-09-21T19:11:30.810485+0000","last_fullsized":"2024-09-21T19:11:30.810485+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.111305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.111305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.568023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"444'7445","reported_seq":7516,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.114821+0000","last_change":"2024-09-21T19:10:01.250359+0000","last_active":"2024-09-21T19:11:33.114821+0000","last_peered":"2024-09-21T19:11:33.114821+0000","last_clean":"2024-09-21T19:11:33.114821+0000","last_became_active":"2024-09-21T19:10:01.250079+0000","last_became_peered":"2024-09-21T19:10:01.250079+0000","last_unstale":"2024-09-21T19:11:33.114821+0000","last_undegraded":"2024-09-21T19:11:33.114821+0000","last_fullsized":"2024-09-21T19:11:33.114821+0000","mapping_epoch":414,"log_start":"427'3000","ondisk_log_start":"427'3000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487915520,"num_objects":7445,"num_object_clones":0,"num_object_copies":22335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7445,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":581,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829883+0000","last_change":"2024-09-21T19:11:07.477321+0000","last_active":"2024-09-21T19:11:30.829883+0000","last_peered":"2024-09-21T19:11:30.829883+0000","last_clean":"2024-09-21T19:11:30.829883+0000","last_became_active":"2024-09-21T19:06:54.701930+0000","last_became_peered":"2024-09-21T19:06:54.701930+0000","last_unstale":"2024-09-21T19:11:30.829883+0000","last_undegraded":"2024-09-21T19:11:30.829883+0000","last_fullsized":"2024-09-21T19:11:30.829883+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:07.476444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.476444+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:35.128950+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1166,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.814041+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:11:30.814041+0000","last_peered":"2024-09-21T19:11:30.814041+0000","last_clean":"2024-09-21T19:11:30.814041+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:11:30.814041+0000","last_undegraded":"2024-09-21T19:11:30.814041+0000","last_fullsized":"2024-09-21T19:11:30.814041+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.035088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":856,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809720+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:11:30.809720+0000","last_peered":"2024-09-21T19:11:30.809720+0000","last_clean":"2024-09-21T19:11:30.809720+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:11:30.809720+0000","last_undegraded":"2024-09-21T19:11:30.809720+0000","last_fullsized":"2024-09-21T19:11:30.809720+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.906766+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":404,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811385+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:11:30.811385+0000","last_peered":"2024-09-21T19:11:30.811385+0000","last_clean":"2024-09-21T19:11:30.811385+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:11:30.811385+0000","last_undegraded":"2024-09-21T19:11:30.811385+0000","last_fullsized":"2024-09-21T19:11:30.811385+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.115247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":710,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829570+0000","last_change":"2024-09-21T19:11:05.417598+0000","last_active":"2024-09-21T19:11:30.829570+0000","last_peered":"2024-09-21T19:11:30.829570+0000","last_clean":"2024-09-21T19:11:30.829570+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:11:30.829570+0000","last_undegraded":"2024-09-21T19:11:30.829570+0000","last_fullsized":"2024-09-21T19:11:30.829570+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.417365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.417365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.055191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":812,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.815848+0000","last_change":"2024-09-21T19:10:28.717418+0000","last_active":"2024-09-21T19:11:30.815848+0000","last_peered":"2024-09-21T19:11:30.815848+0000","last_clean":"2024-09-21T19:11:30.815848+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:11:30.815848+0000","last_undegraded":"2024-09-21T19:11:30.815848+0000","last_fullsized":"2024-09-21T19:11:30.815848+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:28.717310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.717310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.163329+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"444'7352","reported_seq":7423,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.106185+0000","last_change":"2024-09-21T19:10:01.259760+0000","last_active":"2024-09-21T19:11:33.106185+0000","last_peered":"2024-09-21T19:11:33.106185+0000","last_clean":"2024-09-21T19:11:33.106185+0000","last_became_active":"2024-09-21T19:10:01.259451+0000","last_became_peered":"2024-09-21T19:10:01.259451+0000","last_unstale":"2024-09-21T19:11:33.106185+0000","last_undegraded":"2024-09-21T19:11:33.106185+0000","last_fullsized":"2024-09-21T19:11:33.106185+0000","mapping_epoch":414,"log_start":"427'3000","ondisk_log_start":"427'3000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481820672,"num_objects":7352,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7352,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":917,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811527+0000","last_change":"2024-09-21T19:09:55.484547+0000","last_active":"2024-09-21T19:11:30.811527+0000","last_peered":"2024-09-21T19:11:30.811527+0000","last_clean":"2024-09-21T19:11:30.811527+0000","last_became_active":"2024-09-21T18:58:40.528548+0000","last_became_peered":"2024-09-21T18:58:40.528548+0000","last_unstale":"2024-09-21T19:11:30.811527+0000","last_undegraded":"2024-09-21T19:11:30.811527+0000","last_fullsized":"2024-09-21T19:11:30.811527+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:22.755065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":330,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820330+0000","last_change":"2024-09-21T19:10:20.080922+0000","last_active":"2024-09-21T19:11:30.820330+0000","last_peered":"2024-09-21T19:11:30.820330+0000","last_clean":"2024-09-21T19:11:30.820330+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:11:30.820330+0000","last_undegraded":"2024-09-21T19:11:30.820330+0000","last_fullsized":"2024-09-21T19:11:30.820330+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:10:20.080874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:20.080874+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:45.892100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":879,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810148+0000","last_change":"2024-09-21T19:09:02.141807+0000","last_active":"2024-09-21T19:11:30.810148+0000","last_peered":"2024-09-21T19:11:30.810148+0000","last_clean":"2024-09-21T19:11:30.810148+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:11:30.810148+0000","last_undegraded":"2024-09-21T19:11:30.810148+0000","last_fullsized":"2024-09-21T19:11:30.810148+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:09:02.141723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:02.141723+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:35.138769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":280,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:32.736397+0000","last_change":"2024-09-21T19:11:32.736397+0000","last_active":"2024-09-21T19:11:32.736397+0000","last_peered":"2024-09-21T19:11:32.736397+0000","last_clean":"2024-09-21T19:11:32.736397+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:11:32.736397+0000","last_undegraded":"2024-09-21T19:11:32.736397+0000","last_fullsized":"2024-09-21T19:11:32.736397+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:32.736299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:11:32.736299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:49.298535+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":279,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.817828+0000","last_change":"2024-09-21T19:09:09.724873+0000","last_active":"2024-09-21T19:11:30.817828+0000","last_peered":"2024-09-21T19:11:30.817828+0000","last_clean":"2024-09-21T19:11:30.817828+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:11:30.817828+0000","last_undegraded":"2024-09-21T19:11:30.817828+0000","last_fullsized":"2024-09-21T19:11:30.817828+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:09.724802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:09.724802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:43.358042+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":841,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812843+0000","last_change":"2024-09-21T19:10:45.720231+0000","last_active":"2024-09-21T19:11:30.812843+0000","last_peered":"2024-09-21T19:11:30.812843+0000","last_clean":"2024-09-21T19:11:30.812843+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:11:30.812843+0000","last_undegraded":"2024-09-21T19:11:30.812843+0000","last_fullsized":"2024-09-21T19:11:30.812843+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:45.720186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.720186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.576025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"444'7479","reported_seq":7550,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.116535+0000","last_change":"2024-09-21T19:10:01.250406+0000","last_active":"2024-09-21T19:11:33.116535+0000","last_peered":"2024-09-21T19:11:33.116535+0000","last_clean":"2024-09-21T19:11:33.116535+0000","last_became_active":"2024-09-21T19:10:01.250100+0000","last_became_peered":"2024-09-21T19:10:01.250100+0000","last_unstale":"2024-09-21T19:11:33.116535+0000","last_undegraded":"2024-09-21T19:11:33.116535+0000","last_fullsized":"2024-09-21T19:11:33.116535+0000","mapping_epoch":414,"log_start":"427'3100","ondisk_log_start":"427'3100","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490143744,"num_objects":7479,"num_object_clones":0,"num_object_copies":22437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":623,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811503+0000","last_change":"2024-09-21T19:10:28.147523+0000","last_active":"2024-09-21T19:11:30.811503+0000","last_peered":"2024-09-21T19:11:30.811503+0000","last_clean":"2024-09-21T19:11:30.811503+0000","last_became_active":"2024-09-21T19:02:40.878806+0000","last_became_peered":"2024-09-21T19:02:40.878806+0000","last_unstale":"2024-09-21T19:11:30.811503+0000","last_undegraded":"2024-09-21T19:11:30.811503+0000","last_fullsized":"2024-09-21T19:11:30.811503+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:28.147274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.147274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:39.517401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":824,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830835+0000","last_change":"2024-09-21T19:10:26.510494+0000","last_active":"2024-09-21T19:11:30.830835+0000","last_peered":"2024-09-21T19:11:30.830835+0000","last_clean":"2024-09-21T19:11:30.830835+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:11:30.830835+0000","last_undegraded":"2024-09-21T19:11:30.830835+0000","last_fullsized":"2024-09-21T19:11:30.830835+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:10:26.510232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:26.510232+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":710,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830200+0000","last_change":"2024-09-21T19:10:18.502592+0000","last_active":"2024-09-21T19:11:30.830200+0000","last_peered":"2024-09-21T19:11:30.830200+0000","last_clean":"2024-09-21T19:11:30.830200+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:11:30.830200+0000","last_undegraded":"2024-09-21T19:11:30.830200+0000","last_fullsized":"2024-09-21T19:11:30.830200+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:18.502526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:18.502526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.162083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1958,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:34.956059+0000","last_change":"2024-09-21T19:10:28.526838+0000","last_active":"2024-09-21T19:11:34.956059+0000","last_peered":"2024-09-21T19:11:34.956059+0000","last_clean":"2024-09-21T19:11:34.956059+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:11:34.956059+0000","last_undegraded":"2024-09-21T19:11:34.956059+0000","last_fullsized":"2024-09-21T19:11:34.956059+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:10:28.526790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.526790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:50.333975+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1097,"num_read_kb":1092,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":783,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820920+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:11:30.820920+0000","last_peered":"2024-09-21T19:11:30.820920+0000","last_clean":"2024-09-21T19:11:30.820920+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:11:30.820920+0000","last_undegraded":"2024-09-21T19:11:30.820920+0000","last_fullsized":"2024-09-21T19:11:30.820920+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.851635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":838,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810872+0000","last_change":"2024-09-21T19:10:17.073561+0000","last_active":"2024-09-21T19:11:30.810872+0000","last_peered":"2024-09-21T19:11:30.810872+0000","last_clean":"2024-09-21T19:11:30.810872+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:11:30.810872+0000","last_undegraded":"2024-09-21T19:11:30.810872+0000","last_fullsized":"2024-09-21T19:11:30.810872+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.073521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.073521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"444'7412","reported_seq":7483,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.090877+0000","last_change":"2024-09-21T19:10:01.260090+0000","last_active":"2024-09-21T19:11:33.090877+0000","last_peered":"2024-09-21T19:11:33.090877+0000","last_clean":"2024-09-21T19:11:33.090877+0000","last_became_active":"2024-09-21T19:10:01.259620+0000","last_became_peered":"2024-09-21T19:10:01.259620+0000","last_unstale":"2024-09-21T19:11:33.090877+0000","last_undegraded":"2024-09-21T19:11:33.090877+0000","last_fullsized":"2024-09-21T19:11:33.090877+0000","mapping_epoch":414,"log_start":"427'3000","ondisk_log_start":"427'3000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485752832,"num_objects":7412,"num_object_clones":0,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":1034,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.823182+0000","last_change":"2024-09-21T19:11:02.540038+0000","last_active":"2024-09-21T19:11:30.823182+0000","last_peered":"2024-09-21T19:11:30.823182+0000","last_clean":"2024-09-21T19:11:30.823182+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:11:30.823182+0000","last_undegraded":"2024-09-21T19:11:30.823182+0000","last_fullsized":"2024-09-21T19:11:30.823182+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:02.539974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.539974+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.492636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":838,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816768+0000","last_change":"2024-09-21T19:06:54.697048+0000","last_active":"2024-09-21T19:11:30.816768+0000","last_peered":"2024-09-21T19:11:30.816768+0000","last_clean":"2024-09-21T19:11:30.816768+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:11:30.816768+0000","last_undegraded":"2024-09-21T19:11:30.816768+0000","last_fullsized":"2024-09-21T19:11:30.816768+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.757120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.757120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:34.702481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":923,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812482+0000","last_change":"2024-09-21T19:09:46.543580+0000","last_active":"2024-09-21T19:11:30.812482+0000","last_peered":"2024-09-21T19:11:30.812482+0000","last_clean":"2024-09-21T19:11:30.812482+0000","last_became_active":"2024-09-21T19:02:40.886789+0000","last_became_peered":"2024-09-21T19:02:40.886789+0000","last_unstale":"2024-09-21T19:11:30.812482+0000","last_undegraded":"2024-09-21T19:11:30.812482+0000","last_fullsized":"2024-09-21T19:11:30.812482+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:46.543525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:46.543525+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:17.736334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":847,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.813983+0000","last_change":"2024-09-21T19:10:17.553053+0000","last_active":"2024-09-21T19:11:30.813983+0000","last_peered":"2024-09-21T19:11:30.813983+0000","last_clean":"2024-09-21T19:11:30.813983+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:11:30.813983+0000","last_undegraded":"2024-09-21T19:11:30.813983+0000","last_fullsized":"2024-09-21T19:11:30.813983+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.553007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.553007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:47.373270+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":373,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816043+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:11:30.816043+0000","last_peered":"2024-09-21T19:11:30.816043+0000","last_clean":"2024-09-21T19:11:30.816043+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:11:30.816043+0000","last_undegraded":"2024-09-21T19:11:30.816043+0000","last_fullsized":"2024-09-21T19:11:30.816043+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"444'7562","reported_seq":7633,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:34.367268+0000","last_change":"2024-09-21T19:10:01.250432+0000","last_active":"2024-09-21T19:11:34.367268+0000","last_peered":"2024-09-21T19:11:34.367268+0000","last_clean":"2024-09-21T19:11:34.367268+0000","last_became_active":"2024-09-21T19:10:01.250116+0000","last_became_peered":"2024-09-21T19:10:01.250116+0000","last_unstale":"2024-09-21T19:11:34.367268+0000","last_undegraded":"2024-09-21T19:11:34.367268+0000","last_fullsized":"2024-09-21T19:11:34.367268+0000","mapping_epoch":414,"log_start":"425'2200","ondisk_log_start":"425'2200","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495583232,"num_objects":7562,"num_object_clones":0,"num_object_copies":22686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":483968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":885,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810625+0000","last_change":"2024-09-21T19:10:23.782154+0000","last_active":"2024-09-21T19:11:30.810625+0000","last_peered":"2024-09-21T19:11:30.810625+0000","last_clean":"2024-09-21T19:11:30.810625+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:11:30.810625+0000","last_undegraded":"2024-09-21T19:11:30.810625+0000","last_fullsized":"2024-09-21T19:11:30.810625+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:10:23.782107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.782107+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":853,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809379+0000","last_change":"2024-09-21T19:10:19.729765+0000","last_active":"2024-09-21T19:11:30.809379+0000","last_peered":"2024-09-21T19:11:30.809379+0000","last_clean":"2024-09-21T19:11:30.809379+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:11:30.809379+0000","last_undegraded":"2024-09-21T19:11:30.809379+0000","last_fullsized":"2024-09-21T19:11:30.809379+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:19.729685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.729685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":837,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810034+0000","last_change":"2024-09-21T19:10:58.999039+0000","last_active":"2024-09-21T19:11:30.810034+0000","last_peered":"2024-09-21T19:11:30.810034+0000","last_clean":"2024-09-21T19:11:30.810034+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:11:30.810034+0000","last_undegraded":"2024-09-21T19:11:30.810034+0000","last_fullsized":"2024-09-21T19:11:30.810034+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.998847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.998847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:59.614433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":810,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.817075+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:11:30.817075+0000","last_peered":"2024-09-21T19:11:30.817075+0000","last_clean":"2024-09-21T19:11:30.817075+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:11:30.817075+0000","last_undegraded":"2024-09-21T19:11:30.817075+0000","last_fullsized":"2024-09-21T19:11:30.817075+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.968888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"444'7301","reported_seq":7438,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:33.083655+0000","last_change":"2024-09-21T19:11:09.898977+0000","last_active":"2024-09-21T19:11:33.083655+0000","last_peered":"2024-09-21T19:11:33.083655+0000","last_clean":"2024-09-21T19:11:33.083655+0000","last_became_active":"2024-09-21T19:10:01.247852+0000","last_became_peered":"2024-09-21T19:10:01.247852+0000","last_unstale":"2024-09-21T19:11:33.083655+0000","last_undegraded":"2024-09-21T19:11:33.083655+0000","last_fullsized":"2024-09-21T19:11:33.083655+0000","mapping_epoch":414,"log_start":"420'1000","ondisk_log_start":"420'1000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'5812","last_scrub_stamp":"2024-09-21T19:11:09.898694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.898694+0000","objects_scrubbed":5735,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:34.977808+0000","scrub_duration":2494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478478336,"num_objects":7301,"num_object_clones":0,"num_object_copies":21903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7301,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":283,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.815885+0000","last_change":"2024-09-21T19:10:14.570018+0000","last_active":"2024-09-21T19:11:30.815885+0000","last_peered":"2024-09-21T19:11:30.815885+0000","last_clean":"2024-09-21T19:11:30.815885+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:11:30.815885+0000","last_undegraded":"2024-09-21T19:11:30.815885+0000","last_fullsized":"2024-09-21T19:11:30.815885+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:14.569931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:14.569931+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:24.555695+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":926,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811415+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:11:30.811415+0000","last_peered":"2024-09-21T19:11:30.811415+0000","last_clean":"2024-09-21T19:11:30.811415+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:11:30.811415+0000","last_undegraded":"2024-09-21T19:11:30.811415+0000","last_fullsized":"2024-09-21T19:11:30.811415+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:41.851179+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":752,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821632+0000","last_change":"2024-09-21T19:10:17.814004+0000","last_active":"2024-09-21T19:11:30.821632+0000","last_peered":"2024-09-21T19:11:30.821632+0000","last_clean":"2024-09-21T19:11:30.821632+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:11:30.821632+0000","last_undegraded":"2024-09-21T19:11:30.821632+0000","last_fullsized":"2024-09-21T19:11:30.821632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.813917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.813917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:46.777123+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":284,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:31.656763+0000","last_change":"2024-09-21T19:11:31.656763+0000","last_active":"2024-09-21T19:11:31.656763+0000","last_peered":"2024-09-21T19:11:31.656763+0000","last_clean":"2024-09-21T19:11:31.656763+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:11:31.656763+0000","last_undegraded":"2024-09-21T19:11:31.656763+0000","last_fullsized":"2024-09-21T19:11:31.656763+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.656689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.656689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.381458+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":284,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818189+0000","last_change":"2024-09-21T19:10:45.402838+0000","last_active":"2024-09-21T19:11:30.818189+0000","last_peered":"2024-09-21T19:11:30.818189+0000","last_clean":"2024-09-21T19:11:30.818189+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:11:30.818189+0000","last_undegraded":"2024-09-21T19:11:30.818189+0000","last_fullsized":"2024-09-21T19:11:30.818189+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:45.402769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.402769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:57.226877+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"444'7540","reported_seq":7608,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:34.432285+0000","last_change":"2024-09-21T19:10:01.252391+0000","last_active":"2024-09-21T19:11:34.432285+0000","last_peered":"2024-09-21T19:11:34.432285+0000","last_clean":"2024-09-21T19:11:34.432285+0000","last_became_active":"2024-09-21T19:10:01.251894+0000","last_became_peered":"2024-09-21T19:10:01.251894+0000","last_unstale":"2024-09-21T19:11:34.432285+0000","last_undegraded":"2024-09-21T19:11:34.432285+0000","last_fullsized":"2024-09-21T19:11:34.432285+0000","mapping_epoch":414,"log_start":"425'2200","ondisk_log_start":"425'2200","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.727577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494141440,"num_objects":7540,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":482560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":938,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811993+0000","last_change":"2024-09-21T19:10:51.731370+0000","last_active":"2024-09-21T19:11:30.811993+0000","last_peered":"2024-09-21T19:11:30.811993+0000","last_clean":"2024-09-21T19:11:30.811993+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:11:30.811993+0000","last_undegraded":"2024-09-21T19:11:30.811993+0000","last_fullsized":"2024-09-21T19:11:30.811993+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:51.731299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.731299+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:56.746842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":852,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810356+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:11:30.810356+0000","last_peered":"2024-09-21T19:11:30.810356+0000","last_clean":"2024-09-21T19:11:30.810356+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:11:30.810356+0000","last_undegraded":"2024-09-21T19:11:30.810356+0000","last_fullsized":"2024-09-21T19:11:30.810356+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:40.839786+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":860,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818285+0000","last_change":"2024-09-21T19:10:55.909242+0000","last_active":"2024-09-21T19:11:30.818285+0000","last_peered":"2024-09-21T19:11:30.818285+0000","last_clean":"2024-09-21T19:11:30.818285+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:11:30.818285+0000","last_undegraded":"2024-09-21T19:11:30.818285+0000","last_fullsized":"2024-09-21T19:11:30.818285+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:55.909172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:55.909172+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.738817+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":356,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.823942+0000","last_change":"2024-09-21T19:10:24.479930+0000","last_active":"2024-09-21T19:11:30.823942+0000","last_peered":"2024-09-21T19:11:30.823942+0000","last_clean":"2024-09-21T19:11:30.823942+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:11:30.823942+0000","last_undegraded":"2024-09-21T19:11:30.823942+0000","last_fullsized":"2024-09-21T19:11:30.823942+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:10:24.479847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:24.479847+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":825,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809725+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:11:30.809725+0000","last_peered":"2024-09-21T19:11:30.809725+0000","last_clean":"2024-09-21T19:11:30.809725+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:11:30.809725+0000","last_undegraded":"2024-09-21T19:11:30.809725+0000","last_fullsized":"2024-09-21T19:11:30.809725+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:42.815657+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"444'7662","reported_seq":7732,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:36.950130+0000","last_change":"2024-09-21T19:10:01.250343+0000","last_active":"2024-09-21T19:11:36.950130+0000","last_peered":"2024-09-21T19:11:36.950130+0000","last_clean":"2024-09-21T19:11:36.950130+0000","last_became_active":"2024-09-21T19:10:01.249931+0000","last_became_peered":"2024-09-21T19:10:01.249931+0000","last_unstale":"2024-09-21T19:11:36.950130+0000","last_undegraded":"2024-09-21T19:11:36.950130+0000","last_fullsized":"2024-09-21T19:11:36.950130+0000","mapping_epoch":414,"log_start":"427'2800","ondisk_log_start":"427'2800","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4862,"log_dups_size":0,"ondisk_log_size":4862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.706001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502136832,"num_objects":7662,"num_object_clones":0,"num_object_copies":22986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":490368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":712,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829747+0000","last_change":"2024-09-21T19:06:09.060446+0000","last_active":"2024-09-21T19:11:30.829747+0000","last_peered":"2024-09-21T19:11:30.829747+0000","last_clean":"2024-09-21T19:11:30.829747+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:11:30.829747+0000","last_undegraded":"2024-09-21T19:11:30.829747+0000","last_fullsized":"2024-09-21T19:11:30.829747+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:06:09.060391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:09.060391+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.499806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":813,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.819025+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:11:30.819025+0000","last_peered":"2024-09-21T19:11:30.819025+0000","last_clean":"2024-09-21T19:11:30.819025+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:11:30.819025+0000","last_undegraded":"2024-09-21T19:11:30.819025+0000","last_fullsized":"2024-09-21T19:11:30.819025+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.953609+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":276,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820464+0000","last_change":"2024-09-21T19:10:27.397361+0000","last_active":"2024-09-21T19:11:30.820464+0000","last_peered":"2024-09-21T19:11:30.820464+0000","last_clean":"2024-09-21T19:11:30.820464+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:11:30.820464+0000","last_undegraded":"2024-09-21T19:11:30.820464+0000","last_fullsized":"2024-09-21T19:11:30.820464+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.397188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.397188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:57.228642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":724,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:31.588989+0000","last_change":"2024-09-21T19:11:31.588989+0000","last_active":"2024-09-21T19:11:31.588989+0000","last_peered":"2024-09-21T19:11:31.588989+0000","last_clean":"2024-09-21T19:11:31.588989+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:11:31.588989+0000","last_undegraded":"2024-09-21T19:11:31.588989+0000","last_fullsized":"2024-09-21T19:11:31.588989+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.588820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.588820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:42.713349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"444'7642","reported_seq":7713,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:36.936935+0000","last_change":"2024-09-21T19:10:01.260078+0000","last_active":"2024-09-21T19:11:36.936935+0000","last_peered":"2024-09-21T19:11:36.936935+0000","last_clean":"2024-09-21T19:11:36.936935+0000","last_became_active":"2024-09-21T19:10:01.259752+0000","last_became_peered":"2024-09-21T19:10:01.259752+0000","last_unstale":"2024-09-21T19:11:36.936935+0000","last_undegraded":"2024-09-21T19:11:36.936935+0000","last_fullsized":"2024-09-21T19:11:36.936935+0000","mapping_epoch":414,"log_start":"427'2800","ondisk_log_start":"427'2800","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.779638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500826112,"num_objects":7642,"num_object_clones":0,"num_object_copies":22926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":771,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821490+0000","last_change":"2024-09-21T19:10:12.835830+0000","last_active":"2024-09-21T19:11:30.821490+0000","last_peered":"2024-09-21T19:11:30.821490+0000","last_clean":"2024-09-21T19:11:30.821490+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:11:30.821490+0000","last_undegraded":"2024-09-21T19:11:30.821490+0000","last_fullsized":"2024-09-21T19:11:30.821490+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:10:12.835752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:12.835752+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.728578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":852,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.813350+0000","last_change":"2024-09-21T19:10:29.539254+0000","last_active":"2024-09-21T19:11:30.813350+0000","last_peered":"2024-09-21T19:11:30.813350+0000","last_clean":"2024-09-21T19:11:30.813350+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:11:30.813350+0000","last_undegraded":"2024-09-21T19:11:30.813350+0000","last_fullsized":"2024-09-21T19:11:30.813350+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:29.539210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:29.539210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.989413+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":279,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816053+0000","last_change":"2024-09-21T19:08:58.758437+0000","last_active":"2024-09-21T19:11:30.816053+0000","last_peered":"2024-09-21T19:11:30.816053+0000","last_clean":"2024-09-21T19:11:30.816053+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:11:30.816053+0000","last_undegraded":"2024-09-21T19:11:30.816053+0000","last_fullsized":"2024-09-21T19:11:30.816053+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:08:58.758328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:08:58.758328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:32.430448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":832,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812403+0000","last_change":"2024-09-21T19:10:58.720745+0000","last_active":"2024-09-21T19:11:30.812403+0000","last_peered":"2024-09-21T19:11:30.812403+0000","last_clean":"2024-09-21T19:11:30.812403+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:11:30.812403+0000","last_undegraded":"2024-09-21T19:11:30.812403+0000","last_fullsized":"2024-09-21T19:11:30.812403+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.720634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.720634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.021358+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"444'7510","reported_seq":7581,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:32.910102+0000","last_change":"2024-09-21T19:10:01.252118+0000","last_active":"2024-09-21T19:11:32.910102+0000","last_peered":"2024-09-21T19:11:32.910102+0000","last_clean":"2024-09-21T19:11:32.910102+0000","last_became_active":"2024-09-21T19:10:01.251738+0000","last_became_peered":"2024-09-21T19:10:01.251738+0000","last_unstale":"2024-09-21T19:11:32.910102+0000","last_undegraded":"2024-09-21T19:11:32.910102+0000","last_fullsized":"2024-09-21T19:11:32.910102+0000","mapping_epoch":414,"log_start":"424'2000","ondisk_log_start":"424'2000","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.479895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492175360,"num_objects":7510,"num_object_clones":0,"num_object_copies":22530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":480640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":889,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:31.160427+0000","last_change":"2024-09-21T19:11:31.160427+0000","last_active":"2024-09-21T19:11:31.160427+0000","last_peered":"2024-09-21T19:11:31.160427+0000","last_clean":"2024-09-21T19:11:31.160427+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:11:31.160427+0000","last_undegraded":"2024-09-21T19:11:31.160427+0000","last_fullsized":"2024-09-21T19:11:31.160427+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:11:31.160369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.160369+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.166557+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":854,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.814442+0000","last_change":"2024-09-21T19:10:57.731366+0000","last_active":"2024-09-21T19:11:30.814442+0000","last_peered":"2024-09-21T19:11:30.814442+0000","last_clean":"2024-09-21T19:11:30.814442+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:11:30.814442+0000","last_undegraded":"2024-09-21T19:11:30.814442+0000","last_fullsized":"2024-09-21T19:11:30.814442+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:57.731319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.731319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.882041+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":276,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.815795+0000","last_change":"2024-09-21T19:11:06.454799+0000","last_active":"2024-09-21T19:11:30.815795+0000","last_peered":"2024-09-21T19:11:30.815795+0000","last_clean":"2024-09-21T19:11:30.815795+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:11:30.815795+0000","last_undegraded":"2024-09-21T19:11:30.815795+0000","last_fullsized":"2024-09-21T19:11:30.815795+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:06.454727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.454727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.197782+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":804,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.817179+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:11:30.817179+0000","last_peered":"2024-09-21T19:11:30.817179+0000","last_clean":"2024-09-21T19:11:30.817179+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:11:30.817179+0000","last_undegraded":"2024-09-21T19:11:30.817179+0000","last_fullsized":"2024-09-21T19:11:30.817179+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.112801+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"444'7551","reported_seq":7690,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:32.785831+0000","last_change":"2024-09-21T19:11:09.208009+0000","last_active":"2024-09-21T19:11:32.785831+0000","last_peered":"2024-09-21T19:11:32.785831+0000","last_clean":"2024-09-21T19:11:32.785831+0000","last_became_active":"2024-09-21T19:10:01.259826+0000","last_became_peered":"2024-09-21T19:10:01.259826+0000","last_unstale":"2024-09-21T19:11:32.785831+0000","last_undegraded":"2024-09-21T19:11:32.785831+0000","last_fullsized":"2024-09-21T19:11:32.785831+0000","mapping_epoch":414,"log_start":"426'2700","ondisk_log_start":"426'2700","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'5961","last_scrub_stamp":"2024-09-21T19:11:09.207825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.207825+0000","objects_scrubbed":5889,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.058679+0000","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494862336,"num_objects":7551,"num_object_clones":0,"num_object_copies":22653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":444,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811855+0000","last_change":"2024-09-21T19:10:20.596047+0000","last_active":"2024-09-21T19:11:30.811855+0000","last_peered":"2024-09-21T19:11:30.811855+0000","last_clean":"2024-09-21T19:11:30.811855+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:11:30.811855+0000","last_undegraded":"2024-09-21T19:11:30.811855+0000","last_fullsized":"2024-09-21T19:11:30.811855+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:10:20.595995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:20.595995+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":860,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.813894+0000","last_change":"2024-09-21T19:09:30.608553+0000","last_active":"2024-09-21T19:11:30.813894+0000","last_peered":"2024-09-21T19:11:30.813894+0000","last_clean":"2024-09-21T19:11:30.813894+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:11:30.813894+0000","last_undegraded":"2024-09-21T19:11:30.813894+0000","last_fullsized":"2024-09-21T19:11:30.813894+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:09:30.608469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:30.608469+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:31.541007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":352,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.815728+0000","last_change":"2024-09-21T19:10:59.827898+0000","last_active":"2024-09-21T19:11:30.815728+0000","last_peered":"2024-09-21T19:11:30.815728+0000","last_clean":"2024-09-21T19:11:30.815728+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:11:30.815728+0000","last_undegraded":"2024-09-21T19:11:30.815728+0000","last_fullsized":"2024-09-21T19:11:30.815728+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:10:59.827833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.827833+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.990995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":847,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821276+0000","last_change":"2024-09-21T19:10:59.672382+0000","last_active":"2024-09-21T19:11:30.821276+0000","last_peered":"2024-09-21T19:11:30.821276+0000","last_clean":"2024-09-21T19:11:30.821276+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:11:30.821276+0000","last_undegraded":"2024-09-21T19:11:30.821276+0000","last_fullsized":"2024-09-21T19:11:30.821276+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:59.672320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.672320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:59.920295+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":782,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821218+0000","last_change":"2024-09-21T19:11:08.539733+0000","last_active":"2024-09-21T19:11:30.821218+0000","last_peered":"2024-09-21T19:11:30.821218+0000","last_clean":"2024-09-21T19:11:30.821218+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:11:30.821218+0000","last_undegraded":"2024-09-21T19:11:30.821218+0000","last_fullsized":"2024-09-21T19:11:30.821218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.539689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.539689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:35.735538+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":845,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816385+0000","last_change":"2024-09-21T19:10:24.873768+0000","last_active":"2024-09-21T19:11:30.816385+0000","last_peered":"2024-09-21T19:11:30.816385+0000","last_clean":"2024-09-21T19:11:30.816385+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:11:30.816385+0000","last_undegraded":"2024-09-21T19:11:30.816385+0000","last_fullsized":"2024-09-21T19:11:30.816385+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:10:24.873701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:24.873701+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.273198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":373,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.815768+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:11:30.815768+0000","last_peered":"2024-09-21T19:11:30.815768+0000","last_clean":"2024-09-21T19:11:30.815768+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:11:30.815768+0000","last_undegraded":"2024-09-21T19:11:30.815768+0000","last_fullsized":"2024-09-21T19:11:30.815768+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.027866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":820,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812752+0000","last_change":"2024-09-21T19:10:22.627798+0000","last_active":"2024-09-21T19:11:30.812752+0000","last_peered":"2024-09-21T19:11:30.812752+0000","last_clean":"2024-09-21T19:11:30.812752+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:11:30.812752+0000","last_undegraded":"2024-09-21T19:11:30.812752+0000","last_fullsized":"2024-09-21T19:11:30.812752+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:22.627740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:22.627740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":720,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830938+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:11:30.830938+0000","last_peered":"2024-09-21T19:11:30.830938+0000","last_clean":"2024-09-21T19:11:30.830938+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:11:30.830938+0000","last_undegraded":"2024-09-21T19:11:30.830938+0000","last_fullsized":"2024-09-21T19:11:30.830938+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.467919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":832,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812168+0000","last_change":"2024-09-21T19:10:18.539646+0000","last_active":"2024-09-21T19:11:30.812168+0000","last_peered":"2024-09-21T19:11:30.812168+0000","last_clean":"2024-09-21T19:11:30.812168+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:11:30.812168+0000","last_undegraded":"2024-09-21T19:11:30.812168+0000","last_fullsized":"2024-09-21T19:11:30.812168+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:18.539586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:18.539586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":815,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816751+0000","last_change":"2024-09-21T19:10:46.660901+0000","last_active":"2024-09-21T19:11:30.816751+0000","last_peered":"2024-09-21T19:11:30.816751+0000","last_clean":"2024-09-21T19:11:30.816751+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:11:30.816751+0000","last_undegraded":"2024-09-21T19:11:30.816751+0000","last_fullsized":"2024-09-21T19:11:30.816751+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.660838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.660838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:07.972413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":268,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818572+0000","last_change":"2024-09-21T19:09:01.758906+0000","last_active":"2024-09-21T19:11:30.818572+0000","last_peered":"2024-09-21T19:11:30.818572+0000","last_clean":"2024-09-21T19:11:30.818572+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:11:30.818572+0000","last_undegraded":"2024-09-21T19:11:30.818572+0000","last_fullsized":"2024-09-21T19:11:30.818572+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:01.758839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:01.758839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:40.410888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":434,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:34.827737+0000","last_change":"2024-09-21T19:11:34.827737+0000","last_active":"2024-09-21T19:11:34.827737+0000","last_peered":"2024-09-21T19:11:34.827737+0000","last_clean":"2024-09-21T19:11:34.827737+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:11:34.827737+0000","last_undegraded":"2024-09-21T19:11:34.827737+0000","last_fullsized":"2024-09-21T19:11:34.827737+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:34.827659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.827659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.075348+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":709,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830001+0000","last_change":"2024-09-21T19:06:16.040258+0000","last_active":"2024-09-21T19:11:30.830001+0000","last_peered":"2024-09-21T19:11:30.830001+0000","last_clean":"2024-09-21T19:11:30.830001+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:11:30.830001+0000","last_undegraded":"2024-09-21T19:11:30.830001+0000","last_fullsized":"2024-09-21T19:11:30.830001+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:16.040190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:16.040190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:37.462446+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1068,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818473+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:11:30.818473+0000","last_peered":"2024-09-21T19:11:30.818473+0000","last_clean":"2024-09-21T19:11:30.818473+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:11:30.818473+0000","last_undegraded":"2024-09-21T19:11:30.818473+0000","last_fullsized":"2024-09-21T19:11:30.818473+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":835,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.809926+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:11:30.809926+0000","last_peered":"2024-09-21T19:11:30.809926+0000","last_clean":"2024-09-21T19:11:30.809926+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:11:30.809926+0000","last_undegraded":"2024-09-21T19:11:30.809926+0000","last_fullsized":"2024-09-21T19:11:30.809926+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:29.127237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":860,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818749+0000","last_change":"2024-09-21T19:10:39.003954+0000","last_active":"2024-09-21T19:11:30.818749+0000","last_peered":"2024-09-21T19:11:30.818749+0000","last_clean":"2024-09-21T19:11:30.818749+0000","last_became_active":"2024-09-21T19:10:38.923267+0000","last_became_peered":"2024-09-21T19:10:38.923267+0000","last_unstale":"2024-09-21T19:11:30.818749+0000","last_undegraded":"2024-09-21T19:11:30.818749+0000","last_fullsized":"2024-09-21T19:11:30.818749+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:19.876033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.876033+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":843,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812977+0000","last_change":"2024-09-21T19:10:46.686915+0000","last_active":"2024-09-21T19:11:30.812977+0000","last_peered":"2024-09-21T19:11:30.812977+0000","last_clean":"2024-09-21T19:11:30.812977+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:11:30.812977+0000","last_undegraded":"2024-09-21T19:11:30.812977+0000","last_fullsized":"2024-09-21T19:11:30.812977+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.686843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.686843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.453957+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":518,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811950+0000","last_change":"2024-09-21T19:10:18.103484+0000","last_active":"2024-09-21T19:11:30.811950+0000","last_peered":"2024-09-21T19:11:30.811950+0000","last_clean":"2024-09-21T19:11:30.811950+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:11:30.811950+0000","last_undegraded":"2024-09-21T19:11:30.811950+0000","last_fullsized":"2024-09-21T19:11:30.811950+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:10:18.103406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:18.103406+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":709,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.831202+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:11:30.831202+0000","last_peered":"2024-09-21T19:11:30.831202+0000","last_clean":"2024-09-21T19:11:30.831202+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:11:30.831202+0000","last_undegraded":"2024-09-21T19:11:30.831202+0000","last_fullsized":"2024-09-21T19:11:30.831202+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.496959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":906,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.808617+0000","last_change":"2024-09-21T19:06:17.839499+0000","last_active":"2024-09-21T19:11:30.808617+0000","last_peered":"2024-09-21T19:11:30.808617+0000","last_clean":"2024-09-21T19:11:30.808617+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:11:30.808617+0000","last_undegraded":"2024-09-21T19:11:30.808617+0000","last_fullsized":"2024-09-21T19:11:30.808617+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:06:17.839433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:17.839433+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:37.861523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":825,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810598+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:11:30.810598+0000","last_peered":"2024-09-21T19:11:30.810598+0000","last_clean":"2024-09-21T19:11:30.810598+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:11:30.810598+0000","last_undegraded":"2024-09-21T19:11:30.810598+0000","last_fullsized":"2024-09-21T19:11:30.810598+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.166063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":833,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.810872+0000","last_change":"2024-09-21T19:09:33.104038+0000","last_active":"2024-09-21T19:11:30.810872+0000","last_peered":"2024-09-21T19:11:30.810872+0000","last_clean":"2024-09-21T19:11:30.810872+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:11:30.810872+0000","last_undegraded":"2024-09-21T19:11:30.810872+0000","last_fullsized":"2024-09-21T19:11:30.810872+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:33.103968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:33.103968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:39.125799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816122+0000","last_change":"2024-09-21T19:10:27.770820+0000","last_active":"2024-09-21T19:11:30.816122+0000","last_peered":"2024-09-21T19:11:30.816122+0000","last_clean":"2024-09-21T19:11:30.816122+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:11:30.816122+0000","last_undegraded":"2024-09-21T19:11:30.816122+0000","last_fullsized":"2024-09-21T19:11:30.816122+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.770751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.770751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:45.709466+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":912,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811196+0000","last_change":"2024-09-21T19:11:06.137061+0000","last_active":"2024-09-21T19:11:30.811196+0000","last_peered":"2024-09-21T19:11:30.811196+0000","last_clean":"2024-09-21T19:11:30.811196+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:11:30.811196+0000","last_undegraded":"2024-09-21T19:11:30.811196+0000","last_fullsized":"2024-09-21T19:11:30.811196+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:06.136973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.136973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.021915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":837,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.813134+0000","last_change":"2024-09-21T19:11:07.755091+0000","last_active":"2024-09-21T19:11:30.813134+0000","last_peered":"2024-09-21T19:11:30.813134+0000","last_clean":"2024-09-21T19:11:30.813134+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:11:30.813134+0000","last_undegraded":"2024-09-21T19:11:30.813134+0000","last_fullsized":"2024-09-21T19:11:30.813134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:07.755026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.755026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.561489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":829,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818052+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:11:30.818052+0000","last_peered":"2024-09-21T19:11:30.818052+0000","last_clean":"2024-09-21T19:11:30.818052+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:11:30.818052+0000","last_undegraded":"2024-09-21T19:11:30.818052+0000","last_fullsized":"2024-09-21T19:11:30.818052+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:26.155553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":720,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.829519+0000","last_change":"2024-09-21T19:06:54.695552+0000","last_active":"2024-09-21T19:11:30.829519+0000","last_peered":"2024-09-21T19:11:30.829519+0000","last_clean":"2024-09-21T19:11:30.829519+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:11:30.829519+0000","last_undegraded":"2024-09-21T19:11:30.829519+0000","last_fullsized":"2024-09-21T19:11:30.829519+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:51.135723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:05:51.135723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:33.411163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":853,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.816652+0000","last_change":"2024-09-21T19:05:30.713724+0000","last_active":"2024-09-21T19:11:30.816652+0000","last_peered":"2024-09-21T19:11:30.816652+0000","last_clean":"2024-09-21T19:11:30.816652+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:11:30.816652+0000","last_undegraded":"2024-09-21T19:11:30.816652+0000","last_fullsized":"2024-09-21T19:11:30.816652+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:05:30.713543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:05:30.713543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:35.695151+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":774,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.819880+0000","last_change":"2024-09-21T19:11:03.004430+0000","last_active":"2024-09-21T19:11:30.819880+0000","last_peered":"2024-09-21T19:11:30.819880+0000","last_clean":"2024-09-21T19:11:30.819880+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:11:30.819880+0000","last_undegraded":"2024-09-21T19:11:30.819880+0000","last_fullsized":"2024-09-21T19:11:30.819880+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.004386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.004386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.746086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":856,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811645+0000","last_change":"2024-09-21T19:11:01.724461+0000","last_active":"2024-09-21T19:11:30.811645+0000","last_peered":"2024-09-21T19:11:30.811645+0000","last_clean":"2024-09-21T19:11:30.811645+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:11:30.811645+0000","last_undegraded":"2024-09-21T19:11:30.811645+0000","last_fullsized":"2024-09-21T19:11:30.811645+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:01.724417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:01.724417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:02.723695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":836,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.818194+0000","last_change":"2024-09-21T19:10:16.861750+0000","last_active":"2024-09-21T19:11:30.818194+0000","last_peered":"2024-09-21T19:11:30.818194+0000","last_clean":"2024-09-21T19:11:30.818194+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:11:30.818194+0000","last_undegraded":"2024-09-21T19:11:30.818194+0000","last_fullsized":"2024-09-21T19:11:30.818194+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.861671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.861671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.883789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":860,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.820796+0000","last_change":"2024-09-21T19:10:52.788414+0000","last_active":"2024-09-21T19:11:30.820796+0000","last_peered":"2024-09-21T19:11:30.820796+0000","last_clean":"2024-09-21T19:11:30.820796+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:11:30.820796+0000","last_undegraded":"2024-09-21T19:11:30.820796+0000","last_fullsized":"2024-09-21T19:11:30.820796+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:10:52.788342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:52.788342+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:04.702000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":712,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.830782+0000","last_change":"2024-09-21T19:11:02.421016+0000","last_active":"2024-09-21T19:11:30.830782+0000","last_peered":"2024-09-21T19:11:30.830782+0000","last_clean":"2024-09-21T19:11:30.830782+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:11:30.830782+0000","last_undegraded":"2024-09-21T19:11:30.830782+0000","last_fullsized":"2024-09-21T19:11:30.830782+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:02.420940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.420940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.532139+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":741,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821308+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:11:30.821308+0000","last_peered":"2024-09-21T19:11:30.821308+0000","last_clean":"2024-09-21T19:11:30.821308+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:11:30.821308+0000","last_undegraded":"2024-09-21T19:11:30.821308+0000","last_fullsized":"2024-09-21T19:11:30.821308+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.662425+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":831,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.812056+0000","last_change":"2024-09-21T19:11:09.805596+0000","last_active":"2024-09-21T19:11:30.812056+0000","last_peered":"2024-09-21T19:11:30.812056+0000","last_clean":"2024-09-21T19:11:30.812056+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:11:30.812056+0000","last_undegraded":"2024-09-21T19:11:30.812056+0000","last_fullsized":"2024-09-21T19:11:30.812056+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:09.805527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.805527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:24.581512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":829,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.811783+0000","last_change":"2024-09-21T19:09:00.105884+0000","last_active":"2024-09-21T19:11:30.811783+0000","last_peered":"2024-09-21T19:11:30.811783+0000","last_clean":"2024-09-21T19:11:30.811783+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:11:30.811783+0000","last_undegraded":"2024-09-21T19:11:30.811783+0000","last_fullsized":"2024-09-21T19:11:30.811783+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:00.105826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:00.105826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:38.129647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":845,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821055+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:11:30.821055+0000","last_peered":"2024-09-21T19:11:30.821055+0000","last_clean":"2024-09-21T19:11:30.821055+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:11:30.821055+0000","last_undegraded":"2024-09-21T19:11:30.821055+0000","last_fullsized":"2024-09-21T19:11:30.821055+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.009111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":823,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.821511+0000","last_change":"2024-09-21T19:11:00.629261+0000","last_active":"2024-09-21T19:11:30.821511+0000","last_peered":"2024-09-21T19:11:30.821511+0000","last_clean":"2024-09-21T19:11:30.821511+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:11:30.821511+0000","last_undegraded":"2024-09-21T19:11:30.821511+0000","last_fullsized":"2024-09-21T19:11:30.821511+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:00.629191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.629191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.169952+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":732,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T19:11:30.831121+0000","last_change":"2024-09-21T19:10:23.500631+0000","last_active":"2024-09-21T19:11:30.831121+0000","last_peered":"2024-09-21T19:11:30.831121+0000","last_clean":"2024-09-21T19:11:30.831121+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:11:30.831121+0000","last_undegraded":"2024-09-21T19:11:30.831121+0000","last_fullsized":"2024-09-21T19:11:30.831121+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:23.500562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.500562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:42.786621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7869497344,"num_objects":120079,"num_object_clones":0,"num_object_copies":360237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120079,"num_write_kb":7685056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23581687808,"data_stored":23581687808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80379,"ondisk_log_size":80379,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":5446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1097,"num_read_kb":1092,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266914,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3608548,"kb_used_data":2941524,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90136604,"statfs":{"total":95995035648,"available":92299882496,"internally_reserved":0,"allocated":3012120576,"data_stored":3011508794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":683014602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464364,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4221460,"kb_used_data":3816004,"kb_used_omap":4,"kb_used_meta":405435,"kb_avail":89523692,"statfs":{"total":95995035648,"available":91672260608,"internally_reserved":0,"allocated":3907588096,"data_stored":3906992049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":415166116},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619065,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2207196,"kb_used_data":1901196,"kb_used_omap":4,"kb_used_meta":305979,"kb_avail":91537956,"statfs":{"total":95995035648,"available":93734866944,"internally_reserved":0,"allocated":1946824704,"data_stored":1946221139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":313323171},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019120,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2549100,"kb_used_data":1954268,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91196052,"statfs":{"total":95995035648,"available":93384757248,"internally_reserved":0,"allocated":2001170432,"data_stored":2000550958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182644,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4130684,"kb_used_data":3315116,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":89614468,"statfs":{"total":95995035648,"available":91765215232,"internally_reserved":0,"allocated":3394678784,"data_stored":3394043769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411578,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3478856,"kb_used_data":2856064,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":90266296,"statfs":{"total":95995035648,"available":92432687104,"internally_reserved":0,"allocated":2924609536,"data_stored":2924018264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575102,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4595848,"kb_used_data":3798328,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":89149304,"statfs":{"total":95995035648,"available":91288887296,"internally_reserved":0,"allocated":3889487872,"data_stored":3888840129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738628,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3260840,"kb_used_data":2377240,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90484312,"statfs":{"total":95995035648,"available":92655935488,"internally_reserved":0,"allocated":2434293760,"data_stored":2433678571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2437349376,"data_stored":2437349376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3912433664,"data_stored":3912433664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2933325824,"data_stored":2933325824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3396534272,"data_stored":3396534272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2007236608,"data_stored":2007236608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1951858688,"data_stored":1951858688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3929341952,"data_stored":3929341952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3013607424,"data_stored":3013607424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:11:42.321 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T19:11:42.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-21T19:11:42.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:42.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:42.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:42.492 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:42.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:42.623 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:42.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:42.759 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:42.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:42.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:42.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:42.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:11:42.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:43.030 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:43.052 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:43.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:43.175 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: pgmap v917: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 929 B/s rd, 58 MiB/s wr, 991 op/s 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: 1.4 scrub starts 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: 1.4 scrub ok 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: from='client.26473 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: 5.15 scrub starts 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: 5.15 scrub ok 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: pgmap v917: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 929 B/s rd, 58 MiB/s wr, 991 op/s 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: 1.4 scrub starts 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: 1.4 scrub ok 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: from='client.26473 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: 5.15 scrub starts 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: 5.15 scrub ok 2024-09-21T19:11:43.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T19:11:43.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:43 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T19:11:43.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:43.304 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:43.405 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:43.429 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:43.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: pgmap v917: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 929 B/s rd, 58 MiB/s wr, 991 op/s 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: 1.4 scrub starts 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: 1.4 scrub ok 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: from='client.26473 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: 5.15 scrub starts 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: 5.15 scrub ok 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T19:11:43.499 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:43 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T19:11:43.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:43.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:11:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.570 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:43.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:43.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:43.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:43.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:43.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:43.840 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:43.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:43.980 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:43.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:44.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:44.116 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:44.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:44.250 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:11:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:11:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T19:11:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T19:11:44.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:11:44.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:11:44.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T19:11:44.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:44 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T19:11:44.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:44.362 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:44 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:11:44.362 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:44 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:11:44.362 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:44 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T19:11:44.362 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:44 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T19:11:44.396 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:44.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:44.510 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:44.510 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:44.536 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:44.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:44.666 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:44.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:44.804 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:44.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:44.935 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:44.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:44.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:11:45.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:45.058 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 1 pg_num to 21 2024-09-21T19:11:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:45.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:45.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:45.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:45.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:45.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:45.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:45.203 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:45.249 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: pgmap v919: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 1.03k op/s 2024-09-21T19:11:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2590041028' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: from='client.36632 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T19:11:45.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:45 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1658727797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: pgmap v919: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 1.03k op/s 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2590041028' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: from='client.36632 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T19:11:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1658727797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: pgmap v919: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 1.03k op/s 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2590041028' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: from='client.36632 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T19:11:45.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:45 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1658727797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:45.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.372 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:45.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:45.402 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:11:45.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:45.507 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:45.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:45.646 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:45.748 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:45.748 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:45.776 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:45.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:45.909 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:45.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:46.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:46.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:46.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: 3.11 scrub starts 2024-09-21T19:11:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: 3.11 scrub ok 2024-09-21T19:11:46.172 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: 4.0 scrub starts 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: 4.0 scrub ok 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: from='client.36632 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:46.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:46 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:46.183 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:46.285 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:46.325 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:46.326 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:11:46.326 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:11:46.326 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:11:46.326 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:46.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:46.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: 3.11 scrub starts 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: 3.11 scrub ok 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: 4.0 scrub starts 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: 4.0 scrub ok 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: from='client.36632 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:46.515 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: 3.11 scrub starts 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: 3.11 scrub ok 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: 4.0 scrub starts 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: 4.0 scrub ok 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: from='client.36632 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:46.516 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:46 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:46.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:46.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:46.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:46.853 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:46.955 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:46.977 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:46.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:47.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:47.103 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:47.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: pgmap v922: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 2.5 MiB/s wr, 906 op/s 2024-09-21T19:11:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:11:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: pgmap v922: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 2.5 MiB/s wr, 906 op/s 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T19:11:47.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T19:11:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T19:11:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T19:11:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T19:11:47.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T19:11:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:47.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.366 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:47.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:47.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:47.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:47.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: pgmap v922: 158 pgs: 158 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 2.5 MiB/s wr, 906 op/s 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T19:11:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:47 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T19:11:47.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:11:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:11:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:11:47.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:47.630 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:47.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.771 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:47.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:47.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:47.904 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:47.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:48.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:48.017 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:48.029 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:48.131 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:48.159 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:48.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:48.291 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:48.371 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 1.11 scrub starts 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: 1.11 scrub ok 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:48.372 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 1.11 scrub starts 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: 1.11 scrub ok 2024-09-21T19:11:48.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 1.11 scrub starts 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: 1.11 scrub ok 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:48.378 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:48 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T19:11:48.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:48.416 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:11:48.422 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:48.523 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:48.558 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:48.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:48.700 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:48.783 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:48.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.843 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:48.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:48.945 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:48.984 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:48.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:49.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:11:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:11:49.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:49.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:49.119 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:11:49.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:49.250 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:49.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:49.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: pgmap v925: 168 pgs: 1 active+clean+scrubbing, 3 peering, 10 unknown, 154 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 499 op/s 2024-09-21T19:11:49.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: 1.5 scrub starts 2024-09-21T19:11:49.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: 1.5 scrub ok 2024-09-21T19:11:49.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:11:49.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:11:49.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T19:11:49.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T19:11:49.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T19:11:49.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T19:11:49.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:49 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1826478847' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:49.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:49.353 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149397 10.a 10:5708e5f1:::benchmark_data_smithi081_155596_object31924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.14008297500001, 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56206911699999995, 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.354 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149397 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207080+0000", 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3900000000000005e-07 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207083+0000", 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.142e-06 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207087+0000", 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6809999999999997e-06 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207092+0000", 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4989999999999996e-06 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207456+0000", 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036441799999999998 2024-09-21T19:11:49.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207464+0000", 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2269999999999993e-06 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.765864+0000", 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.55839969 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.765987+0000", 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012294199999999999 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.766111+0000", 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124268 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.768991+0000", 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002879904 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769047+0000", 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5457000000000001e-05 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769062+0000", 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5401999999999999e-05 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769108+0000", 2024-09-21T19:11:49.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5580000000000001e-05 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769149+0000", 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1368000000000001e-05 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.362 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149401 10.a 10:5176693a:::benchmark_data_smithi081_155596_object31928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.207800+0000", 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.139363011, 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.57158363599999995, 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.363 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149401 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207800+0000", 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207800+0000", 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207801+0000", 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2899999999999999e-06 2024-09-21T19:11:49.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207806+0000", 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9529999999999998e-06 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207818+0000", 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1945e-05 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207825+0000", 2024-09-21T19:11:49.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6839999999999999e-06 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207872+0000", 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6999000000000002e-05 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.207879+0000", 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.379e-06 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769179+0000", 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.56129959100000004 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769253+0000", 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3997000000000003e-05 2024-09-21T19:11:49.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.769326+0000", 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2886000000000007e-05 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774586+0000", 2024-09-21T19:11:49.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052604319999999998 2024-09-21T19:11:49.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774601+0000", 2024-09-21T19:11:49.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4745e-05 2024-09-21T19:11:49.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779292+0000", 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046908790000000002 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779342+0000", 2024-09-21T19:11:49.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0544000000000003e-05 2024-09-21T19:11:49.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779384+0000", 2024-09-21T19:11:49.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1312e-05 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149415 10.a 10:58e095b4:::benchmark_data_smithi081_155596_object31942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.374 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.221767+0000", 2024-09-21T19:11:49.375 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.125396287, 2024-09-21T19:11:49.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.55778803399999999, 2024-09-21T19:11:49.375 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.375 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.375 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.375 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149415 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221767+0000", 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221767+0000", 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221768+0000", 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.728e-06 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221774+0000", 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2020000000000003e-06 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221787+0000", 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3413e-05 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221795+0000", 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9840000000000001e-06 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221956+0000", 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016143 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.221966+0000", 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0108999999999999e-05 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.771078+0000", 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.54911152699999999 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.771155+0000", 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7137000000000004e-05 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.379 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.771223+0000", 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8151000000000005e-05 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774649+0000", 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034256159999999998 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774713+0000", 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3899000000000006e-05 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.380 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779438+0000", 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047251480000000002 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779491+0000", 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3065000000000001e-05 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779555+0000", 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3625000000000002e-05 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149439 10.a 10:598e2f20:::benchmark_data_smithi081_155596_object31966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.242682+0000", 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.10448068400001, 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.537047512, 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.382 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149439 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242682+0000", 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242682+0000", 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242683+0000", 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5000000000000001e-07 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242686+0000", 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9239999999999999e-06 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242710+0000", 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4128e-05 2024-09-21T19:11:49.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242716+0000", 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5300000000000004e-06 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242772+0000", 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5874000000000001e-05 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.242781+0000", 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1039999999999996e-06 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.772708+0000", 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.529927284 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.772783+0000", 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5451999999999995e-05 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.772849+0000", 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5692999999999996e-05 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774701+0000", 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001851669 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774719+0000", 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8284999999999999e-05 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779643+0000", 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049241600000000003 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779693+0000", 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9941000000000002e-05 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779730+0000", 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6517999999999998e-05 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.389 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149442 10.a 10:57628efb:::benchmark_data_smithi081_155596_object31969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.246877+0000", 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.100285472, 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.53906758300000002, 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149442 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246877+0000", 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246877+0000", 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246878+0000", 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9100000000000003e-07 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246881+0000", 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9359999999999999e-06 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246897+0000", 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5753e-05 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246908+0000", 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1532000000000001e-05 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246972+0000", 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3788000000000007e-05 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.246985+0000", 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.308e-05 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774753+0000", 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52776764600000003 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774857+0000", 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010378 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.774969+0000", 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011192 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779566+0000", 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045972069999999999 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779600+0000", 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4319999999999997e-05 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785815+0000", 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006214448 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785901+0000", 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6349999999999998e-05 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785945+0000", 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4131999999999997e-05 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149455 10.a 10:5dea2684:::benchmark_data_smithi081_155596_object31982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.260978+0000", 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.08618540200001, 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.52510691499999995, 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.396 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149455 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260978+0000", 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260978+0000", 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260980+0000", 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0310000000000001e-06 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260988+0000", 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6880000000000005e-06 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.260999+0000", 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0530000000000001e-05 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.261007+0000", 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0239999999999997e-06 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.261093+0000", 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6580000000000001e-05 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.261103+0000", 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4380000000000001e-06 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.777104+0000", 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51600082300000005 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.777218+0000", 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113892 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.777337+0000", 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000119475 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779740+0000", 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002402829 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779759+0000", 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9296999999999998e-05 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.785986+0000", 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062272100000000004 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786052+0000", 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5537999999999995e-05 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786084+0000", 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.256e-05 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149476 10.a 10:54c43eb7:::benchmark_data_smithi081_155596_object32003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.279885+0000", 2024-09-21T19:11:49.402 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.06727783100001, 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51528070100000001, 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149476 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279885+0000", 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279885+0000", 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279886+0000", 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.285e-06 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279891+0000", 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4580000000000003e-06 2024-09-21T19:11:49.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279900+0000", 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7649999999999999e-06 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.279907+0000", 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1690000000000003e-06 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.280422+0000", 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00051480700000000005 2024-09-21T19:11:49.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.280427+0000", 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6879999999999996e-06 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779798+0000", 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49937083500000001 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.779910+0000", 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111414 2024-09-21T19:11:49.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.780020+0000", 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011003899999999999 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786130+0000", 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006110959 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792039+0000", 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059088609999999996 2024-09-21T19:11:49.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795063+0000", 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030232560000000002 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795120+0000", 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7145999999999998e-05 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795166+0000", 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6019000000000001e-05 2024-09-21T19:11:49.408 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149487 10.a 10:5fee5d06:::benchmark_data_smithi081_155596_object32014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.288631+0000", 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.05853158900001, 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51888262600000001, 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.409 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149487 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288631+0000", 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288631+0000", 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288632+0000", 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3900000000000003e-07 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288636+0000", 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6569999999999999e-06 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288642+0000", 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7769999999999999e-06 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288646+0000", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1169999999999997e-06 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288720+0000", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3895999999999999e-05 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.288725+0000", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5419999999999996e-06 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.782607+0000", 2024-09-21T19:11:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49388191799999998 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.782847+0000", 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00023967699999999999 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.782968+0000", 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120704 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786166+0000", 2024-09-21T19:11:49.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031983300000000001 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794941+0000", 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.008774703 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807354+0000", 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012413258999999999 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807455+0000", 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010142800000000001 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807514+0000", 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8678999999999998e-05 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149495 10.a 10:5fba34ce:::benchmark_data_smithi081_155596_object32022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.295642+0000", 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.051520815, 2024-09-21T19:11:49.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.51213065599999996, 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149495 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295642+0000", 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295642+0000", 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295643+0000", 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.328e-06 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295648+0000", 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3159999999999998e-06 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295655+0000", 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9800000000000001e-06 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295661+0000", 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9309999999999996e-06 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295735+0000", 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4820999999999994e-05 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.295746+0000", 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0433e-05 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786197+0000", 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49045062099999998 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786299+0000", 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102802 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.786409+0000", 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109434 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794946+0000", 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0085368019999999996 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794987+0000", 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1854000000000003e-05 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807610+0000", 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012622301000000001 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807731+0000", 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120884 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807773+0000", 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2148999999999999e-05 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.421 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149497 10.a 10:5debb899:::benchmark_data_smithi081_155596_object32024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.298977+0000", 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.04818630099999, 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50910686000000005, 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149497 2024-09-21T19:11:49.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298977+0000", 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298977+0000", 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298978+0000", 2024-09-21T19:11:49.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.373e-06 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298983+0000", 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8189999999999999e-06 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298992+0000", 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9209999999999998e-06 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.298998+0000", 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3199999999999996e-06 2024-09-21T19:11:49.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.299066+0000", 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7823000000000004e-05 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.299076+0000", 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0282000000000001e-05 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.789200+0000", 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.490123859 2024-09-21T19:11:49.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.789337+0000", 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013721999999999999 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.789456+0000", 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001186 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.794998+0000", 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055423909999999998 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801375+0000", 2024-09-21T19:11:49.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063767989999999998 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807817+0000", 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064422699999999999 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808046+0000", 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022911599999999999 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808083+0000", 2024-09-21T19:11:49.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7067000000000003e-05 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149511 10.a 10:59ccc354:::benchmark_data_smithi081_155596_object32038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.312604+0000", 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.03455877299999, 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.49560010599999998, 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.430 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149511 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312604+0000", 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312604+0000", 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312606+0000", 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6130000000000001e-06 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312611+0000", 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9359999999999998e-06 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312623+0000", 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.259e-05 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312631+0000", 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3100000000000003e-06 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312692+0000", 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0998000000000001e-05 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.312703+0000", 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1439e-05 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792059+0000", 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47935586499999999 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792135+0000", 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6219999999999999e-05 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.792258+0000", 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000122569 2024-09-21T19:11:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801418+0000", 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091600880000000003 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807547+0000", 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061290930000000004 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808125+0000", 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057815500000000001 2024-09-21T19:11:49.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808172+0000", 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6820999999999998e-05 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808204+0000", 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2409000000000001e-05 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.436 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149512 10.a 10:5cee4d35:::benchmark_data_smithi081_155596_object32039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.313008+0000", 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.034155196, 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50836416399999995, 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149512 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313008+0000", 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313008+0000", 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313009+0000", 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.215e-06 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313013+0000", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.121e-06 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313023+0000", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0436e-05 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313029+0000", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2979999999999999e-06 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313065+0000", 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6677000000000001e-05 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.313071+0000", 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.152e-06 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795199+0000", 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48212796699999999 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795319+0000", 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120629 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.795444+0000", 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124337 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807553+0000", 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012109488999999999 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808248+0000", 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069539600000000003 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821240+0000", 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012991864000000001 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821311+0000", 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0576999999999999e-05 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821372+0000", 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1005999999999998e-05 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149517 10.a 10:5b635e0f:::benchmark_data_smithi081_155596_object32044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.319426+0000", 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.02773738299999, 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50211374600000003, 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.443 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149517 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319426+0000", 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319426+0000", 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319427+0000", 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5349999999999999e-06 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319432+0000", 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7779999999999998e-06 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319445+0000", 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3652000000000001e-05 2024-09-21T19:11:49.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319451+0000", 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9599999999999997e-06 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319518+0000", 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6574000000000003e-05 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.319535+0000", 2024-09-21T19:11:49.446 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6835999999999999e-05 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.798217+0000", 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.47868188699999997 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.798331+0000", 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000114641 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.798460+0000", 2024-09-21T19:11:49.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012896199999999999 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807558+0000", 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0090974139999999998 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808283+0000", 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072537299999999997 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821438+0000", 2024-09-21T19:11:49.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013154654 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821490+0000", 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2448999999999999e-05 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821539+0000", 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9030999999999999e-05 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149518 10.a 10:51048542:::benchmark_data_smithi081_155596_object32045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.320165+0000", 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 210.026998381, 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.50150713199999997, 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149518 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320165+0000", 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320165+0000", 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320166+0000", 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0330000000000001e-06 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320172+0000", 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6839999999999999e-06 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320181+0000", 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0950000000000002e-06 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320187+0000", 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4330000000000003e-06 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320231+0000", 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3899000000000001e-05 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.320236+0000", 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2859999999999999e-06 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801454+0000", 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.48121767599999998 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801604+0000", 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000150123 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.801737+0000", 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000133153 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807562+0000", 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058254550000000002 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.808500+0000", 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093748900000000001 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821600+0000", 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013100067999999999 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821650+0000", 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0327000000000001e-05 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821672+0000", 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1410999999999999e-05 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149533 10.a 10:506d36d3:::benchmark_data_smithi081_155596_object32060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.355055+0000", 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 209.992107839, 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.46668358700000001, 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.456 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149533 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355055+0000", 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355055+0000", 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355058+0000", 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7420000000000002e-06 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355066+0000", 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.208e-06 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355084+0000", 2024-09-21T19:11:49.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8371000000000001e-05 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355094+0000", 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8689999999999999e-06 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355167+0000", 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3161e-05 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.355180+0000", 2024-09-21T19:11:49.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2319e-05 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.804086+0000", 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.44890657899999997 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.804216+0000", 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013010100000000001 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,5", 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.804346+0000", 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012922000000000001 2024-09-21T19:11:49.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.807567+0000", 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032212080000000001 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.814723+0000", 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0071559900000000001 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821698+0000", 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069747979999999999 2024-09-21T19:11:49.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821721+0000", 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2918999999999999e-05 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.821739+0000", 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8102e-05 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.462 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:149663 10.0 10:0e2e1098:::benchmark_data_smithi081_155596_object32190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:08:19.887455+0000", 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 209.45970815999999, 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.31268661800000003, 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 149663 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887455+0000", 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887455+0000", 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887457+0000", 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.046e-06 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887463+0000", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9909999999999999e-06 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887474+0000", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1377e-05 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887481+0000", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1149999999999996e-06 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887632+0000", 2024-09-21T19:11:49.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015040799999999999 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887749+0000", 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000117589 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.887859+0000", 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109308 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.890594+0000", 2024-09-21T19:11:49.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027349150000000001 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:19.894705+0000", 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041116779999999997 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200077+0000", 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.30537224099999999 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200115+0000", 2024-09-21T19:11:49.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8031000000000002e-05 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:08:20.200141+0000", 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5919e-05 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36077.0:189632 10.7 10:e3409a5e:::benchmark_data_smithi081_155596_object72159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 167.06607106000001, 2024-09-21T19:11:49.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.310254274, 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36077", 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 189632 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281092+0000", 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4799999999999998e-07 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281095+0000", 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3769999999999999e-06 2024-09-21T19:11:49.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281103+0000", 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8269999999999998e-06 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.281107+0000", 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6480000000000002e-06 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.292262+0000", 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011154181000000001 2024-09-21T19:11:49.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.292359+0000", 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7398999999999994e-05 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.304951+0000", 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012592348 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.305035+0000", 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3821999999999998e-05 2024-09-21T19:11:49.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.305160+0000", 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012451500000000001 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.311340+0000", 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061803090000000001 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.319670+0000", 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083303030000000007 2024-09-21T19:11:49.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.590170+0000", 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.27049980099999998 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.590223+0000", 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.304e-05 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:09:02.591346+0000", 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011230560000000001 2024-09-21T19:11:49.474 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:95336 11.1 11:89358cba:::benchmark_data_smithi081_181703_object95335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:10.591248+0000", 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.755915383999998, 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.40602611, 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:49.475 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95336 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591248+0000", 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591248+0000", 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591249+0000", 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4589999999999999e-06 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591302+0000", 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3298999999999999e-05 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591313+0000", 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0725e-05 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591320+0000", 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2960000000000001e-06 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591377+0000", 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6789999999999997e-05 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591489+0000", 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111832 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:49.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.591593+0000", 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103959 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.597984+0000", 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063915259999999998 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.598087+0000", 2024-09-21T19:11:49.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102971 2024-09-21T19:11:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:49.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:49.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:49.480 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: pgmap v925: 168 pgs: 1 active+clean+scrubbing, 3 peering, 10 unknown, 154 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 499 op/s 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: 1.5 scrub starts 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: 1.5 scrub ok 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1826478847' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: pgmap v925: 168 pgs: 1 active+clean+scrubbing, 3 peering, 10 unknown, 154 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 499 op/s 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: 1.5 scrub starts 2024-09-21T19:11:49.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: 1.5 scrub ok 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T19:11:49.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:49 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1826478847' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:11:49.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.997156+0000", 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.39906866600000002 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.997214+0000", 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8048999999999999e-05 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.997274+0000", 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9537999999999998e-05 2024-09-21T19:11:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:95361 11.1 11:8d53b4d7:::benchmark_data_smithi081_181703_object95360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:10.613151+0000", 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 38.734011445, 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38821530999999998, 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:49.484 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 95361 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613151+0000", 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613151+0000", 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613153+0000", 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.266e-06 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613217+0000", 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3825000000000006e-05 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613227+0000", 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0779e-05 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613233+0000", 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4480000000000002e-06 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613291+0000", 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8e-05 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613400+0000", 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109319 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.613497+0000", 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6518e-05 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.616180+0000", 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026830410000000002 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:10.616311+0000", 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013171600000000001 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:11.001264+0000", 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.38495236399999999 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:11.001328+0000", 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4430999999999993e-05 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:11.001367+0000", 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8603000000000003e-05 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:100773 11.3 11:cc170bc3:::benchmark_data_smithi081_181703_object100772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:11:16.217160+0000", 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.130002476999998, 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.353879954, 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 100773 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.490 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217160+0000", 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217160+0000", 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217162+0000", 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829999999999999e-06 2024-09-21T19:11:49.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217262+0000", 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100076 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217278+0000", 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6039e-05 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217286+0000", 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7610000000000007e-06 2024-09-21T19:11:49.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217346+0000", 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9719000000000002e-05 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217470+0000", 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000124738 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.217638+0000", 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016798099999999999 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.221211+0000", 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035725650000000002 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.222943+0000", 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0017320110000000001 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.570823+0000", 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.34788053299999999 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.570979+0000", 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015552900000000001 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:11:16.571040+0000", 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1419000000000003e-05 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:11:49.495 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:11:49.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:49.619 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:49.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:49.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:49.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:49.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:49.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:49.897 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:49.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:49.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:11:49.999 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:50.027 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:50.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:50.156 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: 1.4 scrub starts 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: 1.4 scrub ok 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1826478847' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:11:50.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:50 smithi104 ceph-mon[32438]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T19:11:50.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:50.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:50.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:11:50.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:50.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: 1.4 scrub starts 2024-09-21T19:11:50.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: 1.4 scrub ok 2024-09-21T19:11:50.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T19:11:50.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T19:11:50.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:50.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1826478847' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:11:50.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[29384]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: 1.4 scrub starts 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: 1.4 scrub ok 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1826478847' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:11:50.320 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:50 smithi081 ceph-mon[39192]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T19:11:50.388 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:50.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:50.443 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:50.545 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:50.573 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:50.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:50.697 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:50.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:50.866 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:50.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:50.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:51.062 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:51.062 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:51.101 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:51.101 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:51.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:51.134 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:51.134 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:51.147 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:51.170 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:51.170 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:51.202 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:51.202 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:51.241 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:51.241 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:51.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:51.262 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:51.262 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:51.276 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:51.276 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:51.294 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:51.311 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:51.312 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:51.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:51.426 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:51 smithi104 ceph-mon[32438]: pgmap v928: 168 pgs: 1 active+clean+scrubbing, 6 peering, 3 unknown, 158 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 4.0 KiB/s rd, 18 KiB/s wr, 1.34k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-21T19:11:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:51 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:11:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:51 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/215898683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:51.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:51.559 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:51.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:51 smithi081 ceph-mon[39192]: pgmap v928: 168 pgs: 1 active+clean+scrubbing, 6 peering, 3 unknown, 158 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 4.0 KiB/s rd, 18 KiB/s wr, 1.34k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-21T19:11:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:51 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:11:51.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:51 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/215898683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:51 smithi081 ceph-mon[29384]: pgmap v928: 168 pgs: 1 active+clean+scrubbing, 6 peering, 3 unknown, 158 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 4.0 KiB/s rd, 18 KiB/s wr, 1.34k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-21T19:11:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:51 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:11:51.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:51 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/215898683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:51.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:51.700 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:51.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:51.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:51.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:51.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:11:52.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:11:52.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:52.133 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:52.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:52.266 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:52.307 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:11:52.307 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:11:52.308 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-21T19:11:52.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:11:52.356 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:52.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:52.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.430 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:52.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:52.515 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:52.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:52.574 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:52.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:52.703 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:52.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.844 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:52.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:52.944 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:52.945 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:52.945 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:52.984 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:52.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:53.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:53.125 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:53.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:53.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:53.391 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:53.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[29384]: pgmap v929: 168 pgs: 1 active+clean+scrubbing, 5 peering, 2 unknown, 160 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 4.7 KiB/s rd, 26 KiB/s wr, 1.01k op/s; 18 KiB/s, 0 objects/s recovering 2024-09-21T19:11:53.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[39192]: pgmap v929: 168 pgs: 1 active+clean+scrubbing, 5 peering, 2 unknown, 160 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 4.7 KiB/s rd, 26 KiB/s wr, 1.01k op/s; 18 KiB/s, 0 objects/s recovering 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:11:53.393 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:53 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:11:53.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:53 smithi104 ceph-mon[32438]: pgmap v929: 168 pgs: 1 active+clean+scrubbing, 5 peering, 2 unknown, 160 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 4.7 KiB/s rd, 26 KiB/s wr, 1.01k op/s; 18 KiB/s, 0 objects/s recovering 2024-09-21T19:11:53.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:53 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:53.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:11:53.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:53 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:11:53.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:53 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:11:53.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:53.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.656 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:53.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:53.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:11:53.734 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:53.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:53.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:53.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:53.920 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:53.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:54.020 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:11:54.021 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:54.057 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:54.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:54.187 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:54.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:11:54.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.347 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:54.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:54.379 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:11:54.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:54.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:54 smithi104 ceph-mon[32438]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T19:11:54.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:54 smithi104 ceph-mon[32438]: pgmap v931: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 4.5 KiB/s rd, 25 KiB/s wr, 1.60k op/s; 71 KiB/s, 0 objects/s recovering 2024-09-21T19:11:54.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:54 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2336628526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:54.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:54 smithi104 ceph-mon[32438]: from='client.26494 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[39192]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[39192]: pgmap v931: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 4.5 KiB/s rd, 25 KiB/s wr, 1.60k op/s; 71 KiB/s, 0 objects/s recovering 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2336628526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[39192]: from='client.26494 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[29384]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[29384]: pgmap v931: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 4.5 KiB/s rd, 25 KiB/s wr, 1.60k op/s; 71 KiB/s, 0 objects/s recovering 2024-09-21T19:11:54.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2336628526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:54.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:54 smithi081 ceph-mon[29384]: from='client.26494 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:11:54.495 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:54.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:54.604 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:11:54.605 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:11:54.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:54.716 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:11:54.719 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:11:54.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:54.876 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:54.907 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:54.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:55.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.040 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:55.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:55.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:55.180 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:55.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:55.314 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:55.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T19:11:55.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:55.442 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:55.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T19:11:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[29384]: from='client.26494 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[29384]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T19:11:55.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[29384]: from='client.16791 v1:172.21.15.81:0/1642865068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T19:11:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[39192]: from='client.26494 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[39192]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T19:11:55.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:55 smithi081 ceph-mon[39192]: from='client.16791 v1:172.21.15.81:0/1642865068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:55.596 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:55.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:55.724 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:55.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:55 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:11:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:55 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T19:11:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:55 smithi104 ceph-mon[32438]: from='client.26494 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:11:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:55 smithi104 ceph-mon[32438]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T19:11:55.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:55 smithi104 ceph-mon[32438]: from='client.16791 v1:172.21.15.81:0/1642865068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:11:55.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:55.862 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:55.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:55.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:55.998 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:56.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:56.102 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":932,"stamp":"2024-09-21T19:11:54.371930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7020888367,"num_objects":107323,"num_object_clones":0,"num_object_copies":321969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107323,"num_whiteouts":0,"num_read":2264,"num_read_kb":2026,"num_write":140848,"num_write_kb":7911746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":707872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77960,"ondisk_log_size":77960,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":508,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25876736,"kb_used_data":20746556,"kb_used_omap":18,"kb_used_meta":5129965,"kb_avail":724084480,"statfs":{"total":767960285184,"available":741462507520,"internally_reserved":0,"allocated":21244473344,"data_stored":21241184229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":5253085168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,4,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-541645060,"num_objects":-8263,"num_object_clones":0,"num_object_copies":-24789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8263,"num_whiteouts":0,"num_read":17,"num_read_kb":27,"num_write":8275,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":435020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.255436"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":749,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.134623+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:11:49.134623+0000","last_peered":"2024-09-21T19:11:49.134623+0000","last_clean":"2024-09-21T19:11:49.134623+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:11:49.134623+0000","last_undegraded":"2024-09-21T19:11:49.134623+0000","last_fullsized":"2024-09-21T19:11:49.134623+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.472742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":918,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122786+0000","last_change":"2024-09-21T19:10:17.698193+0000","last_active":"2024-09-21T19:11:49.122786+0000","last_peered":"2024-09-21T19:11:49.122786+0000","last_clean":"2024-09-21T19:11:49.122786+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:11:49.122786+0000","last_undegraded":"2024-09-21T19:11:49.122786+0000","last_fullsized":"2024-09-21T19:11:49.122786+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:10:17.698093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.698093+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121231+0000","last_change":"2024-09-21T19:11:47.058089+0000","last_active":"2024-09-21T19:11:49.121231+0000","last_peered":"2024-09-21T19:11:49.121231+0000","last_clean":"2024-09-21T19:11:49.121231+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:11:49.121231+0000","last_undegraded":"2024-09-21T19:11:49.121231+0000","last_fullsized":"2024-09-21T19:11:49.121231+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:47.058027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.058027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:50.497460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":835,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122929+0000","last_change":"2024-09-21T19:11:33.079408+0000","last_active":"2024-09-21T19:11:49.122929+0000","last_peered":"2024-09-21T19:11:49.122929+0000","last_clean":"2024-09-21T19:11:49.122929+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:11:49.122929+0000","last_undegraded":"2024-09-21T19:11:49.122929+0000","last_fullsized":"2024-09-21T19:11:49.122929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:33.079348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:33.079348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:00.185027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":844,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122148+0000","last_change":"2024-09-21T19:10:25.846651+0000","last_active":"2024-09-21T19:11:49.122148+0000","last_peered":"2024-09-21T19:11:49.122148+0000","last_clean":"2024-09-21T19:11:49.122148+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:11:49.122148+0000","last_undegraded":"2024-09-21T19:11:49.122148+0000","last_fullsized":"2024-09-21T19:11:49.122148+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:25.846276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.846276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":872,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.124825+0000","last_change":"2024-09-21T19:10:54.811144+0000","last_active":"2024-09-21T19:11:49.124825+0000","last_peered":"2024-09-21T19:11:49.124825+0000","last_clean":"2024-09-21T19:11:49.124825+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:11:49.124825+0000","last_undegraded":"2024-09-21T19:11:49.124825+0000","last_fullsized":"2024-09-21T19:11:49.124825+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:54.811103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:54.811103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.471846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":960,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122694+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:11:49.122694+0000","last_peered":"2024-09-21T19:11:49.122694+0000","last_clean":"2024-09-21T19:11:49.122694+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:11:49.122694+0000","last_undegraded":"2024-09-21T19:11:49.122694+0000","last_fullsized":"2024-09-21T19:11:49.122694+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122761+0000","last_change":"2024-09-21T19:10:16.657183+0000","last_active":"2024-09-21T19:11:49.122761+0000","last_peered":"2024-09-21T19:11:49.122761+0000","last_clean":"2024-09-21T19:11:49.122761+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:11:49.122761+0000","last_undegraded":"2024-09-21T19:11:49.122761+0000","last_fullsized":"2024-09-21T19:11:49.122761+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.657131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.657131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":842,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.124016+0000","last_change":"2024-09-21T19:10:13.614795+0000","last_active":"2024-09-21T19:11:49.124016+0000","last_peered":"2024-09-21T19:11:49.124016+0000","last_clean":"2024-09-21T19:11:49.124016+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:11:49.124016+0000","last_undegraded":"2024-09-21T19:11:49.124016+0000","last_fullsized":"2024-09-21T19:11:49.124016+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.614720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.614720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":891,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123268+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:11:49.123268+0000","last_peered":"2024-09-21T19:11:49.123268+0000","last_clean":"2024-09-21T19:11:49.123268+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:11:49.123268+0000","last_undegraded":"2024-09-21T19:11:49.123268+0000","last_fullsized":"2024-09-21T19:11:49.123268+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.138431+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":804,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126070+0000","last_change":"2024-09-21T19:11:00.940146+0000","last_active":"2024-09-21T19:11:49.126070+0000","last_peered":"2024-09-21T19:11:49.126070+0000","last_clean":"2024-09-21T19:11:49.126070+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:11:49.126070+0000","last_undegraded":"2024-09-21T19:11:49.126070+0000","last_fullsized":"2024-09-21T19:11:49.126070+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.940070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.940070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.146532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":715,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.129580+0000","last_change":"2024-09-21T19:11:03.440073+0000","last_active":"2024-09-21T19:11:49.129580+0000","last_peered":"2024-09-21T19:11:49.129580+0000","last_clean":"2024-09-21T19:11:49.129580+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:11:49.129580+0000","last_undegraded":"2024-09-21T19:11:49.129580+0000","last_fullsized":"2024-09-21T19:11:49.129580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.439988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.439988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:07.256937+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":395,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123690+0000","last_change":"2024-09-21T19:11:08.134154+0000","last_active":"2024-09-21T19:11:49.123690+0000","last_peered":"2024-09-21T19:11:49.123690+0000","last_clean":"2024-09-21T19:11:49.123690+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:11:49.123690+0000","last_undegraded":"2024-09-21T19:11:49.123690+0000","last_fullsized":"2024-09-21T19:11:49.123690+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.134097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.134097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.482578+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123065+0000","last_change":"2024-09-21T19:11:39.159512+0000","last_active":"2024-09-21T19:11:49.123065+0000","last_peered":"2024-09-21T19:11:49.123065+0000","last_clean":"2024-09-21T19:11:49.123065+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:11:49.123065+0000","last_undegraded":"2024-09-21T19:11:49.123065+0000","last_fullsized":"2024-09-21T19:11:49.123065+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:11:39.159436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:39.159436+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:01.228931+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":835,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128427+0000","last_change":"2024-09-21T19:11:40.758093+0000","last_active":"2024-09-21T19:11:49.128427+0000","last_peered":"2024-09-21T19:11:49.128427+0000","last_clean":"2024-09-21T19:11:49.128427+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:11:49.128427+0000","last_undegraded":"2024-09-21T19:11:49.128427+0000","last_fullsized":"2024-09-21T19:11:49.128427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:40.758049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:40.758049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:41.348208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":411,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121037+0000","last_change":"2024-09-21T19:11:44.104002+0000","last_active":"2024-09-21T19:11:49.121037+0000","last_peered":"2024-09-21T19:11:49.121037+0000","last_clean":"2024-09-21T19:11:49.121037+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:11:49.121037+0000","last_undegraded":"2024-09-21T19:11:49.121037+0000","last_fullsized":"2024-09-21T19:11:49.121037+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:44.103926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.103926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.036090+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":864,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123504+0000","last_change":"2024-09-21T19:10:26.819197+0000","last_active":"2024-09-21T19:11:49.123504+0000","last_peered":"2024-09-21T19:11:49.123504+0000","last_clean":"2024-09-21T19:11:49.123504+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:11:49.123504+0000","last_undegraded":"2024-09-21T19:11:49.123504+0000","last_fullsized":"2024-09-21T19:11:49.123504+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:26.819128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:26.819128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":731,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132440+0000","last_change":"2024-09-21T19:11:42.596370+0000","last_active":"2024-09-21T19:11:49.132440+0000","last_peered":"2024-09-21T19:11:49.132440+0000","last_clean":"2024-09-21T19:11:49.132440+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:11:49.132440+0000","last_undegraded":"2024-09-21T19:11:49.132440+0000","last_fullsized":"2024-09-21T19:11:49.132440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:42.596058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:42.596058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:56.734805+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":314,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.130144+0000","last_change":"2024-09-21T19:11:37.697054+0000","last_active":"2024-09-21T19:11:49.130144+0000","last_peered":"2024-09-21T19:11:49.130144+0000","last_clean":"2024-09-21T19:11:49.130144+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:11:49.130144+0000","last_undegraded":"2024-09-21T19:11:49.130144+0000","last_fullsized":"2024-09-21T19:11:49.130144+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:11:37.697000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.697000+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.365077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":866,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121621+0000","last_change":"2024-09-21T19:11:31.048129+0000","last_active":"2024-09-21T19:11:49.121621+0000","last_peered":"2024-09-21T19:11:49.121621+0000","last_clean":"2024-09-21T19:11:49.121621+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:11:49.121621+0000","last_undegraded":"2024-09-21T19:11:49.121621+0000","last_fullsized":"2024-09-21T19:11:49.121621+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.048064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.048064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:39.917247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":840,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123303+0000","last_change":"2024-09-21T19:10:13.916666+0000","last_active":"2024-09-21T19:11:49.123303+0000","last_peered":"2024-09-21T19:11:49.123303+0000","last_clean":"2024-09-21T19:11:49.123303+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:11:49.123303+0000","last_undegraded":"2024-09-21T19:11:49.123303+0000","last_fullsized":"2024-09-21T19:11:49.123303+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.916604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.916604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123430+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:11:49.123430+0000","last_peered":"2024-09-21T19:11:49.123430+0000","last_clean":"2024-09-21T19:11:49.123430+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:11:49.123430+0000","last_undegraded":"2024-09-21T19:11:49.123430+0000","last_fullsized":"2024-09-21T19:11:49.123430+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.111619+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":930,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123639+0000","last_change":"2024-09-21T19:11:34.009972+0000","last_active":"2024-09-21T19:11:49.123639+0000","last_peered":"2024-09-21T19:11:49.123639+0000","last_clean":"2024-09-21T19:11:49.123639+0000","last_became_active":"2024-09-21T19:10:52.013122+0000","last_became_peered":"2024-09-21T19:10:52.013122+0000","last_unstale":"2024-09-21T19:11:49.123639+0000","last_undegraded":"2024-09-21T19:11:49.123639+0000","last_fullsized":"2024-09-21T19:11:49.123639+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:11:34.009898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.009898+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:37.167818+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123559+0000","last_change":"2024-09-21T19:11:34.072955+0000","last_active":"2024-09-21T19:11:49.123559+0000","last_peered":"2024-09-21T19:11:49.123559+0000","last_clean":"2024-09-21T19:11:49.123559+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:11:49.123559+0000","last_undegraded":"2024-09-21T19:11:49.123559+0000","last_fullsized":"2024-09-21T19:11:49.123559+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:34.072904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.072904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.755481+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":873,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128019+0000","last_change":"2024-09-21T19:10:13.592189+0000","last_active":"2024-09-21T19:11:49.128019+0000","last_peered":"2024-09-21T19:11:49.128019+0000","last_clean":"2024-09-21T19:11:49.128019+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:11:49.128019+0000","last_undegraded":"2024-09-21T19:11:49.128019+0000","last_fullsized":"2024-09-21T19:11:49.128019+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:13.592116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.592116+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.755272+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":872,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121312+0000","last_change":"2024-09-21T19:10:27.115013+0000","last_active":"2024-09-21T19:11:49.121312+0000","last_peered":"2024-09-21T19:11:49.121312+0000","last_clean":"2024-09-21T19:11:49.121312+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:11:49.121312+0000","last_undegraded":"2024-09-21T19:11:49.121312+0000","last_fullsized":"2024-09-21T19:11:49.121312+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:27.114925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.114925+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:50.719198+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":760,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132540+0000","last_change":"2024-09-21T19:10:27.501434+0000","last_active":"2024-09-21T19:11:49.132540+0000","last_peered":"2024-09-21T19:11:49.132540+0000","last_clean":"2024-09-21T19:11:49.132540+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:11:49.132540+0000","last_undegraded":"2024-09-21T19:11:49.132540+0000","last_fullsized":"2024-09-21T19:11:49.132540+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:27.500959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.500959+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":968,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.131269+0000","last_change":"2024-09-21T19:11:38.797906+0000","last_active":"2024-09-21T19:11:49.131269+0000","last_peered":"2024-09-21T19:11:49.131269+0000","last_clean":"2024-09-21T19:11:49.131269+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:11:49.131269+0000","last_undegraded":"2024-09-21T19:11:49.131269+0000","last_fullsized":"2024-09-21T19:11:49.131269+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:11:38.797839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.797839+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:08.395366+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"451'8741","reported_seq":8826,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:52.838083+0000","last_change":"2024-09-21T19:10:01.248307+0000","last_active":"2024-09-21T19:11:52.838083+0000","last_peered":"2024-09-21T19:11:52.838083+0000","last_clean":"2024-09-21T19:11:52.838083+0000","last_became_active":"2024-09-21T19:10:01.248029+0000","last_became_peered":"2024-09-21T19:10:01.248029+0000","last_unstale":"2024-09-21T19:11:52.838083+0000","last_undegraded":"2024-09-21T19:11:52.838083+0000","last_fullsized":"2024-09-21T19:11:52.838083+0000","mapping_epoch":414,"log_start":"430'4192","ondisk_log_start":"430'4192","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:21.027444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429195264,"num_objects":6549,"num_object_clones":0,"num_object_copies":19647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":489280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":304,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.131789+0000","last_change":"2024-09-21T19:10:21.515835+0000","last_active":"2024-09-21T19:11:49.131789+0000","last_peered":"2024-09-21T19:11:49.131789+0000","last_clean":"2024-09-21T19:11:49.131789+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:11:49.131789+0000","last_undegraded":"2024-09-21T19:11:49.131789+0000","last_fullsized":"2024-09-21T19:11:49.131789+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:21.515713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.515713+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":900,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123263+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:11:49.123263+0000","last_peered":"2024-09-21T19:11:49.123263+0000","last_clean":"2024-09-21T19:11:49.123263+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:11:49.123263+0000","last_undegraded":"2024-09-21T19:11:49.123263+0000","last_fullsized":"2024-09-21T19:11:49.123263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.082123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122274+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:11:49.122274+0000","last_peered":"2024-09-21T19:11:49.122274+0000","last_clean":"2024-09-21T19:11:49.122274+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:11:49.122274+0000","last_undegraded":"2024-09-21T19:11:49.122274+0000","last_fullsized":"2024-09-21T19:11:49.122274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1113,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123604+0000","last_change":"2024-09-21T19:11:10.165309+0000","last_active":"2024-09-21T19:11:49.123604+0000","last_peered":"2024-09-21T19:11:49.123604+0000","last_clean":"2024-09-21T19:11:49.123604+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:11:49.123604+0000","last_undegraded":"2024-09-21T19:11:49.123604+0000","last_fullsized":"2024-09-21T19:11:49.123604+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:11:10.165236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:10.165236+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.702978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":535,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121386+0000","last_change":"2024-09-21T19:11:49.094049+0000","last_active":"2024-09-21T19:11:49.121386+0000","last_peered":"2024-09-21T19:11:49.121386+0000","last_clean":"2024-09-21T19:11:49.121386+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:11:49.121386+0000","last_undegraded":"2024-09-21T19:11:49.121386+0000","last_fullsized":"2024-09-21T19:11:49.121386+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:49.093991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:49.093991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.399096+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"451'8916","reported_seq":9000,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:51.949925+0000","last_change":"2024-09-21T19:10:01.252308+0000","last_active":"2024-09-21T19:11:51.949925+0000","last_peered":"2024-09-21T19:11:51.949925+0000","last_clean":"2024-09-21T19:11:51.949925+0000","last_became_active":"2024-09-21T19:10:01.252017+0000","last_became_peered":"2024-09-21T19:10:01.252017+0000","last_unstale":"2024-09-21T19:11:51.949925+0000","last_undegraded":"2024-09-21T19:11:51.949925+0000","last_fullsized":"2024-09-21T19:11:51.949925+0000","mapping_epoch":414,"log_start":"430'4326","ondisk_log_start":"430'4326","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.828286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448266240,"num_objects":6840,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":644,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123919+0000","last_change":"2024-09-21T19:11:48.057060+0000","last_active":"2024-09-21T19:11:49.123919+0000","last_peered":"2024-09-21T19:11:49.123919+0000","last_clean":"2024-09-21T19:11:49.123919+0000","last_became_active":"2024-09-21T19:11:47.120840+0000","last_became_peered":"2024-09-21T19:11:47.120840+0000","last_unstale":"2024-09-21T19:11:49.123919+0000","last_undegraded":"2024-09-21T19:11:49.123919+0000","last_fullsized":"2024-09-21T19:11:49.123919+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.056992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.056992+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:08.932687+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":935,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122000+0000","last_change":"2024-09-21T19:11:44.959245+0000","last_active":"2024-09-21T19:11:49.122000+0000","last_peered":"2024-09-21T19:11:49.122000+0000","last_clean":"2024-09-21T19:11:49.122000+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:11:49.122000+0000","last_undegraded":"2024-09-21T19:11:49.122000+0000","last_fullsized":"2024-09-21T19:11:49.122000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:44.959061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.959061+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:11.599555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":860,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123684+0000","last_change":"2024-09-21T19:10:56.831121+0000","last_active":"2024-09-21T19:11:49.123684+0000","last_peered":"2024-09-21T19:11:49.123684+0000","last_clean":"2024-09-21T19:11:49.123684+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:11:49.123684+0000","last_undegraded":"2024-09-21T19:11:49.123684+0000","last_fullsized":"2024-09-21T19:11:49.123684+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:56.831042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:56.831042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:00.224419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":837,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122820+0000","last_change":"2024-09-21T19:11:05.056988+0000","last_active":"2024-09-21T19:11:49.122820+0000","last_peered":"2024-09-21T19:11:49.122820+0000","last_clean":"2024-09-21T19:11:49.122820+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:11:49.122820+0000","last_undegraded":"2024-09-21T19:11:49.122820+0000","last_fullsized":"2024-09-21T19:11:49.122820+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.056931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.056931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:22.842104+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":854,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123482+0000","last_change":"2024-09-21T19:10:51.864691+0000","last_active":"2024-09-21T19:11:49.123482+0000","last_peered":"2024-09-21T19:11:49.123482+0000","last_clean":"2024-09-21T19:11:49.123482+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:11:49.123482+0000","last_undegraded":"2024-09-21T19:11:49.123482+0000","last_fullsized":"2024-09-21T19:11:49.123482+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:51.864627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.864627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:01.755943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"451'8650","reported_seq":8735,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:51.450865+0000","last_change":"2024-09-21T19:10:01.248578+0000","last_active":"2024-09-21T19:11:51.450865+0000","last_peered":"2024-09-21T19:11:51.450865+0000","last_clean":"2024-09-21T19:11:51.450865+0000","last_became_active":"2024-09-21T19:10:01.248286+0000","last_became_peered":"2024-09-21T19:10:01.248286+0000","last_unstale":"2024-09-21T19:11:51.450865+0000","last_undegraded":"2024-09-21T19:11:51.450865+0000","last_fullsized":"2024-09-21T19:11:51.450865+0000","mapping_epoch":414,"log_start":"430'4231","ondisk_log_start":"430'4231","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.205302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435159040,"num_objects":6640,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":489280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":635,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121039+0000","last_change":"2024-09-21T19:11:48.886612+0000","last_active":"2024-09-21T19:11:49.121039+0000","last_peered":"2024-09-21T19:11:49.121039+0000","last_clean":"2024-09-21T19:11:49.121039+0000","last_became_active":"2024-09-21T19:11:47.119064+0000","last_became_peered":"2024-09-21T19:11:47.119064+0000","last_unstale":"2024-09-21T19:11:49.121039+0000","last_undegraded":"2024-09-21T19:11:49.121039+0000","last_fullsized":"2024-09-21T19:11:49.121039+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.886536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.886536+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.061161+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":906,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122922+0000","last_change":"2024-09-21T19:10:11.634360+0000","last_active":"2024-09-21T19:11:49.122922+0000","last_peered":"2024-09-21T19:11:49.122922+0000","last_clean":"2024-09-21T19:11:49.122922+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:11:49.122922+0000","last_undegraded":"2024-09-21T19:11:49.122922+0000","last_fullsized":"2024-09-21T19:11:49.122922+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:10:11.634280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:11.634280+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":728,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132914+0000","last_change":"2024-09-21T19:11:04.465550+0000","last_active":"2024-09-21T19:11:49.132914+0000","last_peered":"2024-09-21T19:11:49.132914+0000","last_clean":"2024-09-21T19:11:49.132914+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:11:49.132914+0000","last_undegraded":"2024-09-21T19:11:49.132914+0000","last_fullsized":"2024-09-21T19:11:49.132914+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:04.464221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:04.464221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:31.185556+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":1057,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121866+0000","last_change":"2024-09-21T19:10:21.865360+0000","last_active":"2024-09-21T19:11:49.121866+0000","last_peered":"2024-09-21T19:11:49.121866+0000","last_clean":"2024-09-21T19:11:49.121866+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:11:49.121866+0000","last_undegraded":"2024-09-21T19:11:49.121866+0000","last_fullsized":"2024-09-21T19:11:49.121866+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:10:21.865158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.865158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":857,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126145+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:11:49.126145+0000","last_peered":"2024-09-21T19:11:49.126145+0000","last_clean":"2024-09-21T19:11:49.126145+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:11:49.126145+0000","last_undegraded":"2024-09-21T19:11:49.126145+0000","last_fullsized":"2024-09-21T19:11:49.126145+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.789154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"451'8735","reported_seq":8821,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:53.132535+0000","last_change":"2024-09-21T19:10:01.251501+0000","last_active":"2024-09-21T19:11:53.132535+0000","last_peered":"2024-09-21T19:11:53.132535+0000","last_clean":"2024-09-21T19:11:53.132535+0000","last_became_active":"2024-09-21T19:10:01.250354+0000","last_became_peered":"2024-09-21T19:10:01.250354+0000","last_unstale":"2024-09-21T19:11:53.132535+0000","last_undegraded":"2024-09-21T19:11:53.132535+0000","last_fullsized":"2024-09-21T19:11:53.132535+0000","mapping_epoch":414,"log_start":"426'2569","ondisk_log_start":"426'2569","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.606410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430702616,"num_objects":6573,"num_object_clones":0,"num_object_copies":19719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8735,"num_write_kb":489793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":651,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.131477+0000","last_change":"2024-09-21T19:11:47.126158+0000","last_active":"2024-09-21T19:11:49.131477+0000","last_peered":"2024-09-21T19:11:49.131477+0000","last_clean":"2024-09-21T19:11:49.131477+0000","last_became_active":"2024-09-21T19:11:47.123128+0000","last_became_peered":"2024-09-21T19:11:47.123128+0000","last_unstale":"2024-09-21T19:11:49.131477+0000","last_undegraded":"2024-09-21T19:11:49.131477+0000","last_fullsized":"2024-09-21T19:11:49.131477+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":890,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122945+0000","last_change":"2024-09-21T19:10:57.167898+0000","last_active":"2024-09-21T19:11:49.122945+0000","last_peered":"2024-09-21T19:11:49.122945+0000","last_clean":"2024-09-21T19:11:49.122945+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:11:49.122945+0000","last_undegraded":"2024-09-21T19:11:49.122945+0000","last_fullsized":"2024-09-21T19:11:49.122945+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:57.167826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.167826+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:01.720799+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":402,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123822+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:11:49.123822+0000","last_peered":"2024-09-21T19:11:49.123822+0000","last_clean":"2024-09-21T19:11:49.123822+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:11:49.123822+0000","last_undegraded":"2024-09-21T19:11:49.123822+0000","last_fullsized":"2024-09-21T19:11:49.123822+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":939,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132389+0000","last_change":"2024-09-21T19:11:32.614783+0000","last_active":"2024-09-21T19:11:49.132389+0000","last_peered":"2024-09-21T19:11:49.132389+0000","last_clean":"2024-09-21T19:11:49.132389+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:11:49.132389+0000","last_undegraded":"2024-09-21T19:11:49.132389+0000","last_fullsized":"2024-09-21T19:11:49.132389+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'14","last_scrub_stamp":"2024-09-21T19:11:32.614707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:32.614707+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:45.115750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":739,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.129754+0000","last_change":"2024-09-21T19:11:08.473248+0000","last_active":"2024-09-21T19:11:49.129754+0000","last_peered":"2024-09-21T19:11:49.129754+0000","last_clean":"2024-09-21T19:11:49.129754+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:11:49.129754+0000","last_undegraded":"2024-09-21T19:11:49.129754+0000","last_fullsized":"2024-09-21T19:11:49.129754+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.473184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.473184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:38.292717+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"451'8943","reported_seq":9113,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:52.798780+0000","last_change":"2024-09-21T19:11:35.874862+0000","last_active":"2024-09-21T19:11:52.798780+0000","last_peered":"2024-09-21T19:11:52.798780+0000","last_clean":"2024-09-21T19:11:52.798780+0000","last_became_active":"2024-09-21T19:10:01.243730+0000","last_became_peered":"2024-09-21T19:10:01.243730+0000","last_unstale":"2024-09-21T19:11:52.798780+0000","last_undegraded":"2024-09-21T19:11:52.798780+0000","last_fullsized":"2024-09-21T19:11:52.798780+0000","mapping_epoch":414,"log_start":"430'4388","ondisk_log_start":"430'4388","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'7719","last_scrub_stamp":"2024-09-21T19:11:35.874735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:35.874735+0000","objects_scrubbed":7633,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:46.639059+0000","scrub_duration":2835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440991744,"num_objects":6729,"num_object_clones":0,"num_object_copies":20187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8943,"num_write_kb":501504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":629,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.130946+0000","last_change":"2024-09-21T19:11:47.128871+0000","last_active":"2024-09-21T19:11:49.130946+0000","last_peered":"2024-09-21T19:11:49.130946+0000","last_clean":"2024-09-21T19:11:49.130946+0000","last_became_active":"2024-09-21T19:11:47.128147+0000","last_became_peered":"2024-09-21T19:11:47.128147+0000","last_unstale":"2024-09-21T19:11:49.130946+0000","last_undegraded":"2024-09-21T19:11:49.130946+0000","last_fullsized":"2024-09-21T19:11:49.130946+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":776,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132532+0000","last_change":"2024-09-21T19:10:25.488039+0000","last_active":"2024-09-21T19:11:49.132532+0000","last_peered":"2024-09-21T19:11:49.132532+0000","last_clean":"2024-09-21T19:11:49.132532+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:11:49.132532+0000","last_undegraded":"2024-09-21T19:11:49.132532+0000","last_fullsized":"2024-09-21T19:11:49.132532+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:10:25.487878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.487878+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":870,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122489+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:11:49.122489+0000","last_peered":"2024-09-21T19:11:49.122489+0000","last_clean":"2024-09-21T19:11:49.122489+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:11:49.122489+0000","last_undegraded":"2024-09-21T19:11:49.122489+0000","last_fullsized":"2024-09-21T19:11:49.122489+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.041720+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":747,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132011+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:11:49.132011+0000","last_peered":"2024-09-21T19:11:49.132011+0000","last_clean":"2024-09-21T19:11:49.132011+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:11:49.132011+0000","last_undegraded":"2024-09-21T19:11:49.132011+0000","last_fullsized":"2024-09-21T19:11:49.132011+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:17.702612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121140+0000","last_change":"2024-09-21T19:11:00.111371+0000","last_active":"2024-09-21T19:11:49.121140+0000","last_peered":"2024-09-21T19:11:49.121140+0000","last_clean":"2024-09-21T19:11:49.121140+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:11:49.121140+0000","last_undegraded":"2024-09-21T19:11:49.121140+0000","last_fullsized":"2024-09-21T19:11:49.121140+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.111305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.111305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.568023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"451'8778","reported_seq":8863,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:53.133144+0000","last_change":"2024-09-21T19:10:01.250359+0000","last_active":"2024-09-21T19:11:53.133144+0000","last_peered":"2024-09-21T19:11:53.133144+0000","last_clean":"2024-09-21T19:11:53.133144+0000","last_became_active":"2024-09-21T19:10:01.250079+0000","last_became_peered":"2024-09-21T19:10:01.250079+0000","last_unstale":"2024-09-21T19:11:53.133144+0000","last_undegraded":"2024-09-21T19:11:53.133144+0000","last_fullsized":"2024-09-21T19:11:53.133144+0000","mapping_epoch":414,"log_start":"433'4631","ondisk_log_start":"433'4631","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438304768,"num_objects":6688,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8778,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":604,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.129554+0000","last_change":"2024-09-21T19:11:47.114755+0000","last_active":"2024-09-21T19:11:49.129554+0000","last_peered":"2024-09-21T19:11:49.129554+0000","last_clean":"2024-09-21T19:11:49.129554+0000","last_became_active":"2024-09-21T19:11:47.114145+0000","last_became_peered":"2024-09-21T19:11:47.114145+0000","last_unstale":"2024-09-21T19:11:49.129554+0000","last_undegraded":"2024-09-21T19:11:49.129554+0000","last_fullsized":"2024-09-21T19:11:49.129554+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:07.476444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.476444+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1184,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121906+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:11:49.121906+0000","last_peered":"2024-09-21T19:11:49.121906+0000","last_clean":"2024-09-21T19:11:49.121906+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:11:49.121906+0000","last_undegraded":"2024-09-21T19:11:49.121906+0000","last_fullsized":"2024-09-21T19:11:49.121906+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.035088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":870,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123455+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:11:49.123455+0000","last_peered":"2024-09-21T19:11:49.123455+0000","last_clean":"2024-09-21T19:11:49.123455+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:11:49.123455+0000","last_undegraded":"2024-09-21T19:11:49.123455+0000","last_fullsized":"2024-09-21T19:11:49.123455+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":418,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123233+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:11:49.123233+0000","last_peered":"2024-09-21T19:11:49.123233+0000","last_clean":"2024-09-21T19:11:49.123233+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:11:49.123233+0000","last_undegraded":"2024-09-21T19:11:49.123233+0000","last_fullsized":"2024-09-21T19:11:49.123233+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.115247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":724,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132157+0000","last_change":"2024-09-21T19:11:05.417598+0000","last_active":"2024-09-21T19:11:49.132157+0000","last_peered":"2024-09-21T19:11:49.132157+0000","last_clean":"2024-09-21T19:11:49.132157+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:11:49.132157+0000","last_undegraded":"2024-09-21T19:11:49.132157+0000","last_fullsized":"2024-09-21T19:11:49.132157+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.417365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.417365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.055191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":826,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122829+0000","last_change":"2024-09-21T19:10:28.717418+0000","last_active":"2024-09-21T19:11:49.122829+0000","last_peered":"2024-09-21T19:11:49.122829+0000","last_clean":"2024-09-21T19:11:49.122829+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:11:49.122829+0000","last_undegraded":"2024-09-21T19:11:49.122829+0000","last_fullsized":"2024-09-21T19:11:49.122829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:28.717310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.717310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"451'8728","reported_seq":8813,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:53.125449+0000","last_change":"2024-09-21T19:10:01.259760+0000","last_active":"2024-09-21T19:11:53.125449+0000","last_peered":"2024-09-21T19:11:53.125449+0000","last_clean":"2024-09-21T19:11:53.125449+0000","last_became_active":"2024-09-21T19:10:01.259451+0000","last_became_peered":"2024-09-21T19:10:01.259451+0000","last_unstale":"2024-09-21T19:11:53.125449+0000","last_undegraded":"2024-09-21T19:11:53.125449+0000","last_fullsized":"2024-09-21T19:11:53.125449+0000","mapping_epoch":414,"log_start":"432'4599","ondisk_log_start":"432'4599","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428081152,"num_objects":6532,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8728,"num_write_kb":488320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":940,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128244+0000","last_change":"2024-09-21T19:11:47.118419+0000","last_active":"2024-09-21T19:11:49.128244+0000","last_peered":"2024-09-21T19:11:49.128244+0000","last_clean":"2024-09-21T19:11:49.128244+0000","last_became_active":"2024-09-21T19:11:47.117467+0000","last_became_peered":"2024-09-21T19:11:47.117467+0000","last_unstale":"2024-09-21T19:11:49.128244+0000","last_undegraded":"2024-09-21T19:11:49.128244+0000","last_fullsized":"2024-09-21T19:11:49.128244+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":344,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122741+0000","last_change":"2024-09-21T19:10:20.080922+0000","last_active":"2024-09-21T19:11:49.122741+0000","last_peered":"2024-09-21T19:11:49.122741+0000","last_clean":"2024-09-21T19:11:49.122741+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:11:49.122741+0000","last_undegraded":"2024-09-21T19:11:49.122741+0000","last_fullsized":"2024-09-21T19:11:49.122741+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:10:20.080874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:20.080874+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":901,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121543+0000","last_change":"2024-09-21T19:11:37.043422+0000","last_active":"2024-09-21T19:11:49.121543+0000","last_peered":"2024-09-21T19:11:49.121543+0000","last_clean":"2024-09-21T19:11:49.121543+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:11:49.121543+0000","last_undegraded":"2024-09-21T19:11:49.121543+0000","last_fullsized":"2024-09-21T19:11:49.121543+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:37.043234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.043234+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:52.960859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":294,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127540+0000","last_change":"2024-09-21T19:11:32.736397+0000","last_active":"2024-09-21T19:11:49.127540+0000","last_peered":"2024-09-21T19:11:49.127540+0000","last_clean":"2024-09-21T19:11:49.127540+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:11:49.127540+0000","last_undegraded":"2024-09-21T19:11:49.127540+0000","last_fullsized":"2024-09-21T19:11:49.127540+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:32.736299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:11:32.736299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:49.298535+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":301,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128727+0000","last_change":"2024-09-21T19:11:47.541937+0000","last_active":"2024-09-21T19:11:49.128727+0000","last_peered":"2024-09-21T19:11:49.128727+0000","last_clean":"2024-09-21T19:11:49.128727+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:11:49.128727+0000","last_undegraded":"2024-09-21T19:11:49.128727+0000","last_fullsized":"2024-09-21T19:11:49.128727+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:47.541386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.541386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:56.449155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125758+0000","last_change":"2024-09-21T19:10:45.720231+0000","last_active":"2024-09-21T19:11:49.125758+0000","last_peered":"2024-09-21T19:11:49.125758+0000","last_clean":"2024-09-21T19:11:49.125758+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:11:49.125758+0000","last_undegraded":"2024-09-21T19:11:49.125758+0000","last_fullsized":"2024-09-21T19:11:49.125758+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:45.720186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.720186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.576025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"451'8906","reported_seq":8991,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:53.114809+0000","last_change":"2024-09-21T19:10:01.250406+0000","last_active":"2024-09-21T19:11:53.114809+0000","last_peered":"2024-09-21T19:11:53.114809+0000","last_clean":"2024-09-21T19:11:53.114809+0000","last_became_active":"2024-09-21T19:10:01.250100+0000","last_became_peered":"2024-09-21T19:10:01.250100+0000","last_unstale":"2024-09-21T19:11:53.114809+0000","last_undegraded":"2024-09-21T19:11:53.114809+0000","last_fullsized":"2024-09-21T19:11:53.114809+0000","mapping_epoch":414,"log_start":"433'4794","ondisk_log_start":"433'4794","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435159040,"num_objects":6640,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8906,"num_write_kb":497472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":654,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.124559+0000","last_change":"2024-09-21T19:11:48.108418+0000","last_active":"2024-09-21T19:11:49.124559+0000","last_peered":"2024-09-21T19:11:49.124559+0000","last_clean":"2024-09-21T19:11:49.124559+0000","last_became_active":"2024-09-21T19:11:47.124560+0000","last_became_peered":"2024-09-21T19:11:47.124560+0000","last_unstale":"2024-09-21T19:11:49.124559+0000","last_undegraded":"2024-09-21T19:11:49.124559+0000","last_fullsized":"2024-09-21T19:11:49.124559+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.108284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.108284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:52.461410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":846,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132716+0000","last_change":"2024-09-21T19:11:45.692291+0000","last_active":"2024-09-21T19:11:49.132716+0000","last_peered":"2024-09-21T19:11:49.132716+0000","last_clean":"2024-09-21T19:11:49.132716+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:11:49.132716+0000","last_undegraded":"2024-09-21T19:11:49.132716+0000","last_fullsized":"2024-09-21T19:11:49.132716+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:11:45.692118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.692118+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:58.276257+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":732,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.130309+0000","last_change":"2024-09-21T19:11:46.659335+0000","last_active":"2024-09-21T19:11:49.130309+0000","last_peered":"2024-09-21T19:11:49.130309+0000","last_clean":"2024-09-21T19:11:49.130309+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:11:49.130309+0000","last_undegraded":"2024-09-21T19:11:49.130309+0000","last_fullsized":"2024-09-21T19:11:49.130309+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:46.659194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:46.659194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.723509+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":1987,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.978908+0000","last_change":"2024-09-21T19:10:28.526838+0000","last_active":"2024-09-21T19:11:49.978908+0000","last_peered":"2024-09-21T19:11:49.978908+0000","last_clean":"2024-09-21T19:11:49.978908+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:11:49.978908+0000","last_undegraded":"2024-09-21T19:11:49.978908+0000","last_fullsized":"2024-09-21T19:11:49.978908+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:10:28.526790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.526790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1112,"num_read_kb":1107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":797,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128399+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:11:49.128399+0000","last_peered":"2024-09-21T19:11:49.128399+0000","last_clean":"2024-09-21T19:11:49.128399+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:11:49.128399+0000","last_undegraded":"2024-09-21T19:11:49.128399+0000","last_fullsized":"2024-09-21T19:11:49.128399+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.851635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":860,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121713+0000","last_change":"2024-09-21T19:11:43.150420+0000","last_active":"2024-09-21T19:11:49.121713+0000","last_peered":"2024-09-21T19:11:49.121713+0000","last_clean":"2024-09-21T19:11:49.121713+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:11:49.121713+0000","last_undegraded":"2024-09-21T19:11:49.121713+0000","last_fullsized":"2024-09-21T19:11:49.121713+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:43.150353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:43.150353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:10.609375+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"451'8812","reported_seq":8897,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:53.128841+0000","last_change":"2024-09-21T19:10:01.260090+0000","last_active":"2024-09-21T19:11:53.128841+0000","last_peered":"2024-09-21T19:11:53.128841+0000","last_clean":"2024-09-21T19:11:53.128841+0000","last_became_active":"2024-09-21T19:10:01.259620+0000","last_became_peered":"2024-09-21T19:10:01.259620+0000","last_unstale":"2024-09-21T19:11:53.128841+0000","last_undegraded":"2024-09-21T19:11:53.128841+0000","last_fullsized":"2024-09-21T19:11:53.128841+0000","mapping_epoch":414,"log_start":"432'4629","ondisk_log_start":"432'4629","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429260800,"num_objects":6550,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":491584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":1051,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.135270+0000","last_change":"2024-09-21T19:11:02.540038+0000","last_active":"2024-09-21T19:11:49.135270+0000","last_peered":"2024-09-21T19:11:49.135270+0000","last_clean":"2024-09-21T19:11:49.135270+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:11:49.135270+0000","last_undegraded":"2024-09-21T19:11:49.135270+0000","last_fullsized":"2024-09-21T19:11:49.135270+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:02.539974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.539974+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.492636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":860,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121053+0000","last_change":"2024-09-21T19:11:36.018987+0000","last_active":"2024-09-21T19:11:49.121053+0000","last_peered":"2024-09-21T19:11:49.121053+0000","last_clean":"2024-09-21T19:11:49.121053+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:11:49.121053+0000","last_undegraded":"2024-09-21T19:11:49.121053+0000","last_fullsized":"2024-09-21T19:11:49.121053+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:36.018911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:36.018911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.421244+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":954,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127357+0000","last_change":"2024-09-21T19:11:47.120442+0000","last_active":"2024-09-21T19:11:49.127357+0000","last_peered":"2024-09-21T19:11:49.127357+0000","last_clean":"2024-09-21T19:11:49.127357+0000","last_became_active":"2024-09-21T19:11:47.120134+0000","last_became_peered":"2024-09-21T19:11:47.120134+0000","last_unstale":"2024-09-21T19:11:49.127357+0000","last_undegraded":"2024-09-21T19:11:49.127357+0000","last_fullsized":"2024-09-21T19:11:49.127357+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:45.911119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.911119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":861,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128163+0000","last_change":"2024-09-21T19:10:17.553053+0000","last_active":"2024-09-21T19:11:49.128163+0000","last_peered":"2024-09-21T19:11:49.128163+0000","last_clean":"2024-09-21T19:11:49.128163+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:11:49.128163+0000","last_undegraded":"2024-09-21T19:11:49.128163+0000","last_fullsized":"2024-09-21T19:11:49.128163+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.553007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.553007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":388,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123209+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:11:49.123209+0000","last_peered":"2024-09-21T19:11:49.123209+0000","last_clean":"2024-09-21T19:11:49.123209+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:11:49.123209+0000","last_undegraded":"2024-09-21T19:11:49.123209+0000","last_fullsized":"2024-09-21T19:11:49.123209+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":652,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.133096+0000","last_change":"2024-09-21T19:11:49.133015+0000","last_active":"2024-09-21T19:11:49.133096+0000","last_peered":"2024-09-21T19:11:49.133096+0000","last_clean":"2024-09-21T19:11:49.133096+0000","last_became_active":"2024-09-21T19:11:49.131805+0000","last_became_peered":"2024-09-21T19:11:49.131805+0000","last_unstale":"2024-09-21T19:11:49.133096+0000","last_undegraded":"2024-09-21T19:11:49.133096+0000","last_fullsized":"2024-09-21T19:11:49.133096+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:28.147274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.147274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"451'8595","reported_seq":8766,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.401339+0000","last_change":"2024-09-21T19:11:38.535387+0000","last_active":"2024-09-21T19:11:49.401339+0000","last_peered":"2024-09-21T19:11:49.401339+0000","last_clean":"2024-09-21T19:11:49.401339+0000","last_became_active":"2024-09-21T19:10:01.250116+0000","last_became_peered":"2024-09-21T19:10:01.250116+0000","last_unstale":"2024-09-21T19:11:49.401339+0000","last_undegraded":"2024-09-21T19:11:49.401339+0000","last_fullsized":"2024-09-21T19:11:49.401339+0000","mapping_epoch":414,"log_start":"429'3792","ondisk_log_start":"429'3792","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'7836","last_scrub_stamp":"2024-09-21T19:11:38.534990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.534990+0000","objects_scrubbed":7739,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:54.342857+0000","scrub_duration":2790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457113600,"num_objects":6975,"num_object_clones":0,"num_object_copies":20925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":498240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":899,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123691+0000","last_change":"2024-09-21T19:10:23.782154+0000","last_active":"2024-09-21T19:11:49.123691+0000","last_peered":"2024-09-21T19:11:49.123691+0000","last_clean":"2024-09-21T19:11:49.123691+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:11:49.123691+0000","last_undegraded":"2024-09-21T19:11:49.123691+0000","last_fullsized":"2024-09-21T19:11:49.123691+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:10:23.782107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.782107+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":867,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123068+0000","last_change":"2024-09-21T19:10:19.729765+0000","last_active":"2024-09-21T19:11:49.123068+0000","last_peered":"2024-09-21T19:11:49.123068+0000","last_clean":"2024-09-21T19:11:49.123068+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:11:49.123068+0000","last_undegraded":"2024-09-21T19:11:49.123068+0000","last_fullsized":"2024-09-21T19:11:49.123068+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:19.729685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.729685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123786+0000","last_change":"2024-09-21T19:10:58.999039+0000","last_active":"2024-09-21T19:11:49.123786+0000","last_peered":"2024-09-21T19:11:49.123786+0000","last_clean":"2024-09-21T19:11:49.123786+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:11:49.123786+0000","last_undegraded":"2024-09-21T19:11:49.123786+0000","last_fullsized":"2024-09-21T19:11:49.123786+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.998847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.998847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:59.614433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":824,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121141+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:11:49.121141+0000","last_peered":"2024-09-21T19:11:49.121141+0000","last_clean":"2024-09-21T19:11:49.121141+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:11:49.121141+0000","last_undegraded":"2024-09-21T19:11:49.121141+0000","last_fullsized":"2024-09-21T19:11:49.121141+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.968888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"451'8642","reported_seq":8792,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:53.143989+0000","last_change":"2024-09-21T19:11:09.898977+0000","last_active":"2024-09-21T19:11:53.143989+0000","last_peered":"2024-09-21T19:11:53.143989+0000","last_clean":"2024-09-21T19:11:53.143989+0000","last_became_active":"2024-09-21T19:10:01.247852+0000","last_became_peered":"2024-09-21T19:10:01.247852+0000","last_unstale":"2024-09-21T19:11:53.143989+0000","last_undegraded":"2024-09-21T19:11:53.143989+0000","last_fullsized":"2024-09-21T19:11:53.143989+0000","mapping_epoch":414,"log_start":"426'2434","ondisk_log_start":"426'2434","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'5812","last_scrub_stamp":"2024-09-21T19:11:09.898694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.898694+0000","objects_scrubbed":5735,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:34.977808+0000","scrub_duration":2494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427556864,"num_objects":6524,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":308,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126858+0000","last_change":"2024-09-21T19:11:48.520793+0000","last_active":"2024-09-21T19:11:49.126858+0000","last_peered":"2024-09-21T19:11:49.126858+0000","last_clean":"2024-09-21T19:11:49.126858+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:11:49.126858+0000","last_undegraded":"2024-09-21T19:11:49.126858+0000","last_fullsized":"2024-09-21T19:11:49.126858+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.520711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.520711+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.918449+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":940,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123818+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:11:49.123818+0000","last_peered":"2024-09-21T19:11:49.123818+0000","last_clean":"2024-09-21T19:11:49.123818+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:11:49.123818+0000","last_undegraded":"2024-09-21T19:11:49.123818+0000","last_fullsized":"2024-09-21T19:11:49.123818+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":774,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132671+0000","last_change":"2024-09-21T19:11:46.781966+0000","last_active":"2024-09-21T19:11:49.132671+0000","last_peered":"2024-09-21T19:11:49.132671+0000","last_clean":"2024-09-21T19:11:49.132671+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:11:49.132671+0000","last_undegraded":"2024-09-21T19:11:49.132671+0000","last_fullsized":"2024-09-21T19:11:49.132671+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:46.781896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:46.781896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:50.353466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":298,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.131819+0000","last_change":"2024-09-21T19:11:31.656763+0000","last_active":"2024-09-21T19:11:49.131819+0000","last_peered":"2024-09-21T19:11:49.131819+0000","last_clean":"2024-09-21T19:11:49.131819+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:11:49.131819+0000","last_undegraded":"2024-09-21T19:11:49.131819+0000","last_fullsized":"2024-09-21T19:11:49.131819+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.656689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.656689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.381458+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":298,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128033+0000","last_change":"2024-09-21T19:10:45.402838+0000","last_active":"2024-09-21T19:11:49.128033+0000","last_peered":"2024-09-21T19:11:49.128033+0000","last_clean":"2024-09-21T19:11:49.128033+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:11:49.128033+0000","last_undegraded":"2024-09-21T19:11:49.128033+0000","last_fullsized":"2024-09-21T19:11:49.128033+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:45.402769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.402769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:57.226877+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"451'8517","reported_seq":8602,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.421099+0000","last_change":"2024-09-21T19:10:01.252391+0000","last_active":"2024-09-21T19:11:49.421099+0000","last_peered":"2024-09-21T19:11:49.421099+0000","last_clean":"2024-09-21T19:11:49.421099+0000","last_became_active":"2024-09-21T19:10:01.251894+0000","last_became_peered":"2024-09-21T19:10:01.251894+0000","last_unstale":"2024-09-21T19:11:49.421099+0000","last_undegraded":"2024-09-21T19:11:49.421099+0000","last_fullsized":"2024-09-21T19:11:49.421099+0000","mapping_epoch":414,"log_start":"429'3865","ondisk_log_start":"429'3865","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.727577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454098944,"num_objects":6929,"num_object_clones":0,"num_object_copies":20787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8517,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":956,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125731+0000","last_change":"2024-09-21T19:10:51.731370+0000","last_active":"2024-09-21T19:11:49.125731+0000","last_peered":"2024-09-21T19:11:49.125731+0000","last_clean":"2024-09-21T19:11:49.125731+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:11:49.125731+0000","last_undegraded":"2024-09-21T19:11:49.125731+0000","last_fullsized":"2024-09-21T19:11:49.125731+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:51.731299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.731299+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:03.710169+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":866,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123620+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:11:49.123620+0000","last_peered":"2024-09-21T19:11:49.123620+0000","last_clean":"2024-09-21T19:11:49.123620+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:11:49.123620+0000","last_undegraded":"2024-09-21T19:11:49.123620+0000","last_fullsized":"2024-09-21T19:11:49.123620+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":874,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122678+0000","last_change":"2024-09-21T19:10:55.909242+0000","last_active":"2024-09-21T19:11:49.122678+0000","last_peered":"2024-09-21T19:11:49.122678+0000","last_clean":"2024-09-21T19:11:49.122678+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:11:49.122678+0000","last_undegraded":"2024-09-21T19:11:49.122678+0000","last_fullsized":"2024-09-21T19:11:49.122678+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:55.909172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:55.909172+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.738817+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":382,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128086+0000","last_change":"2024-09-21T19:11:38.701095+0000","last_active":"2024-09-21T19:11:49.128086+0000","last_peered":"2024-09-21T19:11:49.128086+0000","last_clean":"2024-09-21T19:11:49.128086+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:11:49.128086+0000","last_undegraded":"2024-09-21T19:11:49.128086+0000","last_fullsized":"2024-09-21T19:11:49.128086+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:11:38.700917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.700917+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:01.033044+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":839,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123406+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:11:49.123406+0000","last_peered":"2024-09-21T19:11:49.123406+0000","last_clean":"2024-09-21T19:11:49.123406+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:11:49.123406+0000","last_undegraded":"2024-09-21T19:11:49.123406+0000","last_fullsized":"2024-09-21T19:11:49.123406+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":12,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.148265+0000","last_change":"2024-09-21T19:11:49.147463+0000","last_active":"2024-09-21T19:11:49.148265+0000","last_peered":"2024-09-21T19:11:49.148265+0000","last_clean":"2024-09-21T19:11:49.148265+0000","last_became_active":"2024-09-21T19:11:49.147110+0000","last_became_peered":"2024-09-21T19:11:49.147110+0000","last_unstale":"2024-09-21T19:11:49.148265+0000","last_undegraded":"2024-09-21T19:11:49.148265+0000","last_fullsized":"2024-09-21T19:11:49.148265+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"451'8689","reported_seq":8774,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:51.868077+0000","last_change":"2024-09-21T19:10:01.250343+0000","last_active":"2024-09-21T19:11:51.868077+0000","last_peered":"2024-09-21T19:11:51.868077+0000","last_clean":"2024-09-21T19:11:51.868077+0000","last_became_active":"2024-09-21T19:10:01.249931+0000","last_became_peered":"2024-09-21T19:10:01.249931+0000","last_unstale":"2024-09-21T19:11:51.868077+0000","last_undegraded":"2024-09-21T19:11:51.868077+0000","last_fullsized":"2024-09-21T19:11:51.868077+0000","mapping_epoch":414,"log_start":"430'4120","ondisk_log_start":"430'4120","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.706001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440729600,"num_objects":6725,"num_object_clones":0,"num_object_copies":20175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":735,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132622+0000","last_change":"2024-09-21T19:11:41.665090+0000","last_active":"2024-09-21T19:11:49.132622+0000","last_peered":"2024-09-21T19:11:49.132622+0000","last_clean":"2024-09-21T19:11:49.132622+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:11:49.132622+0000","last_undegraded":"2024-09-21T19:11:49.132622+0000","last_fullsized":"2024-09-21T19:11:49.132622+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:41.665004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.665004+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.270163+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":827,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125559+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:11:49.125559+0000","last_peered":"2024-09-21T19:11:49.125559+0000","last_clean":"2024-09-21T19:11:49.125559+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:11:49.125559+0000","last_undegraded":"2024-09-21T19:11:49.125559+0000","last_fullsized":"2024-09-21T19:11:49.125559+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.953609+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":290,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127630+0000","last_change":"2024-09-21T19:10:27.397361+0000","last_active":"2024-09-21T19:11:49.127630+0000","last_peered":"2024-09-21T19:11:49.127630+0000","last_clean":"2024-09-21T19:11:49.127630+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:11:49.127630+0000","last_undegraded":"2024-09-21T19:11:49.127630+0000","last_fullsized":"2024-09-21T19:11:49.127630+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.397188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.397188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:57.228642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":738,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.130194+0000","last_change":"2024-09-21T19:11:31.588989+0000","last_active":"2024-09-21T19:11:49.130194+0000","last_peered":"2024-09-21T19:11:49.130194+0000","last_clean":"2024-09-21T19:11:49.130194+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:11:49.130194+0000","last_undegraded":"2024-09-21T19:11:49.130194+0000","last_fullsized":"2024-09-21T19:11:49.130194+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.588820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.588820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:42.713349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":657,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.146226+0000","last_change":"2024-09-21T19:11:49.143261+0000","last_active":"2024-09-21T19:11:49.146226+0000","last_peered":"2024-09-21T19:11:49.146226+0000","last_clean":"2024-09-21T19:11:49.146226+0000","last_became_active":"2024-09-21T19:11:49.142607+0000","last_became_peered":"2024-09-21T19:11:49.142607+0000","last_unstale":"2024-09-21T19:11:49.146226+0000","last_undegraded":"2024-09-21T19:11:49.146226+0000","last_fullsized":"2024-09-21T19:11:49.146226+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"451'8663","reported_seq":8748,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:51.943113+0000","last_change":"2024-09-21T19:10:01.260078+0000","last_active":"2024-09-21T19:11:51.943113+0000","last_peered":"2024-09-21T19:11:51.943113+0000","last_clean":"2024-09-21T19:11:51.943113+0000","last_became_active":"2024-09-21T19:10:01.259752+0000","last_became_peered":"2024-09-21T19:10:01.259752+0000","last_unstale":"2024-09-21T19:11:51.943113+0000","last_undegraded":"2024-09-21T19:11:51.943113+0000","last_fullsized":"2024-09-21T19:11:51.943113+0000","mapping_epoch":414,"log_start":"429'4059","ondisk_log_start":"429'4059","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.779638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440467456,"num_objects":6721,"num_object_clones":0,"num_object_copies":20163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8663,"num_write_kb":492288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":785,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132541+0000","last_change":"2024-09-21T19:10:12.835830+0000","last_active":"2024-09-21T19:11:49.132541+0000","last_peered":"2024-09-21T19:11:49.132541+0000","last_clean":"2024-09-21T19:11:49.132541+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:11:49.132541+0000","last_undegraded":"2024-09-21T19:11:49.132541+0000","last_fullsized":"2024-09-21T19:11:49.132541+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:10:12.835752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:12.835752+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.728578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":866,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126261+0000","last_change":"2024-09-21T19:10:29.539254+0000","last_active":"2024-09-21T19:11:49.126261+0000","last_peered":"2024-09-21T19:11:49.126261+0000","last_clean":"2024-09-21T19:11:49.126261+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:11:49.126261+0000","last_undegraded":"2024-09-21T19:11:49.126261+0000","last_fullsized":"2024-09-21T19:11:49.126261+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:29.539210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:29.539210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":301,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127153+0000","last_change":"2024-09-21T19:11:34.718818+0000","last_active":"2024-09-21T19:11:49.127153+0000","last_peered":"2024-09-21T19:11:49.127153+0000","last_clean":"2024-09-21T19:11:49.127153+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:11:49.127153+0000","last_undegraded":"2024-09-21T19:11:49.127153+0000","last_fullsized":"2024-09-21T19:11:49.127153+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:34.718771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.718771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:58.941434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":846,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127280+0000","last_change":"2024-09-21T19:10:58.720745+0000","last_active":"2024-09-21T19:11:49.127280+0000","last_peered":"2024-09-21T19:11:49.127280+0000","last_clean":"2024-09-21T19:11:49.127280+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:11:49.127280+0000","last_undegraded":"2024-09-21T19:11:49.127280+0000","last_fullsized":"2024-09-21T19:11:49.127280+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.720634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.720634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.021358+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":639,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.161173+0000","last_change":"2024-09-21T19:11:49.161093+0000","last_active":"2024-09-21T19:11:49.161173+0000","last_peered":"2024-09-21T19:11:49.161173+0000","last_clean":"2024-09-21T19:11:49.161173+0000","last_became_active":"2024-09-21T19:11:49.145230+0000","last_became_peered":"2024-09-21T19:11:49.145230+0000","last_unstale":"2024-09-21T19:11:49.161173+0000","last_undegraded":"2024-09-21T19:11:49.161173+0000","last_fullsized":"2024-09-21T19:11:49.161173+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:41.967849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.967849+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"451'8891","reported_seq":8976,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:52.878711+0000","last_change":"2024-09-21T19:10:01.252118+0000","last_active":"2024-09-21T19:11:52.878711+0000","last_peered":"2024-09-21T19:11:52.878711+0000","last_clean":"2024-09-21T19:11:52.878711+0000","last_became_active":"2024-09-21T19:10:01.251738+0000","last_became_peered":"2024-09-21T19:10:01.251738+0000","last_unstale":"2024-09-21T19:11:52.878711+0000","last_undegraded":"2024-09-21T19:11:52.878711+0000","last_fullsized":"2024-09-21T19:11:52.878711+0000","mapping_epoch":414,"log_start":"428'3686","ondisk_log_start":"428'3686","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.479895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439549952,"num_objects":6707,"num_object_clones":0,"num_object_copies":20121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":499136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":903,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123143+0000","last_change":"2024-09-21T19:11:31.160427+0000","last_active":"2024-09-21T19:11:49.123143+0000","last_peered":"2024-09-21T19:11:49.123143+0000","last_clean":"2024-09-21T19:11:49.123143+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:11:49.123143+0000","last_undegraded":"2024-09-21T19:11:49.123143+0000","last_fullsized":"2024-09-21T19:11:49.123143+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:11:31.160369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.160369+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.166557+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":868,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.136911+0000","last_change":"2024-09-21T19:10:57.731366+0000","last_active":"2024-09-21T19:11:49.136911+0000","last_peered":"2024-09-21T19:11:49.136911+0000","last_clean":"2024-09-21T19:11:49.136911+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:11:49.136911+0000","last_undegraded":"2024-09-21T19:11:49.136911+0000","last_fullsized":"2024-09-21T19:11:49.136911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:57.731319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.731319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.882041+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":290,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126848+0000","last_change":"2024-09-21T19:11:06.454799+0000","last_active":"2024-09-21T19:11:49.126848+0000","last_peered":"2024-09-21T19:11:49.126848+0000","last_clean":"2024-09-21T19:11:49.126848+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:11:49.126848+0000","last_undegraded":"2024-09-21T19:11:49.126848+0000","last_fullsized":"2024-09-21T19:11:49.126848+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:06.454727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.454727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.197782+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":818,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122367+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:11:49.122367+0000","last_peered":"2024-09-21T19:11:49.122367+0000","last_clean":"2024-09-21T19:11:49.122367+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:11:49.122367+0000","last_undegraded":"2024-09-21T19:11:49.122367+0000","last_fullsized":"2024-09-21T19:11:49.122367+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.112801+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":12,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.141126+0000","last_change":"2024-09-21T19:11:49.139840+0000","last_active":"2024-09-21T19:11:49.141126+0000","last_peered":"2024-09-21T19:11:49.141126+0000","last_clean":"2024-09-21T19:11:49.141126+0000","last_became_active":"2024-09-21T19:11:49.139511+0000","last_became_peered":"2024-09-21T19:11:49.139511+0000","last_unstale":"2024-09-21T19:11:49.141126+0000","last_undegraded":"2024-09-21T19:11:49.141126+0000","last_fullsized":"2024-09-21T19:11:49.141126+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"451'8909","reported_seq":9062,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:52.801268+0000","last_change":"2024-09-21T19:11:09.208009+0000","last_active":"2024-09-21T19:11:52.801268+0000","last_peered":"2024-09-21T19:11:52.801268+0000","last_clean":"2024-09-21T19:11:52.801268+0000","last_became_active":"2024-09-21T19:10:01.259826+0000","last_became_peered":"2024-09-21T19:10:01.259826+0000","last_unstale":"2024-09-21T19:11:52.801268+0000","last_undegraded":"2024-09-21T19:11:52.801268+0000","last_fullsized":"2024-09-21T19:11:52.801268+0000","mapping_epoch":414,"log_start":"430'4289","ondisk_log_start":"430'4289","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'5961","last_scrub_stamp":"2024-09-21T19:11:09.207825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.207825+0000","objects_scrubbed":5889,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.058679+0000","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445710336,"num_objects":6801,"num_object_clones":0,"num_object_copies":20403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8909,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":467,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127079+0000","last_change":"2024-09-21T19:11:47.023173+0000","last_active":"2024-09-21T19:11:49.127079+0000","last_peered":"2024-09-21T19:11:49.127079+0000","last_clean":"2024-09-21T19:11:49.127079+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:11:49.127079+0000","last_undegraded":"2024-09-21T19:11:49.127079+0000","last_fullsized":"2024-09-21T19:11:49.127079+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:11:47.023075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.023075+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.009824+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":882,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127660+0000","last_change":"2024-09-21T19:11:37.790112+0000","last_active":"2024-09-21T19:11:49.127660+0000","last_peered":"2024-09-21T19:11:49.127660+0000","last_clean":"2024-09-21T19:11:49.127660+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:11:49.127660+0000","last_undegraded":"2024-09-21T19:11:49.127660+0000","last_fullsized":"2024-09-21T19:11:49.127660+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:11:37.790039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.790039+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:54.838367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":370,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123525+0000","last_change":"2024-09-21T19:10:59.827898+0000","last_active":"2024-09-21T19:11:49.123525+0000","last_peered":"2024-09-21T19:11:49.123525+0000","last_clean":"2024-09-21T19:11:49.123525+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:11:49.123525+0000","last_undegraded":"2024-09-21T19:11:49.123525+0000","last_fullsized":"2024-09-21T19:11:49.123525+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:10:59.827833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.827833+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.990995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":862,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132133+0000","last_change":"2024-09-21T19:10:59.672382+0000","last_active":"2024-09-21T19:11:49.132133+0000","last_peered":"2024-09-21T19:11:49.132133+0000","last_clean":"2024-09-21T19:11:49.132133+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:11:49.132133+0000","last_undegraded":"2024-09-21T19:11:49.132133+0000","last_fullsized":"2024-09-21T19:11:49.132133+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:59.672320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.672320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:11:59.920295+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":12,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.147183+0000","last_change":"2024-09-21T19:11:49.147092+0000","last_active":"2024-09-21T19:11:49.147183+0000","last_peered":"2024-09-21T19:11:49.147183+0000","last_clean":"2024-09-21T19:11:49.147183+0000","last_became_active":"2024-09-21T19:11:49.146489+0000","last_became_peered":"2024-09-21T19:11:49.146489+0000","last_unstale":"2024-09-21T19:11:49.147183+0000","last_undegraded":"2024-09-21T19:11:49.147183+0000","last_fullsized":"2024-09-21T19:11:49.147183+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":796,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126707+0000","last_change":"2024-09-21T19:11:08.539733+0000","last_active":"2024-09-21T19:11:49.126707+0000","last_peered":"2024-09-21T19:11:49.126707+0000","last_clean":"2024-09-21T19:11:49.126707+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:11:49.126707+0000","last_undegraded":"2024-09-21T19:11:49.126707+0000","last_fullsized":"2024-09-21T19:11:49.126707+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.539689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.539689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:35.735538+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":859,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123545+0000","last_change":"2024-09-21T19:10:24.873768+0000","last_active":"2024-09-21T19:11:49.123545+0000","last_peered":"2024-09-21T19:11:49.123545+0000","last_clean":"2024-09-21T19:11:49.123545+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:11:49.123545+0000","last_undegraded":"2024-09-21T19:11:49.123545+0000","last_fullsized":"2024-09-21T19:11:49.123545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:10:24.873701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:24.873701+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":387,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122752+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:11:49.122752+0000","last_peered":"2024-09-21T19:11:49.122752+0000","last_clean":"2024-09-21T19:11:49.122752+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:11:49.122752+0000","last_undegraded":"2024-09-21T19:11:49.122752+0000","last_fullsized":"2024-09-21T19:11:49.122752+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.027866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":834,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125670+0000","last_change":"2024-09-21T19:10:22.627798+0000","last_active":"2024-09-21T19:11:49.125670+0000","last_peered":"2024-09-21T19:11:49.125670+0000","last_clean":"2024-09-21T19:11:49.125670+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:11:49.125670+0000","last_undegraded":"2024-09-21T19:11:49.125670+0000","last_fullsized":"2024-09-21T19:11:49.125670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:22.627740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:22.627740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":652,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.147692+0000","last_change":"2024-09-21T19:11:49.147602+0000","last_active":"2024-09-21T19:11:49.147692+0000","last_peered":"2024-09-21T19:11:49.147692+0000","last_clean":"2024-09-21T19:11:49.147692+0000","last_became_active":"2024-09-21T19:11:49.138877+0000","last_became_peered":"2024-09-21T19:11:49.138877+0000","last_unstale":"2024-09-21T19:11:49.147692+0000","last_undegraded":"2024-09-21T19:11:49.147692+0000","last_fullsized":"2024-09-21T19:11:49.147692+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:28.147274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.147274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":734,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.132814+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:11:49.132814+0000","last_peered":"2024-09-21T19:11:49.132814+0000","last_clean":"2024-09-21T19:11:49.132814+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:11:49.132814+0000","last_undegraded":"2024-09-21T19:11:49.132814+0000","last_fullsized":"2024-09-21T19:11:49.132814+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.467919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125923+0000","last_change":"2024-09-21T19:11:44.883032+0000","last_active":"2024-09-21T19:11:49.125923+0000","last_peered":"2024-09-21T19:11:49.125923+0000","last_clean":"2024-09-21T19:11:49.125923+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:11:49.125923+0000","last_undegraded":"2024-09-21T19:11:49.125923+0000","last_fullsized":"2024-09-21T19:11:49.125923+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:44.882985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.882985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:07.884044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":829,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123387+0000","last_change":"2024-09-21T19:10:46.660901+0000","last_active":"2024-09-21T19:11:49.123387+0000","last_peered":"2024-09-21T19:11:49.123387+0000","last_clean":"2024-09-21T19:11:49.123387+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:11:49.123387+0000","last_undegraded":"2024-09-21T19:11:49.123387+0000","last_fullsized":"2024-09-21T19:11:49.123387+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.660838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.660838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:07.972413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":290,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.130134+0000","last_change":"2024-09-21T19:11:43.659822+0000","last_active":"2024-09-21T19:11:49.130134+0000","last_peered":"2024-09-21T19:11:49.130134+0000","last_clean":"2024-09-21T19:11:49.130134+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:11:49.130134+0000","last_undegraded":"2024-09-21T19:11:49.130134+0000","last_fullsized":"2024-09-21T19:11:49.130134+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:43.659592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:43.659592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:49.689636+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":960,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.134673+0000","last_change":"2024-09-21T19:11:49.134584+0000","last_active":"2024-09-21T19:11:49.134673+0000","last_peered":"2024-09-21T19:11:49.134673+0000","last_clean":"2024-09-21T19:11:49.134673+0000","last_became_active":"2024-09-21T19:11:49.133497+0000","last_became_peered":"2024-09-21T19:11:49.133497+0000","last_unstale":"2024-09-21T19:11:49.134673+0000","last_undegraded":"2024-09-21T19:11:49.134673+0000","last_fullsized":"2024-09-21T19:11:49.134673+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:45.911119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.911119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":448,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125904+0000","last_change":"2024-09-21T19:11:34.827737+0000","last_active":"2024-09-21T19:11:49.125904+0000","last_peered":"2024-09-21T19:11:49.125904+0000","last_clean":"2024-09-21T19:11:49.125904+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:11:49.125904+0000","last_undegraded":"2024-09-21T19:11:49.125904+0000","last_fullsized":"2024-09-21T19:11:49.125904+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:34.827659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.827659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.075348+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":731,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.133305+0000","last_change":"2024-09-21T19:11:40.646183+0000","last_active":"2024-09-21T19:11:49.133305+0000","last_peered":"2024-09-21T19:11:49.133305+0000","last_clean":"2024-09-21T19:11:49.133305+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:11:49.133305+0000","last_undegraded":"2024-09-21T19:11:49.133305+0000","last_fullsized":"2024-09-21T19:11:49.133305+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:40.645904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:40.645904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:44.505399+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1087,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:52.677352+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:11:52.677352+0000","last_peered":"2024-09-21T19:11:52.677352+0000","last_clean":"2024-09-21T19:11:52.677352+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:11:52.677352+0000","last_undegraded":"2024-09-21T19:11:52.677352+0000","last_fullsized":"2024-09-21T19:11:52.677352+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":849,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123757+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:11:49.123757+0000","last_peered":"2024-09-21T19:11:49.123757+0000","last_clean":"2024-09-21T19:11:49.123757+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:11:49.123757+0000","last_undegraded":"2024-09-21T19:11:49.123757+0000","last_fullsized":"2024-09-21T19:11:49.123757+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:29.127237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":618,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.151455+0000","last_change":"2024-09-21T19:11:49.151010+0000","last_active":"2024-09-21T19:11:49.151455+0000","last_peered":"2024-09-21T19:11:49.151455+0000","last_clean":"2024-09-21T19:11:49.151455+0000","last_became_active":"2024-09-21T19:11:49.149051+0000","last_became_peered":"2024-09-21T19:11:49.149051+0000","last_unstale":"2024-09-21T19:11:49.151455+0000","last_undegraded":"2024-09-21T19:11:49.151455+0000","last_fullsized":"2024-09-21T19:11:49.151455+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:47.627361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.627361+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.786190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":875,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123973+0000","last_change":"2024-09-21T19:10:39.003954+0000","last_active":"2024-09-21T19:11:49.123973+0000","last_peered":"2024-09-21T19:11:49.123973+0000","last_clean":"2024-09-21T19:11:49.123973+0000","last_became_active":"2024-09-21T19:10:38.923267+0000","last_became_peered":"2024-09-21T19:10:38.923267+0000","last_unstale":"2024-09-21T19:11:49.123973+0000","last_undegraded":"2024-09-21T19:11:49.123973+0000","last_fullsized":"2024-09-21T19:11:49.123973+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:19.876033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.876033+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127401+0000","last_change":"2024-09-21T19:10:46.686915+0000","last_active":"2024-09-21T19:11:49.127401+0000","last_peered":"2024-09-21T19:11:49.127401+0000","last_clean":"2024-09-21T19:11:49.127401+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:11:49.127401+0000","last_undegraded":"2024-09-21T19:11:49.127401+0000","last_fullsized":"2024-09-21T19:11:49.127401+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.686843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.686843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.453957+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":544,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121339+0000","last_change":"2024-09-21T19:11:42.105123+0000","last_active":"2024-09-21T19:11:49.121339+0000","last_peered":"2024-09-21T19:11:49.121339+0000","last_clean":"2024-09-21T19:11:49.121339+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:11:49.121339+0000","last_undegraded":"2024-09-21T19:11:49.121339+0000","last_fullsized":"2024-09-21T19:11:49.121339+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:11:42.105053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:42.105053+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:56.107355+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":723,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.133077+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:11:49.133077+0000","last_peered":"2024-09-21T19:11:49.133077+0000","last_clean":"2024-09-21T19:11:49.133077+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:11:49.133077+0000","last_undegraded":"2024-09-21T19:11:49.133077+0000","last_fullsized":"2024-09-21T19:11:49.133077+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.496959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":929,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123387+0000","last_change":"2024-09-21T19:11:41.105439+0000","last_active":"2024-09-21T19:11:49.123387+0000","last_peered":"2024-09-21T19:11:49.123387+0000","last_clean":"2024-09-21T19:11:49.123387+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:11:49.123387+0000","last_undegraded":"2024-09-21T19:11:49.123387+0000","last_fullsized":"2024-09-21T19:11:49.123387+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:41.105362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.105362+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:43.066173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":839,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123705+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:11:49.123705+0000","last_peered":"2024-09-21T19:11:49.123705+0000","last_clean":"2024-09-21T19:11:49.123705+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:11:49.123705+0000","last_undegraded":"2024-09-21T19:11:49.123705+0000","last_fullsized":"2024-09-21T19:11:49.123705+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.166063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":856,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121545+0000","last_change":"2024-09-21T19:11:41.125672+0000","last_active":"2024-09-21T19:11:49.121545+0000","last_peered":"2024-09-21T19:11:49.121545+0000","last_clean":"2024-09-21T19:11:49.121545+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:11:49.121545+0000","last_undegraded":"2024-09-21T19:11:49.121545+0000","last_fullsized":"2024-09-21T19:11:49.121545+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:41.125603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.125603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.999077+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123085+0000","last_change":"2024-09-21T19:10:27.770820+0000","last_active":"2024-09-21T19:11:49.123085+0000","last_peered":"2024-09-21T19:11:49.123085+0000","last_clean":"2024-09-21T19:11:49.123085+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:11:49.123085+0000","last_undegraded":"2024-09-21T19:11:49.123085+0000","last_fullsized":"2024-09-21T19:11:49.123085+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.770751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.770751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":927,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.123130+0000","last_change":"2024-09-21T19:11:06.137061+0000","last_active":"2024-09-21T19:11:49.123130+0000","last_peered":"2024-09-21T19:11:49.123130+0000","last_clean":"2024-09-21T19:11:49.123130+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:11:49.123130+0000","last_undegraded":"2024-09-21T19:11:49.123130+0000","last_fullsized":"2024-09-21T19:11:49.123130+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:06.136973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.136973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.021915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.126009+0000","last_change":"2024-09-21T19:11:07.755091+0000","last_active":"2024-09-21T19:11:49.126009+0000","last_peered":"2024-09-21T19:11:49.126009+0000","last_clean":"2024-09-21T19:11:49.126009+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:11:49.126009+0000","last_undegraded":"2024-09-21T19:11:49.126009+0000","last_fullsized":"2024-09-21T19:11:49.126009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:07.755026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.755026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.561489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":844,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122491+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:11:49.122491+0000","last_peered":"2024-09-21T19:11:49.122491+0000","last_clean":"2024-09-21T19:11:49.122491+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:11:49.122491+0000","last_undegraded":"2024-09-21T19:11:49.122491+0000","last_fullsized":"2024-09-21T19:11:49.122491+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:26.155553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":742,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.129799+0000","last_change":"2024-09-21T19:11:36.632665+0000","last_active":"2024-09-21T19:11:49.129799+0000","last_peered":"2024-09-21T19:11:49.129799+0000","last_clean":"2024-09-21T19:11:49.129799+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:11:49.129799+0000","last_undegraded":"2024-09-21T19:11:49.129799+0000","last_fullsized":"2024-09-21T19:11:49.129799+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:36.632596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:36.632596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.311366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":12,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.141133+0000","last_change":"2024-09-21T19:11:49.141039+0000","last_active":"2024-09-21T19:11:49.141133+0000","last_peered":"2024-09-21T19:11:49.141133+0000","last_clean":"2024-09-21T19:11:49.141133+0000","last_became_active":"2024-09-21T19:11:49.139887+0000","last_became_peered":"2024-09-21T19:11:49.139887+0000","last_unstale":"2024-09-21T19:11:49.141133+0000","last_undegraded":"2024-09-21T19:11:49.141133+0000","last_fullsized":"2024-09-21T19:11:49.141133+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:41.967849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.967849+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":876,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122919+0000","last_change":"2024-09-21T19:11:37.973551+0000","last_active":"2024-09-21T19:11:49.122919+0000","last_peered":"2024-09-21T19:11:49.122919+0000","last_clean":"2024-09-21T19:11:49.122919+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:11:49.122919+0000","last_undegraded":"2024-09-21T19:11:49.122919+0000","last_fullsized":"2024-09-21T19:11:49.122919+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:37.973476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.973476+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.521765+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":789,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.127097+0000","last_change":"2024-09-21T19:11:03.004430+0000","last_active":"2024-09-21T19:11:49.127097+0000","last_peered":"2024-09-21T19:11:49.127097+0000","last_clean":"2024-09-21T19:11:49.127097+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:11:49.127097+0000","last_undegraded":"2024-09-21T19:11:49.127097+0000","last_fullsized":"2024-09-21T19:11:49.127097+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.004386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.004386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.746086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":871,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.125801+0000","last_change":"2024-09-21T19:11:01.724461+0000","last_active":"2024-09-21T19:11:49.125801+0000","last_peered":"2024-09-21T19:11:49.125801+0000","last_clean":"2024-09-21T19:11:49.125801+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:11:49.125801+0000","last_undegraded":"2024-09-21T19:11:49.125801+0000","last_fullsized":"2024-09-21T19:11:49.125801+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:01.724417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:01.724417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:02.723695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":850,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122590+0000","last_change":"2024-09-21T19:10:16.861750+0000","last_active":"2024-09-21T19:11:49.122590+0000","last_peered":"2024-09-21T19:11:49.122590+0000","last_clean":"2024-09-21T19:11:49.122590+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:11:49.122590+0000","last_undegraded":"2024-09-21T19:11:49.122590+0000","last_fullsized":"2024-09-21T19:11:49.122590+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.861671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.861671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":875,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121952+0000","last_change":"2024-09-21T19:10:52.788414+0000","last_active":"2024-09-21T19:11:49.121952+0000","last_peered":"2024-09-21T19:11:49.121952+0000","last_clean":"2024-09-21T19:11:49.121952+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:11:49.121952+0000","last_undegraded":"2024-09-21T19:11:49.121952+0000","last_fullsized":"2024-09-21T19:11:49.121952+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:10:52.788342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:52.788342+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:04.702000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":726,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.134445+0000","last_change":"2024-09-21T19:11:02.421016+0000","last_active":"2024-09-21T19:11:49.134445+0000","last_peered":"2024-09-21T19:11:49.134445+0000","last_clean":"2024-09-21T19:11:49.134445+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:11:49.134445+0000","last_undegraded":"2024-09-21T19:11:49.134445+0000","last_fullsized":"2024-09-21T19:11:49.134445+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:02.420940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.420940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.532139+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":755,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.130777+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:11:49.130777+0000","last_peered":"2024-09-21T19:11:49.130777+0000","last_clean":"2024-09-21T19:11:49.130777+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:11:49.130777+0000","last_undegraded":"2024-09-21T19:11:49.130777+0000","last_fullsized":"2024-09-21T19:11:49.130777+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.662425+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.128340+0000","last_change":"2024-09-21T19:11:09.805596+0000","last_active":"2024-09-21T19:11:49.128340+0000","last_peered":"2024-09-21T19:11:49.128340+0000","last_clean":"2024-09-21T19:11:49.128340+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:11:49.128340+0000","last_undegraded":"2024-09-21T19:11:49.128340+0000","last_fullsized":"2024-09-21T19:11:49.128340+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:09.805527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.805527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:24.581512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.121152+0000","last_change":"2024-09-21T19:11:40.130856+0000","last_active":"2024-09-21T19:11:49.121152+0000","last_peered":"2024-09-21T19:11:49.121152+0000","last_clean":"2024-09-21T19:11:49.121152+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:11:49.121152+0000","last_undegraded":"2024-09-21T19:11:49.121152+0000","last_fullsized":"2024-09-21T19:11:49.121152+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:40.130789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:40.130789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:02.390303+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":859,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.122134+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:11:49.122134+0000","last_peered":"2024-09-21T19:11:49.122134+0000","last_clean":"2024-09-21T19:11:49.122134+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:11:49.122134+0000","last_undegraded":"2024-09-21T19:11:49.122134+0000","last_fullsized":"2024-09-21T19:11:49.122134+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.009111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":837,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.131380+0000","last_change":"2024-09-21T19:11:00.629261+0000","last_active":"2024-09-21T19:11:49.131380+0000","last_peered":"2024-09-21T19:11:49.131380+0000","last_clean":"2024-09-21T19:11:49.131380+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:11:49.131380+0000","last_undegraded":"2024-09-21T19:11:49.131380+0000","last_fullsized":"2024-09-21T19:11:49.131380+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:00.629191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.629191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.169952+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":747,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-09-21T19:11:49.133033+0000","last_change":"2024-09-21T19:10:23.500631+0000","last_active":"2024-09-21T19:11:49.133033+0000","last_peered":"2024-09-21T19:11:49.133033+0000","last_clean":"2024-09-21T19:11:49.133033+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:11:49.133033+0000","last_undegraded":"2024-09-21T19:11:49.133033+0000","last_fullsized":"2024-09-21T19:11:49.133033+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:23.500562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.500562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7020347416,"num_objects":107123,"num_object_clones":0,"num_object_copies":321369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140115,"num_write_kb":7911553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21091659776,"data_stored":21091647560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75511,"ondisk_log_size":75511,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":5446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1112,"num_read_kb":1107,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":534812,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":22,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":697708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266917,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3282052,"kb_used_data":2605300,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":90463100,"statfs":{"total":95995035648,"available":92634214400,"internally_reserved":0,"allocated":2667827200,"data_stored":2667420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":692976074},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464367,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3982448,"kb_used_data":3563100,"kb_used_omap":4,"kb_used_meta":419323,"kb_avail":89762704,"statfs":{"total":95995035648,"available":91917008896,"internally_reserved":0,"allocated":3648614400,"data_stored":3648224275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":429387428},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619069,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2016956,"kb_used_data":1706796,"kb_used_omap":4,"kb_used_meta":310139,"kb_avail":91728196,"statfs":{"total":95995035648,"available":93929672704,"internally_reserved":0,"allocated":1747759104,"data_stored":1747360281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":317583011},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019123,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2335440,"kb_used_data":1740608,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91409712,"statfs":{"total":95995035648,"available":93603545088,"internally_reserved":0,"allocated":1782382592,"data_stored":1781963788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182648,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3781788,"kb_used_data":2966156,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":89963364,"statfs":{"total":95995035648,"available":92122484736,"internally_reserved":0,"allocated":3037343744,"data_stored":3036913471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411582,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3207988,"kb_used_data":2572900,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":90537164,"statfs":{"total":95995035648,"available":92710055936,"internally_reserved":0,"allocated":2634649600,"data_stored":2634258998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4253800,"kb_used_data":3450712,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":89491352,"statfs":{"total":95995035648,"available":91639144448,"internally_reserved":0,"allocated":3533529088,"data_stored":3533086087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3016264,"kb_used_data":2140984,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":90728888,"statfs":{"total":95995035648,"available":92906381312,"internally_reserved":0,"allocated":2192367616,"data_stored":2191957169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2183331840,"data_stored":2183331840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3504799744,"data_stored":3504799744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2614104064,"data_stored":2614099992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3024289792,"data_stored":3024289792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1773998080,"data_stored":1773994008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1733558272,"data_stored":1733558272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3619622912,"data_stored":3619618840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2637955072,"data_stored":2637955072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:11:56.110 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T19:11:56.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-21T19:11:56.122 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:56.126 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:56.199 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:11:56.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:56.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:56.414 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:56.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:56.561 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:56.581 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:56 smithi081 ceph-mon[39192]: pgmap v933: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 2.4 KiB/s rd, 11 KiB/s wr, 1.05k op/s; 55 KiB/s, 0 objects/s recovering 2024-09-21T19:11:56.582 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:56 smithi081 ceph-mon[29384]: pgmap v933: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 2.4 KiB/s rd, 11 KiB/s wr, 1.05k op/s; 55 KiB/s, 0 objects/s recovering 2024-09-21T19:11:56.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:56.703 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:56.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:56 smithi104 ceph-mon[32438]: pgmap v933: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 2.4 KiB/s rd, 11 KiB/s wr, 1.05k op/s; 55 KiB/s, 0 objects/s recovering 2024-09-21T19:11:56.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:56.835 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:56.906 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:11:56.906 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:11:56.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:56.943 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:11:56.943 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:11:56.977 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:56.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:56.981 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:11:56.981 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:11:57.021 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:11:57.021 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:11:57.056 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:11:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:11:57.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:57.088 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:11:57.088 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:11:57.117 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:11:57.120 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:11:57.120 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:11:57.152 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:11:57.153 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:11:57.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:57.293 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:11:57.293 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:11:57.293 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-09-21T19:11:57.293 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:11:57.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:57.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:11:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.384 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:57.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:57.457 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:57 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3911943900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:57.457 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:57 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3911943900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:57.457 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:11:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:11:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:11:57.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:57.513 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:57.615 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:57.640 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:57.742 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:57.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:57 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3911943900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:11:57.771 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:57.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:57.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:58.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:58.046 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:58.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:11:58.180 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:11:58.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:11:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.315 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:58.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:11:58.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T19:11:58.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.470 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:58.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:58.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.628 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:58.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:58.725 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:58 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:11:58.726 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:58 smithi104 ceph-mon[32438]: pgmap v934: 168 pgs: 168 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 2.6 KiB/s rd, 9.2 KiB/s wr, 971 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-21T19:11:58.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:58.757 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:58.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:58 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:11:58.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:58 smithi081 ceph-mon[29384]: pgmap v934: 168 pgs: 168 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 2.6 KiB/s rd, 9.2 KiB/s wr, 971 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-21T19:11:58.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:58 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:11:58.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:58 smithi081 ceph-mon[39192]: pgmap v934: 168 pgs: 168 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 2.6 KiB/s rd, 9.2 KiB/s wr, 971 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-21T19:11:58.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:58.895 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:58.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:58.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:59.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:11:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:11:59.038 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:59.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:11:59.173 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:11:59.275 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:11:59.299 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:11:59.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:11:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.430 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:11:59.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:11:59.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:59.568 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:59.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:11:59.706 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:11:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:59 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:11:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:59 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:11:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:59 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:59.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:11:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:59.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:11:59.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:11:59.815 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:11:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:59.816 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:11:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:11:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:11:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:11:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:11:59.849 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:11:59.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:11:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:11:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:11:59.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:00.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:00.121 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:00.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:00.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:00.253 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:00.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:00.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:00.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:00.515 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:00.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:00.655 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:00.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.792 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:00.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:00.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:00.931 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:00.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:01.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.066 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:01.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:01.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:01.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:00 smithi104 ceph-mon[32438]: pgmap v935: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 40 KiB/s, 0 objects/s recovering 2024-09-21T19:12:01.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:01.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:00 smithi081 ceph-mon[29384]: pgmap v935: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 40 KiB/s, 0 objects/s recovering 2024-09-21T19:12:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:00 smithi081 ceph-mon[39192]: pgmap v935: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 40 KiB/s, 0 objects/s recovering 2024-09-21T19:12:01.326 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:01.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:01.457 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:01.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:01.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:01.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.728 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:01.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:01.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:01.862 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:01.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:01.991 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:01.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:02.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.128 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:02.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:02.229 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:02.256 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:02.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:02.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:02.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:02.515 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:02.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:02.652 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:02.754 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:02.783 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:02.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:02.925 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:02.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:03.027 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:03.102 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:03.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:03.180 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:03.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:02 smithi081 ceph-mon[39192]: pgmap v936: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 604 B/s rd, 0 B/s wr, 837 op/s 2024-09-21T19:12:03.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:02 smithi081 ceph-mon[29384]: pgmap v936: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 604 B/s rd, 0 B/s wr, 837 op/s 2024-09-21T19:12:03.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:02 smithi104 ceph-mon[32438]: pgmap v936: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 604 B/s rd, 0 B/s wr, 837 op/s 2024-09-21T19:12:03.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:03.320 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:03.334 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:03.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:03.461 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:03.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:03.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:03.724 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:03.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:03.851 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:03.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:03.886 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:03.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:04.002 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:04.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:04.134 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:04.235 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:04.272 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:04.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:04.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:04.416 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:04.517 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:04.544 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:04.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:04.667 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:04.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:04.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:04.915 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 11 pg_num to 26 2024-09-21T19:12:04.928 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:04.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:05.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:05.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:12:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.071 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:05.161 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:04 smithi081 ceph-mon[39192]: pgmap v937: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-21T19:12:05.162 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/846987449' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T19:12:05.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:04 smithi081 ceph-mon[29384]: pgmap v937: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-21T19:12:05.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/846987449' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T19:12:05.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:05.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:05.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:04 smithi104 ceph-mon[32438]: pgmap v937: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-21T19:12:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/846987449' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T19:12:05.314 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:05.339 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:05.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:05.468 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:05.570 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:05.600 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:05.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:05.733 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:05.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:05.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:05.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:06.033 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:06.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:06.155 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:06.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:05 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/846987449' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T19:12:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:05 smithi104 ceph-mon[32438]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T19:12:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:05 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:05 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:06.257 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/846987449' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[29384]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/846987449' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[39192]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:06.265 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:05 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:06.278 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:06.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:06.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:06.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:06.652 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:12:06.652 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:12:06.652 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:12:06.652 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:12:06.652 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:12:06.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:06.702 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:06.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.826 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:06.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:06.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:06.938 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:12:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:06.979 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:06.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:07.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[39192]: pgmap v939: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[39192]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1596056570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[39192]: from='client.36671 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[29384]: pgmap v939: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T19:12:07.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[29384]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T19:12:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1596056570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:12:07.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:06 smithi081 ceph-mon[29384]: from='client.36671 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:12:07.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.245 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:07.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:06 smithi104 ceph-mon[32438]: pgmap v939: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-21T19:12:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:06 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T19:12:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:06 smithi104 ceph-mon[32438]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T19:12:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1596056570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:12:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:06 smithi104 ceph-mon[32438]: from='client.36671 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:12:07.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.384 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:07.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:07.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:07.498 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:12:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:12:07.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T19:12:07.523 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:07.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:07.729 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:07.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.811 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:07.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:07.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:08.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:08.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: 11.0 scrub starts 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: 11.2 scrub starts 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:08.174 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: 11.0 scrub starts 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: 11.2 scrub starts 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:08.175 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:07 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:08.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: 11.0 scrub starts 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: 11.2 scrub starts 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:07 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T19:12:08.331 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:08.358 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:08.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:08.487 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:08.589 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:08.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:08.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:08.733 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:08.736 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:12:08.736 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:12:08.752 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:12:08.753 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:12:08.770 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:12:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:12:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:08.787 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:12:08.787 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:12:08.804 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:12:08.804 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:12:08.821 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:12:08.821 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:12:08.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:08.837 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:12:08.837 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:12:08.853 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:12:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:12:08.856 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:08.958 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:08.980 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:08.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:09.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: pgmap v942: 178 pgs: 1 active, 8 unknown, 1 active+recovering, 168 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: 11.0 scrub ok 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: 11.8 scrub starts 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: 11.2 scrub ok 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: 11.8 scrub ok 2024-09-21T19:12:09.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3824014650' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:09.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: pgmap v942: 178 pgs: 1 active, 8 unknown, 1 active+recovering, 168 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: 11.0 scrub ok 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: 11.8 scrub starts 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: 11.2 scrub ok 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: 11.8 scrub ok 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3824014650' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: pgmap v942: 178 pgs: 1 active, 8 unknown, 1 active+recovering, 168 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-21T19:12:09.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: 11.0 scrub ok 2024-09-21T19:12:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: 11.8 scrub starts 2024-09-21T19:12:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T19:12:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T19:12:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: 11.2 scrub ok 2024-09-21T19:12:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: 11.8 scrub ok 2024-09-21T19:12:09.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3824014650' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:09.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:09.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:09.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:09.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.477 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:09.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.599 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:09.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.723 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:09.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:09.881 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:09.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:09.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:10.019 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:10.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:10.155 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:10.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:12:10.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:10.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:09 smithi104 ceph-mon[32438]: 11.7 scrub starts 2024-09-21T19:12:10.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:09 smithi104 ceph-mon[32438]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T19:12:10.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:09 smithi104 ceph-mon[32438]: 11.7 scrub ok 2024-09-21T19:12:10.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:09 smithi104 ceph-mon[32438]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T19:12:10.261 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:10.261 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[39192]: 11.7 scrub starts 2024-09-21T19:12:10.261 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[39192]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T19:12:10.262 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[39192]: 11.7 scrub ok 2024-09-21T19:12:10.262 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[39192]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T19:12:10.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:10.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[29384]: 11.7 scrub starts 2024-09-21T19:12:10.263 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[29384]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T19:12:10.263 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[29384]: 11.7 scrub ok 2024-09-21T19:12:10.263 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:09 smithi081 ceph-mon[29384]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T19:12:10.312 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:10.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:10.456 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:10.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:10.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:10.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:10.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:10.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:10.895 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:12:10.895 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:12:10.895 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T19:12:10.896 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T19:12:10.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:11.108 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:12:11.108 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:12:11.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:11.186 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.6 scrub starts 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 1.f scrub starts 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 1.f scrub ok 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9700 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 456'9768 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9892 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 456'9971 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9668 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9700 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9768 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9892 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9971 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: pgmap v946: 178 pgs: 3 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 1 active+recovering, 171 active+clean; 5.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9931 2024-09-21T19:12:11.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:10 smithi104 ceph-mon[32438]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9931 2024-09-21T19:12:11.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.6 scrub starts 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 1.f scrub starts 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 1.f scrub ok 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9700 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 456'9768 2024-09-21T19:12:11.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9892 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 456'9971 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9668 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9700 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9768 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9892 2024-09-21T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9971 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: pgmap v946: 178 pgs: 3 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 1 active+recovering, 171 active+clean; 5.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9931 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[39192]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9931 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.6 scrub starts 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 1.f scrub starts 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 1.f scrub ok 2024-09-21T19:12:11.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9700 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 456'9768 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9892 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 456'9971 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9668 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9700 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9768 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 456'9785 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.13 starting backfill to osd.5 from (0'0,0'0] MAX to 456'9779 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.10 starting backfill to osd.6 from (0'0,0'0] MAX to 456'9773 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9892 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9971 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: pgmap v946: 178 pgs: 3 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 1 active+recovering, 171 active+clean; 5.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9936 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 456'9931 2024-09-21T19:12:11.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:10 smithi081 ceph-mon[29384]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 456'9931 2024-09-21T19:12:11.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:11.448 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:11.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:11.588 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:11.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:12:11.690 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:11.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:11.859 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:11.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:11.961 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:12.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:12.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:11 smithi081 ceph-mon[29384]: 11.6 scrub ok 2024-09-21T19:12:12.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:11 smithi081 ceph-mon[39192]: 11.6 scrub ok 2024-09-21T19:12:12.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:12.138 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:12.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:12.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:11 smithi104 ceph-mon[32438]: 11.6 scrub ok 2024-09-21T19:12:12.289 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:12.297 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T19:12:12.351 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:12.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:12.418 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:12.453 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:12.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:12.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:12.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:12.711 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:12.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:12.859 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:12.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:12.961 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:13.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:13.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:12 smithi081 ceph-mon[39192]: pgmap v947: 178 pgs: 3 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 1 active+recovering, 171 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:12:13.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:12 smithi081 ceph-mon[29384]: pgmap v947: 178 pgs: 3 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 1 active+recovering, 171 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:12:13.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:13.165 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:13.172 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:12:13.173 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:12:13.223 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:12:13.223 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:12:13.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:12 smithi104 ceph-mon[32438]: pgmap v947: 178 pgs: 3 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 1 active+recovering, 171 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T19:12:13.261 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:12:13.261 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:12:13.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:13.296 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:12:13.296 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:12:13.323 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:13.334 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:12:13.334 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:12:13.365 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:12:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:12:13.401 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:12:13.401 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:12:13.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:13.426 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:12:13.426 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:12:13.446 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:13.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:13.579 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:13.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:13.719 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:13.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:13.862 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:13.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:13.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:13.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:14.004 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:14.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:14.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:14.145 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:14.232 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:13 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/625119714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:14.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:13 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/625119714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:14.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:14.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:13 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/625119714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:14.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:14.395 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:14.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:14.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:14.555 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:14.560 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "epoch": 459, 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "d839dc86-7849-11ef-baf6-efdc52797490", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "created": "2024-09-21T18:47:59.361278+0000", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "modified": "2024-09-21T19:12:09.803397+0000", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "last_up_change": "2024-09-21T19:06:54.357382+0000", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "last_in_change": "2024-09-21T19:05:41.856710+0000", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_num": 5804032, 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_set": [ 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "nodeep-scrub", 2024-09-21T19:12:14.566 INFO:teuthology.orchestra.run.smithi081.stdout: "pglog_hardlimit", 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "purged_snapdirs", 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_deletes", 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "sortbitwise" 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_version": 20, 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_snapshot": "", 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_max": 11, 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "max_osd": 8, 2024-09-21T19:12:14.567 INFO:teuthology.orchestra.run.smithi081.stdout: "require_min_compat_client": "luminous", 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "min_compat_client": "luminous", 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "require_osd_release": "squid", 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "allow_crimson": false, 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "pools": [ 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 1, 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".mgr", 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:50:55.921148+0000", 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.568 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 21, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 21, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 21, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 21, 2024-09-21T19:12:14.569 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 21, 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "450", 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.570 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "448", 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.571 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.572 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.573 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_max": 32, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_min": 1 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": {} 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.8999999761581421, 2024-09-21T19:12:14.574 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.8999999761581421, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.8999999761581421, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.8999999761581421, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 2, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "datapool", 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:04.563630+0000", 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 8193, 2024-09-21T19:12:14.575 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 3, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 3, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 3, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 3, 2024-09-21T19:12:14.576 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 3, 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "60", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.577 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 2, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 60, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.578 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.579 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd": {} 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.6500000953674316, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.6500000953674316, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T19:12:14.580 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 3, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": ".rgw.root", 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:05.494307+0000", 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.581 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.582 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "59", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:12:14.583 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.584 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.585 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:12:14.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 4, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.log", 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:07.470408+0000", 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.588 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.589 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "61", 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.590 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.591 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.592 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2.25, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2.25, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:12:14.593 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2.25, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2.25, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 5, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.control", 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:09.418988+0000", 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.594 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:12:14.595 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "63", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.596 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.598 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.5, 2024-09-21T19:12:14.599 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:12:14.600 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5, 2024-09-21T19:12:14.600 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.5, 2024-09-21T19:12:14.600 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.600 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.600 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.600 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 6, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "default.rgw.meta", 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T18:53:11.462116+0000", 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.601 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 32, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 32, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 32, 2024-09-21T19:12:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 32, 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 32, 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "65", 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T19:12:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.605 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "options": { 2024-09-21T19:12:14.606 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_bias": 4 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw": {} 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 2, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 2, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 2, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 2, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "pool": 11, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_name": "unique_pool_4", 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "create_time": "2024-09-21T19:09:59.705202+0000", 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "flags": 1, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "flags_names": "hashpspool", 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "type": 1, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 3, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "min_size": 2, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_rule": 0, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_count": 0, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_target": 0, 2024-09-21T19:12:14.608 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "is_stretch_pool": false, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "object_hash": 2, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_autoscale_mode": "off", 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num": 26, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num": 26, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_placement_num_target": 26, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_target": 26, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_num_pending": 26, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "last_pg_merge_meta": { 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "source_pgid": "0.0", 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "ready_epoch": 0, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_started": 0, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "last_epoch_clean": 0, 2024-09-21T19:12:14.609 INFO:teuthology.orchestra.run.smithi081.stdout: "source_version": "0'0", 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "target_version": "0'0" 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_change": "457", 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend": "0", 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "last_force_op_resend_preluminous": "455", 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "auid": 0, 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_mode": "selfmanaged", 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_seq": 0, 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_epoch": 0, 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_snaps": [], 2024-09-21T19:12:14.610 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps": "[]", 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_bytes": 0, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "quota_max_objects": 0, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "tiers": [], 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "tier_of": -1, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "read_tier": -1, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "write_tier": -1, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_mode": "none", 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_bytes": 0, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_objects": 0, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_flush_age": 0, 2024-09-21T19:12:14.611 INFO:teuthology.orchestra.run.smithi081.stdout: "cache_min_evict_age": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profile": "", 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_params": { 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "none" 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_period": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_count": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "use_gmt_hitset": true, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "min_read_recency_for_promote": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "min_write_recency_for_promote": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "hit_set_search_last_n": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "grade_table": [], 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "stripe_width": 0, 2024-09-21T19:12:14.612 INFO:teuthology.orchestra.run.smithi081.stdout: "expected_num_objects": 0, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "fast_read": false, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "options": {}, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "application_metadata": { 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "rados": {} 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "read_balance": { 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "score_type": "Fair distribution", 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "score_acting": 1.5399999618530273, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "score_stable": 1.8500000238418579, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "optimal_score": 1, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_acting": 1.5399999618530273, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "raw_score_stable": 1.8500000238418579, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity_weighted": 0.89614105224609375, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-21T19:12:14.613 INFO:teuthology.orchestra.run.smithi081.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "bd6626b0-dc43-42d0-a459-9cf3911457ea", 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:12:14.614 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 8, 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 458, 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6801", 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6802", 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6804", 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6803", 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 390647394 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6801/390647394", 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6802/390647394", 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6804/390647394", 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6803/390647394", 2024-09-21T19:12:14.617 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "2f541064-dc48-4f7f-b29b-065491e148b5", 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:12:14.618 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 13, 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 458, 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6805", 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6806", 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6808", 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.620 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6807", 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3705537065 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6805/3705537065", 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6806/3705537065", 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6808/3705537065", 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6807/3705537065", 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.621 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e7050b14-f463-4de3-8a8d-e05e78507938", 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:12:14.622 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 18, 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 457, 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6809", 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6810", 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6812", 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6811", 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 486671153 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6809/486671153", 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6810/486671153", 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6812/486671153", 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6811/486671153", 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "75b4cb80-138b-445b-a93a-3c4c598b5e0e", 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:12:14.626 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 25, 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 458, 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6813", 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6814", 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6816", 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:6815", 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1516859335 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.81:6813/1516859335", 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.81:6814/1516859335", 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.81:6816/1516859335", 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.81:6815/1516859335", 2024-09-21T19:12:14.629 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "85220357-38ea-4c90-b422-5d703329f106", 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.630 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 30, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 457, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6800", 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:12:14.631 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6801", 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6803", 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6802", 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 1904940409 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.633 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6800/1904940409", 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6801/1904940409", 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6803/1904940409", 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6802/1904940409", 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.634 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c", 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 0.16912841796875, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 37, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 326, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 348, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 458, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 327, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.635 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6804", 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.636 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6805", 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6807", 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.637 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6806", 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3649492645 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.638 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6804/3649492645", 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6805/3649492645", 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6807/3649492645", 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6806/3649492645", 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:12:14.639 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd", 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 45, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 238, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 261, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 458, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 239, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.640 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6808", 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6809", 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.641 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6811", 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6810", 2024-09-21T19:12:14.642 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2795996229 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6808/2795996229", 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6809/2795996229", 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6811/2795996229", 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6810/2795996229", 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:12:14.643 INFO:teuthology.orchestra.run.smithi081.stdout: "uuid": "f15ee432-aa23-4e5a-9df4-478538af2045", 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "up": 1, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "in": 1, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "weight": 1, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_affinity": 1, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_begin": 0, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "last_clean_end": 0, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "up_from": 53, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "up_thru": 457, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "down_at": 0, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "lost_at": 0, 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrs": { 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.644 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6812", 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addrs": { 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6813", 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.645 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addrs": { 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6815", 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addrs": { 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addrvec": [ 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-09-21T19:12:14.646 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.104:6814", 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 385235741 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "172.21.15.104:6812/385235741", 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "172.21.15.104:6813/385235741", 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_back_addr": "172.21.15.104:6815/385235741", 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_front_addr": "172.21.15.104:6814/385235741", 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "state": [ 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "exists", 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: "up" 2024-09-21T19:12:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_xinfo": [ 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:52.444817+0000", 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.648 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:27.545832+0000", 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.649 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:26.401113+0000", 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.650 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:20.750999+0000", 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 4, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:30.618811+0000", 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 5, 2024-09-21T19:12:14.651 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:45.156673+0000", 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 326 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 6, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.652 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:06.571191+0000", 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 238 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 7, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "down_stamp": "0.000000", 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_probability": 0, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "laggy_interval": 0, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "features": 4540719139924082687, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "old_weight": 0, 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "last_purged_snaps_scrub": "2024-09-21T19:11:43.871840+0000", 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: "dead_epoch": 0 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap": [], 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_items": [ 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.6", 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.9", 2024-09-21T19:12:14.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 5 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "4.14", 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 2 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.655 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.9", 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 7, 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 3 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.c", 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.656 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 1, 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 6 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "5.1f", 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 4, 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 3 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "6.2", 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "mappings": [ 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 3, 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 0 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_upmap_primaries": [], 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "pg_temp": [ 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.10", 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.658 INFO:teuthology.orchestra.run.smithi081.stdout: 1, 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: 3, 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: 7 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.11", 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: 0, 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: 3, 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: 6 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.12", 2024-09-21T19:12:14.659 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: 0, 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: 2, 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: 3 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.13", 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: 6, 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: 1, 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: 7 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.660 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.14", 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: 6, 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: 2, 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: 7 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.15", 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: 5, 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: 6, 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: 7 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.16", 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: 0, 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: 1, 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: 2 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.17", 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: 1, 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: 5, 2024-09-21T19:12:14.662 INFO:teuthology.orchestra.run.smithi081.stdout: 4 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.18", 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: 3, 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: 6, 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: 5 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "11.19", 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osds": [ 2024-09-21T19:12:14.663 INFO:teuthology.orchestra.run.smithi081.stdout: 0, 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: 3, 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: 6 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "primary_temp": [], 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklist": { 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/188521944": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/116812683": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/628265626": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1496889754": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/2203644131": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3906790599": "2024-09-22T18:48:21.208275+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3045112983": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:12:14.664 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/806683817": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/3701153923": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/1611463627": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/1508794297": "2024-09-22T18:48:45.625700+0000", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:6800/3426409743": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/198739293": "2024-09-22T18:53:35.352265+0000", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "172.21.15.81:0/4266768293": "2024-09-22T18:53:35.352265+0000" 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "range_blocklist": {}, 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_profiles": { 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "default": { 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "crush-failure-domain": "osd", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "k": "2", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "m": "1", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin": "jerasure", 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: "technique": "reed_sol_van" 2024-09-21T19:12:14.665 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "removed_snaps_queue": [], 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "new_removed_snaps": [], 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "new_purged_snaps": [], 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_node_flags": {}, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "device_class_flags": {}, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode": { 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_enabled": false, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_bucket_count": 0, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "degraded_stretch_mode": 0, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "recovering_stretch_mode": 0, 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: "stretch_mode_bucket": 0 2024-09-21T19:12:14.666 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:14.667 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:12:14.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.702 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:14.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:14.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:14.809 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:12:14.836 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:14.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:14.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:14.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:15.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:15.116 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:15.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: pgmap v948: 178 pgs: 3 active+remapped+backfilling, 2 remapped+peering, 5 active+remapped+backfill_toofull, 168 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 46392/258345 objects misplaced (17.957%); 11 MiB/s, 172 objects/s recovering 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1689645892' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2164858829' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2164858829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[29384]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: pgmap v948: 178 pgs: 3 active+remapped+backfilling, 2 remapped+peering, 5 active+remapped+backfill_toofull, 168 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 46392/258345 objects misplaced (17.957%); 11 MiB/s, 172 objects/s recovering 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1689645892' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2164858829' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:12:15.146 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:12:15.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2164858829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:12:15.147 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:14 smithi081 ceph-mon[39192]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T19:12:15.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:15.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:15.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: pgmap v948: 178 pgs: 3 active+remapped+backfilling, 2 remapped+peering, 5 active+remapped+backfill_toofull, 168 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 46392/258345 objects misplaced (17.957%); 11 MiB/s, 172 objects/s recovering 2024-09-21T19:12:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:12:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1689645892' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T19:12:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2164858829' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:12:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:12:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2164858829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:12:15.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:14 smithi104 ceph-mon[32438]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T19:12:15.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:15.393 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:15.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:15.531 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:15.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:15.675 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:15.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:15.819 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:15.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:15.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:16.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:16.084 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:16.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:16.220 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:16.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:16.360 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:16.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.499 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:16.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:16.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.645 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:16.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:16.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:16.776 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:16.880 INFO:tasks.thrashosds.thrasher:j is {'epoch': 459, 'fsid': 'd839dc86-7849-11ef-baf6-efdc52797490', 'created': '2024-09-21T18:47:59.361278+0000', 'modified': '2024-09-21T19:12:09.803397+0000', 'last_up_change': '2024-09-21T19:06:54.357382+0000', 'last_in_change': '2024-09-21T19:05:41.856710+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T18:50:55.921148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '450', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '448', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T18:53:04.563630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 60, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T18:53:05.494307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T18:53:07.470408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T18:53:09.418988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T18:53:11.462116+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T19:09:59.705202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '457', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '455', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.850000023841858, 'optimal_score': 1, 'raw_score_acting': 1.5399999618530273, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 0.8961410522460938, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': 'bd6626b0-dc43-42d0-a459-9cf3911457ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6801', 'nonce': 390647394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6802', 'nonce': 390647394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6804', 'nonce': 390647394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6803', 'nonce': 390647394}]}, 'public_addr': '172.21.15.81:6801/390647394', 'cluster_addr': '172.21.15.81:6802/390647394', 'heartbeat_back_addr': '172.21.15.81:6804/390647394', 'heartbeat_front_addr': '172.21.15.81:6803/390647394', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2f541064-dc48-4f7f-b29b-065491e148b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6805', 'nonce': 3705537065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6806', 'nonce': 3705537065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6808', 'nonce': 3705537065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6807', 'nonce': 3705537065}]}, 'public_addr': '172.21.15.81:6805/3705537065', 'cluster_addr': '172.21.15.81:6806/3705537065', 'heartbeat_back_addr': '172.21.15.81:6808/3705537065', 'heartbeat_front_addr': '172.21.15.81:6807/3705537065', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e7050b14-f463-4de3-8a8d-e05e78507938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 457, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6809', 'nonce': 486671153}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6810', 'nonce': 486671153}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6812', 'nonce': 486671153}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6811', 'nonce': 486671153}]}, 'public_addr': '172.21.15.81:6809/486671153', 'cluster_addr': '172.21.15.81:6810/486671153', 'heartbeat_back_addr': '172.21.15.81:6812/486671153', 'heartbeat_front_addr': '172.21.15.81:6811/486671153', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '75b4cb80-138b-445b-a93a-3c4c598b5e0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 458, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6813', 'nonce': 1516859335}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6814', 'nonce': 1516859335}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6816', 'nonce': 1516859335}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.81:6815', 'nonce': 1516859335}]}, 'public_addr': '172.21.15.81:6813/1516859335', 'cluster_addr': '172.21.15.81:6814/1516859335', 'heartbeat_back_addr': '172.21.15.81:6816/1516859335', 'heartbeat_front_addr': '172.21.15.81:6815/1516859335', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '85220357-38ea-4c90-b422-5d703329f106', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 457, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6800', 'nonce': 1904940409}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6801', 'nonce': 1904940409}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6803', 'nonce': 1904940409}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6802', 'nonce': 1904940409}]}, 'public_addr': '172.21.15.104:6800/1904940409', 'cluster_addr': '172.21.15.104:6801/1904940409', 'heartbeat_back_addr': '172.21.15.104:6803/1904940409', 'heartbeat_front_addr': '172.21.15.104:6802/1904940409', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.16912841796875, 'last_clean_begin': 37, 'last_clean_end': 326, 'up_from': 348, 'up_thru': 458, 'down_at': 327, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6804', 'nonce': 3649492645}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6805', 'nonce': 3649492645}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6807', 'nonce': 3649492645}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6806', 'nonce': 3649492645}]}, 'public_addr': '172.21.15.104:6804/3649492645', 'cluster_addr': '172.21.15.104:6805/3649492645', 'heartbeat_back_addr': '172.21.15.104:6807/3649492645', 'heartbeat_front_addr': '172.21.15.104:6806/3649492645', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 45, 'last_clean_end': 238, 'up_from': 261, 'up_thru': 458, 'down_at': 239, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6808', 'nonce': 2795996229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6809', 'nonce': 2795996229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6811', 'nonce': 2795996229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6810', 'nonce': 2795996229}]}, 'public_addr': '172.21.15.104:6808/2795996229', 'cluster_addr': '172.21.15.104:6809/2795996229', 'heartbeat_back_addr': '172.21.15.104:6811/2795996229', 'heartbeat_front_addr': '172.21.15.104:6810/2795996229', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f15ee432-aa23-4e5a-9df4-478538af2045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 457, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6812', 'nonce': 385235741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6813', 'nonce': 385235741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6815', 'nonce': 385235741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.104:6814', 'nonce': 385235741}]}, 'public_addr': '172.21.15.104:6812/385235741', 'cluster_addr': '172.21.15.104:6813/385235741', 'heartbeat_back_addr': '172.21.15.104:6815/385235741', 'heartbeat_front_addr': '172.21.15.104:6814/385235741', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:52.444817+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:27.545832+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:26.401113+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:20.750999+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:30.618811+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:45.156673+0000', 'dead_epoch': 326}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:06.571191+0000', 'dead_epoch': 238}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T19:11:43.871840+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.9', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.c', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1f', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.2', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.10', 'osds': [1, 3, 7]}, {'pgid': '11.11', 'osds': [0, 3, 6]}, {'pgid': '11.12', 'osds': [0, 2, 3]}, {'pgid': '11.13', 'osds': [6, 1, 7]}, {'pgid': '11.14', 'osds': [6, 2, 7]}, {'pgid': '11.15', 'osds': [5, 6, 7]}, {'pgid': '11.16', 'osds': [0, 1, 2]}, {'pgid': '11.17', 'osds': [1, 5, 4]}, {'pgid': '11.18', 'osds': [3, 6, 5]}, {'pgid': '11.19', 'osds': [0, 3, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.81:0/188521944': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/116812683': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/628265626': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/1496889754': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:6800/2203644131': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3906790599': '2024-09-22T18:48:21.208275+0000', '172.21.15.81:0/3045112983': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/806683817': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:0/3701153923': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/1611463627': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/1508794297': '2024-09-22T18:48:45.625700+0000', '172.21.15.81:6800/3426409743': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/198739293': '2024-09-22T18:53:35.352265+0000', '172.21.15.81:0/4266768293': '2024-09-22T18:53:35.352265+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T19:12:16.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:12:16.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:16.911 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:16.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:17.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.082 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:17.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:17.117 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:16 smithi081 ceph-mon[29384]: pgmap v950: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 57251/256452 objects misplaced (22.324%); 19 MiB/s, 310 objects/s recovering 2024-09-21T19:12:17.118 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:16 smithi081 ceph-mon[39192]: pgmap v950: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 57251/256452 objects misplaced (22.324%); 19 MiB/s, 310 objects/s recovering 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:166686 11.10 11:0df5290d:::benchmark_data_smithi081_181703_object43065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:17.108811+0000", 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0054236889999999998, 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054567900000000004, 2024-09-21T19:12:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "waiting for sub ops", 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 166686 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108811+0000", 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108811+0000", 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108813+0000", 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1730000000000002e-06 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108819+0000", 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2909999999999998e-06 2024-09-21T19:12:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108833+0000", 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4545999999999999e-05 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108841+0000", 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9109999999999993e-06 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.108903+0000", 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2026000000000006e-05 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.109036+0000", 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000133242 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,2,3,6,7", 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.109326+0000", 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028943799999999999 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.111379+0000", 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0020529799999999998 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:17.111583+0000", 2024-09-21T19:12:17.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020389799999999999 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 1 2024-09-21T19:12:17.124 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:12:17.182 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:17.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:12:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.232 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:17.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:16 smithi104 ceph-mon[32438]: pgmap v950: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 57251/256452 objects misplaced (22.324%); 19 MiB/s, 310 objects/s recovering 2024-09-21T19:12:17.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.364 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:17.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:17.442 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:12:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:12:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:12:17.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:17.504 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:17.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:17.652 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:17.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:17.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.806 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:17.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:17.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:17.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:18.038 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:18.072 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:18.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:18.214 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:18.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:18.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T19:12:18.356 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:18.379 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:18.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:18.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:18.636 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:18.662 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:18.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:18.790 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:18.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:18.926 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:18.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:18.986 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:19.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:18 smithi104 ceph-mon[32438]: pgmap v951: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57251/253530 objects misplaced (22.582%); 16 MiB/s, 248 objects/s recovering 2024-09-21T19:12:19.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:12:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:12:19.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:19.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:18 smithi081 ceph-mon[39192]: pgmap v951: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57251/253530 objects misplaced (22.582%); 16 MiB/s, 248 objects/s recovering 2024-09-21T19:12:19.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:18 smithi081 ceph-mon[29384]: pgmap v951: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57251/253530 objects misplaced (22.582%); 16 MiB/s, 248 objects/s recovering 2024-09-21T19:12:19.107 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:19.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:19.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:19.381 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:19.390 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:12:19.422 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:12:19.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:19.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:12:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:19.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:19.673 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:19.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.817 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:19.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:12:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:12:19.859 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:19.859 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:12:19.859 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:12:19.890 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:12:19.891 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:12:19.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:19.932 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:12:19.932 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:12:19.971 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:12:19.971 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:12:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:19.978 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:19.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:20.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:20.129 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:20.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1579240939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:19 smithi104 ceph-mon[32438]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T19:12:20.269 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:20.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:20.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1579240939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:20.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:19 smithi081 ceph-mon[29384]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T19:12:20.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:20.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1579240939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:20.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:19 smithi081 ceph-mon[39192]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T19:12:20.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:20.403 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:20.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.543 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:20.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:20.644 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:20.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:20.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:20.816 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:20.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:20.954 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:20.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:21.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:21.087 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:21.139 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":951,"stamp":"2024-09-21T19:12:17.848116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5525782831,"num_objects":84510,"num_object_clones":0,"num_object_copies":253530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57251,"num_objects_unfound":0,"num_objects_dirty":84510,"num_whiteouts":0,"num_read":2289,"num_read_kb":2051,"num_write":139111,"num_write_kb":6679362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":131583264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127550,"ondisk_log_size":127550,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":554,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21912356,"kb_used_data":16525088,"kb_used_omap":18,"kb_used_meta":5386797,"kb_avail":728048860,"statfs":{"total":767960285184,"available":745522032640,"internally_reserved":0,"allocated":16921690112,"data_stored":16918063117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":5516081136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-150994944,"num_objects":-2303,"num_object_clones":0,"num_object_copies":-6909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57251,"num_objects_unfound":0,"num_objects_dirty":-2303,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":-14461,"num_write_kb":-983264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":130875392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.042602"},"pg_stats":[{"pgid":"11.17","version":"459'10084","reported_seq":10268,"reported_epoch":459,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T19:12:12.829780+0000","last_change":"2024-09-21T19:12:09.938375+0000","last_active":"2024-09-21T19:12:12.829780+0000","last_peered":"2024-09-21T19:12:12.829780+0000","last_clean":"2024-09-21T19:12:08.364355+0000","last_became_active":"2024-09-21T19:12:09.909495+0000","last_became_peered":"2024-09-21T19:12:09.909495+0000","last_unstale":"2024-09-21T19:12:12.829780+0000","last_undegraded":"2024-09-21T19:12:12.829780+0000","last_fullsized":"2024-09-21T19:12:12.829780+0000","mapping_epoch":458,"log_start":"433'5289","ondisk_log_start":"433'5289","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"436'5961","last_scrub_stamp":"2024-09-21T19:11:09.207825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.207825+0000","objects_scrubbed":5889,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5460,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":251360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":2730}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":765,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812635+0000","last_change":"2024-09-21T19:09:50.182643+0000","last_active":"2024-09-21T19:12:09.812635+0000","last_peered":"2024-09-21T19:12:09.812635+0000","last_clean":"2024-09-21T19:12:09.812635+0000","last_became_active":"2024-09-21T19:06:54.704980+0000","last_became_peered":"2024-09-21T19:06:54.704980+0000","last_unstale":"2024-09-21T19:12:09.812635+0000","last_undegraded":"2024-09-21T19:12:09.812635+0000","last_fullsized":"2024-09-21T19:12:09.812635+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.182598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.182598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.472742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":936,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814242+0000","last_change":"2024-09-21T19:10:17.698193+0000","last_active":"2024-09-21T19:12:14.814242+0000","last_peered":"2024-09-21T19:12:14.814242+0000","last_clean":"2024-09-21T19:12:14.814242+0000","last_became_active":"2024-09-21T19:05:00.734324+0000","last_became_peered":"2024-09-21T19:05:00.734324+0000","last_unstale":"2024-09-21T19:12:14.814242+0000","last_undegraded":"2024-09-21T19:12:14.814242+0000","last_fullsized":"2024-09-21T19:12:14.814242+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:10:17.698093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.698093+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":871,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810653+0000","last_change":"2024-09-21T19:11:47.058089+0000","last_active":"2024-09-21T19:12:09.810653+0000","last_peered":"2024-09-21T19:12:09.810653+0000","last_clean":"2024-09-21T19:12:09.810653+0000","last_became_active":"2024-09-21T19:06:54.693621+0000","last_became_peered":"2024-09-21T19:06:54.693621+0000","last_unstale":"2024-09-21T19:12:09.810653+0000","last_undegraded":"2024-09-21T19:12:09.810653+0000","last_fullsized":"2024-09-21T19:12:09.810653+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:47.058027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.058027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:50.497460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":853,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814060+0000","last_change":"2024-09-21T19:11:33.079408+0000","last_active":"2024-09-21T19:12:14.814060+0000","last_peered":"2024-09-21T19:12:14.814060+0000","last_clean":"2024-09-21T19:12:14.814060+0000","last_became_active":"2024-09-21T19:06:54.703449+0000","last_became_peered":"2024-09-21T19:06:54.703449+0000","last_unstale":"2024-09-21T19:12:14.814060+0000","last_undegraded":"2024-09-21T19:12:14.814060+0000","last_fullsized":"2024-09-21T19:12:14.814060+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:33.079348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:33.079348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:00.185027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"459'10034","reported_seq":10157,"reported_epoch":459,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T19:12:12.874377+0000","last_change":"2024-09-21T19:12:10.023181+0000","last_active":"2024-09-21T19:12:12.874377+0000","last_peered":"2024-09-21T19:12:12.874377+0000","last_clean":"2024-09-21T19:12:08.364766+0000","last_became_active":"2024-09-21T19:12:09.938167+0000","last_became_peered":"2024-09-21T19:12:09.938167+0000","last_unstale":"2024-09-21T19:12:12.874377+0000","last_undegraded":"2024-09-21T19:12:12.874377+0000","last_fullsized":"2024-09-21T19:12:12.874377+0000","mapping_epoch":458,"log_start":"433'4686","ondisk_log_start":"433'4686","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177209344,"num_objects":2704,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8112,"num_objects_unfound":0,"num_objects_dirty":2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5095,"num_write_kb":249568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":2704}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":860,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.806697+0000","last_change":"2024-09-21T19:10:25.846651+0000","last_active":"2024-09-21T19:12:09.806697+0000","last_peered":"2024-09-21T19:12:09.806697+0000","last_clean":"2024-09-21T19:12:09.806697+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:12:09.806697+0000","last_undegraded":"2024-09-21T19:12:09.806697+0000","last_fullsized":"2024-09-21T19:12:09.806697+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:25.846276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.846276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":888,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809242+0000","last_change":"2024-09-21T19:10:54.811144+0000","last_active":"2024-09-21T19:12:09.809242+0000","last_peered":"2024-09-21T19:12:09.809242+0000","last_clean":"2024-09-21T19:12:09.809242+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:12:09.809242+0000","last_undegraded":"2024-09-21T19:12:09.809242+0000","last_fullsized":"2024-09-21T19:12:09.809242+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:54.811103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:54.811103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.471846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":976,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.806639+0000","last_change":"2024-09-21T19:06:53.950730+0000","last_active":"2024-09-21T19:12:09.806639+0000","last_peered":"2024-09-21T19:12:09.806639+0000","last_clean":"2024-09-21T19:12:09.806639+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:12:09.806639+0000","last_undegraded":"2024-09-21T19:12:09.806639+0000","last_fullsized":"2024-09-21T19:12:09.806639+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:06:53.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:53.950646+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:50.846842+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":881,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814154+0000","last_change":"2024-09-21T19:12:07.256589+0000","last_active":"2024-09-21T19:12:14.814154+0000","last_peered":"2024-09-21T19:12:14.814154+0000","last_clean":"2024-09-21T19:12:14.814154+0000","last_became_active":"2024-09-21T19:02:40.880096+0000","last_became_peered":"2024-09-21T19:02:40.880096+0000","last_unstale":"2024-09-21T19:12:14.814154+0000","last_undegraded":"2024-09-21T19:12:14.814154+0000","last_fullsized":"2024-09-21T19:12:14.814154+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:12:07.256522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:12:07.256522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:13.704633+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"459'9909","reported_seq":10024,"reported_epoch":459,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T19:12:13.187237+0000","last_change":"2024-09-21T19:12:09.926961+0000","last_active":"2024-09-21T19:12:13.187237+0000","last_peered":"2024-09-21T19:12:13.187237+0000","last_clean":"2024-09-21T19:12:08.394688+0000","last_became_active":"2024-09-21T19:12:09.901880+0000","last_became_peered":"2024-09-21T19:12:09.901880+0000","last_unstale":"2024-09-21T19:12:13.187237+0000","last_undegraded":"2024-09-21T19:12:13.187237+0000","last_fullsized":"2024-09-21T19:12:13.187237+0000","mapping_epoch":458,"log_start":"433'5159","ondisk_log_start":"433'5159","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7926,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":246144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":2642}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":860,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815241+0000","last_change":"2024-09-21T19:10:13.614795+0000","last_active":"2024-09-21T19:12:14.815241+0000","last_peered":"2024-09-21T19:12:14.815241+0000","last_clean":"2024-09-21T19:12:14.815241+0000","last_became_active":"2024-09-21T19:06:54.700401+0000","last_became_peered":"2024-09-21T19:06:54.700401+0000","last_unstale":"2024-09-21T19:12:14.815241+0000","last_undegraded":"2024-09-21T19:12:14.815241+0000","last_fullsized":"2024-09-21T19:12:14.815241+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.614720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.614720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":909,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814374+0000","last_change":"2024-09-21T19:04:33.670071+0000","last_active":"2024-09-21T19:12:14.814374+0000","last_peered":"2024-09-21T19:12:14.814374+0000","last_clean":"2024-09-21T19:12:14.814374+0000","last_became_active":"2024-09-21T18:55:51.521916+0000","last_became_peered":"2024-09-21T18:55:51.521916+0000","last_unstale":"2024-09-21T19:12:14.814374+0000","last_undegraded":"2024-09-21T19:12:14.814374+0000","last_fullsized":"2024-09-21T19:12:14.814374+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:04:33.669988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:04:33.669988+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":820,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810203+0000","last_change":"2024-09-21T19:11:00.940146+0000","last_active":"2024-09-21T19:12:09.810203+0000","last_peered":"2024-09-21T19:12:09.810203+0000","last_clean":"2024-09-21T19:12:09.810203+0000","last_became_active":"2024-09-21T19:06:54.696261+0000","last_became_peered":"2024-09-21T19:06:54.696261+0000","last_unstale":"2024-09-21T19:12:09.810203+0000","last_undegraded":"2024-09-21T19:12:09.810203+0000","last_fullsized":"2024-09-21T19:12:09.810203+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.940070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.940070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.146532+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":731,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811699+0000","last_change":"2024-09-21T19:11:03.440073+0000","last_active":"2024-09-21T19:12:09.811699+0000","last_peered":"2024-09-21T19:12:09.811699+0000","last_clean":"2024-09-21T19:12:09.811699+0000","last_became_active":"2024-09-21T18:55:51.540043+0000","last_became_peered":"2024-09-21T18:55:51.540043+0000","last_unstale":"2024-09-21T19:12:09.811699+0000","last_undegraded":"2024-09-21T19:12:09.811699+0000","last_fullsized":"2024-09-21T19:12:09.811699+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.439988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.439988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"459'9945","reported_seq":11436,"reported_epoch":459,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:12:14.455847+0000","last_change":"2024-09-21T19:12:09.922071+0000","last_active":"2024-09-21T19:12:14.455847+0000","last_peered":"2024-09-21T19:12:14.455847+0000","last_clean":"2024-09-21T19:12:08.375244+0000","last_became_active":"2024-09-21T19:12:09.896664+0000","last_became_peered":"2024-09-21T19:12:09.896664+0000","last_unstale":"2024-09-21T19:12:14.455847+0000","last_undegraded":"2024-09-21T19:12:14.455847+0000","last_fullsized":"2024-09-21T19:12:14.455847+0000","mapping_epoch":458,"log_start":"433'5120","ondisk_log_start":"433'5120","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171311104,"num_objects":2614,"num_object_clones":0,"num_object_copies":7842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4334,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":246624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":459,"num_bytes_recovered":30081024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":2614}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":413,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815137+0000","last_change":"2024-09-21T19:11:08.134154+0000","last_active":"2024-09-21T19:12:14.815137+0000","last_peered":"2024-09-21T19:12:14.815137+0000","last_clean":"2024-09-21T19:12:14.815137+0000","last_became_active":"2024-09-21T19:06:54.700308+0000","last_became_peered":"2024-09-21T19:06:54.700308+0000","last_unstale":"2024-09-21T19:12:14.815137+0000","last_undegraded":"2024-09-21T19:12:14.815137+0000","last_fullsized":"2024-09-21T19:12:14.815137+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.134097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.134097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.482578+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":863,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814780+0000","last_change":"2024-09-21T19:11:39.159512+0000","last_active":"2024-09-21T19:12:14.814780+0000","last_peered":"2024-09-21T19:12:14.814780+0000","last_clean":"2024-09-21T19:12:14.814780+0000","last_became_active":"2024-09-21T18:53:09.406138+0000","last_became_peered":"2024-09-21T18:53:09.406138+0000","last_unstale":"2024-09-21T19:12:14.814780+0000","last_undegraded":"2024-09-21T19:12:14.814780+0000","last_fullsized":"2024-09-21T19:12:14.814780+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T19:11:39.159436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:39.159436+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:01.228931+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":853,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828381+0000","last_change":"2024-09-21T19:11:40.758093+0000","last_active":"2024-09-21T19:12:14.828381+0000","last_peered":"2024-09-21T19:12:14.828381+0000","last_clean":"2024-09-21T19:12:14.828381+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:12:14.828381+0000","last_undegraded":"2024-09-21T19:12:14.828381+0000","last_fullsized":"2024-09-21T19:12:14.828381+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:40.758049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:40.758049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:41.348208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":427,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810492+0000","last_change":"2024-09-21T19:11:44.104002+0000","last_active":"2024-09-21T19:12:09.810492+0000","last_peered":"2024-09-21T19:12:09.810492+0000","last_clean":"2024-09-21T19:12:09.810492+0000","last_became_active":"2024-09-21T19:06:54.696385+0000","last_became_peered":"2024-09-21T19:06:54.696385+0000","last_unstale":"2024-09-21T19:12:09.810492+0000","last_undegraded":"2024-09-21T19:12:09.810492+0000","last_fullsized":"2024-09-21T19:12:09.810492+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:44.103926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.103926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.036090+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"459'9921","reported_seq":12060,"reported_epoch":459,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:12:14.457551+0000","last_change":"2024-09-21T19:12:09.940539+0000","last_active":"2024-09-21T19:12:14.457551+0000","last_peered":"2024-09-21T19:12:14.457551+0000","last_clean":"2024-09-21T19:12:08.280360+0000","last_became_active":"2024-09-21T19:12:09.905847+0000","last_became_peered":"2024-09-21T19:12:09.905847+0000","last_unstale":"2024-09-21T19:12:14.457551+0000","last_undegraded":"2024-09-21T19:12:14.457551+0000","last_fullsized":"2024-09-21T19:12:14.457551+0000","mapping_epoch":458,"log_start":"433'4765","ondisk_log_start":"433'4765","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5156,"log_dups_size":0,"ondisk_log_size":5156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174686208,"num_objects":2666,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6525,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5058,"num_write_kb":247136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":33030144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6,7","objects":2665},{"shards":"5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":882,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815722+0000","last_change":"2024-09-21T19:10:26.819197+0000","last_active":"2024-09-21T19:12:14.815722+0000","last_peered":"2024-09-21T19:12:14.815722+0000","last_clean":"2024-09-21T19:12:14.815722+0000","last_became_active":"2024-09-21T19:06:54.703822+0000","last_became_peered":"2024-09-21T19:06:54.703822+0000","last_unstale":"2024-09-21T19:12:14.815722+0000","last_undegraded":"2024-09-21T19:12:14.815722+0000","last_fullsized":"2024-09-21T19:12:14.815722+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:26.819128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:26.819128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":747,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.813436+0000","last_change":"2024-09-21T19:11:42.596370+0000","last_active":"2024-09-21T19:12:09.813436+0000","last_peered":"2024-09-21T19:12:09.813436+0000","last_clean":"2024-09-21T19:12:09.813436+0000","last_became_active":"2024-09-21T18:55:51.520440+0000","last_became_peered":"2024-09-21T18:55:51.520440+0000","last_unstale":"2024-09-21T19:12:09.813436+0000","last_undegraded":"2024-09-21T19:12:09.813436+0000","last_fullsized":"2024-09-21T19:12:09.813436+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:42.596058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:42.596058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:56.734805+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":330,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811888+0000","last_change":"2024-09-21T19:11:37.697054+0000","last_active":"2024-09-21T19:12:09.811888+0000","last_peered":"2024-09-21T19:12:09.811888+0000","last_clean":"2024-09-21T19:12:09.811888+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:12:09.811888+0000","last_undegraded":"2024-09-21T19:12:09.811888+0000","last_fullsized":"2024-09-21T19:12:09.811888+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:11:37.697000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.697000+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.365077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":882,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812412+0000","last_change":"2024-09-21T19:11:31.048129+0000","last_active":"2024-09-21T19:12:09.812412+0000","last_peered":"2024-09-21T19:12:09.812412+0000","last_clean":"2024-09-21T19:12:09.812412+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:12:09.812412+0000","last_undegraded":"2024-09-21T19:12:09.812412+0000","last_fullsized":"2024-09-21T19:12:09.812412+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.048064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.048064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:39.917247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"459'9772","reported_seq":10807,"reported_epoch":459,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:12:12.952641+0000","last_change":"2024-09-21T19:12:09.941852+0000","last_active":"2024-09-21T19:12:12.952641+0000","last_peered":"2024-09-21T19:12:12.952641+0000","last_clean":"2024-09-21T19:12:08.393836+0000","last_became_active":"2024-09-21T19:12:09.922360+0000","last_became_peered":"2024-09-21T19:12:09.922360+0000","last_unstale":"2024-09-21T19:12:12.952641+0000","last_undegraded":"2024-09-21T19:12:12.952641+0000","last_fullsized":"2024-09-21T19:12:12.952641+0000","mapping_epoch":458,"log_start":"427'3434","ondisk_log_start":"427'3434","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"437'5812","last_scrub_stamp":"2024-09-21T19:11:09.898694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.898694+0000","objects_scrubbed":5735,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2183,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":242656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":28049408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[0,2,3],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":2607}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":857,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809564+0000","last_change":"2024-09-21T19:10:13.916666+0000","last_active":"2024-09-21T19:12:09.809564+0000","last_peered":"2024-09-21T19:12:09.809564+0000","last_clean":"2024-09-21T19:12:09.809564+0000","last_became_active":"2024-09-21T19:06:54.695177+0000","last_became_peered":"2024-09-21T19:06:54.695177+0000","last_unstale":"2024-09-21T19:12:09.809564+0000","last_undegraded":"2024-09-21T19:12:09.809564+0000","last_fullsized":"2024-09-21T19:12:09.809564+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:13.916604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.916604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":876,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814478+0000","last_change":"2024-09-21T19:05:42.894738+0000","last_active":"2024-09-21T19:12:14.814478+0000","last_peered":"2024-09-21T19:12:14.814478+0000","last_clean":"2024-09-21T19:12:14.814478+0000","last_became_active":"2024-09-21T19:05:42.894490+0000","last_became_peered":"2024-09-21T19:05:42.894490+0000","last_unstale":"2024-09-21T19:12:14.814478+0000","last_undegraded":"2024-09-21T19:12:14.814478+0000","last_fullsized":"2024-09-21T19:12:14.814478+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:35.694554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:35.694554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.111619+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":946,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809655+0000","last_change":"2024-09-21T19:11:34.009972+0000","last_active":"2024-09-21T19:12:09.809655+0000","last_peered":"2024-09-21T19:12:09.809655+0000","last_clean":"2024-09-21T19:12:09.809655+0000","last_became_active":"2024-09-21T19:10:52.013122+0000","last_became_peered":"2024-09-21T19:10:52.013122+0000","last_unstale":"2024-09-21T19:12:09.809655+0000","last_undegraded":"2024-09-21T19:12:09.809655+0000","last_fullsized":"2024-09-21T19:12:09.809655+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:11:34.009898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.009898+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:37.167818+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":868,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814728+0000","last_change":"2024-09-21T19:11:34.072955+0000","last_active":"2024-09-21T19:12:14.814728+0000","last_peered":"2024-09-21T19:12:14.814728+0000","last_clean":"2024-09-21T19:12:14.814728+0000","last_became_active":"2024-09-21T19:05:42.897146+0000","last_became_peered":"2024-09-21T19:05:42.897146+0000","last_unstale":"2024-09-21T19:12:14.814728+0000","last_undegraded":"2024-09-21T19:12:14.814728+0000","last_fullsized":"2024-09-21T19:12:14.814728+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:34.072904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.072904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.755481+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"459'10001","reported_seq":11021,"reported_epoch":459,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:12:12.946219+0000","last_change":"2024-09-21T19:12:09.949151+0000","last_active":"2024-09-21T19:12:12.946219+0000","last_peered":"2024-09-21T19:12:12.946219+0000","last_clean":"2024-09-21T19:12:08.357486+0000","last_became_active":"2024-09-21T19:12:09.919728+0000","last_became_peered":"2024-09-21T19:12:09.919728+0000","last_unstale":"2024-09-21T19:12:12.946219+0000","last_undegraded":"2024-09-21T19:12:12.946219+0000","last_fullsized":"2024-09-21T19:12:12.946219+0000","mapping_epoch":458,"log_start":"433'4892","ondisk_log_start":"433'4892","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"444'7836","last_scrub_stamp":"2024-09-21T19:11:38.534990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.534990+0000","objects_scrubbed":7739,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176848896,"num_objects":2699,"num_object_clones":0,"num_object_copies":8097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4856,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":249120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":17891328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":2699}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":891,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.829634+0000","last_change":"2024-09-21T19:10:13.592189+0000","last_active":"2024-09-21T19:12:14.829634+0000","last_peered":"2024-09-21T19:12:14.829634+0000","last_clean":"2024-09-21T19:12:14.829634+0000","last_became_active":"2024-09-21T19:06:54.692781+0000","last_became_peered":"2024-09-21T19:06:54.692781+0000","last_unstale":"2024-09-21T19:12:14.829634+0000","last_undegraded":"2024-09-21T19:12:14.829634+0000","last_fullsized":"2024-09-21T19:12:14.829634+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:13.592116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:13.592116+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.755272+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":888,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810730+0000","last_change":"2024-09-21T19:10:27.115013+0000","last_active":"2024-09-21T19:12:09.810730+0000","last_peered":"2024-09-21T19:12:09.810730+0000","last_clean":"2024-09-21T19:12:09.810730+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:12:09.810730+0000","last_undegraded":"2024-09-21T19:12:09.810730+0000","last_fullsized":"2024-09-21T19:12:09.810730+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:27.114925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.114925+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:01.151227+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":776,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811999+0000","last_change":"2024-09-21T19:10:27.501434+0000","last_active":"2024-09-21T19:12:09.811999+0000","last_peered":"2024-09-21T19:12:09.811999+0000","last_clean":"2024-09-21T19:12:09.811999+0000","last_became_active":"2024-09-21T18:55:51.543808+0000","last_became_peered":"2024-09-21T18:55:51.543808+0000","last_unstale":"2024-09-21T19:12:09.811999+0000","last_undegraded":"2024-09-21T19:12:09.811999+0000","last_fullsized":"2024-09-21T19:12:09.811999+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:27.500959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.500959+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:53.653616+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":989,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809048+0000","last_change":"2024-09-21T19:11:38.797906+0000","last_active":"2024-09-21T19:12:09.809048+0000","last_peered":"2024-09-21T19:12:09.809048+0000","last_clean":"2024-09-21T19:12:09.809048+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:12:09.809048+0000","last_undegraded":"2024-09-21T19:12:09.809048+0000","last_fullsized":"2024-09-21T19:12:09.809048+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:11:38.797839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.797839+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:08.395366+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"459'9891","reported_seq":10038,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:12.770140+0000","last_change":"2024-09-21T19:12:08.004217+0000","last_active":"2024-09-21T19:12:12.770140+0000","last_peered":"2024-09-21T19:12:12.770140+0000","last_clean":"2024-09-21T19:12:12.770140+0000","last_became_active":"2024-09-21T19:12:06.968714+0000","last_became_peered":"2024-09-21T19:12:06.968714+0000","last_unstale":"2024-09-21T19:12:12.770140+0000","last_undegraded":"2024-09-21T19:12:12.770140+0000","last_fullsized":"2024-09-21T19:12:12.770140+0000","mapping_epoch":414,"log_start":"433'5092","ondisk_log_start":"433'5092","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9766","last_scrub_stamp":"2024-09-21T19:12:08.004009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:12:08.004009+0000","objects_scrubbed":2811,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:22.468525+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174325760,"num_objects":2660,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":151,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":321,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811205+0000","last_change":"2024-09-21T19:10:21.515835+0000","last_active":"2024-09-21T19:12:09.811205+0000","last_peered":"2024-09-21T19:12:09.811205+0000","last_clean":"2024-09-21T19:12:09.811205+0000","last_became_active":"2024-09-21T19:06:54.693001+0000","last_became_peered":"2024-09-21T19:06:54.693001+0000","last_unstale":"2024-09-21T19:12:09.811205+0000","last_undegraded":"2024-09-21T19:12:09.811205+0000","last_fullsized":"2024-09-21T19:12:09.811205+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:21.515713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.515713+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.467018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":916,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.874326+0000","last_change":"2024-09-21T19:09:56.168284+0000","last_active":"2024-09-21T19:12:09.874326+0000","last_peered":"2024-09-21T19:12:09.874326+0000","last_clean":"2024-09-21T19:12:09.874326+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:12:09.874326+0000","last_undegraded":"2024-09-21T19:12:09.874326+0000","last_fullsized":"2024-09-21T19:12:09.874326+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:09:56.168199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.168199+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.082123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":854,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809589+0000","last_change":"2024-09-21T19:09:50.708334+0000","last_active":"2024-09-21T19:12:09.809589+0000","last_peered":"2024-09-21T19:12:09.809589+0000","last_clean":"2024-09-21T19:12:09.809589+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:12:09.809589+0000","last_undegraded":"2024-09-21T19:12:09.809589+0000","last_fullsized":"2024-09-21T19:12:09.809589+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:50.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:50.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.933893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"350'12","reported_seq":1136,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815812+0000","last_change":"2024-09-21T19:11:10.165309+0000","last_active":"2024-09-21T19:12:14.815812+0000","last_peered":"2024-09-21T19:12:14.815812+0000","last_clean":"2024-09-21T19:12:14.815812+0000","last_became_active":"2024-09-21T19:06:54.705501+0000","last_became_peered":"2024-09-21T19:06:54.705501+0000","last_unstale":"2024-09-21T19:12:14.815812+0000","last_undegraded":"2024-09-21T19:12:14.815812+0000","last_fullsized":"2024-09-21T19:12:14.815812+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:11:10.165236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:10.165236+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.702978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":552,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812369+0000","last_change":"2024-09-21T19:11:49.094049+0000","last_active":"2024-09-21T19:12:09.812369+0000","last_peered":"2024-09-21T19:12:09.812369+0000","last_clean":"2024-09-21T19:12:09.812369+0000","last_became_active":"2024-09-21T19:06:54.703052+0000","last_became_peered":"2024-09-21T19:06:54.703052+0000","last_unstale":"2024-09-21T19:12:09.812369+0000","last_undegraded":"2024-09-21T19:12:09.812369+0000","last_fullsized":"2024-09-21T19:12:09.812369+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:49.093991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:49.093991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.399096+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"460'10581","reported_seq":10686,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:16.989632+0000","last_change":"2024-09-21T19:10:01.252308+0000","last_active":"2024-09-21T19:12:16.989632+0000","last_peered":"2024-09-21T19:12:16.989632+0000","last_clean":"2024-09-21T19:12:16.989632+0000","last_became_active":"2024-09-21T19:10:01.252017+0000","last_became_peered":"2024-09-21T19:10:01.252017+0000","last_unstale":"2024-09-21T19:12:16.989632+0000","last_undegraded":"2024-09-21T19:12:16.989632+0000","last_fullsized":"2024-09-21T19:12:16.989632+0000","mapping_epoch":414,"log_start":"439'6496","ondisk_log_start":"439'6496","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10581,"num_write_kb":504192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":662,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815136+0000","last_change":"2024-09-21T19:11:48.057060+0000","last_active":"2024-09-21T19:12:14.815136+0000","last_peered":"2024-09-21T19:12:14.815136+0000","last_clean":"2024-09-21T19:12:14.815136+0000","last_became_active":"2024-09-21T19:11:47.120840+0000","last_became_peered":"2024-09-21T19:11:47.120840+0000","last_unstale":"2024-09-21T19:12:14.815136+0000","last_undegraded":"2024-09-21T19:12:14.815136+0000","last_fullsized":"2024-09-21T19:12:14.815136+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.056992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.056992+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:08.932687+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":951,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808008+0000","last_change":"2024-09-21T19:11:44.959245+0000","last_active":"2024-09-21T19:12:09.808008+0000","last_peered":"2024-09-21T19:12:09.808008+0000","last_clean":"2024-09-21T19:12:09.808008+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:12:09.808008+0000","last_undegraded":"2024-09-21T19:12:09.808008+0000","last_fullsized":"2024-09-21T19:12:09.808008+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:44.959061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.959061+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:11.599555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":876,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809899+0000","last_change":"2024-09-21T19:10:56.831121+0000","last_active":"2024-09-21T19:12:09.809899+0000","last_peered":"2024-09-21T19:12:09.809899+0000","last_clean":"2024-09-21T19:12:09.809899+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:12:09.809899+0000","last_undegraded":"2024-09-21T19:12:09.809899+0000","last_fullsized":"2024-09-21T19:12:09.809899+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:56.831042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:56.831042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":856,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814052+0000","last_change":"2024-09-21T19:11:05.056988+0000","last_active":"2024-09-21T19:12:14.814052+0000","last_peered":"2024-09-21T19:12:14.814052+0000","last_clean":"2024-09-21T19:12:14.814052+0000","last_became_active":"2024-09-21T18:53:11.411312+0000","last_became_peered":"2024-09-21T18:53:11.411312+0000","last_unstale":"2024-09-21T19:12:14.814052+0000","last_undegraded":"2024-09-21T19:12:14.814052+0000","last_fullsized":"2024-09-21T19:12:14.814052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.056931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.056931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:22.842104+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":873,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814318+0000","last_change":"2024-09-21T19:10:51.864691+0000","last_active":"2024-09-21T19:12:14.814318+0000","last_peered":"2024-09-21T19:12:14.814318+0000","last_clean":"2024-09-21T19:12:14.814318+0000","last_became_active":"2024-09-21T19:05:42.892504+0000","last_became_peered":"2024-09-21T19:05:42.892504+0000","last_unstale":"2024-09-21T19:12:14.814318+0000","last_undegraded":"2024-09-21T19:12:14.814318+0000","last_fullsized":"2024-09-21T19:12:14.814318+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:51.864627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.864627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"460'10263","reported_seq":10367,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:16.471702+0000","last_change":"2024-09-21T19:10:01.248578+0000","last_active":"2024-09-21T19:12:16.471702+0000","last_peered":"2024-09-21T19:12:16.471702+0000","last_clean":"2024-09-21T19:12:16.471702+0000","last_became_active":"2024-09-21T19:10:01.248286+0000","last_became_peered":"2024-09-21T19:10:01.248286+0000","last_unstale":"2024-09-21T19:12:16.471702+0000","last_undegraded":"2024-09-21T19:12:16.471702+0000","last_fullsized":"2024-09-21T19:12:16.471702+0000","mapping_epoch":414,"log_start":"438'6180","ondisk_log_start":"438'6180","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329449472,"num_objects":5027,"num_object_clones":0,"num_object_copies":15081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10263,"num_write_kb":489280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":651,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.806697+0000","last_change":"2024-09-21T19:11:48.886612+0000","last_active":"2024-09-21T19:12:09.806697+0000","last_peered":"2024-09-21T19:12:09.806697+0000","last_clean":"2024-09-21T19:12:09.806697+0000","last_became_active":"2024-09-21T19:11:47.119064+0000","last_became_peered":"2024-09-21T19:11:47.119064+0000","last_unstale":"2024-09-21T19:12:09.806697+0000","last_undegraded":"2024-09-21T19:12:09.806697+0000","last_fullsized":"2024-09-21T19:12:09.806697+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.886536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.886536+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.061161+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":924,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814208+0000","last_change":"2024-09-21T19:10:11.634360+0000","last_active":"2024-09-21T19:12:14.814208+0000","last_peered":"2024-09-21T19:12:14.814208+0000","last_clean":"2024-09-21T19:12:14.814208+0000","last_became_active":"2024-09-21T19:06:54.699507+0000","last_became_peered":"2024-09-21T19:06:54.699507+0000","last_unstale":"2024-09-21T19:12:14.814208+0000","last_undegraded":"2024-09-21T19:12:14.814208+0000","last_fullsized":"2024-09-21T19:12:14.814208+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T19:10:11.634280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:11.634280+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":744,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812294+0000","last_change":"2024-09-21T19:11:04.465550+0000","last_active":"2024-09-21T19:12:09.812294+0000","last_peered":"2024-09-21T19:12:09.812294+0000","last_clean":"2024-09-21T19:12:09.812294+0000","last_became_active":"2024-09-21T18:55:51.539758+0000","last_became_peered":"2024-09-21T18:55:51.539758+0000","last_unstale":"2024-09-21T19:12:09.812294+0000","last_undegraded":"2024-09-21T19:12:09.812294+0000","last_fullsized":"2024-09-21T19:12:09.812294+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:04.464221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:04.464221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:31.185556+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":1077,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807851+0000","last_change":"2024-09-21T19:10:21.865360+0000","last_active":"2024-09-21T19:12:09.807851+0000","last_peered":"2024-09-21T19:12:09.807851+0000","last_clean":"2024-09-21T19:12:09.807851+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:12:09.807851+0000","last_undegraded":"2024-09-21T19:12:09.807851+0000","last_fullsized":"2024-09-21T19:12:09.807851+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:10:21.865158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:21.865158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":876,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828469+0000","last_change":"2024-09-21T19:09:48.513901+0000","last_active":"2024-09-21T19:12:14.828469+0000","last_peered":"2024-09-21T19:12:14.828469+0000","last_clean":"2024-09-21T19:12:14.828469+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:12:14.828469+0000","last_undegraded":"2024-09-21T19:12:14.828469+0000","last_fullsized":"2024-09-21T19:12:14.828469+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:48.513857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:48.513857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:15.789154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"459'10083","reported_seq":10188,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:13.087811+0000","last_change":"2024-09-21T19:10:01.251501+0000","last_active":"2024-09-21T19:12:13.087811+0000","last_peered":"2024-09-21T19:12:13.087811+0000","last_clean":"2024-09-21T19:12:13.087811+0000","last_became_active":"2024-09-21T19:10:01.250354+0000","last_became_peered":"2024-09-21T19:10:01.250354+0000","last_unstale":"2024-09-21T19:12:13.087811+0000","last_undegraded":"2024-09-21T19:12:13.087811+0000","last_fullsized":"2024-09-21T19:12:13.087811+0000","mapping_epoch":414,"log_start":"433'4786","ondisk_log_start":"433'4786","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342360088,"num_objects":5225,"num_object_clones":0,"num_object_copies":15675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10083,"num_write_kb":489793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":667,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809254+0000","last_change":"2024-09-21T19:11:47.126158+0000","last_active":"2024-09-21T19:12:09.809254+0000","last_peered":"2024-09-21T19:12:09.809254+0000","last_clean":"2024-09-21T19:12:09.809254+0000","last_became_active":"2024-09-21T19:11:47.123128+0000","last_became_peered":"2024-09-21T19:11:47.123128+0000","last_unstale":"2024-09-21T19:12:09.809254+0000","last_undegraded":"2024-09-21T19:12:09.809254+0000","last_fullsized":"2024-09-21T19:12:09.809254+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.906970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.906970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:51.764634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":907,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812901+0000","last_change":"2024-09-21T19:10:57.167898+0000","last_active":"2024-09-21T19:12:09.812901+0000","last_peered":"2024-09-21T19:12:09.812901+0000","last_clean":"2024-09-21T19:12:09.812901+0000","last_became_active":"2024-09-21T19:06:54.703992+0000","last_became_peered":"2024-09-21T19:06:54.703992+0000","last_unstale":"2024-09-21T19:12:09.812901+0000","last_undegraded":"2024-09-21T19:12:09.812901+0000","last_fullsized":"2024-09-21T19:12:09.812901+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:57.167826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.167826+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:06.188884+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":418,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809677+0000","last_change":"2024-09-21T19:09:54.781125+0000","last_active":"2024-09-21T19:12:09.809677+0000","last_peered":"2024-09-21T19:12:09.809677+0000","last_clean":"2024-09-21T19:12:09.809677+0000","last_became_active":"2024-09-21T19:06:54.691223+0000","last_became_peered":"2024-09-21T19:06:54.691223+0000","last_unstale":"2024-09-21T19:12:09.809677+0000","last_undegraded":"2024-09-21T19:12:09.809677+0000","last_fullsized":"2024-09-21T19:12:09.809677+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:54.781069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.781069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:06.070674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"350'14","reported_seq":959,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.813376+0000","last_change":"2024-09-21T19:11:32.614783+0000","last_active":"2024-09-21T19:12:09.813376+0000","last_peered":"2024-09-21T19:12:09.813376+0000","last_clean":"2024-09-21T19:12:09.813376+0000","last_became_active":"2024-09-21T19:06:54.700292+0000","last_became_peered":"2024-09-21T19:06:54.700292+0000","last_unstale":"2024-09-21T19:12:09.813376+0000","last_undegraded":"2024-09-21T19:12:09.813376+0000","last_fullsized":"2024-09-21T19:12:09.813376+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'14","last_scrub_stamp":"2024-09-21T19:11:32.614707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:32.614707+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:45.115750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":755,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811846+0000","last_change":"2024-09-21T19:11:08.473248+0000","last_active":"2024-09-21T19:12:09.811846+0000","last_peered":"2024-09-21T19:12:09.811846+0000","last_clean":"2024-09-21T19:12:09.811846+0000","last_became_active":"2024-09-21T19:05:42.900821+0000","last_became_peered":"2024-09-21T19:05:42.900821+0000","last_unstale":"2024-09-21T19:12:09.811846+0000","last_undegraded":"2024-09-21T19:12:09.811846+0000","last_fullsized":"2024-09-21T19:12:09.811846+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.473184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.473184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:38.292717+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"459'10321","reported_seq":10510,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:12.825394+0000","last_change":"2024-09-21T19:11:35.874862+0000","last_active":"2024-09-21T19:12:12.825394+0000","last_peered":"2024-09-21T19:12:12.825394+0000","last_clean":"2024-09-21T19:12:12.825394+0000","last_became_active":"2024-09-21T19:10:01.243730+0000","last_became_peered":"2024-09-21T19:10:01.243730+0000","last_unstale":"2024-09-21T19:12:12.825394+0000","last_undegraded":"2024-09-21T19:12:12.825394+0000","last_fullsized":"2024-09-21T19:12:12.825394+0000","mapping_epoch":414,"log_start":"438'6150","ondisk_log_start":"438'6150","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'7719","last_scrub_stamp":"2024-09-21T19:11:35.874735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:35.874735+0000","objects_scrubbed":7633,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:03.056396+0000","scrub_duration":2835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350683136,"num_objects":5351,"num_object_clones":0,"num_object_copies":16053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10321,"num_write_kb":501504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":645,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809272+0000","last_change":"2024-09-21T19:11:47.128871+0000","last_active":"2024-09-21T19:12:09.809272+0000","last_peered":"2024-09-21T19:12:09.809272+0000","last_clean":"2024-09-21T19:12:09.809272+0000","last_became_active":"2024-09-21T19:11:47.128147+0000","last_became_peered":"2024-09-21T19:11:47.128147+0000","last_unstale":"2024-09-21T19:12:09.809272+0000","last_undegraded":"2024-09-21T19:12:09.809272+0000","last_fullsized":"2024-09-21T19:12:09.809272+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:13:03.751655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":792,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.813510+0000","last_change":"2024-09-21T19:10:25.488039+0000","last_active":"2024-09-21T19:12:09.813510+0000","last_peered":"2024-09-21T19:12:09.813510+0000","last_clean":"2024-09-21T19:12:09.813510+0000","last_became_active":"2024-09-21T19:06:54.699910+0000","last_became_peered":"2024-09-21T19:06:54.699910+0000","last_unstale":"2024-09-21T19:12:09.813510+0000","last_undegraded":"2024-09-21T19:12:09.813510+0000","last_fullsized":"2024-09-21T19:12:09.813510+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T19:10:25.487878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:25.487878+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.652098+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":887,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.814790+0000","last_change":"2024-09-21T19:09:42.077976+0000","last_active":"2024-09-21T19:12:09.814790+0000","last_peered":"2024-09-21T19:12:09.814790+0000","last_clean":"2024-09-21T19:12:09.814790+0000","last_became_active":"2024-09-21T19:06:54.695116+0000","last_became_peered":"2024-09-21T19:06:54.695116+0000","last_unstale":"2024-09-21T19:12:09.814790+0000","last_undegraded":"2024-09-21T19:12:09.814790+0000","last_fullsized":"2024-09-21T19:12:09.814790+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:42.077896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:09:42.077896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:14.041720+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":763,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808349+0000","last_change":"2024-09-21T19:06:54.701549+0000","last_active":"2024-09-21T19:12:09.808349+0000","last_peered":"2024-09-21T19:12:09.808349+0000","last_clean":"2024-09-21T19:12:09.808349+0000","last_became_active":"2024-09-21T19:06:54.700868+0000","last_became_peered":"2024-09-21T19:06:54.700868+0000","last_unstale":"2024-09-21T19:12:09.808349+0000","last_undegraded":"2024-09-21T19:12:09.808349+0000","last_fullsized":"2024-09-21T19:12:09.808349+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:42.700352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:42.700352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:17.702612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":875,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810577+0000","last_change":"2024-09-21T19:11:00.111371+0000","last_active":"2024-09-21T19:12:09.810577+0000","last_peered":"2024-09-21T19:12:09.810577+0000","last_clean":"2024-09-21T19:12:09.810577+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:12:09.810577+0000","last_undegraded":"2024-09-21T19:12:09.810577+0000","last_fullsized":"2024-09-21T19:12:09.810577+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:00.111305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.111305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.568023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"459'10150","reported_seq":10254,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:13.159872+0000","last_change":"2024-09-21T19:10:01.250359+0000","last_active":"2024-09-21T19:12:13.159872+0000","last_peered":"2024-09-21T19:12:13.159872+0000","last_clean":"2024-09-21T19:12:13.159872+0000","last_became_active":"2024-09-21T19:10:01.250079+0000","last_became_peered":"2024-09-21T19:10:01.250079+0000","last_unstale":"2024-09-21T19:12:13.159872+0000","last_undegraded":"2024-09-21T19:12:13.159872+0000","last_fullsized":"2024-09-21T19:12:13.159872+0000","mapping_epoch":414,"log_start":"439'6339","ondisk_log_start":"439'6339","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348389376,"num_objects":5316,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":620,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.889338+0000","last_change":"2024-09-21T19:11:47.114755+0000","last_active":"2024-09-21T19:12:09.889338+0000","last_peered":"2024-09-21T19:12:09.889338+0000","last_clean":"2024-09-21T19:12:09.889338+0000","last_became_active":"2024-09-21T19:11:47.114145+0000","last_became_peered":"2024-09-21T19:11:47.114145+0000","last_unstale":"2024-09-21T19:12:09.889338+0000","last_undegraded":"2024-09-21T19:12:09.889338+0000","last_fullsized":"2024-09-21T19:12:09.889338+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:07.476444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.476444+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:49.613083+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"321'42","reported_seq":1205,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812911+0000","last_change":"2024-09-21T19:09:51.228432+0000","last_active":"2024-09-21T19:12:09.812911+0000","last_peered":"2024-09-21T19:12:09.812911+0000","last_clean":"2024-09-21T19:12:09.812911+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:12:09.812911+0000","last_undegraded":"2024-09-21T19:12:09.812911+0000","last_fullsized":"2024-09-21T19:12:09.812911+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'42","last_scrub_stamp":"2024-09-21T19:09:51.228370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.228370+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.035088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":889,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815117+0000","last_change":"2024-09-21T19:06:23.837162+0000","last_active":"2024-09-21T19:12:14.815117+0000","last_peered":"2024-09-21T19:12:14.815117+0000","last_clean":"2024-09-21T19:12:14.815117+0000","last_became_active":"2024-09-21T19:02:40.881576+0000","last_became_peered":"2024-09-21T19:02:40.881576+0000","last_unstale":"2024-09-21T19:12:14.815117+0000","last_undegraded":"2024-09-21T19:12:14.815117+0000","last_fullsized":"2024-09-21T19:12:14.815117+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:23.837098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:23.837098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.083552+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":434,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.814743+0000","last_change":"2024-09-21T19:08:19.209964+0000","last_active":"2024-09-21T19:12:09.814743+0000","last_peered":"2024-09-21T19:12:09.814743+0000","last_clean":"2024-09-21T19:12:09.814743+0000","last_became_active":"2024-09-21T19:06:54.706398+0000","last_became_peered":"2024-09-21T19:06:54.706398+0000","last_unstale":"2024-09-21T19:12:09.814743+0000","last_undegraded":"2024-09-21T19:12:09.814743+0000","last_fullsized":"2024-09-21T19:12:09.814743+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:08:19.209872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:08:19.209872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.115247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":740,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812223+0000","last_change":"2024-09-21T19:11:05.417598+0000","last_active":"2024-09-21T19:12:09.812223+0000","last_peered":"2024-09-21T19:12:09.812223+0000","last_clean":"2024-09-21T19:12:09.812223+0000","last_became_active":"2024-09-21T18:55:51.538705+0000","last_became_peered":"2024-09-21T18:55:51.538705+0000","last_unstale":"2024-09-21T19:12:09.812223+0000","last_undegraded":"2024-09-21T19:12:09.812223+0000","last_fullsized":"2024-09-21T19:12:09.812223+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:05.417365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:05.417365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.055191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":842,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808989+0000","last_change":"2024-09-21T19:10:28.717418+0000","last_active":"2024-09-21T19:12:09.808989+0000","last_peered":"2024-09-21T19:12:09.808989+0000","last_clean":"2024-09-21T19:12:09.808989+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:12:09.808989+0000","last_undegraded":"2024-09-21T19:12:09.808989+0000","last_fullsized":"2024-09-21T19:12:09.808989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:28.717310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.717310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"459'9804","reported_seq":9947,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:13.169749+0000","last_change":"2024-09-21T19:12:08.613880+0000","last_active":"2024-09-21T19:12:13.169749+0000","last_peered":"2024-09-21T19:12:13.169749+0000","last_clean":"2024-09-21T19:12:13.169749+0000","last_became_active":"2024-09-21T19:12:06.940410+0000","last_became_peered":"2024-09-21T19:12:06.940410+0000","last_unstale":"2024-09-21T19:12:13.169749+0000","last_undegraded":"2024-09-21T19:12:13.169749+0000","last_fullsized":"2024-09-21T19:12:13.169749+0000","mapping_epoch":414,"log_start":"434'5499","ondisk_log_start":"434'5499","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9689","last_scrub_stamp":"2024-09-21T19:12:08.613786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:12:08.613786+0000","objects_scrubbed":2738,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:32.556223+0000","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171769856,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":958,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828160+0000","last_change":"2024-09-21T19:11:47.118419+0000","last_active":"2024-09-21T19:12:14.828160+0000","last_peered":"2024-09-21T19:12:14.828160+0000","last_clean":"2024-09-21T19:12:14.828160+0000","last_became_active":"2024-09-21T19:11:47.117467+0000","last_became_peered":"2024-09-21T19:11:47.117467+0000","last_unstale":"2024-09-21T19:12:14.828160+0000","last_undegraded":"2024-09-21T19:12:14.828160+0000","last_fullsized":"2024-09-21T19:12:14.828160+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:51.014661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":360,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.816288+0000","last_change":"2024-09-21T19:10:20.080922+0000","last_active":"2024-09-21T19:12:09.816288+0000","last_peered":"2024-09-21T19:12:09.816288+0000","last_clean":"2024-09-21T19:12:09.816288+0000","last_became_active":"2024-09-21T19:06:54.704662+0000","last_became_peered":"2024-09-21T19:06:54.704662+0000","last_unstale":"2024-09-21T19:12:09.816288+0000","last_undegraded":"2024-09-21T19:12:09.816288+0000","last_fullsized":"2024-09-21T19:12:09.816288+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T19:10:20.080874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:20.080874+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.166179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":917,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.814808+0000","last_change":"2024-09-21T19:11:37.043422+0000","last_active":"2024-09-21T19:12:09.814808+0000","last_peered":"2024-09-21T19:12:09.814808+0000","last_clean":"2024-09-21T19:12:09.814808+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:12:09.814808+0000","last_undegraded":"2024-09-21T19:12:09.814808+0000","last_fullsized":"2024-09-21T19:12:09.814808+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:37.043234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.043234+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:52.960859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":311,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809723+0000","last_change":"2024-09-21T19:11:32.736397+0000","last_active":"2024-09-21T19:12:09.809723+0000","last_peered":"2024-09-21T19:12:09.809723+0000","last_clean":"2024-09-21T19:12:09.809723+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:12:09.809723+0000","last_undegraded":"2024-09-21T19:12:09.809723+0000","last_fullsized":"2024-09-21T19:12:09.809723+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:11:32.736299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:11:32.736299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:49.298535+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":318,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812510+0000","last_change":"2024-09-21T19:11:47.541937+0000","last_active":"2024-09-21T19:12:09.812510+0000","last_peered":"2024-09-21T19:12:09.812510+0000","last_clean":"2024-09-21T19:12:09.812510+0000","last_became_active":"2024-09-21T19:06:54.694285+0000","last_became_peered":"2024-09-21T19:06:54.694285+0000","last_unstale":"2024-09-21T19:12:09.812510+0000","last_undegraded":"2024-09-21T19:12:09.812510+0000","last_fullsized":"2024-09-21T19:12:09.812510+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:47.541386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.541386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:56.449155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":874,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828429+0000","last_change":"2024-09-21T19:10:45.720231+0000","last_active":"2024-09-21T19:12:14.828429+0000","last_peered":"2024-09-21T19:12:14.828429+0000","last_clean":"2024-09-21T19:12:14.828429+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:12:14.828429+0000","last_undegraded":"2024-09-21T19:12:14.828429+0000","last_fullsized":"2024-09-21T19:12:14.828429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:10:45.720186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.720186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"459'10045","reported_seq":10152,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:13.171813+0000","last_change":"2024-09-21T19:12:06.971773+0000","last_active":"2024-09-21T19:12:13.171813+0000","last_peered":"2024-09-21T19:12:13.171813+0000","last_clean":"2024-09-21T19:12:13.171813+0000","last_became_active":"2024-09-21T19:12:06.963154+0000","last_became_peered":"2024-09-21T19:12:06.963154+0000","last_unstale":"2024-09-21T19:12:13.171813+0000","last_undegraded":"2024-09-21T19:12:13.171813+0000","last_fullsized":"2024-09-21T19:12:13.171813+0000","mapping_epoch":414,"log_start":"435'5694","ondisk_log_start":"435'5694","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174358528,"num_objects":2660,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":248736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":670,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.818689+0000","last_change":"2024-09-21T19:11:48.108418+0000","last_active":"2024-09-21T19:12:09.818689+0000","last_peered":"2024-09-21T19:12:09.818689+0000","last_clean":"2024-09-21T19:12:09.818689+0000","last_became_active":"2024-09-21T19:11:47.124560+0000","last_became_peered":"2024-09-21T19:11:47.124560+0000","last_unstale":"2024-09-21T19:12:09.818689+0000","last_undegraded":"2024-09-21T19:12:09.818689+0000","last_fullsized":"2024-09-21T19:12:09.818689+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.108284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.108284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:52.461410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":862,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812101+0000","last_change":"2024-09-21T19:11:45.692291+0000","last_active":"2024-09-21T19:12:09.812101+0000","last_peered":"2024-09-21T19:12:09.812101+0000","last_clean":"2024-09-21T19:12:09.812101+0000","last_became_active":"2024-09-21T19:06:54.705348+0000","last_became_peered":"2024-09-21T19:06:54.705348+0000","last_unstale":"2024-09-21T19:12:09.812101+0000","last_undegraded":"2024-09-21T19:12:09.812101+0000","last_fullsized":"2024-09-21T19:12:09.812101+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:11:45.692118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.692118+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:58.276257+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":748,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.889873+0000","last_change":"2024-09-21T19:11:46.659335+0000","last_active":"2024-09-21T19:12:09.889873+0000","last_peered":"2024-09-21T19:12:09.889873+0000","last_clean":"2024-09-21T19:12:09.889873+0000","last_became_active":"2024-09-21T18:55:51.538123+0000","last_became_peered":"2024-09-21T18:55:51.538123+0000","last_unstale":"2024-09-21T19:12:09.889873+0000","last_undegraded":"2024-09-21T19:12:09.889873+0000","last_fullsized":"2024-09-21T19:12:09.889873+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:46.659194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:46.659194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.723509+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":2030,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:15.017000+0000","last_change":"2024-09-21T19:10:28.526838+0000","last_active":"2024-09-21T19:12:15.017000+0000","last_peered":"2024-09-21T19:12:15.017000+0000","last_clean":"2024-09-21T19:12:15.017000+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:12:15.017000+0000","last_undegraded":"2024-09-21T19:12:15.017000+0000","last_fullsized":"2024-09-21T19:12:15.017000+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:10:28.526790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.526790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.000273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1137,"num_read_kb":1132,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":813,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811914+0000","last_change":"2024-09-21T19:09:53.733554+0000","last_active":"2024-09-21T19:12:09.811914+0000","last_peered":"2024-09-21T19:12:09.811914+0000","last_clean":"2024-09-21T19:12:09.811914+0000","last_became_active":"2024-09-21T19:06:54.695371+0000","last_became_peered":"2024-09-21T19:06:54.695371+0000","last_unstale":"2024-09-21T19:12:09.811914+0000","last_undegraded":"2024-09-21T19:12:09.811914+0000","last_fullsized":"2024-09-21T19:12:09.811914+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.733495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.733495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:20.851635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":877,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812716+0000","last_change":"2024-09-21T19:11:43.150420+0000","last_active":"2024-09-21T19:12:09.812716+0000","last_peered":"2024-09-21T19:12:09.812716+0000","last_clean":"2024-09-21T19:12:09.812716+0000","last_became_active":"2024-09-21T19:06:54.695216+0000","last_became_peered":"2024-09-21T19:06:54.695216+0000","last_unstale":"2024-09-21T19:12:09.812716+0000","last_undegraded":"2024-09-21T19:12:09.812716+0000","last_fullsized":"2024-09-21T19:12:09.812716+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:43.150353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:43.150353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:10.609375+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"459'10118","reported_seq":10223,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:13.177992+0000","last_change":"2024-09-21T19:10:01.260090+0000","last_active":"2024-09-21T19:12:13.177992+0000","last_peered":"2024-09-21T19:12:13.177992+0000","last_clean":"2024-09-21T19:12:13.177992+0000","last_became_active":"2024-09-21T19:10:01.259620+0000","last_became_peered":"2024-09-21T19:10:01.259620+0000","last_unstale":"2024-09-21T19:12:13.177992+0000","last_undegraded":"2024-09-21T19:12:13.177992+0000","last_fullsized":"2024-09-21T19:12:13.177992+0000","mapping_epoch":414,"log_start":"439'6301","ondisk_log_start":"439'6301","created":414,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":3817,"log_dups_size":0,"ondisk_log_size":3817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343670784,"num_objects":5244,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":491584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"276'30","reported_seq":1072,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809290+0000","last_change":"2024-09-21T19:11:02.540038+0000","last_active":"2024-09-21T19:12:09.809290+0000","last_peered":"2024-09-21T19:12:09.809290+0000","last_clean":"2024-09-21T19:12:09.809290+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:12:09.809290+0000","last_undegraded":"2024-09-21T19:12:09.809290+0000","last_fullsized":"2024-09-21T19:12:09.809290+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:02.539974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.539974+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":877,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807057+0000","last_change":"2024-09-21T19:11:36.018987+0000","last_active":"2024-09-21T19:12:09.807057+0000","last_peered":"2024-09-21T19:12:09.807057+0000","last_clean":"2024-09-21T19:12:09.807057+0000","last_became_active":"2024-09-21T19:06:54.696754+0000","last_became_peered":"2024-09-21T19:06:54.696754+0000","last_unstale":"2024-09-21T19:12:09.807057+0000","last_undegraded":"2024-09-21T19:12:09.807057+0000","last_fullsized":"2024-09-21T19:12:09.807057+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:36.018911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:36.018911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.421244+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":972,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828715+0000","last_change":"2024-09-21T19:11:47.120442+0000","last_active":"2024-09-21T19:12:14.828715+0000","last_peered":"2024-09-21T19:12:14.828715+0000","last_clean":"2024-09-21T19:12:14.828715+0000","last_became_active":"2024-09-21T19:11:47.120134+0000","last_became_peered":"2024-09-21T19:11:47.120134+0000","last_unstale":"2024-09-21T19:12:14.828715+0000","last_undegraded":"2024-09-21T19:12:14.828715+0000","last_fullsized":"2024-09-21T19:12:14.828715+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:45.911119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.911119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:53.014722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":880,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.829787+0000","last_change":"2024-09-21T19:10:17.553053+0000","last_active":"2024-09-21T19:12:14.829787+0000","last_peered":"2024-09-21T19:12:14.829787+0000","last_clean":"2024-09-21T19:12:14.829787+0000","last_became_active":"2024-09-21T19:06:54.691826+0000","last_became_peered":"2024-09-21T19:06:54.691826+0000","last_unstale":"2024-09-21T19:12:14.829787+0000","last_undegraded":"2024-09-21T19:12:14.829787+0000","last_fullsized":"2024-09-21T19:12:14.829787+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:17.553007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:17.553007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:00.041731+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":404,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809475+0000","last_change":"2024-09-21T19:09:53.768537+0000","last_active":"2024-09-21T19:12:09.809475+0000","last_peered":"2024-09-21T19:12:09.809475+0000","last_clean":"2024-09-21T19:12:09.809475+0000","last_became_active":"2024-09-21T19:06:54.707849+0000","last_became_peered":"2024-09-21T19:06:54.707849+0000","last_unstale":"2024-09-21T19:12:09.809475+0000","last_undegraded":"2024-09-21T19:12:09.809475+0000","last_fullsized":"2024-09-21T19:12:09.809475+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:53.768487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.768487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":668,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.818343+0000","last_change":"2024-09-21T19:11:49.133015+0000","last_active":"2024-09-21T19:12:09.818343+0000","last_peered":"2024-09-21T19:12:09.818343+0000","last_clean":"2024-09-21T19:12:09.818343+0000","last_became_active":"2024-09-21T19:11:49.131805+0000","last_became_peered":"2024-09-21T19:11:49.131805+0000","last_unstale":"2024-09-21T19:12:09.818343+0000","last_undegraded":"2024-09-21T19:12:09.818343+0000","last_fullsized":"2024-09-21T19:12:09.818343+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:28.147274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.147274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:54.129069+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"459'10067","reported_seq":10265,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.452466+0000","last_change":"2024-09-21T19:12:06.963804+0000","last_active":"2024-09-21T19:12:14.452466+0000","last_peered":"2024-09-21T19:12:14.452466+0000","last_clean":"2024-09-21T19:12:14.452466+0000","last_became_active":"2024-09-21T19:12:06.951191+0000","last_became_peered":"2024-09-21T19:12:06.951191+0000","last_unstale":"2024-09-21T19:12:14.452466+0000","last_undegraded":"2024-09-21T19:12:14.452466+0000","last_fullsized":"2024-09-21T19:12:14.452466+0000","mapping_epoch":414,"log_start":"433'4892","ondisk_log_start":"433'4892","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'7836","last_scrub_stamp":"2024-09-21T19:11:38.534990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.534990+0000","objects_scrubbed":7739,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172523520,"num_objects":2632,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5152,"num_write_kb":249120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":917,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815867+0000","last_change":"2024-09-21T19:10:23.782154+0000","last_active":"2024-09-21T19:12:14.815867+0000","last_peered":"2024-09-21T19:12:14.815867+0000","last_clean":"2024-09-21T19:12:14.815867+0000","last_became_active":"2024-09-21T19:02:40.880681+0000","last_became_peered":"2024-09-21T19:02:40.880681+0000","last_unstale":"2024-09-21T19:12:14.815867+0000","last_undegraded":"2024-09-21T19:12:14.815867+0000","last_fullsized":"2024-09-21T19:12:14.815867+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:10:23.782107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.782107+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":885,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814505+0000","last_change":"2024-09-21T19:10:19.729765+0000","last_active":"2024-09-21T19:12:14.814505+0000","last_peered":"2024-09-21T19:12:14.814505+0000","last_clean":"2024-09-21T19:12:14.814505+0000","last_became_active":"2024-09-21T19:05:42.891496+0000","last_became_peered":"2024-09-21T19:05:42.891496+0000","last_unstale":"2024-09-21T19:12:14.814505+0000","last_undegraded":"2024-09-21T19:12:14.814505+0000","last_fullsized":"2024-09-21T19:12:14.814505+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:19.729685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.729685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":870,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815122+0000","last_change":"2024-09-21T19:10:58.999039+0000","last_active":"2024-09-21T19:12:14.815122+0000","last_peered":"2024-09-21T19:12:14.815122+0000","last_clean":"2024-09-21T19:12:14.815122+0000","last_became_active":"2024-09-21T19:06:54.702648+0000","last_became_peered":"2024-09-21T19:06:54.702648+0000","last_unstale":"2024-09-21T19:12:14.815122+0000","last_undegraded":"2024-09-21T19:12:14.815122+0000","last_fullsized":"2024-09-21T19:12:14.815122+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.998847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.998847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":840,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807317+0000","last_change":"2024-09-21T19:06:54.698442+0000","last_active":"2024-09-21T19:12:09.807317+0000","last_peered":"2024-09-21T19:12:09.807317+0000","last_clean":"2024-09-21T19:12:09.807317+0000","last_became_active":"2024-09-21T19:06:54.691749+0000","last_became_peered":"2024-09-21T19:06:54.691749+0000","last_unstale":"2024-09-21T19:12:09.807317+0000","last_undegraded":"2024-09-21T19:12:09.807317+0000","last_fullsized":"2024-09-21T19:12:09.807317+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.521707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.521707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:13.968888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"459'9747","reported_seq":9960,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:13.080471+0000","last_change":"2024-09-21T19:12:08.429792+0000","last_active":"2024-09-21T19:12:13.080471+0000","last_peered":"2024-09-21T19:12:13.080471+0000","last_clean":"2024-09-21T19:12:13.080471+0000","last_became_active":"2024-09-21T19:12:06.951545+0000","last_became_peered":"2024-09-21T19:12:06.951545+0000","last_unstale":"2024-09-21T19:12:13.080471+0000","last_undegraded":"2024-09-21T19:12:13.080471+0000","last_fullsized":"2024-09-21T19:12:13.080471+0000","mapping_epoch":414,"log_start":"427'3434","ondisk_log_start":"427'3434","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9652","last_scrub_stamp":"2024-09-21T19:12:08.429621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:12:08.429621+0000","objects_scrubbed":2687,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:31.615080+0000","scrub_duration":832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169279488,"num_objects":2583,"num_object_clones":0,"num_object_copies":7749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":325,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809142+0000","last_change":"2024-09-21T19:11:48.520793+0000","last_active":"2024-09-21T19:12:09.809142+0000","last_peered":"2024-09-21T19:12:09.809142+0000","last_clean":"2024-09-21T19:12:09.809142+0000","last_became_active":"2024-09-21T19:06:54.689889+0000","last_became_peered":"2024-09-21T19:06:54.689889+0000","last_unstale":"2024-09-21T19:12:09.809142+0000","last_undegraded":"2024-09-21T19:12:09.809142+0000","last_fullsized":"2024-09-21T19:12:09.809142+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.520711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.520711+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.918449+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":958,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815023+0000","last_change":"2024-09-21T19:06:47.067429+0000","last_active":"2024-09-21T19:12:14.815023+0000","last_peered":"2024-09-21T19:12:14.815023+0000","last_clean":"2024-09-21T19:12:14.815023+0000","last_became_active":"2024-09-21T19:05:42.895962+0000","last_became_peered":"2024-09-21T19:05:42.895962+0000","last_unstale":"2024-09-21T19:12:14.815023+0000","last_undegraded":"2024-09-21T19:12:14.815023+0000","last_fullsized":"2024-09-21T19:12:14.815023+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T19:06:47.067341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:47.067341+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:03.109162+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":790,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809133+0000","last_change":"2024-09-21T19:11:46.781966+0000","last_active":"2024-09-21T19:12:09.809133+0000","last_peered":"2024-09-21T19:12:09.809133+0000","last_clean":"2024-09-21T19:12:09.809133+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:12:09.809133+0000","last_undegraded":"2024-09-21T19:12:09.809133+0000","last_fullsized":"2024-09-21T19:12:09.809133+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:46.781896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:46.781896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:50.353466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":314,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808662+0000","last_change":"2024-09-21T19:11:31.656763+0000","last_active":"2024-09-21T19:12:09.808662+0000","last_peered":"2024-09-21T19:12:09.808662+0000","last_clean":"2024-09-21T19:12:09.808662+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:12:09.808662+0000","last_undegraded":"2024-09-21T19:12:09.808662+0000","last_fullsized":"2024-09-21T19:12:09.808662+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.656689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.656689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.381458+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":314,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811449+0000","last_change":"2024-09-21T19:10:45.402838+0000","last_active":"2024-09-21T19:12:09.811449+0000","last_peered":"2024-09-21T19:12:09.811449+0000","last_clean":"2024-09-21T19:12:09.811449+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:12:09.811449+0000","last_undegraded":"2024-09-21T19:12:09.811449+0000","last_fullsized":"2024-09-21T19:12:09.811449+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:45.402769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:45.402769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.591823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"459'9936","reported_seq":10047,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.432341+0000","last_change":"2024-09-21T19:12:06.963506+0000","last_active":"2024-09-21T19:12:14.432341+0000","last_peered":"2024-09-21T19:12:14.432341+0000","last_clean":"2024-09-21T19:12:14.432341+0000","last_became_active":"2024-09-21T19:12:06.948907+0000","last_became_peered":"2024-09-21T19:12:06.948907+0000","last_unstale":"2024-09-21T19:12:14.432341+0000","last_undegraded":"2024-09-21T19:12:14.432341+0000","last_fullsized":"2024-09-21T19:12:14.432341+0000","mapping_epoch":414,"log_start":"433'4765","ondisk_log_start":"433'4765","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173703168,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5072,"num_write_kb":247136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":974,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828250+0000","last_change":"2024-09-21T19:10:51.731370+0000","last_active":"2024-09-21T19:12:14.828250+0000","last_peered":"2024-09-21T19:12:14.828250+0000","last_clean":"2024-09-21T19:12:14.828250+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:12:14.828250+0000","last_undegraded":"2024-09-21T19:12:14.828250+0000","last_fullsized":"2024-09-21T19:12:14.828250+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:51.731299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:51.731299+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":884,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814892+0000","last_change":"2024-09-21T19:06:22.819758+0000","last_active":"2024-09-21T19:12:14.814892+0000","last_peered":"2024-09-21T19:12:14.814892+0000","last_clean":"2024-09-21T19:12:14.814892+0000","last_became_active":"2024-09-21T19:02:40.883489+0000","last_became_peered":"2024-09-21T19:02:40.883489+0000","last_unstale":"2024-09-21T19:12:14.814892+0000","last_undegraded":"2024-09-21T19:12:14.814892+0000","last_fullsized":"2024-09-21T19:12:14.814892+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:06:22.819685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:06:22.819685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.115802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":890,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807167+0000","last_change":"2024-09-21T19:10:55.909242+0000","last_active":"2024-09-21T19:12:09.807167+0000","last_peered":"2024-09-21T19:12:09.807167+0000","last_clean":"2024-09-21T19:12:09.807167+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:12:09.807167+0000","last_undegraded":"2024-09-21T19:12:09.807167+0000","last_fullsized":"2024-09-21T19:12:09.807167+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:55.909172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:55.909172+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.738817+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"350'12","reported_seq":402,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811679+0000","last_change":"2024-09-21T19:11:38.701095+0000","last_active":"2024-09-21T19:12:09.811679+0000","last_peered":"2024-09-21T19:12:09.811679+0000","last_clean":"2024-09-21T19:12:09.811679+0000","last_became_active":"2024-09-21T19:06:54.694246+0000","last_became_peered":"2024-09-21T19:06:54.694246+0000","last_unstale":"2024-09-21T19:12:09.811679+0000","last_undegraded":"2024-09-21T19:12:09.811679+0000","last_fullsized":"2024-09-21T19:12:09.811679+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'12","last_scrub_stamp":"2024-09-21T19:11:38.700917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:38.700917+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:01.033044+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":857,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815634+0000","last_change":"2024-09-21T19:06:48.105808+0000","last_active":"2024-09-21T19:12:14.815634+0000","last_peered":"2024-09-21T19:12:14.815634+0000","last_clean":"2024-09-21T19:12:14.815634+0000","last_became_active":"2024-09-21T19:05:42.898484+0000","last_became_peered":"2024-09-21T19:05:42.898484+0000","last_unstale":"2024-09-21T19:12:14.815634+0000","last_undegraded":"2024-09-21T19:12:14.815634+0000","last_fullsized":"2024-09-21T19:12:14.815634+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:48.105255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:48.105255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:06.220680+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":30,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815543+0000","last_change":"2024-09-21T19:11:49.147463+0000","last_active":"2024-09-21T19:12:14.815543+0000","last_peered":"2024-09-21T19:12:14.815543+0000","last_clean":"2024-09-21T19:12:14.815543+0000","last_became_active":"2024-09-21T19:11:49.147110+0000","last_became_peered":"2024-09-21T19:11:49.147110+0000","last_unstale":"2024-09-21T19:12:14.815543+0000","last_undegraded":"2024-09-21T19:12:14.815543+0000","last_fullsized":"2024-09-21T19:12:14.815543+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:54.937815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:54.937815+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:54.067093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"460'10007","reported_seq":10122,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:16.944776+0000","last_change":"2024-09-21T19:12:06.990445+0000","last_active":"2024-09-21T19:12:16.944776+0000","last_peered":"2024-09-21T19:12:16.944776+0000","last_clean":"2024-09-21T19:12:16.944776+0000","last_became_active":"2024-09-21T19:12:06.965006+0000","last_became_peered":"2024-09-21T19:12:06.965006+0000","last_unstale":"2024-09-21T19:12:16.944776+0000","last_undegraded":"2024-09-21T19:12:16.944776+0000","last_fullsized":"2024-09-21T19:12:16.944776+0000","mapping_epoch":414,"log_start":"433'5120","ondisk_log_start":"433'5120","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":246624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":751,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.813608+0000","last_change":"2024-09-21T19:11:41.665090+0000","last_active":"2024-09-21T19:12:09.813608+0000","last_peered":"2024-09-21T19:12:09.813608+0000","last_clean":"2024-09-21T19:12:09.813608+0000","last_became_active":"2024-09-21T18:55:51.530369+0000","last_became_peered":"2024-09-21T18:55:51.530369+0000","last_unstale":"2024-09-21T19:12:09.813608+0000","last_undegraded":"2024-09-21T19:12:09.813608+0000","last_fullsized":"2024-09-21T19:12:09.813608+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:41.665004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.665004+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.270163+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":844,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808002+0000","last_change":"2024-09-21T19:06:54.699716+0000","last_active":"2024-09-21T19:12:09.808002+0000","last_peered":"2024-09-21T19:12:09.808002+0000","last_clean":"2024-09-21T19:12:09.808002+0000","last_became_active":"2024-09-21T19:06:54.696228+0000","last_became_peered":"2024-09-21T19:06:54.696228+0000","last_unstale":"2024-09-21T19:12:09.808002+0000","last_undegraded":"2024-09-21T19:12:09.808002+0000","last_fullsized":"2024-09-21T19:12:09.808002+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:02:46.171205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:02:46.171205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":307,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811149+0000","last_change":"2024-09-21T19:10:27.397361+0000","last_active":"2024-09-21T19:12:09.811149+0000","last_peered":"2024-09-21T19:12:09.811149+0000","last_clean":"2024-09-21T19:12:09.811149+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:12:09.811149+0000","last_undegraded":"2024-09-21T19:12:09.811149+0000","last_fullsized":"2024-09-21T19:12:09.811149+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.397188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.397188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:00.641341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":754,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.889793+0000","last_change":"2024-09-21T19:11:31.588989+0000","last_active":"2024-09-21T19:12:09.889793+0000","last_peered":"2024-09-21T19:12:09.889793+0000","last_clean":"2024-09-21T19:12:09.889793+0000","last_became_active":"2024-09-21T19:05:42.902206+0000","last_became_peered":"2024-09-21T19:05:42.902206+0000","last_unstale":"2024-09-21T19:12:09.889793+0000","last_undegraded":"2024-09-21T19:12:09.889793+0000","last_fullsized":"2024-09-21T19:12:09.889793+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.588820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.588820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:42.713349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":681,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809121+0000","last_change":"2024-09-21T19:12:09.780060+0000","last_active":"2024-09-21T19:12:09.809121+0000","last_peered":"2024-09-21T19:12:09.809121+0000","last_clean":"2024-09-21T19:12:09.809121+0000","last_became_active":"2024-09-21T19:11:49.142607+0000","last_became_peered":"2024-09-21T19:11:49.142607+0000","last_unstale":"2024-09-21T19:12:09.809121+0000","last_undegraded":"2024-09-21T19:12:09.809121+0000","last_fullsized":"2024-09-21T19:12:09.809121+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:12:09.779988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:12:09.779988+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:33.221475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"460'10041","reported_seq":10154,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:16.938985+0000","last_change":"2024-09-21T19:12:06.957833+0000","last_active":"2024-09-21T19:12:16.938985+0000","last_peered":"2024-09-21T19:12:16.938985+0000","last_clean":"2024-09-21T19:12:16.938985+0000","last_became_active":"2024-09-21T19:12:06.946611+0000","last_became_peered":"2024-09-21T19:12:06.946611+0000","last_unstale":"2024-09-21T19:12:16.938985+0000","last_undegraded":"2024-09-21T19:12:16.938985+0000","last_fullsized":"2024-09-21T19:12:16.938985+0000","mapping_epoch":414,"log_start":"433'5159","ondisk_log_start":"433'5159","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164495360,"num_objects":2510,"num_object_clones":0,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5182,"num_write_kb":246144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":801,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808942+0000","last_change":"2024-09-21T19:10:12.835830+0000","last_active":"2024-09-21T19:12:09.808942+0000","last_peered":"2024-09-21T19:12:09.808942+0000","last_clean":"2024-09-21T19:12:09.808942+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:12:09.808942+0000","last_undegraded":"2024-09-21T19:12:09.808942+0000","last_fullsized":"2024-09-21T19:12:09.808942+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T19:10:12.835752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:12.835752+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:25.728578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":884,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.829340+0000","last_change":"2024-09-21T19:10:29.539254+0000","last_active":"2024-09-21T19:12:14.829340+0000","last_peered":"2024-09-21T19:12:14.829340+0000","last_clean":"2024-09-21T19:12:14.829340+0000","last_became_active":"2024-09-21T19:06:54.691565+0000","last_became_peered":"2024-09-21T19:06:54.691565+0000","last_unstale":"2024-09-21T19:12:14.829340+0000","last_undegraded":"2024-09-21T19:12:14.829340+0000","last_fullsized":"2024-09-21T19:12:14.829340+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:29.539210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:29.539210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.076755+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":317,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811283+0000","last_change":"2024-09-21T19:11:34.718818+0000","last_active":"2024-09-21T19:12:09.811283+0000","last_peered":"2024-09-21T19:12:09.811283+0000","last_clean":"2024-09-21T19:12:09.811283+0000","last_became_active":"2024-09-21T19:06:54.696825+0000","last_became_peered":"2024-09-21T19:06:54.696825+0000","last_unstale":"2024-09-21T19:12:09.811283+0000","last_undegraded":"2024-09-21T19:12:09.811283+0000","last_fullsized":"2024-09-21T19:12:09.811283+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:34.718771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.718771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:58.941434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":864,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828624+0000","last_change":"2024-09-21T19:10:58.720745+0000","last_active":"2024-09-21T19:12:14.828624+0000","last_peered":"2024-09-21T19:12:14.828624+0000","last_clean":"2024-09-21T19:12:14.828624+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:12:14.828624+0000","last_undegraded":"2024-09-21T19:12:14.828624+0000","last_fullsized":"2024-09-21T19:12:14.828624+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:58.720634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:58.720634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:23.021358+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":655,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807537+0000","last_change":"2024-09-21T19:11:49.161093+0000","last_active":"2024-09-21T19:12:09.807537+0000","last_peered":"2024-09-21T19:12:09.807537+0000","last_clean":"2024-09-21T19:12:09.807537+0000","last_became_active":"2024-09-21T19:11:49.145230+0000","last_became_peered":"2024-09-21T19:11:49.145230+0000","last_unstale":"2024-09-21T19:12:09.807537+0000","last_undegraded":"2024-09-21T19:12:09.807537+0000","last_fullsized":"2024-09-21T19:12:09.807537+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:41.967849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.967849+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:49.855850+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"459'10081","reported_seq":10228,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:12.910873+0000","last_change":"2024-09-21T19:12:10.739606+0000","last_active":"2024-09-21T19:12:12.910873+0000","last_peered":"2024-09-21T19:12:12.910873+0000","last_clean":"2024-09-21T19:12:12.910873+0000","last_became_active":"2024-09-21T19:12:06.961524+0000","last_became_peered":"2024-09-21T19:12:06.961524+0000","last_unstale":"2024-09-21T19:12:12.910873+0000","last_undegraded":"2024-09-21T19:12:12.910873+0000","last_fullsized":"2024-09-21T19:12:12.910873+0000","mapping_epoch":414,"log_start":"433'4686","ondisk_log_start":"433'4686","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'9987","last_scrub_stamp":"2024-09-21T19:12:10.739447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:12:10.739447+0000","objects_scrubbed":2876,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:37.038945+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":921,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814357+0000","last_change":"2024-09-21T19:11:31.160427+0000","last_active":"2024-09-21T19:12:14.814357+0000","last_peered":"2024-09-21T19:12:14.814357+0000","last_clean":"2024-09-21T19:12:14.814357+0000","last_became_active":"2024-09-21T19:06:54.699242+0000","last_became_peered":"2024-09-21T19:06:54.699242+0000","last_unstale":"2024-09-21T19:12:14.814357+0000","last_undegraded":"2024-09-21T19:12:14.814357+0000","last_fullsized":"2024-09-21T19:12:14.814357+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T19:11:31.160369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.160369+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:51.166557+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":886,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.829911+0000","last_change":"2024-09-21T19:10:57.731366+0000","last_active":"2024-09-21T19:12:14.829911+0000","last_peered":"2024-09-21T19:12:14.829911+0000","last_clean":"2024-09-21T19:12:14.829911+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:12:14.829911+0000","last_undegraded":"2024-09-21T19:12:14.829911+0000","last_fullsized":"2024-09-21T19:12:14.829911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:57.731319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:57.731319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":307,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809050+0000","last_change":"2024-09-21T19:11:06.454799+0000","last_active":"2024-09-21T19:12:09.809050+0000","last_peered":"2024-09-21T19:12:09.809050+0000","last_clean":"2024-09-21T19:12:09.809050+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:12:09.809050+0000","last_undegraded":"2024-09-21T19:12:09.809050+0000","last_fullsized":"2024-09-21T19:12:09.809050+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:06.454727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.454727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.197782+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":834,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.806846+0000","last_change":"2024-09-21T19:06:44.832495+0000","last_active":"2024-09-21T19:12:09.806846+0000","last_peered":"2024-09-21T19:12:09.806846+0000","last_clean":"2024-09-21T19:12:09.806846+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:12:09.806846+0000","last_undegraded":"2024-09-21T19:12:09.806846+0000","last_fullsized":"2024-09-21T19:12:09.806846+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:44.832359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:06:44.832359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:27.112801+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":30,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828316+0000","last_change":"2024-09-21T19:11:49.139840+0000","last_active":"2024-09-21T19:12:14.828316+0000","last_peered":"2024-09-21T19:12:14.828316+0000","last_clean":"2024-09-21T19:12:14.828316+0000","last_became_active":"2024-09-21T19:11:49.139511+0000","last_became_peered":"2024-09-21T19:11:49.139511+0000","last_unstale":"2024-09-21T19:12:14.828316+0000","last_undegraded":"2024-09-21T19:12:14.828316+0000","last_fullsized":"2024-09-21T19:12:14.828316+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:53.595833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.595833+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:56.049812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"459'10101","reported_seq":10319,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:12.825983+0000","last_change":"2024-09-21T19:12:09.581965+0000","last_active":"2024-09-21T19:12:12.825983+0000","last_peered":"2024-09-21T19:12:12.825983+0000","last_clean":"2024-09-21T19:12:12.825983+0000","last_became_active":"2024-09-21T19:12:06.986935+0000","last_became_peered":"2024-09-21T19:12:06.986935+0000","last_unstale":"2024-09-21T19:12:12.825983+0000","last_undegraded":"2024-09-21T19:12:12.825983+0000","last_fullsized":"2024-09-21T19:12:12.825983+0000","mapping_epoch":414,"log_start":"433'5289","ondisk_log_start":"433'5289","created":414,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9997","last_scrub_stamp":"2024-09-21T19:12:09.581880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:12:09.581880+0000","objects_scrubbed":2849,"log_size":4812,"log_dups_size":0,"ondisk_log_size":4812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:30.165410+0000","scrub_duration":338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179896320,"num_objects":2745,"num_object_clones":0,"num_object_copies":8235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":485,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828281+0000","last_change":"2024-09-21T19:11:47.023173+0000","last_active":"2024-09-21T19:12:14.828281+0000","last_peered":"2024-09-21T19:12:14.828281+0000","last_clean":"2024-09-21T19:12:14.828281+0000","last_became_active":"2024-09-21T19:06:54.689091+0000","last_became_peered":"2024-09-21T19:06:54.689091+0000","last_unstale":"2024-09-21T19:12:14.828281+0000","last_undegraded":"2024-09-21T19:12:14.828281+0000","last_fullsized":"2024-09-21T19:12:14.828281+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T19:11:47.023075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.023075+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.009824+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":900,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.832781+0000","last_change":"2024-09-21T19:11:37.790112+0000","last_active":"2024-09-21T19:12:14.832781+0000","last_peered":"2024-09-21T19:12:14.832781+0000","last_clean":"2024-09-21T19:12:14.832781+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:12:14.832781+0000","last_undegraded":"2024-09-21T19:12:14.832781+0000","last_fullsized":"2024-09-21T19:12:14.832781+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:11:37.790039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.790039+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:54.838367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":391,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807599+0000","last_change":"2024-09-21T19:10:59.827898+0000","last_active":"2024-09-21T19:12:09.807599+0000","last_peered":"2024-09-21T19:12:09.807599+0000","last_clean":"2024-09-21T19:12:09.807599+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:12:09.807599+0000","last_undegraded":"2024-09-21T19:12:09.807599+0000","last_fullsized":"2024-09-21T19:12:09.807599+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:10:59.827833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.827833+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":878,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808461+0000","last_change":"2024-09-21T19:10:59.672382+0000","last_active":"2024-09-21T19:12:09.808461+0000","last_peered":"2024-09-21T19:12:09.808461+0000","last_clean":"2024-09-21T19:12:09.808461+0000","last_became_active":"2024-09-21T19:06:54.701536+0000","last_became_peered":"2024-09-21T19:06:54.701536+0000","last_unstale":"2024-09-21T19:12:09.808461+0000","last_undegraded":"2024-09-21T19:12:09.808461+0000","last_fullsized":"2024-09-21T19:12:09.808461+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:59.672320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:59.672320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"459'9800","reported_seq":10904,"reported_epoch":459,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T19:12:13.179599+0000","last_change":"2024-09-21T19:12:09.892976+0000","last_active":"2024-09-21T19:12:13.179599+0000","last_peered":"2024-09-21T19:12:13.179599+0000","last_clean":"2024-09-21T19:12:08.383779+0000","last_became_active":"2024-09-21T19:12:09.877456+0000","last_became_peered":"2024-09-21T19:12:09.877456+0000","last_unstale":"2024-09-21T19:12:13.179599+0000","last_undegraded":"2024-09-21T19:12:13.179599+0000","last_fullsized":"2024-09-21T19:12:13.179599+0000","mapping_epoch":458,"log_start":"434'5499","ondisk_log_start":"434'5499","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173080576,"num_objects":2641,"num_object_clones":0,"num_object_copies":7923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4640,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":244160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":21626880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":2641}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":28,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810170+0000","last_change":"2024-09-21T19:11:49.147092+0000","last_active":"2024-09-21T19:12:09.810170+0000","last_peered":"2024-09-21T19:12:09.810170+0000","last_clean":"2024-09-21T19:12:09.810170+0000","last_became_active":"2024-09-21T19:11:49.146489+0000","last_became_peered":"2024-09-21T19:11:49.146489+0000","last_unstale":"2024-09-21T19:12:09.810170+0000","last_undegraded":"2024-09-21T19:12:09.810170+0000","last_fullsized":"2024-09-21T19:12:09.810170+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:09:55.484493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:09:55.484493+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:49.796560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":813,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808153+0000","last_change":"2024-09-21T19:11:08.539733+0000","last_active":"2024-09-21T19:12:09.808153+0000","last_peered":"2024-09-21T19:12:09.808153+0000","last_clean":"2024-09-21T19:12:09.808153+0000","last_became_active":"2024-09-21T19:06:54.695687+0000","last_became_peered":"2024-09-21T19:06:54.695687+0000","last_unstale":"2024-09-21T19:12:09.808153+0000","last_undegraded":"2024-09-21T19:12:09.808153+0000","last_fullsized":"2024-09-21T19:12:09.808153+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:08.539689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:08.539689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:35.735538+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":875,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809823+0000","last_change":"2024-09-21T19:10:24.873768+0000","last_active":"2024-09-21T19:12:09.809823+0000","last_peered":"2024-09-21T19:12:09.809823+0000","last_clean":"2024-09-21T19:12:09.809823+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:12:09.809823+0000","last_undegraded":"2024-09-21T19:12:09.809823+0000","last_fullsized":"2024-09-21T19:12:09.809823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T19:10:24.873701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:24.873701+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":403,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808273+0000","last_change":"2024-09-21T19:06:54.691785+0000","last_active":"2024-09-21T19:12:09.808273+0000","last_peered":"2024-09-21T19:12:09.808273+0000","last_clean":"2024-09-21T19:12:09.808273+0000","last_became_active":"2024-09-21T19:06:54.691208+0000","last_became_peered":"2024-09-21T19:06:54.691208+0000","last_unstale":"2024-09-21T19:12:09.808273+0000","last_undegraded":"2024-09-21T19:12:09.808273+0000","last_fullsized":"2024-09-21T19:12:09.808273+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:03:11.520829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:03:11.520829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.027866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":852,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828387+0000","last_change":"2024-09-21T19:10:22.627798+0000","last_active":"2024-09-21T19:12:14.828387+0000","last_peered":"2024-09-21T19:12:14.828387+0000","last_clean":"2024-09-21T19:12:14.828387+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:12:14.828387+0000","last_undegraded":"2024-09-21T19:12:14.828387+0000","last_fullsized":"2024-09-21T19:12:14.828387+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:22.627740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:22.627740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:57.096538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.19","version":"459'10042","reported_seq":10157,"reported_epoch":459,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T19:12:12.941474+0000","last_change":"2024-09-21T19:12:09.945139+0000","last_active":"2024-09-21T19:12:12.941474+0000","last_peered":"2024-09-21T19:12:12.941474+0000","last_clean":"2024-09-21T19:12:08.346293+0000","last_became_active":"2024-09-21T19:12:09.920710+0000","last_became_peered":"2024-09-21T19:12:09.920710+0000","last_unstale":"2024-09-21T19:12:12.941474+0000","last_undegraded":"2024-09-21T19:12:12.941474+0000","last_fullsized":"2024-09-21T19:12:12.941474+0000","mapping_epoch":458,"log_start":"435'5694","ondisk_log_start":"435'5694","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174555136,"num_objects":2664,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5328,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5110,"num_write_kb":248736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":2664}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":668,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.818857+0000","last_change":"2024-09-21T19:11:49.147602+0000","last_active":"2024-09-21T19:12:09.818857+0000","last_peered":"2024-09-21T19:12:09.818857+0000","last_clean":"2024-09-21T19:12:09.818857+0000","last_became_active":"2024-09-21T19:11:49.138877+0000","last_became_peered":"2024-09-21T19:11:49.138877+0000","last_unstale":"2024-09-21T19:12:09.818857+0000","last_undegraded":"2024-09-21T19:12:09.818857+0000","last_fullsized":"2024-09-21T19:12:09.818857+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:10:28.147274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:10:28.147274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:56.135180+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":750,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812196+0000","last_change":"2024-09-21T19:09:53.171148+0000","last_active":"2024-09-21T19:12:09.812196+0000","last_peered":"2024-09-21T19:12:09.812196+0000","last_clean":"2024-09-21T19:12:09.812196+0000","last_became_active":"2024-09-21T18:55:51.544363+0000","last_became_peered":"2024-09-21T18:55:51.544363+0000","last_unstale":"2024-09-21T19:12:09.812196+0000","last_undegraded":"2024-09-21T19:12:09.812196+0000","last_fullsized":"2024-09-21T19:12:09.812196+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:09:53.171076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:09:53.171076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:12.467919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":873,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828706+0000","last_change":"2024-09-21T19:11:44.883032+0000","last_active":"2024-09-21T19:12:14.828706+0000","last_peered":"2024-09-21T19:12:14.828706+0000","last_clean":"2024-09-21T19:12:14.828706+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:12:14.828706+0000","last_undegraded":"2024-09-21T19:12:14.828706+0000","last_fullsized":"2024-09-21T19:12:14.828706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:44.882985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.882985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:07.884044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":845,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809653+0000","last_change":"2024-09-21T19:10:46.660901+0000","last_active":"2024-09-21T19:12:09.809653+0000","last_peered":"2024-09-21T19:12:09.809653+0000","last_clean":"2024-09-21T19:12:09.809653+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:12:09.809653+0000","last_undegraded":"2024-09-21T19:12:09.809653+0000","last_fullsized":"2024-09-21T19:12:09.809653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.660838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.660838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":306,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811991+0000","last_change":"2024-09-21T19:11:43.659822+0000","last_active":"2024-09-21T19:12:09.811991+0000","last_peered":"2024-09-21T19:12:09.811991+0000","last_clean":"2024-09-21T19:12:09.811991+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:12:09.811991+0000","last_undegraded":"2024-09-21T19:12:09.811991+0000","last_fullsized":"2024-09-21T19:12:09.811991+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:43.659592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:43.659592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:49.689636+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":978,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.827994+0000","last_change":"2024-09-21T19:11:49.134584+0000","last_active":"2024-09-21T19:12:14.827994+0000","last_peered":"2024-09-21T19:12:14.827994+0000","last_clean":"2024-09-21T19:12:14.827994+0000","last_became_active":"2024-09-21T19:11:49.133497+0000","last_became_peered":"2024-09-21T19:11:49.133497+0000","last_unstale":"2024-09-21T19:12:14.827994+0000","last_undegraded":"2024-09-21T19:12:14.827994+0000","last_fullsized":"2024-09-21T19:12:14.827994+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:45.911119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.911119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:54.009753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":466,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828599+0000","last_change":"2024-09-21T19:11:34.827737+0000","last_active":"2024-09-21T19:12:14.828599+0000","last_peered":"2024-09-21T19:12:14.828599+0000","last_clean":"2024-09-21T19:12:14.828599+0000","last_became_active":"2024-09-21T19:06:54.691982+0000","last_became_peered":"2024-09-21T19:06:54.691982+0000","last_unstale":"2024-09-21T19:12:14.828599+0000","last_undegraded":"2024-09-21T19:12:14.828599+0000","last_fullsized":"2024-09-21T19:12:14.828599+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:34.827659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:34.827659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.075348+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":748,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.877326+0000","last_change":"2024-09-21T19:11:40.646183+0000","last_active":"2024-09-21T19:12:09.877326+0000","last_peered":"2024-09-21T19:12:09.877326+0000","last_clean":"2024-09-21T19:12:09.877326+0000","last_became_active":"2024-09-21T18:55:51.522731+0000","last_became_peered":"2024-09-21T18:55:51.522731+0000","last_unstale":"2024-09-21T19:12:09.877326+0000","last_undegraded":"2024-09-21T19:12:09.877326+0000","last_fullsized":"2024-09-21T19:12:09.877326+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:40.645904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:40.645904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:44.505399+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1107,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807438+0000","last_change":"2024-09-21T19:09:52.767913+0000","last_active":"2024-09-21T19:12:09.807438+0000","last_peered":"2024-09-21T19:12:09.807438+0000","last_clean":"2024-09-21T19:12:09.807438+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:12:09.807438+0000","last_undegraded":"2024-09-21T19:12:09.807438+0000","last_fullsized":"2024-09-21T19:12:09.807438+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:09:52.767831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:09:52.767831+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":867,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814979+0000","last_change":"2024-09-21T19:09:51.533226+0000","last_active":"2024-09-21T19:12:14.814979+0000","last_peered":"2024-09-21T19:12:14.814979+0000","last_clean":"2024-09-21T19:12:14.814979+0000","last_became_active":"2024-09-21T19:06:54.699037+0000","last_became_peered":"2024-09-21T19:06:54.699037+0000","last_unstale":"2024-09-21T19:12:14.814979+0000","last_undegraded":"2024-09-21T19:12:14.814979+0000","last_fullsized":"2024-09-21T19:12:14.814979+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:51.533169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:51.533169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:29.127237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":634,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.889432+0000","last_change":"2024-09-21T19:11:49.151010+0000","last_active":"2024-09-21T19:12:09.889432+0000","last_peered":"2024-09-21T19:12:09.889432+0000","last_clean":"2024-09-21T19:12:09.889432+0000","last_became_active":"2024-09-21T19:11:49.149051+0000","last_became_peered":"2024-09-21T19:11:49.149051+0000","last_unstale":"2024-09-21T19:12:09.889432+0000","last_undegraded":"2024-09-21T19:12:09.889432+0000","last_fullsized":"2024-09-21T19:12:09.889432+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:47.627361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:47.627361+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:04.786190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":891,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807781+0000","last_change":"2024-09-21T19:10:39.003954+0000","last_active":"2024-09-21T19:12:09.807781+0000","last_peered":"2024-09-21T19:12:09.807781+0000","last_clean":"2024-09-21T19:12:09.807781+0000","last_became_active":"2024-09-21T19:10:38.923267+0000","last_became_peered":"2024-09-21T19:10:38.923267+0000","last_unstale":"2024-09-21T19:12:09.807781+0000","last_undegraded":"2024-09-21T19:12:09.807781+0000","last_fullsized":"2024-09-21T19:12:09.807781+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T19:10:19.876033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:19.876033+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":876,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.829488+0000","last_change":"2024-09-21T19:10:46.686915+0000","last_active":"2024-09-21T19:12:14.829488+0000","last_peered":"2024-09-21T19:12:14.829488+0000","last_clean":"2024-09-21T19:12:14.829488+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:12:14.829488+0000","last_undegraded":"2024-09-21T19:12:14.829488+0000","last_fullsized":"2024-09-21T19:12:14.829488+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:46.686843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:46.686843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"350'13","reported_seq":564,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812287+0000","last_change":"2024-09-21T19:11:42.105123+0000","last_active":"2024-09-21T19:12:09.812287+0000","last_peered":"2024-09-21T19:12:09.812287+0000","last_clean":"2024-09-21T19:12:09.812287+0000","last_became_active":"2024-09-21T19:06:54.702611+0000","last_became_peered":"2024-09-21T19:06:54.702611+0000","last_unstale":"2024-09-21T19:12:09.812287+0000","last_undegraded":"2024-09-21T19:12:09.812287+0000","last_fullsized":"2024-09-21T19:12:09.812287+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'13","last_scrub_stamp":"2024-09-21T19:11:42.105053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:42.105053+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:56.107355+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":739,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.813069+0000","last_change":"2024-09-21T19:09:56.159164+0000","last_active":"2024-09-21T19:12:09.813069+0000","last_peered":"2024-09-21T19:12:09.813069+0000","last_clean":"2024-09-21T19:12:09.813069+0000","last_became_active":"2024-09-21T18:55:51.546169+0000","last_became_peered":"2024-09-21T18:55:51.546169+0000","last_unstale":"2024-09-21T19:12:09.813069+0000","last_undegraded":"2024-09-21T19:12:09.813069+0000","last_fullsized":"2024-09-21T19:12:09.813069+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:09:56.159116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:09:56.159116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.496959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":947,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.814882+0000","last_change":"2024-09-21T19:11:41.105439+0000","last_active":"2024-09-21T19:12:14.814882+0000","last_peered":"2024-09-21T19:12:14.814882+0000","last_clean":"2024-09-21T19:12:14.814882+0000","last_became_active":"2024-09-21T19:05:00.729609+0000","last_became_peered":"2024-09-21T19:05:00.729609+0000","last_unstale":"2024-09-21T19:12:14.814882+0000","last_undegraded":"2024-09-21T19:12:14.814882+0000","last_fullsized":"2024-09-21T19:12:14.814882+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:41.105362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.105362+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:43.066173+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":858,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.815233+0000","last_change":"2024-09-21T19:04:21.869466+0000","last_active":"2024-09-21T19:12:14.815233+0000","last_peered":"2024-09-21T19:12:14.815233+0000","last_clean":"2024-09-21T19:12:14.815233+0000","last_became_active":"2024-09-21T19:02:40.876345+0000","last_became_peered":"2024-09-21T19:02:40.876345+0000","last_unstale":"2024-09-21T19:12:14.815233+0000","last_undegraded":"2024-09-21T19:12:14.815233+0000","last_fullsized":"2024-09-21T19:12:14.815233+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:04:21.869414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:04:21.869414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":872,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812653+0000","last_change":"2024-09-21T19:11:41.125672+0000","last_active":"2024-09-21T19:12:09.812653+0000","last_peered":"2024-09-21T19:12:09.812653+0000","last_clean":"2024-09-21T19:12:09.812653+0000","last_became_active":"2024-09-21T19:06:54.699637+0000","last_became_peered":"2024-09-21T19:06:54.699637+0000","last_unstale":"2024-09-21T19:12:09.812653+0000","last_undegraded":"2024-09-21T19:12:09.812653+0000","last_fullsized":"2024-09-21T19:12:09.812653+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:41.125603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.125603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:47.999077+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":867,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807183+0000","last_change":"2024-09-21T19:10:27.770820+0000","last_active":"2024-09-21T19:12:09.807183+0000","last_peered":"2024-09-21T19:12:09.807183+0000","last_clean":"2024-09-21T19:12:09.807183+0000","last_became_active":"2024-09-21T19:06:54.698820+0000","last_became_peered":"2024-09-21T19:06:54.698820+0000","last_unstale":"2024-09-21T19:12:09.807183+0000","last_undegraded":"2024-09-21T19:12:09.807183+0000","last_fullsized":"2024-09-21T19:12:09.807183+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:27.770751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.770751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":943,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.814594+0000","last_change":"2024-09-21T19:11:06.137061+0000","last_active":"2024-09-21T19:12:09.814594+0000","last_peered":"2024-09-21T19:12:09.814594+0000","last_clean":"2024-09-21T19:12:09.814594+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:12:09.814594+0000","last_undegraded":"2024-09-21T19:12:09.814594+0000","last_fullsized":"2024-09-21T19:12:09.814594+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:06.136973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:06.136973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:11.021915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":869,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828640+0000","last_change":"2024-09-21T19:11:07.755091+0000","last_active":"2024-09-21T19:12:14.828640+0000","last_peered":"2024-09-21T19:12:14.828640+0000","last_clean":"2024-09-21T19:12:14.828640+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:12:14.828640+0000","last_undegraded":"2024-09-21T19:12:14.828640+0000","last_fullsized":"2024-09-21T19:12:14.828640+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:07.755026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:07.755026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.561489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":860,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.806901+0000","last_change":"2024-09-21T19:06:15.957106+0000","last_active":"2024-09-21T19:12:09.806901+0000","last_peered":"2024-09-21T19:12:09.806901+0000","last_clean":"2024-09-21T19:12:09.806901+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:12:09.806901+0000","last_undegraded":"2024-09-21T19:12:09.806901+0000","last_fullsized":"2024-09-21T19:12:09.806901+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:15.957028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:06:15.957028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:26.155553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":759,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811993+0000","last_change":"2024-09-21T19:11:36.632665+0000","last_active":"2024-09-21T19:12:09.811993+0000","last_peered":"2024-09-21T19:12:09.811993+0000","last_clean":"2024-09-21T19:12:09.811993+0000","last_became_active":"2024-09-21T19:06:54.695108+0000","last_became_peered":"2024-09-21T19:06:54.695108+0000","last_unstale":"2024-09-21T19:12:09.811993+0000","last_undegraded":"2024-09-21T19:12:09.811993+0000","last_fullsized":"2024-09-21T19:12:09.811993+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:36.632596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:36.632596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.311366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":28,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.811399+0000","last_change":"2024-09-21T19:11:49.141039+0000","last_active":"2024-09-21T19:12:09.811399+0000","last_peered":"2024-09-21T19:12:09.811399+0000","last_clean":"2024-09-21T19:12:09.811399+0000","last_became_active":"2024-09-21T19:11:49.139887+0000","last_became_peered":"2024-09-21T19:11:49.139887+0000","last_unstale":"2024-09-21T19:12:09.811399+0000","last_undegraded":"2024-09-21T19:12:09.811399+0000","last_fullsized":"2024-09-21T19:12:09.811399+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:41.967849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.967849+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T19:12:50.474975+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":892,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.806977+0000","last_change":"2024-09-21T19:11:37.973551+0000","last_active":"2024-09-21T19:12:09.806977+0000","last_peered":"2024-09-21T19:12:09.806977+0000","last_clean":"2024-09-21T19:12:09.806977+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:12:09.806977+0000","last_undegraded":"2024-09-21T19:12:09.806977+0000","last_fullsized":"2024-09-21T19:12:09.806977+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:37.973476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:37.973476+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:59.521765+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":805,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.810506+0000","last_change":"2024-09-21T19:11:03.004430+0000","last_active":"2024-09-21T19:12:09.810506+0000","last_peered":"2024-09-21T19:12:09.810506+0000","last_clean":"2024-09-21T19:12:09.810506+0000","last_became_active":"2024-09-21T19:06:54.691694+0000","last_became_peered":"2024-09-21T19:06:54.691694+0000","last_unstale":"2024-09-21T19:12:09.810506+0000","last_undegraded":"2024-09-21T19:12:09.810506+0000","last_fullsized":"2024-09-21T19:12:09.810506+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:03.004386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:03.004386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:28.746086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":889,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828138+0000","last_change":"2024-09-21T19:11:01.724461+0000","last_active":"2024-09-21T19:12:14.828138+0000","last_peered":"2024-09-21T19:12:14.828138+0000","last_clean":"2024-09-21T19:12:14.828138+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:12:14.828138+0000","last_undegraded":"2024-09-21T19:12:14.828138+0000","last_fullsized":"2024-09-21T19:12:14.828138+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:01.724417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:11:01.724417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":867,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807087+0000","last_change":"2024-09-21T19:10:16.861750+0000","last_active":"2024-09-21T19:12:09.807087+0000","last_peered":"2024-09-21T19:12:09.807087+0000","last_clean":"2024-09-21T19:12:09.807087+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:12:09.807087+0000","last_undegraded":"2024-09-21T19:12:09.807087+0000","last_fullsized":"2024-09-21T19:12:09.807087+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:16.861671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:16.861671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":891,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.807910+0000","last_change":"2024-09-21T19:10:52.788414+0000","last_active":"2024-09-21T19:12:09.807910+0000","last_peered":"2024-09-21T19:12:09.807910+0000","last_clean":"2024-09-21T19:12:09.807910+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:12:09.807910+0000","last_undegraded":"2024-09-21T19:12:09.807910+0000","last_fullsized":"2024-09-21T19:12:09.807910+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T19:10:52.788342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:10:52.788342+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":742,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812498+0000","last_change":"2024-09-21T19:11:02.421016+0000","last_active":"2024-09-21T19:12:09.812498+0000","last_peered":"2024-09-21T19:12:09.812498+0000","last_clean":"2024-09-21T19:12:09.812498+0000","last_became_active":"2024-09-21T19:06:54.701787+0000","last_became_peered":"2024-09-21T19:06:54.701787+0000","last_unstale":"2024-09-21T19:12:09.812498+0000","last_undegraded":"2024-09-21T19:12:09.812498+0000","last_fullsized":"2024-09-21T19:12:09.812498+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:02.420940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.420940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:18.532139+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":772,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.808284+0000","last_change":"2024-09-21T19:05:33.720677+0000","last_active":"2024-09-21T19:12:09.808284+0000","last_peered":"2024-09-21T19:12:09.808284+0000","last_clean":"2024-09-21T19:12:09.808284+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:12:09.808284+0000","last_undegraded":"2024-09-21T19:12:09.808284+0000","last_fullsized":"2024-09-21T19:12:09.808284+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:05:33.720610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:05:33.720610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:16.662425+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":864,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-21T19:12:14.828269+0000","last_change":"2024-09-21T19:11:09.805596+0000","last_active":"2024-09-21T19:12:14.828269+0000","last_peered":"2024-09-21T19:12:14.828269+0000","last_clean":"2024-09-21T19:12:14.828269+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:12:14.828269+0000","last_undegraded":"2024-09-21T19:12:14.828269+0000","last_fullsized":"2024-09-21T19:12:14.828269+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:09.805527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:09.805527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:24.581512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"459'9869","reported_seq":10006,"reported_epoch":459,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T19:12:12.715966+0000","last_change":"2024-09-21T19:12:09.992655+0000","last_active":"2024-09-21T19:12:12.715966+0000","last_peered":"2024-09-21T19:12:12.715966+0000","last_clean":"2024-09-21T19:12:08.345814+0000","last_became_active":"2024-09-21T19:12:09.900044+0000","last_became_peered":"2024-09-21T19:12:09.900044+0000","last_unstale":"2024-09-21T19:12:12.715966+0000","last_undegraded":"2024-09-21T19:12:12.715966+0000","last_fullsized":"2024-09-21T19:12:12.715966+0000","mapping_epoch":458,"log_start":"433'5092","ondisk_log_start":"433'5092","created":455,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T19:10:00.224748+0000","last_clean_scrub_stamp":"2024-09-21T19:10:00.224748+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172392448,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7887,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5015,"num_write_kb":244640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":2631}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":867,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812099+0000","last_change":"2024-09-21T19:11:40.130856+0000","last_active":"2024-09-21T19:12:09.812099+0000","last_peered":"2024-09-21T19:12:09.812099+0000","last_clean":"2024-09-21T19:12:09.812099+0000","last_became_active":"2024-09-21T19:06:54.700439+0000","last_became_peered":"2024-09-21T19:06:54.700439+0000","last_unstale":"2024-09-21T19:12:09.812099+0000","last_undegraded":"2024-09-21T19:12:09.812099+0000","last_fullsized":"2024-09-21T19:12:09.812099+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:40.130789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:40.130789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:13:02.390303+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":875,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809450+0000","last_change":"2024-09-21T19:06:51.874035+0000","last_active":"2024-09-21T19:12:09.809450+0000","last_peered":"2024-09-21T19:12:09.809450+0000","last_clean":"2024-09-21T19:12:09.809450+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:12:09.809450+0000","last_undegraded":"2024-09-21T19:12:09.809450+0000","last_fullsized":"2024-09-21T19:12:09.809450+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:06:51.873964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:06:51.873964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.009111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":853,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.809127+0000","last_change":"2024-09-21T19:11:00.629261+0000","last_active":"2024-09-21T19:12:09.809127+0000","last_peered":"2024-09-21T19:12:09.809127+0000","last_clean":"2024-09-21T19:12:09.809127+0000","last_became_active":"2024-09-21T19:06:54.693853+0000","last_became_peered":"2024-09-21T19:06:54.693853+0000","last_unstale":"2024-09-21T19:12:09.809127+0000","last_undegraded":"2024-09-21T19:12:09.809127+0000","last_fullsized":"2024-09-21T19:12:09.809127+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T19:11:00.629191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:00.629191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:30.169952+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":763,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-21T19:12:09.812384+0000","last_change":"2024-09-21T19:10:23.500631+0000","last_active":"2024-09-21T19:12:09.812384+0000","last_peered":"2024-09-21T19:12:09.812384+0000","last_clean":"2024-09-21T19:12:09.812384+0000","last_became_active":"2024-09-21T19:05:42.904118+0000","last_became_peered":"2024-09-21T19:05:42.904118+0000","last_unstale":"2024-09-21T19:12:09.812384+0000","last_undegraded":"2024-09-21T19:12:09.812384+0000","last_fullsized":"2024-09-21T19:12:09.812384+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:23.500562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:10:23.500562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:12:55.701008+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":5525241880,"num_objects":84310,"num_object_clones":0,"num_object_copies":252930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57251,"num_objects_unfound":0,"num_objects_dirty":84310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138378,"num_write_kb":6679169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":130875392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16855674880,"data_stored":16855662664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125101,"ondisk_log_size":125101,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":5446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1137,"num_read_kb":1132,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":534812,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":22,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":697708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266922,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2777392,"kb_used_data":2042336,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":90967760,"statfs":{"total":95995035648,"available":93150986240,"internally_reserved":0,"allocated":2091352064,"data_stored":2090899706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464372,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3157848,"kb_used_data":2698184,"kb_used_omap":4,"kb_used_meta":459579,"kb_avail":90587304,"statfs":{"total":95995035648,"available":92761399296,"internally_reserved":0,"allocated":2762940416,"data_stored":2762504707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":470609572},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619073,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1726684,"kb_used_data":1388684,"kb_used_omap":4,"kb_used_meta":337915,"kb_avail":92018468,"statfs":{"total":95995035648,"available":94226911232,"internally_reserved":0,"allocated":1422012416,"data_stored":1421573261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":346025635},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019128,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031168,"kb_used_data":1426604,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91713984,"statfs":{"total":95995035648,"available":93915119616,"internally_reserved":0,"allocated":1460842496,"data_stored":1460378374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182652,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3222204,"kb_used_data":2370540,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":90522948,"statfs":{"total":95995035648,"available":92695498752,"internally_reserved":0,"allocated":2427432960,"data_stored":2426962355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":872085962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411586,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2761428,"kb_used_data":2090116,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":90983724,"statfs":{"total":95995035648,"available":93167333376,"internally_reserved":0,"allocated":2140278784,"data_stored":2139847850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575110,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3602568,"kb_used_data":2777272,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":90142584,"statfs":{"total":95995035648,"available":92306006016,"internally_reserved":0,"allocated":2843926528,"data_stored":2843443195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738636,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2633064,"kb_used_data":1731352,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91112088,"statfs":{"total":95995035648,"available":93298778112,"internally_reserved":0,"allocated":1772904448,"data_stored":1772453669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1768882176,"data_stored":1768882176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2829385728,"data_stored":2829385728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2130382848,"data_stored":2130378776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2422734848,"data_stored":2422734848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1459294208,"data_stored":1459290136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1414660096,"data_stored":1414660096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2740195328,"data_stored":2740191256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2090139648,"data_stored":2090139648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:12:21.145 INFO:tasks.thrashosds.thrasher:Setting 1.4 to [4, 3, 1, 2, 7, 6] 2024-09-21T19:12:21.145 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.4', '4', '3', '1', '2', '7', '6'] 2024-09-21T19:12:21.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 1.4 4 3 1 2 7 6 2024-09-21T19:12:21.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.237 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:21.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:20 smithi104 ceph-mon[32438]: from='client.36698 v1:172.21.15.81:0/3553364754' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:12:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:20 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:20 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:20 smithi104 ceph-mon[32438]: pgmap v953: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 47938/239481 objects misplaced (20.017%); 38 MiB/s, 603 objects/s recovering 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[39192]: from='client.36698 v1:172.21.15.81:0/3553364754' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[39192]: pgmap v953: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 47938/239481 objects misplaced (20.017%); 38 MiB/s, 603 objects/s recovering 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[29384]: from='client.36698 v1:172.21.15.81:0/3553364754' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:21.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:20 smithi081 ceph-mon[29384]: pgmap v953: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 47938/239481 objects misplaced (20.017%); 38 MiB/s, 603 objects/s recovering 2024-09-21T19:12:21.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.386 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:21.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:21.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:21.540 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:21.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:21.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:21.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:21.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:21.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:21.944 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:12:21.944 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:12:21.990 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:21.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:22.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:22.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:22.212 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:22.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:22.277 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:22.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:22.517 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:22.556 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:22.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.695 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:22.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:22.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:22.832 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:22.899 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T19:12:22.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:22.994 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:22.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:23.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.155 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:23.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:23.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:23.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:22 smithi104 ceph-mon[32438]: pgmap v954: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 47938/238428 objects misplaced (20.106%); 30 MiB/s, 473 objects/s recovering 2024-09-21T19:12:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.305 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:23.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:23.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:22 smithi081 ceph-mon[39192]: pgmap v954: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 47938/238428 objects misplaced (20.106%); 30 MiB/s, 473 objects/s recovering 2024-09-21T19:12:23.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:23.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:22 smithi081 ceph-mon[29384]: pgmap v954: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 47938/238428 objects misplaced (20.106%); 30 MiB/s, 473 objects/s recovering 2024-09-21T19:12:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:23.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:23.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:23.441 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:23.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:23.572 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:23.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:23.710 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:23.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.849 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:23.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:23.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:23.979 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:23.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:24.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:24.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:24.217 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:24.221 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:24.271 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:24.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.419 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:24.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:24.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:24.560 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:24.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:24.697 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:24.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:24.816 INFO:teuthology.orchestra.run.smithi081.stderr:set 1.4 pg_upmap_items mapping to [4->3,1->2,7->6] 2024-09-21T19:12:24.829 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:24.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:24.962 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:24.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:25.064 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:25.072 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:12:25.072 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:12:25.092 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:25.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.228 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:25.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:25.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:24 smithi104 ceph-mon[32438]: pgmap v955: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 42458/224451 objects misplaced (18.916%); 31 MiB/s, 494 objects/s recovering 2024-09-21T19:12:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/934407757' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.4", "id": [4, 3, 1, 2, 7, 6]} : dispatch 2024-09-21T19:12:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/934407757' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.4", "id": [4, 3, 1, 2, 7, 6]}]': finished 2024-09-21T19:12:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:24 smithi104 ceph-mon[32438]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T19:12:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[29384]: pgmap v955: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 42458/224451 objects misplaced (18.916%); 31 MiB/s, 494 objects/s recovering 2024-09-21T19:12:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/934407757' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.4", "id": [4, 3, 1, 2, 7, 6]} : dispatch 2024-09-21T19:12:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/934407757' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.4", "id": [4, 3, 1, 2, 7, 6]}]': finished 2024-09-21T19:12:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[29384]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T19:12:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[39192]: pgmap v955: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 42458/224451 objects misplaced (18.916%); 31 MiB/s, 494 objects/s recovering 2024-09-21T19:12:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/934407757' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.4", "id": [4, 3, 1, 2, 7, 6]} : dispatch 2024-09-21T19:12:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/934407757' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.4", "id": [4, 3, 1, 2, 7, 6]}]': finished 2024-09-21T19:12:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:24 smithi081 ceph-mon[39192]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T19:12:25.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:25.359 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:25.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:25.495 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:25.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:12:25.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:25.651 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:25.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:25.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:25.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:26.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:26.090 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:26.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:26.227 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:26.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:26.349 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:26.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:26.483 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:26.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:26.710 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:26.751 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:26.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:26.886 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:26.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:26.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:27.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:27.130 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.171 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:27.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:26 smithi081 ceph-mon[39192]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T19:12:27.208 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:26 smithi081 ceph-mon[39192]: pgmap v958: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 845 B/s rd, 0 B/s wr, 950 op/s; 39234/222228 objects misplaced (17.655%); 29 MiB/s, 456 objects/s recovering 2024-09-21T19:12:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:26 smithi081 ceph-mon[29384]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T19:12:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:26 smithi081 ceph-mon[29384]: pgmap v958: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 845 B/s rd, 0 B/s wr, 950 op/s; 39234/222228 objects misplaced (17.655%); 29 MiB/s, 456 objects/s recovering 2024-09-21T19:12:27.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:26 smithi104 ceph-mon[32438]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T19:12:27.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:26 smithi104 ceph-mon[32438]: pgmap v958: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 845 B/s rd, 0 B/s wr, 950 op/s; 39234/222228 objects misplaced (17.655%); 29 MiB/s, 456 objects/s recovering 2024-09-21T19:12:27.272 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:27.296 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:27.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:27.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:27.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:12:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:12:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:12:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:27.680 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:27.710 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:27.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:27.852 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:27.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:27.983 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:27.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:28.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:28.114 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:28.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:28.242 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:28.344 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:28.371 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:28.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:28.495 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:28.597 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:28.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:28.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:28.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:28.885 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:28.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:28.987 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.018 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:29.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:29.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:28 smithi081 ceph-mon[39192]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T19:12:29.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:28 smithi081 ceph-mon[39192]: pgmap v960: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.09k op/s; 39234/218850 objects misplaced (17.927%); 29 MiB/s, 459 objects/s recovering 2024-09-21T19:12:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:28 smithi081 ceph-mon[29384]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T19:12:29.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:28 smithi081 ceph-mon[29384]: pgmap v960: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.09k op/s; 39234/218850 objects misplaced (17.927%); 29 MiB/s, 459 objects/s recovering 2024-09-21T19:12:29.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:29.162 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:29.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:28 smithi104 ceph-mon[32438]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T19:12:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:28 smithi104 ceph-mon[32438]: pgmap v960: 178 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.09k op/s; 39234/218850 objects misplaced (17.927%); 29 MiB/s, 459 objects/s recovering 2024-09-21T19:12:29.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:12:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:12:29.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:29.300 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:29.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:29.437 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:29.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:29.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:29.686 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:29.719 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:29.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:29.858 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:29.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:29.988 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:29.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:30.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:30.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:30.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:30.239 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:29 smithi104 ceph-mon[32438]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T19:12:30.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:29 smithi104 ceph-mon[32438]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T19:12:30.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:29 smithi081 ceph-mon[29384]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T19:12:30.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:29 smithi081 ceph-mon[29384]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T19:12:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:29 smithi081 ceph-mon[39192]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T19:12:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:29 smithi081 ceph-mon[39192]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T19:12:30.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:30.363 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:30.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:30.486 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:30.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.610 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:30.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:30.711 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:30.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.863 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:30.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:30.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:30.996 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:30.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:31.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:31.119 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:31.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:31.242 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:30 smithi104 ceph-mon[32438]: pgmap v963: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 30808/205827 objects misplaced (14.968%); 37 MiB/s, 590 objects/s recovering 2024-09-21T19:12:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:30 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T19:12:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:30 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T19:12:31.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:30 smithi104 ceph-mon[32438]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T19:12:31.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[29384]: pgmap v963: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 30808/205827 objects misplaced (14.968%); 37 MiB/s, 590 objects/s recovering 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[29384]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[39192]: pgmap v963: 178 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 168 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 30808/205827 objects misplaced (14.968%); 37 MiB/s, 590 objects/s recovering 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T19:12:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:30 smithi081 ceph-mon[39192]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T19:12:31.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:31.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:31.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:31.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:31.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:31.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:31.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:31.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:31.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:32.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.138 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:32.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:32.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:31 smithi081 ceph-mon[29384]: 11.4 scrub starts 2024-09-21T19:12:32.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:31 smithi081 ceph-mon[29384]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T19:12:32.221 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:31 smithi081 ceph-mon[39192]: 11.4 scrub starts 2024-09-21T19:12:32.221 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:31 smithi081 ceph-mon[39192]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T19:12:32.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:32.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:31 smithi104 ceph-mon[32438]: 11.4 scrub starts 2024-09-21T19:12:32.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:31 smithi104 ceph-mon[32438]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T19:12:32.267 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:32.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:32.405 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:32.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:32.535 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:32.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:32.676 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:32.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:32.814 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:32.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:32.949 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:32.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:33.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:33.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:33.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:33.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:33.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:32 smithi104 ceph-mon[32438]: 11.4 scrub ok 2024-09-21T19:12:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:32 smithi104 ceph-mon[32438]: 11.1 scrub starts 2024-09-21T19:12:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:32 smithi104 ceph-mon[32438]: pgmap v966: 178 pgs: 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 169 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 30808/205032 objects misplaced (15.026%); 37 MiB/s, 596 objects/s recovering 2024-09-21T19:12:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:32 smithi104 ceph-mon[32438]: 11.9 scrub starts 2024-09-21T19:12:33.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:32 smithi104 ceph-mon[32438]: 11.9 scrub ok 2024-09-21T19:12:33.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:33.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[39192]: 11.4 scrub ok 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[39192]: 11.1 scrub starts 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[39192]: pgmap v966: 178 pgs: 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 169 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 30808/205032 objects misplaced (15.026%); 37 MiB/s, 596 objects/s recovering 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[39192]: 11.9 scrub starts 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[39192]: 11.9 scrub ok 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[29384]: 11.4 scrub ok 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[29384]: 11.1 scrub starts 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[29384]: pgmap v966: 178 pgs: 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 169 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 30808/205032 objects misplaced (15.026%); 37 MiB/s, 596 objects/s recovering 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[29384]: 11.9 scrub starts 2024-09-21T19:12:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:32 smithi081 ceph-mon[29384]: 11.9 scrub ok 2024-09-21T19:12:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.341 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:33.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:33.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:33.479 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:33.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:33.614 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:33.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:33.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:33.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:33.887 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:33.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:33.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:34.009 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:34.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:34.148 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:34.250 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:33 smithi104 ceph-mon[32438]: 11.1 scrub ok 2024-09-21T19:12:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:33 smithi104 ceph-mon[32438]: 11.3 scrub starts 2024-09-21T19:12:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:33 smithi104 ceph-mon[32438]: 11.5 scrub starts 2024-09-21T19:12:34.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:33 smithi104 ceph-mon[32438]: 11.13 scrub starts 2024-09-21T19:12:34.272 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[39192]: 11.1 scrub ok 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[39192]: 11.3 scrub starts 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[39192]: 11.5 scrub starts 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[39192]: 11.13 scrub starts 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[29384]: 11.1 scrub ok 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[29384]: 11.3 scrub starts 2024-09-21T19:12:34.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[29384]: 11.5 scrub starts 2024-09-21T19:12:34.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:33 smithi081 ceph-mon[29384]: 11.13 scrub starts 2024-09-21T19:12:34.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:34.411 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:34.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:34.549 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:34.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.688 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:34.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:34.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:34.817 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:34.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:34.956 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:34.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:35.058 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:35.082 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:35.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:35.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 11.3 scrub ok 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 11.5 scrub ok 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 11.13 scrub ok 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: pgmap v967: 178 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 172 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s; 26466/196578 objects misplaced (13.463%); 41 MiB/s, 653 objects/s recovering 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 3.c scrub starts 2024-09-21T19:12:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 3.c scrub ok 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 4.1a scrub starts 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: 4.1a scrub ok 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:35.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:34 smithi104 ceph-mon[32438]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T19:12:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 11.3 scrub ok 2024-09-21T19:12:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 11.5 scrub ok 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 11.13 scrub ok 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: pgmap v967: 178 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 172 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s; 26466/196578 objects misplaced (13.463%); 41 MiB/s, 653 objects/s recovering 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 3.c scrub starts 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 3.c scrub ok 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 4.1a scrub starts 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: 4.1a scrub ok 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[29384]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T19:12:35.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 11.3 scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 11.5 scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 11.13 scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: pgmap v967: 178 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 172 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s; 26466/196578 objects misplaced (13.463%); 41 MiB/s, 653 objects/s recovering 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 3.c scrub starts 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 3.c scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 4.1a scrub starts 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: 4.1a scrub ok 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:35.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:35.319 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:34 smithi081 ceph-mon[39192]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T19:12:35.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:35.346 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:35.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:35.487 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:35.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:35.621 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:35.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:35.768 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:35.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:35.906 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:35.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:36.008 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:36.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:36.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:36.167 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:36.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 11.11 scrub starts 2024-09-21T19:12:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 11.14 scrub starts 2024-09-21T19:12:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 11.12 scrub starts 2024-09-21T19:12:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 3.14 scrub starts 2024-09-21T19:12:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 3.14 scrub ok 2024-09-21T19:12:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:12:36.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:35 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:12:36.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:36.289 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:36.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 11.11 scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 11.14 scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 11.12 scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 3.14 scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 3.14 scrub ok 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 11.11 scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 11.14 scrub starts 2024-09-21T19:12:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 11.12 scrub starts 2024-09-21T19:12:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 3.14 scrub starts 2024-09-21T19:12:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 3.14 scrub ok 2024-09-21T19:12:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:12:36.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:35 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:12:36.391 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:36.427 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:36.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:36.557 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:36.659 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:36.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:36.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:36.815 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:36.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:36.948 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:36.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:36.984 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:12:37.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:37.087 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:37.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:37.204 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: 11.11 scrub ok 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: 11.14 scrub ok 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: 11.18 scrub starts 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: 11.12 scrub ok 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: pgmap v970: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s; 26346/195414 objects misplaced (13.482%); 23 MiB/s, 362 objects/s recovering 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1798811999' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[39192]: from='client.26545 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: 11.11 scrub ok 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: 11.14 scrub ok 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: 11.18 scrub starts 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: 11.12 scrub ok 2024-09-21T19:12:37.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: pgmap v970: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s; 26346/195414 objects misplaced (13.482%); 23 MiB/s, 362 objects/s recovering 2024-09-21T19:12:37.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1798811999' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:12:37.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:36 smithi081 ceph-mon[29384]: from='client.26545 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:12:37.210 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:37.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: 11.11 scrub ok 2024-09-21T19:12:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: 11.14 scrub ok 2024-09-21T19:12:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: 11.18 scrub starts 2024-09-21T19:12:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: 11.12 scrub ok 2024-09-21T19:12:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: pgmap v970: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s; 26346/195414 objects misplaced (13.482%); 23 MiB/s, 362 objects/s recovering 2024-09-21T19:12:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1798811999' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:12:37.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:36 smithi104 ceph-mon[32438]: from='client.26545 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:12:37.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:37.333 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:37.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:37.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:12:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:12:37.567 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:37.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.710 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:37.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:37.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.845 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:37.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:37.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:37.981 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:37.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:38.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:38.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:38.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.238 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:38.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: 11.18 scrub ok 2024-09-21T19:12:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: 4.16 scrub starts 2024-09-21T19:12:38.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: 4.16 scrub ok 2024-09-21T19:12:38.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:38.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: from='client.26545 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:12:38.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T19:12:38.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:38.259 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:37 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: 11.18 scrub ok 2024-09-21T19:12:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: 4.16 scrub starts 2024-09-21T19:12:38.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: 4.16 scrub ok 2024-09-21T19:12:38.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:38.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: from='client.26545 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:12:38.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T19:12:38.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:38.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:38.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: 11.18 scrub ok 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: 4.16 scrub starts 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: 4.16 scrub ok 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: from='client.26545 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:12:38.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:37 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:38.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.362 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:38.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:38.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.501 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:38.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:38.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:38.638 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:38.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:38.761 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:38.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:38.901 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:38.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:39.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:39.035 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:39.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.171 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:39.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:39.225 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:38 smithi081 ceph-mon[39192]: pgmap v972: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 26346/190824 objects misplaced (13.806%) 2024-09-21T19:12:39.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:38 smithi081 ceph-mon[29384]: pgmap v972: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 26346/190824 objects misplaced (13.806%) 2024-09-21T19:12:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:38 smithi104 ceph-mon[32438]: pgmap v972: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 26346/190824 objects misplaced (13.806%) 2024-09-21T19:12:39.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:12:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:12:39.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:39.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:39.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:39.427 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:39.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.584 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:39.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:39.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.721 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:39.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:39.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.856 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:39.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:39.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:39.986 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:39.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:40.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:40.117 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:40.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:40.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:40.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:39 smithi104 ceph-mon[32438]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T19:12:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[29384]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:40.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:39 smithi081 ceph-mon[39192]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T19:12:40.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:40.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:40.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:40.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:40.676 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:40.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.815 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:40.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:40.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:40.937 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:40.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:41.039 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:41.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:40 smithi081 ceph-mon[29384]: pgmap v974: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 23964/177690 objects misplaced (13.486%) 2024-09-21T19:12:41.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:40 smithi081 ceph-mon[39192]: pgmap v974: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 23964/177690 objects misplaced (13.486%) 2024-09-21T19:12:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:41.157 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:40 smithi104 ceph-mon[32438]: pgmap v974: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 23964/177690 objects misplaced (13.486%) 2024-09-21T19:12:41.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:41.200 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:41.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:41.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:41.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:41.467 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:41.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:41.597 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:41.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:41.729 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:41.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:41.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:41.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:41.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:42.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:42.139 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:42.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:42.280 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:42.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.421 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:42.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:42.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.559 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:42.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:42.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:42.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:42.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:42.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:42.968 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:42.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:43.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:43.094 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:43.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:43.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:42 smithi104 ceph-mon[32438]: pgmap v975: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 725 B/s rd, 0 op/s; 23964/175986 objects misplaced (13.617%) 2024-09-21T19:12:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:43.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:42 smithi081 ceph-mon[29384]: pgmap v975: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 725 B/s rd, 0 op/s; 23964/175986 objects misplaced (13.617%) 2024-09-21T19:12:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:42 smithi081 ceph-mon[39192]: pgmap v975: 178 pgs: 5 active+remapped+backfill_toofull, 173 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 725 B/s rd, 0 op/s; 23964/175986 objects misplaced (13.617%) 2024-09-21T19:12:43.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:43.377 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:43.479 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:43.530 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:43.632 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:43.666 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:43.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:43.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:43.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:43.940 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:43.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:44.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:44.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:44.220 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:44.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:43 smithi104 ceph-mon[32438]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:12:44.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:43 smithi081 ceph-mon[29384]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:12:44.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:43 smithi081 ceph-mon[39192]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:12:44.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:44.356 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:44.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:44.497 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:44.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:44.636 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:44.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:44.780 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:44.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:44.913 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:44.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:45.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:45.052 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:45.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:45.188 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:45.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:44 smithi104 ceph-mon[32438]: pgmap v976: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19265/162768 objects misplaced (11.836%) 2024-09-21T19:12:45.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:45.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:44 smithi081 ceph-mon[39192]: pgmap v976: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19265/162768 objects misplaced (11.836%) 2024-09-21T19:12:45.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:44 smithi081 ceph-mon[29384]: pgmap v976: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 19265/162768 objects misplaced (11.836%) 2024-09-21T19:12:45.325 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:45.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:45.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:45.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:45.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:45.749 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:45.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:45.882 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:45.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:45.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:46.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:46.142 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:46.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:45 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:46.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:45 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:45 smithi104 ceph-mon[32438]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T19:12:46.280 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:45 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:45 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:45 smithi081 ceph-mon[29384]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T19:12:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:45 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:45 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:45 smithi081 ceph-mon[39192]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T19:12:46.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:46.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:46.511 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:46.555 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:46.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:46.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:46.827 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:46.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:46.972 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:46.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:47.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:47.104 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:47.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:47.244 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:47.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:46 smithi104 ceph-mon[32438]: pgmap v978: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 19265/161235 objects misplaced (11.948%) 2024-09-21T19:12:47.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:46 smithi081 ceph-mon[29384]: pgmap v978: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 19265/161235 objects misplaced (11.948%) 2024-09-21T19:12:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:46 smithi081 ceph-mon[39192]: pgmap v978: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 19265/161235 objects misplaced (11.948%) 2024-09-21T19:12:47.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:12:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:12:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:12:47.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:47.380 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:47.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:47.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:47.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:47.654 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:47.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T19:12:47.679 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:47.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:47.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:47.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:47.922 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:47.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:48.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:48.078 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:48.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:48.212 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:48.314 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.342 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:48.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:48.421 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:48.438 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:12:48.438 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:12:48.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:48.469 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:48.480 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:12:48.480 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:12:48.522 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:12:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:12:48.562 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:12:48.562 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:12:48.571 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:48.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:12:48.603 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:12:48.603 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:12:48.605 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:48.638 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:12:48.639 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:12:48.673 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:12:48.673 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:12:48.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:48.708 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:12:48.709 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:12:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:48.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:48.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:48.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:49.042 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:49.075 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:49.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:49.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:12:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:12:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:48 smithi104 ceph-mon[32438]: pgmap v979: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 19265/156861 objects misplaced (12.282%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T19:12:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/584952198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:49.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:48 smithi081 ceph-mon[29384]: pgmap v979: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 19265/156861 objects misplaced (12.282%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T19:12:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/584952198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:48 smithi081 ceph-mon[39192]: pgmap v979: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 3.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 19265/156861 objects misplaced (12.282%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T19:12:49.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/584952198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:49.341 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:49.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:49.480 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:49.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:49.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:49.720 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:49.758 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:49.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:49.895 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:49.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:49.997 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:50.042 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:50.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:50.179 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:50.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:50.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:49 smithi104 ceph-mon[32438]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T19:12:50.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:50.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:50.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:49 smithi081 ceph-mon[29384]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T19:12:50.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:12:50.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:49 smithi081 ceph-mon[39192]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T19:12:50.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:50.434 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:50.490 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:12:50.490 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:12:50.490 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T19:12:50.490 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-21T19:12:50.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 0 2024-09-21T19:12:50.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.603 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:50.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:50.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:50.735 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:50.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:50.884 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:50.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:50.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:51.130 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:51.162 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:51.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:50 smithi104 ceph-mon[32438]: pgmap v981: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 13002/144405 objects misplaced (9.004%); 22 MiB/s, 352 objects/s recovering 2024-09-21T19:12:51.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:51.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:50 smithi081 ceph-mon[29384]: pgmap v981: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 13002/144405 objects misplaced (9.004%); 22 MiB/s, 352 objects/s recovering 2024-09-21T19:12:51.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:50 smithi081 ceph-mon[39192]: pgmap v981: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 13002/144405 objects misplaced (9.004%); 22 MiB/s, 352 objects/s recovering 2024-09-21T19:12:51.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:51.408 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:202720 11.c e474/414 11:3d1e7db7:::benchmark_data_smithi081_181703_object79099:head v 474'12837, mlcod=474'12835)", 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:50.994693+0000", 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.017983882999999999, 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018012222000000001, 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994693+0000", 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994693+0000", 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994694+0000", 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6099999999999999e-07 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994700+0000", 2024-09-21T19:12:51.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7270000000000004e-06 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994714+0000", 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3817e-05 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994722+0000", 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.653e-06 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:51.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994820+0000", 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7839999999999998e-05 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.994838+0000", 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7859e-05 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:202723 11.c e474/414 11:331f755e:::benchmark_data_smithi081_181703_object79102:head v 474'12838, mlcod=474'12835)", 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:50.998324+0000", 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.014353288000000001, 2024-09-21T19:12:51.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014437391000000001, 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.998324+0000", 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.998324+0000", 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.998325+0000", 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.299e-06 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.998332+0000", 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5229999999999998e-06 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.998346+0000", 2024-09-21T19:12:51.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3191e-05 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:50.998353+0000", 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1389999999999998e-06 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.003737+0000", 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053846120000000004 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.003898+0000", 2024-09-21T19:12:51.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016064900000000001 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:202726 11.c e474/414 11:3741625f:::benchmark_data_smithi081_181703_object79105:head v 474'12839, mlcod=474'12835)", 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:51.002609+0000", 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.010067558000000001, 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010203863000000001, 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:51.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.002609+0000", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.002609+0000", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.002611+0000", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6390000000000001e-06 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.002619+0000", 2024-09-21T19:12:51.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4160000000000004e-06 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.002631+0000", 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.146e-05 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.002638+0000", 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1579999999999999e-06 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:51.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.007152+0000", 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045134880000000004 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.007173+0000", 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1268999999999999e-05 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:202728 11.c e474/414 11:32727f67:::benchmark_data_smithi081_181703_object79107:head v 474'12840, mlcod=474'12835)", 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:51.004925+0000", 2024-09-21T19:12:51.419 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0077524079999999997, 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0079388930000000007, 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.004925+0000", 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.004925+0000", 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.004926+0000", 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.485e-06 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.004933+0000", 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3039999999999999e-06 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.004944+0000", 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.082e-05 2024-09-21T19:12:51.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.004951+0000", 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6270000000000002e-06 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010151+0000", 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0052006420000000001 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010173+0000", 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1242000000000001e-05 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:202740 11.16 e474/458 11:6b642365:::benchmark_data_smithi081_181703_object79119:head v 474'11246, mlcod=474'11244)", 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:51.008555+0000", 2024-09-21T19:12:51.423 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0041220299999999996, 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043606959999999998, 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.008555+0000", 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.008555+0000", 2024-09-21T19:12:51.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.008556+0000", 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.294e-06 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.008564+0000", 2024-09-21T19:12:51.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4000000000000003e-06 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.008582+0000", 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.806e-05 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.008589+0000", 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0509999999999999e-06 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:51.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.009689+0000", 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001099851 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.009710+0000", 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1467999999999998e-05 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.36497.0:202732 11.c e474/414)", 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:51.010924+0000", 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0017525520000000001, 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002051186, 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "queued for pg", 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010924+0000", 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010924+0000", 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010926+0000", 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3060000000000001e-06 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010933+0000", 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7540000000000006e-06 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010948+0000", 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4399000000000001e-05 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:51.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:51.010955+0000", 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8399999999999997e-06 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 6 2024-09-21T19:12:51.431 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:12:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:51.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:51.571 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:51.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:51.703 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:51.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:51.837 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:51.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:51.977 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:51.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:52.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:52.115 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:52.216 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:52.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:52.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T19:12:52.352 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:52.382 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:52.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:52.519 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:52.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:52.684 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:52.786 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.822 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:52.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:52.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:52.957 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:52.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:53.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.113 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:53.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:53.116 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:53.185 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:53.213 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:53.242 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:53.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[39192]: pgmap v982: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 13002/142866 objects misplaced (9.101%); 14 MiB/s, 225 objects/s recovering 2024-09-21T19:12:53.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:53.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:53.279 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:53.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[29384]: pgmap v982: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 13002/142866 objects misplaced (9.101%); 14 MiB/s, 225 objects/s recovering 2024-09-21T19:12:53.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:53.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:53.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:53.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:53.391 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:53.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:53.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:53 smithi104 ceph-mon[32438]: pgmap v982: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s; 13002/142866 objects misplaced (9.101%); 14 MiB/s, 225 objects/s recovering 2024-09-21T19:12:53.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:53 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:53 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:12:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.538 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:53.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:53.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:53.668 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:53.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.800 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:53.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:53.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:53.935 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:53.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:53.979 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:54.037 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:54.074 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:54.113 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:54 smithi081 ceph-mon[39192]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T19:12:54.113 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:54 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:12:54.113 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:54 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:12:54.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:54 smithi081 ceph-mon[29384]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T19:12:54.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:54 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:12:54.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:54 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:12:54.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:54.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:54.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.359 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:54.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:54.460 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:54.498 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:54 smithi104 ceph-mon[32438]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T19:12:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:54 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:12:54.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:54 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:12:54.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:54.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:54.787 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:54.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:54.927 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:54.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:54.996 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:12:54.996 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-09-21T19:12:54.996 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-09-21T19:12:54.996 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-09-21T19:12:54.996 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:12:55.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:55.054 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:12:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:55.064 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-21T19:12:55.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[39192]: pgmap v984: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 7293/129609 objects misplaced (5.627%); 28 MiB/s, 454 objects/s recovering 2024-09-21T19:12:55.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[39192]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T19:12:55.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1891057605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:12:55.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/351766977' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T19:12:55.065 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[39192]: from='client.26560 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T19:12:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[29384]: pgmap v984: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 7293/129609 objects misplaced (5.627%); 28 MiB/s, 454 objects/s recovering 2024-09-21T19:12:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[29384]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T19:12:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1891057605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:12:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/351766977' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T19:12:55.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:55 smithi081 ceph-mon[29384]: from='client.26560 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T19:12:55.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:12:55.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:55.193 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:55.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:55.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:55 smithi104 ceph-mon[32438]: pgmap v984: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 7293/129609 objects misplaced (5.627%); 28 MiB/s, 454 objects/s recovering 2024-09-21T19:12:55.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:55 smithi104 ceph-mon[32438]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T19:12:55.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1891057605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:12:55.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/351766977' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T19:12:55.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:55 smithi104 ceph-mon[32438]: from='client.26560 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T19:12:55.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:55.451 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:55.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:55.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:55.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:55.697 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:55.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:55.823 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:55.925 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:55.947 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:55.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:56.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:56.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:56.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:56.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:56.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1891057605' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:12:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[39192]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-21T19:12:56.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[39192]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T19:12:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1891057605' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:12:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[29384]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-21T19:12:56.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:56 smithi081 ceph-mon[29384]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T19:12:56.325 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:56.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T19:12:56.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:56.447 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:56 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:12:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1891057605' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:12:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:56 smithi104 ceph-mon[32438]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-21T19:12:56.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:56 smithi104 ceph-mon[32438]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T19:12:56.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:56.570 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:56.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:56.693 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:56.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:56.816 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:56.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:56.940 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:56.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:57.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:57.065 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:57.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:57.179 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:57.198 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:57.212 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:12:57.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:57.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:57 smithi081 ceph-mon[39192]: pgmap v987: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s; 7293/128139 objects misplaced (5.691%); 19 MiB/s, 306 objects/s recovering 2024-09-21T19:12:57.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:57 smithi081 ceph-mon[39192]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T19:12:57.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:12:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:12:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:12:57.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:57 smithi081 ceph-mon[29384]: pgmap v987: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s; 7293/128139 objects misplaced (5.691%); 19 MiB/s, 306 objects/s recovering 2024-09-21T19:12:57.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:57 smithi081 ceph-mon[29384]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T19:12:57.342 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:12:57.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:57.474 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:57.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:57 smithi104 ceph-mon[32438]: pgmap v987: 178 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s; 7293/128139 objects misplaced (5.691%); 19 MiB/s, 306 objects/s recovering 2024-09-21T19:12:57.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:57 smithi104 ceph-mon[32438]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T19:12:57.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:57.611 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:57.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:57.743 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:57.846 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:57.881 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:57.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244006 9.6 9:6e3a6723:::benchmark_data_smithi081_131274_object116151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 429.96985701099999, 2024-09-21T19:12:57.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0195242950000001, 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244006 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947844+0000", 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947847+0000", 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3649999999999998e-06 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947854+0000", 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5519999999999999e-06 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947872+0000", 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7999000000000002e-05 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.947880+0000", 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4200000000000001e-06 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948077+0000", 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019761400000000001 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948118+0000", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0926000000000001e-05 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959355+0000", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.011236561 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959470+0000", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115578 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:12:57.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959574+0000", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010335700000000001 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962650+0000", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030760499999999999 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962661+0000", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1022000000000001e-05 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967081+0000", 2024-09-21T19:12:57.932 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044202369999999996 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967163+0000", 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1880999999999998e-05 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967369+0000", 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020573299999999999 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.933 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244020 9.6 9:649892a9:::benchmark_data_smithi081_131274_object116165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 429.96134332700001, 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0117105959999999, 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:12:57.934 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244020 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956358+0000", 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956361+0000", 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2320000000000001e-06 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956369+0000", 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7789999999999996e-06 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.936 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956389+0000", 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9944000000000001e-05 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956398+0000", 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1600000000000004e-06 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956553+0000", 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015503399999999999 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.937 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.956577+0000", 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3506000000000002e-05 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964359+0000", 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0077825030000001 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964473+0000", 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113613 2024-09-21T19:12:57.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.939 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.939 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,5", 2024-09-21T19:12:57.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964573+0000", 2024-09-21T19:12:57.939 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9993000000000003e-05 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967381+0000", 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028085699999999998 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967707+0000", 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000325555 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967927+0000", 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021985700000000001 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967974+0000", 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.761e-05 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.968069+0000", 2024-09-21T19:12:57.941 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4240000000000006e-05 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189643 10.3 10:cbf6bec4:::benchmark_data_smithi081_155596_object72170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.62874678200001, 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.28335258899999999, 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189643 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288955+0000", 2024-09-21T19:12:57.943 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288956+0000", 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.674e-06 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288961+0000", 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0039999999999999e-06 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288973+0000", 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2011000000000001e-05 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.944 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.288980+0000", 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1729999999999998e-06 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289033+0000", 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2995999999999999e-05 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289155+0000", 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121433 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.945 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.289279+0000", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012419300000000001 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.292626+0000", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033469849999999998 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.295941+0000", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033146389999999999 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572053+0000", 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27611242699999999 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572259+0000", 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020604900000000001 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.572307+0000", 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8004999999999999e-05 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.947 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189647 10.5 10:a120596a:::benchmark_data_smithi081_155596_object72174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.62411741899999, 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29500822199999999, 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189647 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.948 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293584+0000", 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293586+0000", 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8470000000000001e-06 2024-09-21T19:12:57.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293592+0000", 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5810000000000002e-06 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293603+0000", 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0728e-05 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293610+0000", 2024-09-21T19:12:57.950 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2250000000000003e-06 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293666+0000", 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5977999999999998e-05 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293807+0000", 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014065599999999999 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.293923+0000", 2024-09-21T19:12:57.951 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011639800000000001 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.296808+0000", 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028842630000000002 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.297628+0000", 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082086500000000005 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588498+0000", 2024-09-21T19:12:57.952 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.29086982900000002 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588546+0000", 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7862999999999999e-05 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588592+0000", 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5989000000000003e-05 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.953 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189666 10.5 10:afed0067:::benchmark_data_smithi081_155596_object72193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.604150363, 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27515819499999999, 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189666 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.954 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313551+0000", 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313553+0000", 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0530000000000001e-06 2024-09-21T19:12:57.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313559+0000", 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3489999999999997e-06 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313573+0000", 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.324e-05 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313581+0000", 2024-09-21T19:12:57.956 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5320000000000006e-06 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313644+0000", 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3293000000000005e-05 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313760+0000", 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011513799999999999 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.313872+0000", 2024-09-21T19:12:57.957 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011291 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.316982+0000", 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031098860000000001 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.317682+0000", 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000699711 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588631+0000", 2024-09-21T19:12:57.958 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27094870199999999 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588676+0000", 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5027000000000001e-05 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588709+0000", 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3353999999999999e-05 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189672 10.5 10:a358f405:::benchmark_data_smithi081_155596_object72199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.59908914900001, 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27021128500000002, 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189672 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.960 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318612+0000", 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318613+0000", 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1829999999999999e-06 2024-09-21T19:12:57.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318618+0000", 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3819999999999997e-06 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318630+0000", 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2054e-05 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318637+0000", 2024-09-21T19:12:57.962 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0690000000000004e-06 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318711+0000", 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4373000000000001e-05 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318818+0000", 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010637999999999999 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318920+0000", 2024-09-21T19:12:57.963 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010200300000000001 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.324913+0000", 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005993651 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.324929+0000", 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5424999999999999e-05 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588745+0000", 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26381601999999998 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588790+0000", 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5621000000000002e-05 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.588823+0000", 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3124000000000003e-05 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36077.0:189673 10.5 10:a287fbc8:::benchmark_data_smithi081_155596_object72200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 235.59893461799999, 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.27468967399999999, 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36077", 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/684875334", 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 189673 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318767+0000", 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318768+0000", 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318773+0000", 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6940000000000003e-06 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318783+0000", 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0121e-05 2024-09-21T19:12:57.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.318790+0000", 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1210000000000001e-06 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321882+0000", 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030923249999999999 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.321979+0000", 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6842000000000006e-05 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.969 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.322109+0000", 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012934000000000001 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.325016+0000", 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002907251 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.325126+0000", 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110315 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.970 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593359+0000", 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26823260999999998 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593411+0000", 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2719000000000001e-05 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:09:02.593456+0000", 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4946e-05 2024-09-21T19:12:57.971 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:95343 11.2 11:45919c07:::benchmark_data_smithi081_181703_object95342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:10.598376+0000", 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 107.319325042, 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.40489012099999999, 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:57.972 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 95343 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598376+0000", 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598376+0000", 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598379+0000", 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5160000000000001e-06 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598405+0000", 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6591999999999999e-05 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598420+0000", 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4459e-05 2024-09-21T19:12:57.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598427+0000", 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2860000000000002e-06 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598494+0000", 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6602999999999999e-05 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598620+0000", 2024-09-21T19:12:57.975 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000126654 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.598790+0000", 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000169629 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.607003+0000", 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082134900000000004 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.610221+0000", 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032176150000000001 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003178+0000", 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.39295701300000002 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003228+0000", 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0201000000000003e-05 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003266+0000", 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8062999999999998e-05 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:95348 11.2 11:4f564251:::benchmark_data_smithi081_181703_object95347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:10.599297+0000", 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 107.318404722, 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.40408638000000002, 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.978 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 95348 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599297+0000", 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599297+0000", 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599298+0000", 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8050000000000001e-06 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599326+0000", 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7154000000000001e-05 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599335+0000", 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1730000000000001e-06 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.980 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599342+0000", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8909999999999998e-06 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.603296+0000", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039542409999999998 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.603444+0000", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000148026 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.603586+0000", 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014235800000000001 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.607010+0000", 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034238810000000001 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.610270+0000", 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032601890000000001 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003306+0000", 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.39303583600000003 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003351+0000", 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4926999999999999e-05 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003383+0000", 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1899000000000001e-05 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:95349 11.2 11:49060650:::benchmark_data_smithi081_181703_object95348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:10.599357+0000", 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 107.318343868, 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.40414383100000001, 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 95349 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599357+0000", 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599357+0000", 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599358+0000", 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4499999999999996e-07 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.985 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599380+0000", 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1630000000000001e-05 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599409+0000", 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8839e-05 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.599416+0000", 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3440000000000003e-06 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.986 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.607046+0000", 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0076297040000000002 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.607182+0000", 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000135983 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.607313+0000", 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013176299999999999 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.987 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.610355+0000", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00304186 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612192+0000", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018364500000000001 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003417+0000", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.39122558600000001 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003470+0000", 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2905000000000003e-05 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.003501+0000", 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1022000000000002e-05 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:95360 11.2 11:4f5e8eb2:::benchmark_data_smithi081_181703_object95359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:10.612888+0000", 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 107.304813373, 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.39265119100000001, 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.989 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 95360 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612888+0000", 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612888+0000", 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612890+0000", 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5629999999999999e-06 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612911+0000", 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1274999999999999e-05 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.991 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612921+0000", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8390000000000002e-06 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612926+0000", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5040000000000002e-06 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.612965+0000", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8584999999999997e-05 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.613071+0000", 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010587899999999999 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.613167+0000", 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5963000000000003e-05 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.615882+0000", 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027156620000000002 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:10.616878+0000", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00099562499999999998 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.005425+0000", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.38854696999999999 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.005494+0000", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9487999999999996e-05 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:11.005539+0000", 2024-09-21T19:12:57.994 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4838e-05 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:100768 11.2 11:48786ef4:::benchmark_data_smithi081_181703_object100767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:16.215131+0000", 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 101.702570612, 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.36403244699999998, 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:57.995 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 100768 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.215131+0000", 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.215131+0000", 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.996 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.215132+0000", 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7069999999999999e-06 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.215156+0000", 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3102999999999999e-05 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.215169+0000", 2024-09-21T19:12:57.997 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3919e-05 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.215178+0000", 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2109999999999998e-06 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.216132+0000", 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00095474199999999996 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.216432+0000", 2024-09-21T19:12:57.998 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000299155 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.218970+0000", 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025379819999999998 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.225237+0000", 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0062670579999999998 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:57.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.225519+0000", 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000282594 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.579001+0000", 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.35348184799999999 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.579123+0000", 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012211899999999999 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.000 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.579163+0000", 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0009000000000003e-05 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:100778 11.2 11:4df39e7d:::benchmark_data_smithi081_181703_object100777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:11:16.221882+0000", 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 101.695819757, 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.35751396299999999, 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.001 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 100778 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.221882+0000", 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.221882+0000", 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.221883+0000", 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.821e-06 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.221904+0000", 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0925e-05 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.221915+0000", 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0321e-05 2024-09-21T19:12:58.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.221921+0000", 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5579999999999997e-06 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.222287+0000", 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036562600000000003 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.222504+0000", 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000217137 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.004 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 0,3", 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.222658+0000", 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015418499999999999 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.225351+0000", 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026929889999999998 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.005 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.225675+0000", 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00032423800000000002 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.579204+0000", 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.35352856999999999 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.579280+0000", 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6030000000000002e-05 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:11:16.579396+0000", 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115563 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207340 11.15 11:a9a6be57:::benchmark_data_smithi081_181703_object83719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.115075+0000", 2024-09-21T19:12:58.008 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.8026267119999999, 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.004186985, 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207340 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115075+0000", 2024-09-21T19:12:58.009 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115075+0000", 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115077+0000", 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1679999999999998e-06 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115083+0000", 2024-09-21T19:12:58.010 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0809999999999999e-06 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115099+0000", 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6368999999999999e-05 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115108+0000", 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3799999999999994e-06 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115164+0000", 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6691999999999999e-05 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.115173+0000", 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6950000000000006e-06 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.100637+0000", 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98546402600000005 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.100753+0000", 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116302 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.101122+0000", 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036872800000000002 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.105991+0000", 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0048693139999999996 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.013 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.108450+0000", 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024586870000000002 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119106+0000", 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010656331 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119212+0000", 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105219 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.014 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119262+0000", 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9993000000000001e-05 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207385 11.15 11:abdb8199:::benchmark_data_smithi081_181703_object83764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.144982+0000", 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.7727191580000001, 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97447092400000002, 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207385 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.144982+0000", 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.144982+0000", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.144983+0000", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2550000000000001e-06 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.144988+0000", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5299999999999998e-06 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.145000+0000", 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1586000000000001e-05 2024-09-21T19:12:58.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.145009+0000", 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5519999999999993e-06 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.145065+0000", 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5572999999999998e-05 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.145073+0000", 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1380000000000007e-06 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.018 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.103519+0000", 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.95844579399999996 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.103640+0000", 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121359 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.103970+0000", 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00032977100000000002 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.108470+0000", 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045002660000000002 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.108507+0000", 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7058000000000003e-05 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119339+0000", 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010832084000000001 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.020 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119417+0000", 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7658999999999996e-05 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119453+0000", 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6298999999999999e-05 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.021 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207395 11.15 11:ac854341:::benchmark_data_smithi081_181703_object83774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.152523+0000", 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.765178404, 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97961277199999997, 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207395 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.022 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152523+0000", 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152523+0000", 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152524+0000", 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.482e-06 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.023 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152529+0000", 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4479999999999996e-06 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152538+0000", 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5170000000000005e-06 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152545+0000", 2024-09-21T19:12:58.024 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2929999999999997e-06 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152596+0000", 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1096e-05 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.152604+0000", 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2500000000000006e-06 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.106029+0000", 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.95342457599999997 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.106134+0000", 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105797 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.106405+0000", 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027039000000000002 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.026 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.118974+0000", 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012569241 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119025+0000", 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0927999999999998e-05 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132043+0000", 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01301777 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.027 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132096+0000", 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3677999999999997e-05 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132136+0000", 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9305999999999999e-05 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207398 11.15 11:ab08d59e:::benchmark_data_smithi081_181703_object83777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.028 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.154453+0000", 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.763248266, 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97779304700000003, 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207398 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154453+0000", 2024-09-21T19:12:58.029 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154453+0000", 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154454+0000", 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3680000000000001e-06 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154470+0000", 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5183e-05 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154484+0000", 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4187e-05 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154492+0000", 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9689999999999994e-06 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154546+0000", 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4092999999999999e-05 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.154554+0000", 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0800000000000006e-06 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.108536+0000", 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.953982263 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.108633+0000", 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7040000000000006e-05 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.108893+0000", 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026014300000000001 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.118980+0000", 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01008644 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119069+0000", 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9454000000000001e-05 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132175+0000", 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013105916 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132217+0000", 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1369999999999999e-05 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132246+0000", 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9541000000000001e-05 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207424 11.15 11:ad63280a:::benchmark_data_smithi081_181703_object83803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.177052+0000", 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.7406490300000002, 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.95532119900000001, 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.035 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207424 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177052+0000", 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177052+0000", 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.036 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177054+0000", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3480000000000001e-06 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177058+0000", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.189e-06 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177067+0000", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4010000000000003e-06 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177074+0000", 2024-09-21T19:12:58.037 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2459999999999997e-06 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177128+0000", 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3637999999999997e-05 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.177136+0000", 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8939999999999993e-06 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.110826+0000", 2024-09-21T19:12:58.038 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.933689934 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.110920+0000", 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4165000000000004e-05 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.111496+0000", 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00057586300000000003 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119037+0000", 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0075407260000000002 2024-09-21T19:12:58.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119301+0000", 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026435899999999998 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132310+0000", 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013008574 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132345+0000", 2024-09-21T19:12:58.040 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5871000000000003e-05 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132373+0000", 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7991e-05 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207432 11.15 11:a8883608:::benchmark_data_smithi081_181703_object83811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.186333+0000", 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.7313679099999999, 2024-09-21T19:12:58.041 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.94618437300000002, 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207432 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186333+0000", 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186333+0000", 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186335+0000", 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2389999999999999e-06 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186339+0000", 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3320000000000002e-06 2024-09-21T19:12:58.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186348+0000", 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4630000000000007e-06 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186354+0000", 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2970000000000003e-06 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186395+0000", 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1547e-05 2024-09-21T19:12:58.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.186401+0000", 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6099999999999997e-06 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.113956+0000", 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.92755501100000004 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.114608+0000", 2024-09-21T19:12:58.045 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00065184300000000004 2024-09-21T19:12:58.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.046 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.115560+0000", 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00095250499999999997 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.119499+0000", 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039386229999999996 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132404+0000", 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012905197 2024-09-21T19:12:58.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132449+0000", 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4450000000000003e-05 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132489+0000", 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.015e-05 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132518+0000", 2024-09-21T19:12:58.048 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9105999999999999e-05 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:207439 11.15 11:a86657de:::benchmark_data_smithi081_181703_object83818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:56.206034+0000", 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.7116675159999999, 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.92661314100000003, 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:12:58.049 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 207439 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206034+0000", 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206034+0000", 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206036+0000", 2024-09-21T19:12:58.050 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.018e-06 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206041+0000", 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8709999999999996e-06 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206053+0000", 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1925e-05 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206059+0000", 2024-09-21T19:12:58.051 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.156e-06 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206105+0000", 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6159000000000003e-05 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:56.206111+0000", 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6300000000000003e-06 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.121246+0000", 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91513542199999998 2024-09-21T19:12:58.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.121590+0000", 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00034444500000000003 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,4", 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.122935+0000", 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001344952 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132278+0000", 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093429489999999997 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.053 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132438+0000", 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015986700000000001 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132591+0000", 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015249600000000001 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132621+0000", 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0295999999999999e-05 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:12:58.054 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:57.132647+0000", 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5955000000000001e-05 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:12:58.055 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:12:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.056 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:58.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:58.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:58.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:12:58.298 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:58 smithi081 ceph-mon[39192]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T19:12:58.298 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:58 smithi081 ceph-mon[29384]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T19:12:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.339 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:58.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:12:58.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:12:58.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:58 smithi104 ceph-mon[32438]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T19:12:58.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:58.717 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:58.752 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:58.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:58.894 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:58.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:58.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:12:59.024 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:12:59.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:12:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:12:59.094 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:59 smithi081 ceph-mon[29384]: pgmap v990: 178 pgs: 2 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 302 op/s; 5516/125988 objects misplaced (4.378%); 8.5 MiB/s, 135 objects/s recovering 2024-09-21T19:12:59.094 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:12:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2281027827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:59.095 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:59 smithi081 ceph-mon[39192]: pgmap v990: 178 pgs: 2 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 302 op/s; 5516/125988 objects misplaced (4.378%); 8.5 MiB/s, 135 objects/s recovering 2024-09-21T19:12:59.095 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:12:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2281027827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:59.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:59.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:12:59.152 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:59.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:12:59.257 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:12:59.257 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:12:59.257 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:12:59.257 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:12:59.257 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:12:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:12:59.264 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:12:59.265 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:12:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:12:59.392 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:59.420 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:59.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:59 smithi104 ceph-mon[32438]: pgmap v990: 178 pgs: 2 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 302 op/s; 5516/125988 objects misplaced (4.378%); 8.5 MiB/s, 135 objects/s recovering 2024-09-21T19:12:59.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:12:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2281027827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:12:59.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:12:59.566 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:12:59.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:12:59.698 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:12:59.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:12:59.832 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:12:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:12:59.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:12:59.972 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:12:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:12:59.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:00.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:00.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:00.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:00.253 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:00.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:00.387 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:00.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:00 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:00 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:13:00.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:00 smithi104 ceph-mon[32438]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T19:13:00.533 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:00 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:00 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:13:00.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:00 smithi081 ceph-mon[29384]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T19:13:00.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:00 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:00.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:00 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:13:00.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:00 smithi081 ceph-mon[39192]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T19:13:00.636 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:00.663 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:00.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T19:13:00.737 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:00.765 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:00.792 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:00.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:00.918 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:00.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:00.987 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:13:00.987 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:13:00.987 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-21T19:13:00.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd out 5 2024-09-21T19:13:01.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:01.045 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:01.146 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:01.176 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:01.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:01.310 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[39192]: pgmap v992: 178 pgs: 2 peering, 1 active+remapped+backfill_toofull, 175 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 3741/116610 objects misplaced (3.208%); 15 MiB/s, 232 objects/s recovering 2024-09-21T19:13:01.310 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:13:01.310 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:13:01.310 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:01.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[29384]: pgmap v992: 178 pgs: 2 peering, 1 active+remapped+backfill_toofull, 175 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 3741/116610 objects misplaced (3.208%); 15 MiB/s, 232 objects/s recovering 2024-09-21T19:13:01.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:13:01.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:13:01.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:01 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:01.333 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:01.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:01.471 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:01.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:01 smithi104 ceph-mon[32438]: pgmap v992: 178 pgs: 2 peering, 1 active+remapped+backfill_toofull, 175 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 3741/116610 objects misplaced (3.208%); 15 MiB/s, 232 objects/s recovering 2024-09-21T19:13:01.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:01 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:13:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:01 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:13:01.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:01 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:01.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.612 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:01.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:01.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:01.745 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:01.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:01.891 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:01.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:01.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:01.993 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:02.004 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:02.020 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:02.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.157 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:02.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:02.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:02.291 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:02.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:02.424 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:02.526 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:02.564 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:02.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:02.702 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:02.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:02.835 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:02.936 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:02.973 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:02.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:03.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:03.106 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:13:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.118 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:03.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:03.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:03.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:03.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:13:03.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:03.398 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:03.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:03.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:03 smithi104 ceph-mon[32438]: pgmap v993: 178 pgs: 2 peering, 1 active+remapped+backfill_toofull, 175 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 640 op/s; 3741/116610 objects misplaced (3.208%); 12 MiB/s, 187 objects/s recovering 2024-09-21T19:13:03.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:03 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2575964590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.549 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:03.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:03.562 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:03 smithi081 ceph-mon[29384]: pgmap v993: 178 pgs: 2 peering, 1 active+remapped+backfill_toofull, 175 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 640 op/s; 3741/116610 objects misplaced (3.208%); 12 MiB/s, 187 objects/s recovering 2024-09-21T19:13:03.562 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:03 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2575964590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:03.562 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:03 smithi081 ceph-mon[39192]: pgmap v993: 178 pgs: 2 peering, 1 active+remapped+backfill_toofull, 175 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 640 op/s; 3741/116610 objects misplaced (3.208%); 12 MiB/s, 187 objects/s recovering 2024-09-21T19:13:03.562 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:03 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2575964590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:03.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.690 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:03.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:03.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:03.814 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:03.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:03.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:03.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:04.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:04.092 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:04.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:04.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:04.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:04.366 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:04.468 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:04.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2575964590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: 3.1f scrub starts 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: 3.1f scrub ok 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T19:13:04.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:04 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T19:13:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.508 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:04.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2575964590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: 3.1f scrub starts 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: 3.1f scrub ok 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T19:13:04.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2575964590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: 3.1f scrub starts 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: 3.1f scrub ok 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T19:13:04.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:04 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T19:13:04.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:04.655 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:04.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:04.785 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:04.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:04.930 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:04.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:05.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.070 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:05.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:05.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:05.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:05.219 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:05.219 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:05.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:13:05.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:13:05.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: pgmap v995: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s; 3171/99042 objects misplaced (3.202%); 5.5 MiB/s, 84 objects/s recovering 2024-09-21T19:13:05.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: 11.d scrub starts 2024-09-21T19:13:05.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T19:13:05.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: pgmap v995: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s; 3171/99042 objects misplaced (3.202%); 5.5 MiB/s, 84 objects/s recovering 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: 11.d scrub starts 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T19:13:05.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T19:13:05.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:05.331 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:05.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:05.465 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: pgmap v995: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s; 3171/99042 objects misplaced (3.202%); 5.5 MiB/s, 84 objects/s recovering 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: 11.d scrub starts 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T19:13:05.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:05 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T19:13:05.567 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:05.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:05 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T19:13:05.285+0000 7ff1af1de640 -1 mon.a@0(leader).osd e482 definitely_dead 0 2024-09-21T19:13:05.598 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:05.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:05.734 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:05.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:05.869 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:05.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:05.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:05.999 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:06.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:06.131 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:06.140 INFO:teuthology.orchestra.run.smithi081.stderr:marked out osd.5. 2024-09-21T19:13:06.233 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:06.253 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:06.355 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.377 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:06.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.500 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:06.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: 11.d scrub ok 2024-09-21T19:13:06.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1201452638' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T19:13:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T19:13:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T19:13:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: 6.17 scrub starts 2024-09-21T19:13:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: 6.17 scrub ok 2024-09-21T19:13:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: 6.16 scrub starts 2024-09-21T19:13:06.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:06 smithi104 ceph-mon[32438]: 6.16 scrub ok 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: 11.d scrub ok 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1201452638' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: 6.17 scrub starts 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: 6.17 scrub ok 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: 6.16 scrub starts 2024-09-21T19:13:06.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[39192]: 6.16 scrub ok 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: 11.d scrub ok 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1201452638' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: 6.17 scrub starts 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: 6.17 scrub ok 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: 6.16 scrub starts 2024-09-21T19:13:06.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:06 smithi081 ceph-mon[29384]: 6.16 scrub ok 2024-09-21T19:13:06.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:06.622 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:06.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:13:06.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:06.746 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:06.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.870 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:06.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:06.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:06.993 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:06.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:07.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:07.116 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:07.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:07.239 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:07.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:07.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:07.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:07.485 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: pgmap v997: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 844 B/s rd, 0 B/s wr, 1.06k op/s; 3171/97263 objects misplaced (3.260%) 2024-09-21T19:13:07.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T19:13:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: osdmap e483: 8 total, 8 up, 7 in 2024-09-21T19:13:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:13:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:13:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: 6.e scrub starts 2024-09-21T19:13:07.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:07 smithi104 ceph-mon[32438]: 6.e scrub ok 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: pgmap v997: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 844 B/s rd, 0 B/s wr, 1.06k op/s; 3171/97263 objects misplaced (3.260%) 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: osdmap e483: 8 total, 8 up, 7 in 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: 6.e scrub starts 2024-09-21T19:13:07.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[29384]: 6.e scrub ok 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: pgmap v997: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 844 B/s rd, 0 B/s wr, 1.06k op/s; 3171/97263 objects misplaced (3.260%) 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: osdmap e483: 8 total, 8 up, 7 in 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: 6.e scrub starts 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-mon[39192]: 6.e scrub ok 2024-09-21T19:13:07.567 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:13:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:13:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:13:07.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:07.608 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:07.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T19:13:07.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:07.731 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:07.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:07.855 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:07.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:07.978 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:07.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:08.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:08.102 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:08.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: osdmap e484: 8 total, 8 up, 7 in 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[29384]: osdmap e485: 8 total, 8 up, 7 in 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: osdmap e484: 8 total, 8 up, 7 in 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:13:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:13:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T19:13:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T19:13:08.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:08 smithi081 ceph-mon[39192]: osdmap e485: 8 total, 8 up, 7 in 2024-09-21T19:13:08.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:08.348 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:08.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:08.493 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:08.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: osdmap e484: 8 total, 8 up, 7 in 2024-09-21T19:13:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:13:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:13:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T19:13:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T19:13:08.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-mon[32438]: osdmap e485: 8 total, 8 up, 7 in 2024-09-21T19:13:08.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:08.653 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:08.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.787 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:08.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:08.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:08.921 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:08.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:09.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.066 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:09.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:09.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:13:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:13:09.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:09.209 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:09.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:09.335 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:09.405 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:09.430 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:09.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:09.459 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:09.463 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: pgmap v1000: 178 pgs: 10 peering, 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 877 op/s; 3171/93546 objects misplaced (3.390%) 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 482'11588 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.16 starting backfill to osd.1 from (0'0,0'0] MAX to 482'11775 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 482'11674 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.17 starting backfill to osd.1 from (0'0,0'0] MAX to 482'11910 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 482'13656 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 482'11949 2024-09-21T19:13:09.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:13:09.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 11.13 starting backfill to osd.6 from (0'0,0'0] MAX to 482'11722 2024-09-21T19:13:09.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 6.c scrub starts 2024-09-21T19:13:09.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 6.c scrub ok 2024-09-21T19:13:09.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T19:13:09.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:09 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T19:13:09.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: pgmap v1000: 178 pgs: 10 peering, 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 877 op/s; 3171/93546 objects misplaced (3.390%) 2024-09-21T19:13:09.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 482'11588 2024-09-21T19:13:09.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.16 starting backfill to osd.1 from (0'0,0'0] MAX to 482'11775 2024-09-21T19:13:09.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 482'11674 2024-09-21T19:13:09.551 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.17 starting backfill to osd.1 from (0'0,0'0] MAX to 482'11910 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 482'13656 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 482'11949 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 11.13 starting backfill to osd.6 from (0'0,0'0] MAX to 482'11722 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 6.c scrub starts 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 6.c scrub ok 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: pgmap v1000: 178 pgs: 10 peering, 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 877 op/s; 3171/93546 objects misplaced (3.390%) 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 482'11588 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.16 starting backfill to osd.1 from (0'0,0'0] MAX to 482'11775 2024-09-21T19:13:09.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 482'11674 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.17 starting backfill to osd.1 from (0'0,0'0] MAX to 482'11910 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 482'13656 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 482'11949 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 11.13 starting backfill to osd.6 from (0'0,0'0] MAX to 482'11722 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 6.c scrub starts 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 6.c scrub ok 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T19:13:09.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:09 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T19:13:09.565 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.610 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:09.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:09.711 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:09.735 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:09.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:09.874 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:09.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:09.976 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:10.008 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:10.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:10.144 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:10.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:10.284 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:10.386 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:10.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:10 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:10.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:10 smithi081 ceph-mon[29384]: osdmap e486: 8 total, 8 up, 7 in 2024-09-21T19:13:10.413 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:10 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:10.413 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:10 smithi081 ceph-mon[39192]: osdmap e486: 8 total, 8 up, 7 in 2024-09-21T19:13:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:10.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:10 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:10.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:10 smithi104 ceph-mon[32438]: osdmap e486: 8 total, 8 up, 7 in 2024-09-21T19:13:10.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:10.551 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:10.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:10.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:10.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:10.831 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:10.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:10.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:11.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:11.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.241 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:11.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:11.278 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:13:11.278 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:13:11.279 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:13:11.280 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:13:11.280 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:13:11.280 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:13:11.280 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:13:11.280 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:13:11.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:11.368 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:11.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.499 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:11.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:11.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:11 smithi104 ceph-mon[32438]: pgmap v1003: 178 pgs: 1 active+remapped+backfill_wait, 2 remapped+peering, 1 active+remapped, 10 peering, 1 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 5330/84174 objects misplaced (6.332%); 64 KiB/s, 11 objects/s recovering 2024-09-21T19:13:11.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:11 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1847382052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:11 smithi081 ceph-mon[29384]: pgmap v1003: 178 pgs: 1 active+remapped+backfill_wait, 2 remapped+peering, 1 active+remapped, 10 peering, 1 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 5330/84174 objects misplaced (6.332%); 64 KiB/s, 11 objects/s recovering 2024-09-21T19:13:11.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:11 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1847382052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:11.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:11 smithi081 ceph-mon[39192]: pgmap v1003: 178 pgs: 1 active+remapped+backfill_wait, 2 remapped+peering, 1 active+remapped, 10 peering, 1 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 5330/84174 objects misplaced (6.332%); 64 KiB/s, 11 objects/s recovering 2024-09-21T19:13:11.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:11 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1847382052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:11.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:11.624 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:11.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:11.759 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:11.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:11.890 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:11.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:11.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:12.016 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:12.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:12.159 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:12.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:12.299 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:12.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:12.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.445 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:12.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:12.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:12.574 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:12.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:12.703 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:12.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.833 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:12.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:12.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:12.975 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:12.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:13.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:13.112 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:13.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:13.251 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:13.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:13.388 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:13.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:13.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:13 smithi104 ceph-mon[32438]: pgmap v1004: 178 pgs: 1 active+remapped+backfill_wait, 2 remapped+peering, 1 active+remapped, 10 peering, 1 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 759 op/s; 5330/84174 objects misplaced (6.332%); 45 KiB/s, 8 objects/s recovering 2024-09-21T19:13:13.526 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:13.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:13 smithi081 ceph-mon[39192]: pgmap v1004: 178 pgs: 1 active+remapped+backfill_wait, 2 remapped+peering, 1 active+remapped, 10 peering, 1 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 759 op/s; 5330/84174 objects misplaced (6.332%); 45 KiB/s, 8 objects/s recovering 2024-09-21T19:13:13.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:13 smithi081 ceph-mon[29384]: pgmap v1004: 178 pgs: 1 active+remapped+backfill_wait, 2 remapped+peering, 1 active+remapped, 10 peering, 1 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 759 op/s; 5330/84174 objects misplaced (6.332%); 45 KiB/s, 8 objects/s recovering 2024-09-21T19:13:13.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:13.652 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:13.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:13.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:13.915 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:13.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:14.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:14.040 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:14.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:14.168 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:14.183 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:14.184 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:14.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:14.305 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:14.306 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 6, 7, 0, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:13:14.307 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:13:14.307 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T19:13:14.307 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.032138 2024-09-21T19:13:14.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 0.03213840607076435 2024-09-21T19:13:14.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:14.448 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:14.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:14 smithi104 ceph-mon[32438]: osdmap e487: 8 total, 8 up, 7 in 2024-09-21T19:13:14.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:14.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:14 smithi081 ceph-mon[39192]: osdmap e487: 8 total, 8 up, 7 in 2024-09-21T19:13:14.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:14 smithi081 ceph-mon[29384]: osdmap e487: 8 total, 8 up, 7 in 2024-09-21T19:13:14.576 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:14.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:14.699 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:14.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:14.826 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:14.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:14.955 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:14.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:15.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:15.085 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:15.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:15.213 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:15.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:15.339 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:15.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:15.463 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:15.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: pgmap v1006: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 1.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 5537/71940 objects misplaced (7.697%); 27 MiB/s, 444 objects/s recovering 2024-09-21T19:13:15.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:13:15.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:13:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:13:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T19:13:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: osdmap e488: 8 total, 8 up, 7 in 2024-09-21T19:13:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:15 smithi104 ceph-mon[32438]: osdmap e489: 8 total, 8 up, 7 in 2024-09-21T19:13:15.565 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: pgmap v1006: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 1.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 5537/71940 objects misplaced (7.697%); 27 MiB/s, 444 objects/s recovering 2024-09-21T19:13:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:13:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:13:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:13:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T19:13:15.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: osdmap e488: 8 total, 8 up, 7 in 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[29384]: osdmap e489: 8 total, 8 up, 7 in 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: pgmap v1006: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 1.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 5537/71940 objects misplaced (7.697%); 27 MiB/s, 444 objects/s recovering 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: osdmap e488: 8 total, 8 up, 7 in 2024-09-21T19:13:15.567 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:15 smithi081 ceph-mon[39192]: osdmap e489: 8 total, 8 up, 7 in 2024-09-21T19:13:15.596 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:15.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:15.727 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:15.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:15.852 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:15.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:15.976 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:15.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:16.078 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:16.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:16.244 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:16.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:16.374 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:16.476 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:16.505 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:16.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:16.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:16.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:16.774 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:16.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:16.916 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:16.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:17.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:17.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:17.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[39192]: 1.d scrub starts 2024-09-21T19:13:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[39192]: 1.d scrub ok 2024-09-21T19:13:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[39192]: osdmap e490: 8 total, 8 up, 7 in 2024-09-21T19:13:17.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[39192]: pgmap v1010: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 5537/70647 objects misplaced (7.838%); 41 MiB/s, 655 objects/s recovering 2024-09-21T19:13:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[29384]: 1.d scrub starts 2024-09-21T19:13:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[29384]: 1.d scrub ok 2024-09-21T19:13:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[29384]: osdmap e490: 8 total, 8 up, 7 in 2024-09-21T19:13:17.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:16 smithi081 ceph-mon[29384]: pgmap v1010: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 5537/70647 objects misplaced (7.838%); 41 MiB/s, 655 objects/s recovering 2024-09-21T19:13:17.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:16 smithi104 ceph-mon[32438]: 1.d scrub starts 2024-09-21T19:13:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:16 smithi104 ceph-mon[32438]: 1.d scrub ok 2024-09-21T19:13:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:16 smithi104 ceph-mon[32438]: osdmap e490: 8 total, 8 up, 7 in 2024-09-21T19:13:17.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:16 smithi104 ceph-mon[32438]: pgmap v1010: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 5537/70647 objects misplaced (7.838%); 41 MiB/s, 655 objects/s recovering 2024-09-21T19:13:17.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.327 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:17.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:17.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:17.456 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:17.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:17.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:13:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:13:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:13:17.595 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:17.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:17.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.733 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:17.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:17.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:17.861 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:17.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:17.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:18.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:18.127 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:18.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:18.261 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:18.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:18.385 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:18.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:18.508 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:18.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:18.631 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:18.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:18.754 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:18.846 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:13:18.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:18.878 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:18.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:18.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:19.007 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:19.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:19.130 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:19.232 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:19.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:19.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:13:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T19:13:19.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T19:13:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: 5.2 scrub starts 2024-09-21T19:13:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: 5.2 scrub ok 2024-09-21T19:13:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: osdmap e491: 8 total, 8 up, 7 in 2024-09-21T19:13:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1943716802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:18 smithi104 ceph-mon[32438]: pgmap v1012: 178 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 172 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 321 op/s; 4748/68229 objects misplaced (6.959%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: 5.2 scrub starts 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: 5.2 scrub ok 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: osdmap e491: 8 total, 8 up, 7 in 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1943716802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[29384]: pgmap v1012: 178 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 172 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 321 op/s; 4748/68229 objects misplaced (6.959%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:13:19.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T19:13:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T19:13:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: 5.2 scrub starts 2024-09-21T19:13:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: 5.2 scrub ok 2024-09-21T19:13:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: osdmap e491: 8 total, 8 up, 7 in 2024-09-21T19:13:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1943716802' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:19.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:18 smithi081 ceph-mon[39192]: pgmap v1012: 178 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 172 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 321 op/s; 4748/68229 objects misplaced (6.959%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T19:13:19.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:19.378 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:19.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:19.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:19.623 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:19.725 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:19.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:19.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:19.871 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:19.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:19.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:20.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.160 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:20.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:20.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: 4.1e scrub starts 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: 4.1e scrub ok 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1943716802' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[39192]: osdmap e492: 8 total, 8 up, 7 in 2024-09-21T19:13:20.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: 4.1e scrub starts 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: 4.1e scrub ok 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1943716802' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:20.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:19 smithi081 ceph-mon[29384]: osdmap e492: 8 total, 8 up, 7 in 2024-09-21T19:13:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T19:13:20.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: 4.1e scrub starts 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: 4.1e scrub ok 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1943716802' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:20.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:19 smithi104 ceph-mon[32438]: osdmap e492: 8 total, 8 up, 7 in 2024-09-21T19:13:20.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:20.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:20.428 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:20.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:20.575 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:20.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:20.717 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:20.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:20.848 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:20.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:20.982 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:20.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:21.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:21.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T19:13:21.113 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:21.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[29384]: 1.e scrub starts 2024-09-21T19:13:21.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[29384]: 1.e scrub ok 2024-09-21T19:13:21.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[29384]: osdmap e493: 8 total, 8 up, 7 in 2024-09-21T19:13:21.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[29384]: pgmap v1015: 178 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 174 active+clean; 1.2 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 107 KiB/s rd, 0 B/s wr, 1.14k op/s; 1332/58932 objects misplaced (2.260%); 39 MiB/s, 618 objects/s recovering 2024-09-21T19:13:21.133 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[39192]: 1.e scrub starts 2024-09-21T19:13:21.133 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[39192]: 1.e scrub ok 2024-09-21T19:13:21.133 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[39192]: osdmap e493: 8 total, 8 up, 7 in 2024-09-21T19:13:21.133 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:20 smithi081 ceph-mon[39192]: pgmap v1015: 178 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 174 active+clean; 1.2 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 107 KiB/s rd, 0 B/s wr, 1.14k op/s; 1332/58932 objects misplaced (2.260%); 39 MiB/s, 618 objects/s recovering 2024-09-21T19:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:21.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:21.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:20 smithi104 ceph-mon[32438]: 1.e scrub starts 2024-09-21T19:13:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:20 smithi104 ceph-mon[32438]: 1.e scrub ok 2024-09-21T19:13:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:20 smithi104 ceph-mon[32438]: osdmap e493: 8 total, 8 up, 7 in 2024-09-21T19:13:21.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:20 smithi104 ceph-mon[32438]: pgmap v1015: 178 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 174 active+clean; 1.2 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 107 KiB/s rd, 0 B/s wr, 1.14k op/s; 1332/58932 objects misplaced (2.260%); 39 MiB/s, 618 objects/s recovering 2024-09-21T19:13:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.283 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:21.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:21.384 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:21.410 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:21.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:21.540 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:21.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:21.692 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:21.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:21.825 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.840 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:21.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:21.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:13:21.941 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:21.965 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:21.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:21 smithi081 ceph-mon[29384]: 4.c scrub starts 2024-09-21T19:13:22.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:21 smithi081 ceph-mon[29384]: 4.c scrub ok 2024-09-21T19:13:22.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:21 smithi081 ceph-mon[39192]: 4.c scrub starts 2024-09-21T19:13:22.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:21 smithi081 ceph-mon[39192]: 4.c scrub ok 2024-09-21T19:13:22.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.103 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:22.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:22.204 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:21 smithi104 ceph-mon[32438]: 4.c scrub starts 2024-09-21T19:13:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:21 smithi104 ceph-mon[32438]: 4.c scrub ok 2024-09-21T19:13:22.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:22.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:22.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:22.508 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:22.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:22.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:22.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T19:13:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.642 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:22.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:22.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:22.770 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:22.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:22.893 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.0 primary-affinity to 0.0321384 (821062) 2024-09-21T19:13:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:22.902 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:22.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:23.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:23.034 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: 1.b scrub starts 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: 1.b scrub ok 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: 4.1 scrub starts 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: 4.1 scrub ok 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: pgmap v1016: 178 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 174 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 72 KiB/s rd, 0 B/s wr, 770 op/s; 1332/58932 objects misplaced (2.260%); 26 MiB/s, 418 objects/s recovering 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1276228602' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435} : dispatch 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: from='client.36752 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435} : dispatch 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:23.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: 1.b scrub starts 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: 1.b scrub ok 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: 4.1 scrub starts 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: 4.1 scrub ok 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: pgmap v1016: 178 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 174 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 72 KiB/s rd, 0 B/s wr, 770 op/s; 1332/58932 objects misplaced (2.260%); 26 MiB/s, 418 objects/s recovering 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1276228602' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435} : dispatch 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: from='client.36752 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435} : dispatch 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:23.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:22 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:23.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.158 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:23.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:23.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: 1.b scrub starts 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: 1.b scrub ok 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: 4.1 scrub starts 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: 4.1 scrub ok 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: pgmap v1016: 178 pgs: 1 peering, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 174 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 72 KiB/s rd, 0 B/s wr, 770 op/s; 1332/58932 objects misplaced (2.260%); 26 MiB/s, 418 objects/s recovering 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1276228602' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435} : dispatch 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: from='client.36752 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435} : dispatch 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:23.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:22 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:23.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:23.281 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:23.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:23.405 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:23.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:23.529 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:23.531 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-21T19:13:23.531 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-09-21T19:13:23.531 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219766 9.2 9:4fcff26f:::benchmark_data_smithi081_131274_object91911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.749045+0000", 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 477.73809714700002, 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0377334439999999, 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219766 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749045+0000", 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749045+0000", 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749046+0000", 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.511e-06 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.533 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749051+0000", 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2490000000000003e-06 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749058+0000", 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6520000000000006e-06 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.749064+0000", 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5629999999999999e-06 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.534 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.751794+0000", 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027304389999999999 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.751874+0000", 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9385e-05 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2,7", 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.751994+0000", 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000120357 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.535 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.754636+0000", 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026419350000000002 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.754934+0000", 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029846499999999998 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.765492+0000", 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010557676 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.536 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.765502+0000", 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5240000000000006e-06 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.786702+0000", 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.021200541 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.786778+0000", 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6147000000000001e-05 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.537 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:219801 9.2 9:490b2d1b:::benchmark_data_smithi081_131274_object91946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:25.783370+0000", 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 477.70377186899998, 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0096375900000001, 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.538 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 219801 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783370+0000", 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.539 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783370+0000", 2024-09-21T19:13:23.540 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.540 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783372+0000", 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.035e-06 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783378+0000", 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0310000000000004e-06 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783392+0000", 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3685e-05 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783404+0000", 2024-09-21T19:13:23.542 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1819e-05 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783474+0000", 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0130999999999997e-05 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.543 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:25.783482+0000", 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6300000000000004e-06 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.787211+0000", 2024-09-21T19:13:23.544 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0037284470000001 2024-09-21T19:13:23.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.545 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.545 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.787309+0000", 2024-09-21T19:13:23.545 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8555999999999996e-05 2024-09-21T19:13:23.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,7", 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.787420+0000", 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110451 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792822+0000", 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005402095 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792917+0000", 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5041e-05 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792929+0000", 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2169e-05 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.547 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.792972+0000", 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3120000000000001e-05 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:26.793008+0000", 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5379999999999997e-05 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:243993 9.2 9:4d91acac:::benchmark_data_smithi081_131274_object116138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:13:23.548 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.939817+0000", 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 455.54732487199999, 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0272507099999999, 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 243993 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939817+0000", 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939817+0000", 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939819+0000", 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7969999999999999e-06 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939824+0000", 2024-09-21T19:13:23.550 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9339999999999997e-06 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939836+0000", 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2277e-05 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939845+0000", 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3799999999999994e-06 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939906+0000", 2024-09-21T19:13:23.551 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1457999999999993e-05 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.939915+0000", 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6319999999999997e-06 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.956102+0000", 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.016187784 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.956213+0000", 2024-09-21T19:13:23.552 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011089 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.956332+0000", 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118491 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.961700+0000", 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053683489999999997 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966900+0000", 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051996409999999996 2024-09-21T19:13:23.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966975+0000", 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4986000000000003e-05 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967026+0000", 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1731999999999999e-05 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.967068+0000", 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1359000000000002e-05 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244009 9.2 9:43e65944:::benchmark_data_smithi081_131274_object116154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.948852+0000", 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 455.53828977900002, 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0234964339999999, 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.555 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244009 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948852+0000", 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948852+0000", 2024-09-21T19:13:23.556 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948854+0000", 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.407e-06 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948858+0000", 2024-09-21T19:13:23.557 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3919999999999996e-06 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948868+0000", 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8320000000000001e-06 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948873+0000", 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7469999999999996e-06 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948922+0000", 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9359e-05 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.948927+0000", 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.49e-06 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959174+0000", 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.01024611 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.559 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959284+0000", 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000110189 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.959396+0000", 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112038 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966862+0000", 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074661299999999996 2024-09-21T19:13:23.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966940+0000", 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8193000000000002e-05 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972236+0000", 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005295803 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972287+0000", 2024-09-21T19:13:23.561 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0955999999999998e-05 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972349+0000", 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1787999999999999e-05 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244035 9.2 9:4644a85f:::benchmark_data_smithi081_131274_object116180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.965060+0000", 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 455.52208225099997, 2024-09-21T19:13:23.562 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.007491846, 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244035 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965060+0000", 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965060+0000", 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965061+0000", 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.249e-06 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965065+0000", 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1069999999999998e-06 2024-09-21T19:13:23.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965075+0000", 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8849999999999994e-06 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965081+0000", 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4330000000000001e-06 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965132+0000", 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0871999999999997e-05 2024-09-21T19:13:23.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.965140+0000", 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5619999999999998e-06 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.961731+0000", 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.99659121699999997 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.961830+0000", 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8666999999999996e-05 2024-09-21T19:13:23.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.962122+0000", 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029182999999999998 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.966911+0000", 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047890169999999996 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972395+0000", 2024-09-21T19:13:23.567 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054844339999999998 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972469+0000", 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4361000000000002e-05 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972515+0000", 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5766999999999997e-05 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972552+0000", 2024-09-21T19:13:23.568 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6445000000000001e-05 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244062 9.2 9:4e3e292f:::benchmark_data_smithi081_131274_object116207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:47.989381+0000", 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 455.49776051499998, 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98591870999999998, 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.569 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244062 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989381+0000", 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989381+0000", 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989383+0000", 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.206e-06 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989386+0000", 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7110000000000001e-06 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989395+0000", 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.259e-06 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.571 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.989400+0000", 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1950000000000002e-06 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.999601+0000", 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010201412 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:47.999608+0000", 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2019999999999998e-06 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.572 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964532+0000", 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96492351600000004 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964625+0000", 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2520000000000002e-05 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.964724+0000", 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9300000000000001e-05 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.573 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972430+0000", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0077064070000000002 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.972591+0000", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016030600000000001 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975215+0000", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026247470000000002 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975259+0000", 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3557000000000003e-05 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975300+0000", 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1372000000000003e-05 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.15816.0:244072 9.2 9:4b86ab9c:::benchmark_data_smithi081_131274_object116217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:05:48.011542+0000", 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 455.47559951099998, 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.016618896, 2024-09-21T19:13:23.575 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.15816", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3288171005", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 244072 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011542+0000", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011542+0000", 2024-09-21T19:13:23.576 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011544+0000", 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3459999999999999e-06 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011548+0000", 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9339999999999999e-06 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011556+0000", 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3450000000000006e-06 2024-09-21T19:13:23.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011561+0000", 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6229999999999996e-06 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011599+0000", 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8726000000000002e-05 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.011605+0000", 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0749999999999997e-06 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.969646+0000", 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.958041052 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.969767+0000", 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012104600000000001 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,2", 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.969864+0000", 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6974999999999999e-05 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.579 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:48.975402+0000", 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055379100000000001 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.027823+0000", 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052421855000000003 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.027886+0000", 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2778000000000001e-05 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.580 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.028117+0000", 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023073200000000001 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:05:49.028161+0000", 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4499000000000002e-05 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156444 11.19 11:9f6d0501:::benchmark_data_smithi081_181703_object32823:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.581 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935859+0000", 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.551282549999996, 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0501986169999999, 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156444 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935859+0000", 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935859+0000", 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935861+0000", 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.703e-06 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935866+0000", 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1019999999999996e-06 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935878+0000", 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1449e-05 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935886+0000", 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6880000000000005e-06 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.584 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.158759+0000", 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.222872665 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.158770+0000", 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0922e-05 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.985949+0000", 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.82717933499999996 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.585 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.986058+0000", 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010875299999999999 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156454 11.8 11:13baaa4d:::benchmark_data_smithi081_181703_object32833:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935900+0000", 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.55124223, 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0130355579999999, 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.586 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156454 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935900+0000", 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.587 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935900+0000", 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935900+0000", 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8100000000000003e-07 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935903+0000", 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.988e-06 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.588 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935909+0000", 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5799999999999999e-06 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935914+0000", 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.48e-06 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201571+0000", 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26565695299999997 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.589 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201580+0000", 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6789999999999994e-06 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.948807+0000", 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.74722684800000005 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.948935+0000", 2024-09-21T19:13:23.590 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012844899999999999 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156458 11.18 11:1be74b26:::benchmark_data_smithi081_181703_object32837:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935923+0000", 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.551219497000005, 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0577288789999999, 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.591 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156458 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935923+0000", 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935923+0000", 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.592 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935923+0000", 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9599999999999999e-07 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935925+0000", 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3549999999999999e-06 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935931+0000", 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3070000000000002e-06 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.593 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935936+0000", 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1699999999999996e-06 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201602+0000", 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26566605399999998 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201608+0000", 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8109999999999999e-06 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.594 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.993583+0000", 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.791975278 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.993651+0000", 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8408e-05 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156461 11.19 11:98ba99bb:::benchmark_data_smithi081_181703_object32840:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935941+0000", 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.551200761999993, 2024-09-21T19:13:23.595 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.050710335, 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156461 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935941+0000", 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.596 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935941+0000", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935942+0000", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4200000000000001e-07 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935944+0000", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0619999999999999e-06 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935948+0000", 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3760000000000001e-06 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935953+0000", 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7500000000000003e-06 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.158791+0000", 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.22283847600000001 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.158797+0000", 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6099999999999997e-06 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.986588+0000", 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.82779117700000004 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.986652+0000", 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3441999999999993e-05 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.599 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156466 11.19 11:9ebc7e0d:::benchmark_data_smithi081_181703_object32845:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935958+0000", 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.551183551999998, 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0507650019999999, 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156466 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.600 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935958+0000", 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935958+0000", 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935959+0000", 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3000000000000001e-07 2024-09-21T19:13:23.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935961+0000", 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2000000000000001e-06 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935966+0000", 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.69e-06 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935970+0000", 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6990000000000002e-06 2024-09-21T19:13:23.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.158821+0000", 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.22285038700000001 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.158830+0000", 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2839999999999995e-06 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.986677+0000", 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.82784650699999995 2024-09-21T19:13:23.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.986723+0000", 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6805000000000003e-05 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156471 11.8 11:13de6304:::benchmark_data_smithi081_181703_object32850:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935976+0000", 2024-09-21T19:13:23.604 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.551166249999994, 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.015219951, 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156471 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.605 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935976+0000", 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935976+0000", 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935976+0000", 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0699999999999998e-07 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.606 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935978+0000", 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.142e-06 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935983+0000", 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2509999999999996e-06 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935987+0000", 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2579999999999997e-06 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.607 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201629+0000", 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26564200300000002 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201635+0000", 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5030000000000002e-06 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.951132+0000", 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.74949664800000004 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.608 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.951196+0000", 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3738999999999994e-05 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:156476 11.18 11:1ed8e114:::benchmark_data_smithi081_181703_object32855:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:12:05.935991+0000", 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 77.551150707000005, 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0668038179999999, 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.609 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 156476 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935991+0000", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935991+0000", 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935992+0000", 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.96e-07 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935994+0000", 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1679999999999998e-06 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.935998+0000", 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.442e-06 2024-09-21T19:13:23.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:05.936003+0000", 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.87e-06 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201655+0000", 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.26565212999999999 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.201661+0000", 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2619999999999998e-06 2024-09-21T19:13:23.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.994113+0000", 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.79245285600000004 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.994227+0000", 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113199 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.994494+0000", 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026774500000000001 2024-09-21T19:13:23.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:06.997559+0000", 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030648759999999998 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:07.000452+0000", 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028929429999999998 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:07.002708+0000", 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022557050000000002 2024-09-21T19:13:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:07.002755+0000", 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7182000000000003e-05 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:12:07.002795+0000", 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0043999999999998e-05 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.615 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:228454 11.a 11:552e7fbe:::benchmark_data_smithi081_181703_object104833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:13:18.909928+0000", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.577213553, 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.98257016100000005, 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 228454 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.909928+0000", 2024-09-21T19:13:23.616 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.909928+0000", 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.909931+0000", 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1430000000000001e-06 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.909937+0000", 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9270000000000002e-06 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.909949+0000", 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2865e-05 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.909958+0000", 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4659999999999999e-06 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.618 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.910090+0000", 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000132471 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.910099+0000", 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2709999999999992e-06 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.887310+0000", 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.977210947 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.619 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.887443+0000", 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000133256 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.887575+0000", 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013202099999999999 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892315+0000", 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047399529999999999 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.620 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892370+0000", 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5445999999999997e-05 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892387+0000", 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6487999999999999e-05 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892441+0000", 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4694000000000002e-05 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892499+0000", 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7213000000000003e-05 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:228469 11.a 11:5c2d0c85:::benchmark_data_smithi081_181703_object104848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:13:18.924655+0000", 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.5624865779999997, 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.97037099900000001, 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.622 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 228469 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924655+0000", 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924655+0000", 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924658+0000", 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.216e-06 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924663+0000", 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4240000000000001e-06 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.624 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924672+0000", 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2529999999999993e-06 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924680+0000", 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2400000000000001e-06 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924739+0000", 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9024999999999998e-05 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.924746+0000", 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7179999999999996e-06 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.889982+0000", 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.96523525899999996 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.890111+0000", 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012899799999999999 2024-09-21T19:13:23.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.890251+0000", 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140894 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892528+0000", 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002277009 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892640+0000", 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011171999999999999 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.894915+0000", 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022751189999999999 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.894979+0000", 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3940000000000001e-05 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.895026+0000", 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7184000000000001e-05 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:228497 11.a 11:5ea058a2:::benchmark_data_smithi081_181703_object104876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:13:18.949229+0000", 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.5379132030000005, 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.94833198699999999, 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.629 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 228497 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949229+0000", 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949229+0000", 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949231+0000", 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8640000000000001e-06 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949236+0000", 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.755e-06 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.631 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949251+0000", 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4802e-05 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949258+0000", 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.272e-06 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949467+0000", 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020875599999999999 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.632 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.949476+0000", 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2280000000000004e-06 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892680+0000", 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.943203087 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892804+0000", 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012463 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.633 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.892932+0000", 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128279 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897275+0000", 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043421229999999998 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897343+0000", 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8553000000000006e-05 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.634 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897467+0000", 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012397299999999999 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897518+0000", 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0470999999999998e-05 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897561+0000", 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3194e-05 2024-09-21T19:13:23.635 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:228514 11.a 11:5cd495da:::benchmark_data_smithi081_181703_object104893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:13:18.960376+0000", 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.5267661910000001, 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.94499018800000001, 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.636 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 228514 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960376+0000", 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960376+0000", 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960392+0000", 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5695000000000001e-05 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960475+0000", 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3891e-05 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960489+0000", 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3302000000000001e-05 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.638 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960496+0000", 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6179999999999997e-06 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960555+0000", 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9006999999999999e-05 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.960567+0000", 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1250000000000001e-05 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.639 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.895078+0000", 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.93451157299999998 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.895180+0000", 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000102119 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.895268+0000", 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7862999999999995e-05 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897392+0000", 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021242240000000001 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897447+0000", 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4719000000000002e-05 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905261+0000", 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078138989999999992 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905309+0000", 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8211999999999999e-05 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905366+0000", 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6815999999999999e-05 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:228526 11.a 11:532d49c7:::benchmark_data_smithi081_181703_object104905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:13:18.974866+0000", 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.5122760629999998, 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.93062847800000004, 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.643 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 228526 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974866+0000", 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.644 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974866+0000", 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974868+0000", 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.26e-06 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974875+0000", 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3559999999999998e-06 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974889+0000", 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3956e-05 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974896+0000", 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6109999999999996e-06 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974953+0000", 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7238999999999999e-05 2024-09-21T19:13:23.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.974962+0000", 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8699999999999998e-06 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897603+0000", 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.92264108899999997 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897683+0000", 2024-09-21T19:13:23.647 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9314000000000007e-05 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.897753+0000", 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0066000000000003e-05 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905245+0000", 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074927919999999999 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905402+0000", 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015671399999999999 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905415+0000", 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3074000000000001e-05 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905451+0000", 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5357e-05 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.905494+0000", 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3779999999999998e-05 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.36497.0:228551 11.a 11:5f0e449a:::benchmark_data_smithi081_181703_object104930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-21T19:13:18.988821+0000", 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.498321271, 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.92405665199999998, 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-09-21T19:13:23.650 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.36497", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 228551 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988821+0000", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988821+0000", 2024-09-21T19:13:23.651 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988821+0000", 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4199999999999995e-07 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988825+0000", 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1429999999999998e-06 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-09-21T19:13:23.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988831+0000", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1070000000000001e-06 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988836+0000", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2229999999999998e-06 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988891+0000", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5096000000000002e-05 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for peered", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:18.988895+0000", 2024-09-21T19:13:23.653 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6490000000000001e-06 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.908555+0000", 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.91966014399999996 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.908656+0000", 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010078599999999999 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.654 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for subops from 1,6", 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.908772+0000", 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116376 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.912661+0000", 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038886390000000002 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.912778+0000", 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117367 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op_commit", 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.912793+0000", 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5217e-05 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.912838+0000", 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4990000000000001e-05 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-09-21T19:13:19.912877+0000", 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9273000000000002e-05 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-21T19:13:23.657 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-21T19:13:23.657 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-21T19:13:23.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:23.679 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:23.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:23.804 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:23.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:23.941 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:23.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:24.042 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:24.065 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:24.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:24.196 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:24.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:23 smithi104 ceph-mon[32438]: from='client.36752 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435}]': finished 2024-09-21T19:13:24.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:23 smithi104 ceph-mon[32438]: osdmap e494: 8 total, 8 up, 7 in 2024-09-21T19:13:24.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:24.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:23 smithi081 ceph-mon[29384]: from='client.36752 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435}]': finished 2024-09-21T19:13:24.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:23 smithi081 ceph-mon[29384]: osdmap e494: 8 total, 8 up, 7 in 2024-09-21T19:13:24.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:23 smithi081 ceph-mon[39192]: from='client.36752 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.03213840607076435}]': finished 2024-09-21T19:13:24.318 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:23 smithi081 ceph-mon[39192]: osdmap e494: 8 total, 8 up, 7 in 2024-09-21T19:13:24.334 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:24.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:24.476 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:24.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.613 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:24.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:24.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:24.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T19:13:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:24.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:24.908 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:24.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:25.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:25.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[29384]: pgmap v1018: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 982 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 66 KiB/s rd, 0 B/s wr, 1.25k op/s; 1021/47721 objects misplaced (2.140%); 20 MiB/s, 315 objects/s recovering 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[29384]: Health check failed: Reduced data availability: 6 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[29384]: osdmap e495: 8 total, 8 up, 7 in 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[29384]: osdmap e496: 8 total, 8 up, 7 in 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[39192]: pgmap v1018: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 982 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 66 KiB/s rd, 0 B/s wr, 1.25k op/s; 1021/47721 objects misplaced (2.140%); 20 MiB/s, 315 objects/s recovering 2024-09-21T19:13:25.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[39192]: Health check failed: Reduced data availability: 6 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T19:13:25.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[39192]: osdmap e495: 8 total, 8 up, 7 in 2024-09-21T19:13:25.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:24 smithi081 ceph-mon[39192]: osdmap e496: 8 total, 8 up, 7 in 2024-09-21T19:13:25.121 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:25.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:25.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:24 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:13:25.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:24 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:13:25.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:24 smithi104 ceph-mon[32438]: pgmap v1018: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 982 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 66 KiB/s rd, 0 B/s wr, 1.25k op/s; 1021/47721 objects misplaced (2.140%); 20 MiB/s, 315 objects/s recovering 2024-09-21T19:13:25.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:24 smithi104 ceph-mon[32438]: Health check failed: Reduced data availability: 6 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T19:13:25.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:24 smithi104 ceph-mon[32438]: osdmap e495: 8 total, 8 up, 7 in 2024-09-21T19:13:25.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:24 smithi104 ceph-mon[32438]: osdmap e496: 8 total, 8 up, 7 in 2024-09-21T19:13:25.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:25.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:25.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:25.462 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.466 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:25.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:25.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:25.607 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36497.0:235121 11.15 e496/478 11:a94e389b:::benchmark_data_smithi081_181703_object111500:head v 496'12189, mlcod=496'12187)", 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:25.688396+0000", 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.0021137740000000001, 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002144553, 2024-09-21T19:13:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688396+0000", 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688396+0000", 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688397+0000", 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3149999999999999e-06 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688403+0000", 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4910000000000002e-06 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688416+0000", 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2490999999999999e-05 2024-09-21T19:13:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688422+0000", 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5069999999999997e-06 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688486+0000", 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4125e-05 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.688511+0000", 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4913e-05 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_repop(client.36497.0:235124 11.c e496/414)", 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:25.690484+0000", 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 2.5715999999999999e-05, 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000106759, 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:25.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690484+0000", 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690484+0000", 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690485+0000", 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.468e-06 2024-09-21T19:13:25.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690495+0000", 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.781e-06 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690508+0000", 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3648999999999999e-05 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690515+0000", 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8310000000000003e-06 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:25.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:25.690582+0000", 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6797000000000004e-05 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 2 2024-09-21T19:13:25.701 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:25.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:25.751 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:25.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:25.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:25.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:26.021 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:26.033 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:25 smithi081 ceph-mon[39192]: 1.12 scrub starts 2024-09-21T19:13:26.034 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:25 smithi081 ceph-mon[39192]: 1.12 scrub ok 2024-09-21T19:13:26.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:25 smithi081 ceph-mon[29384]: 1.12 scrub starts 2024-09-21T19:13:26.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:25 smithi081 ceph-mon[29384]: 1.12 scrub ok 2024-09-21T19:13:26.109 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:26.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:26.181 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:26.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:25 smithi104 ceph-mon[32438]: 1.12 scrub starts 2024-09-21T19:13:26.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:25 smithi104 ceph-mon[32438]: 1.12 scrub ok 2024-09-21T19:13:26.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:26.310 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:26.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:26.447 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:26.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T19:13:26.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:26.583 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:26.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:26.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:26.856 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:26.941 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is set 2024-09-21T19:13:26.950 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:26.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:26.984 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:26.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:27.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.117 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:27.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: 5.13 scrub starts 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: 5.13 scrub ok 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: 1.10 scrub starts 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: 1.10 scrub ok 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: pgmap v1021: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 942 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 1021/45777 objects misplaced (2.230%) 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2362034663' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:13:27.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[39192]: from='client.36758 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: 5.13 scrub starts 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: 5.13 scrub ok 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: 1.10 scrub starts 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: 1.10 scrub ok 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: pgmap v1021: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 942 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 1021/45777 objects misplaced (2.230%) 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2362034663' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:13:27.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:26 smithi081 ceph-mon[29384]: from='client.36758 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:13:27.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:27.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: 5.13 scrub starts 2024-09-21T19:13:27.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: 5.13 scrub ok 2024-09-21T19:13:27.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: 1.10 scrub starts 2024-09-21T19:13:27.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: 1.10 scrub ok 2024-09-21T19:13:27.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: pgmap v1021: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 942 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 1021/45777 objects misplaced (2.230%) 2024-09-21T19:13:27.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2362034663' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:13:27.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:26 smithi104 ceph-mon[32438]: from='client.36758 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T19:13:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:27.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:27.407 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:27.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:27.556 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:27.556 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:13:27.556 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:13:27.556 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:13:27.556 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.558 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:27.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:27.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:13:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:13:27.659 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.690 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:27.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:27.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:27.820 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:27.921 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:27.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:28.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: 11.17 scrub starts 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: 11.17 scrub ok 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: 3.3 scrub starts 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: 3.3 scrub ok 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: from='client.36758 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[39192]: osdmap e497: 8 total, 8 up, 7 in 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: 11.17 scrub starts 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: 11.17 scrub ok 2024-09-21T19:13:28.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: 3.3 scrub starts 2024-09-21T19:13:28.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: 3.3 scrub ok 2024-09-21T19:13:28.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:28.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: from='client.36758 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:13:28.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:27 smithi081 ceph-mon[29384]: osdmap e497: 8 total, 8 up, 7 in 2024-09-21T19:13:28.072 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:28.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: 11.17 scrub starts 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: 11.17 scrub ok 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: 3.3 scrub starts 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: 3.3 scrub ok 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: from='client.36758 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T19:13:28.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:27 smithi104 ceph-mon[32438]: osdmap e497: 8 total, 8 up, 7 in 2024-09-21T19:13:28.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T19:13:28.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:28.346 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:28.382 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:13:28.382 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:13:28.382 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:13:28.382 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:13:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:13:28.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:13:28.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:13:28.384 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:13:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:13:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:13:28.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:28.481 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:28.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:28.624 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:28.706 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:28.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:28.769 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:28.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:28.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:28.909 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:28.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:29.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.044 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:29.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:29.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:29.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:28 smithi104 ceph-mon[32438]: pgmap v1023: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 850 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 204 B/s rd, 0 op/s; 873/41367 objects misplaced (2.110%) 2024-09-21T19:13:29.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:28 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/887749697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:29.159 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:13:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:13:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:29.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:29.310 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:29.310 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:29.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:28 smithi081 ceph-mon[29384]: pgmap v1023: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 850 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 204 B/s rd, 0 op/s; 873/41367 objects misplaced (2.110%) 2024-09-21T19:13:29.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:28 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/887749697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:28 smithi081 ceph-mon[39192]: pgmap v1023: 178 pgs: 13 peering, 2 active+remapped+backfill_toofull, 163 active+clean; 850 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 204 B/s rd, 0 op/s; 873/41367 objects misplaced (2.110%) 2024-09-21T19:13:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:28 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/887749697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.333 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:29.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:29.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.481 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:29.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:29.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:29.609 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:29.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:29.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:29.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:29.873 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:29.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:29.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:30.008 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:30.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T19:13:30.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:30.179 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:29 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:30.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:29 smithi104 ceph-mon[32438]: osdmap e498: 8 total, 8 up, 7 in 2024-09-21T19:13:30.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:30.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:29 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:30.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:29 smithi081 ceph-mon[29384]: osdmap e498: 8 total, 8 up, 7 in 2024-09-21T19:13:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:29 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:30.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:29 smithi081 ceph-mon[39192]: osdmap e498: 8 total, 8 up, 7 in 2024-09-21T19:13:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:30.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:30.451 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:30.485 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:30.498 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 6, 7, 0, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:13:30.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:13:30.499 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T19:13:30.499 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T19:13:30.499 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T19:13:30.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T19:13:30.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:30.688 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:30.725 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:30.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.866 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:30.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:30.967 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:30.996 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:30.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:31.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:31.135 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:31.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:30 smithi104 ceph-mon[32438]: pgmap v1025: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 714/28524 objects misplaced (2.503%) 2024-09-21T19:13:31.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:30 smithi104 ceph-mon[32438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2024-09-21T19:13:31.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:31.279 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:31.315 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:30 smithi081 ceph-mon[39192]: pgmap v1025: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 714/28524 objects misplaced (2.503%) 2024-09-21T19:13:31.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:30 smithi081 ceph-mon[39192]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2024-09-21T19:13:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:30 smithi081 ceph-mon[29384]: pgmap v1025: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 714/28524 objects misplaced (2.503%) 2024-09-21T19:13:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:30 smithi081 ceph-mon[29384]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2024-09-21T19:13:31.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:31.414 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:31.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T19:13:31.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.561 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:31.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:31.621 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:31.621 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:31.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:31.687 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:31.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:31.906 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:31.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:31.942 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:31.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:31.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T19:13:32.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:32.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:32.197 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:32.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:32.323 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:32.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:32.455 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:32.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:32.593 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:32.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T19:13:32.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:32.702 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:32.746 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:32.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:32.888 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:32.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:32.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:33.019 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:33.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:33.156 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:33.166 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:32 smithi104 ceph-mon[32438]: pgmap v1026: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 714/28524 objects misplaced (2.503%) 2024-09-21T19:13:33.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:33.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:32 smithi081 ceph-mon[29384]: pgmap v1026: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 714/28524 objects misplaced (2.503%) 2024-09-21T19:13:33.282 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:32 smithi081 ceph-mon[39192]: pgmap v1026: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 714/28524 objects misplaced (2.503%) 2024-09-21T19:13:33.304 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:33.318 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:33.318 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "" 2024-09-21T19:13:33.318 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:33.325 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:33.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:33.431 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:33.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:33.573 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:33.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:33.681 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:33.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:33.830 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:33.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:33.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:34.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:34.098 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:34.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:34.221 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:34.323 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:34.377 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:34.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.511 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:34.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:34.553 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:13:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:13:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:13:34.553 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:34.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:34.634 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:34.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:34.772 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:34.809 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Removed 123618 objects 2024-09-21T19:13:34.809 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Clean up completed and total clean up time :117.038 2024-09-21T19:13:34.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:34.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:34.965 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is set 2024-09-21T19:13:34.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:35.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:35.109 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T19:13:35.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T19:13:35.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:35.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:13:35.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:13:35.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:13:35.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:13:35.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:13:35.142 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:13:35.142 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:13:35.142 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:13:35.142 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:13:35.142 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:13:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.143 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:35.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:35.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:35.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:34 smithi104 ceph-mon[32438]: pgmap v1027: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 198 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 547 op/s; 307/10089 objects misplaced (3.043%) 2024-09-21T19:13:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1026173528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:34 smithi104 ceph-mon[32438]: from='client.26605 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:35.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:34 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1924366177' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:35.265 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[29384]: pgmap v1027: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 198 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 547 op/s; 307/10089 objects misplaced (3.043%) 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1026173528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[29384]: from='client.26605 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1924366177' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[39192]: pgmap v1027: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 198 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 547 op/s; 307/10089 objects misplaced (3.043%) 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1026173528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[39192]: from='client.26605 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:35.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:34 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1924366177' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:35.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T19:13:35.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:35.390 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:35.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.515 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:35.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:35.575 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:35.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:35.637 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:35.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:35.762 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:35.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:35.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:35.885 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:35.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:35.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:35 smithi104 ceph-mon[32438]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:35.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:35 smithi104 ceph-mon[32438]: from='client.26605 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:35.952 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:35 smithi104 ceph-mon[32438]: osdmap e499: 8 total, 8 up, 7 in 2024-09-21T19:13:35.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.009 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:36.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:36.061 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:36.061 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2024-09-21T19:13:36.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2024-09-21T19:13:36.061 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:156463 11.14 11:2e2279f1:::benchmark_data_smithi081_181703_object32842:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:05.935709+0000", 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.114584535000006, 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0539894190000001, 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 156463 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.935709+0000", 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.935709+0000", 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.935713+0000", 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4249999999999998e-06 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.935721+0000", 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6559999999999998e-06 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.935743+0000", 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1455999999999999e-05 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.935753+0000", 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0271e-05 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.071802+0000", 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.136049437 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.071810+0000", 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3599999999999998e-06 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.989644+0000", 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.91783409299999996 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.989699+0000", 2024-09-21T19:13:36.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4721e-05 2024-09-21T19:13:36.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.068 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:156467 11.5 11:a448736f:::benchmark_data_smithi081_181703_object32846:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:05.937089+0000", 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.113204820999997, 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0198332910000001, 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.069 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 156467 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937089+0000", 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937089+0000", 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937091+0000", 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7230000000000001e-06 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937097+0000", 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1099999999999999e-06 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937108+0000", 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1496e-05 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937117+0000", 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9400000000000008e-06 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.117314+0000", 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.18019663999999999 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.117324+0000", 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0659000000000001e-05 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.956843+0000", 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.83951884499999996 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.956922+0000", 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8877999999999999e-05 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:156469 11.4 11:27867e9b:::benchmark_data_smithi081_181703_object32848:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:05.937133+0000", 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.113160692999998, 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0369402649999999, 2024-09-21T19:13:36.074 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 156469 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937133+0000", 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937133+0000", 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937134+0000", 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7899999999999998e-07 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937136+0000", 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5639999999999999e-06 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937144+0000", 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6120000000000001e-06 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937149+0000", 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.259e-06 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937241+0000", 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1895e-05 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937247+0000", 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6289999999999998e-06 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.070644+0000", 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13339748700000001 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.070653+0000", 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5699999999999993e-06 2024-09-21T19:13:36.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.973988+0000", 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90333507199999996 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.974073+0000", 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5598000000000003e-05 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.079 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:156470 11.4 11:22dace03:::benchmark_data_smithi081_181703_object32849:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:05.937164+0000", 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.113129584000006, 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0530368400000001, 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 156470 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937164+0000", 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937164+0000", 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937165+0000", 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3099999999999997e-07 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937168+0000", 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8609999999999998e-06 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937172+0000", 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7119999999999999e-06 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937176+0000", 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2210000000000002e-06 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937288+0000", 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011250499999999999 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.937292+0000", 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.031e-06 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.070672+0000", 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.13338034400000001 2024-09-21T19:13:36.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.070679+0000", 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5610000000000004e-06 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.974374+0000", 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.90369481100000004 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for missing object", 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.974505+0000", 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000131199 2024-09-21T19:13:36.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.990131+0000", 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015625943999999999 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.990201+0000", 2024-09-21T19:13:36.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0019999999999997e-05 2024-09-21T19:13:36.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.086 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:156477 11.5 11:a3c79803:::benchmark_data_smithi081_181703_object32856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:05.977658+0000", 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 90.072635969000004, 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99028771299999996, 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 156477 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.088 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.977658+0000", 2024-09-21T19:13:36.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.977658+0000", 2024-09-21T19:13:36.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.977662+0000", 2024-09-21T19:13:36.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1649999999999999e-06 2024-09-21T19:13:36.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.977674+0000", 2024-09-21T19:13:36.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1601e-05 2024-09-21T19:13:36.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.977704+0000", 2024-09-21T19:13:36.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0583000000000002e-05 2024-09-21T19:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:05.977717+0000", 2024-09-21T19:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3169e-05 2024-09-21T19:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.117350+0000", 2024-09-21T19:13:36.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.139633127 2024-09-21T19:13:36.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.117361+0000", 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0657e-05 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.958047+0000", 2024-09-21T19:13:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.84068602199999998 2024-09-21T19:13:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.958166+0000", 2024-09-21T19:13:36.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011885899999999999 2024-09-21T19:13:36.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:13:36.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.958473+0000", 2024-09-21T19:13:36.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030743600000000002 2024-09-21T19:13:36.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.961216+0000", 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002742277 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.964824+0000", 2024-09-21T19:13:36.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036082449999999999 2024-09-21T19:13:36.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.967852+0000", 2024-09-21T19:13:36.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030284330000000001 2024-09-21T19:13:36.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.967912+0000", 2024-09-21T19:13:36.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9914999999999998e-05 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:06.967945+0000", 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3223999999999999e-05 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.101 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.102 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.102 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:207325 11.16 11:6a73c7ac:::benchmark_data_smithi081_181703_object83704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:13:36.102 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:56.107701+0000", 2024-09-21T19:13:36.102 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.942592337000001, 2024-09-21T19:13:36.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99062485099999997, 2024-09-21T19:13:36.103 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.103 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.103 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.103 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.103 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207325 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107701+0000", 2024-09-21T19:13:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107701+0000", 2024-09-21T19:13:36.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107703+0000", 2024-09-21T19:13:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.849e-06 2024-09-21T19:13:36.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107709+0000", 2024-09-21T19:13:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7300000000000002e-06 2024-09-21T19:13:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107719+0000", 2024-09-21T19:13:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5489999999999995e-06 2024-09-21T19:13:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107725+0000", 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9299999999999997e-06 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107768+0000", 2024-09-21T19:13:36.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2184999999999997e-05 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.107775+0000", 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8450000000000003e-06 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.092867+0000", 2024-09-21T19:13:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98509113400000003 2024-09-21T19:13:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.093011+0000", 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014427299999999999 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.093381+0000", 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037017899999999999 2024-09-21T19:13:36.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.095678+0000", 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022967819999999998 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098160+0000", 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024821750000000001 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098219+0000", 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8940000000000002e-05 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098281+0000", 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1722000000000003e-05 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098326+0000", 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5558e-05 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:207330 11.16 11:6e3452c4:::benchmark_data_smithi081_181703_object83709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:56.110578+0000", 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.939715778, 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99469028299999995, 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.115 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207330 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110578+0000", 2024-09-21T19:13:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110578+0000", 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110579+0000", 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5460000000000001e-06 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110584+0000", 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6460000000000001e-06 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110595+0000", 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0849000000000001e-05 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110601+0000", 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2840000000000003e-06 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110653+0000", 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1770000000000001e-05 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110662+0000", 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2499999999999995e-06 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.096139+0000", 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98547634200000001 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.096249+0000", 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110778 2024-09-21T19:13:36.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.096580+0000", 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00033013299999999998 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.100569+0000", 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039891950000000001 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.102612+0000", 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020435169999999999 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105116+0000", 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025039889999999999 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105225+0000", 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108517 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105268+0000", 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3467000000000002e-05 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:207331 11.16 11:6a48145c:::benchmark_data_smithi081_181703_object83710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:56.110706+0000", 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.939588155999999, 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99475101099999996, 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.124 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207331 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110706+0000", 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110706+0000", 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110707+0000", 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.071e-06 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110710+0000", 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4489999999999999e-06 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110718+0000", 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.157e-06 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110724+0000", 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8950000000000003e-06 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110782+0000", 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7747999999999997e-05 2024-09-21T19:13:36.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.110789+0000", 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.576e-06 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098473+0000", 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98768349099999997 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098588+0000", 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011521 2024-09-21T19:13:36.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098861+0000", 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027334599999999999 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.104569+0000", 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057074389999999999 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105313+0000", 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074362799999999998 2024-09-21T19:13:36.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105349+0000", 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6801000000000002e-05 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105420+0000", 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0691999999999993e-05 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105457+0000", 2024-09-21T19:13:36.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6507999999999997e-05 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:207334 11.17 11:e9e4024d:::benchmark_data_smithi081_181703_object83713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:56.111722+0000", 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.938572106999999, 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98984562200000004, 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.131 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207334 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111722+0000", 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111722+0000", 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111723+0000", 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4399999999999998e-07 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111726+0000", 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.777e-06 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111736+0000", 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1779999999999992e-06 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111741+0000", 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.519e-06 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111790+0000", 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9094999999999997e-05 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.111798+0000", 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2409999999999995e-06 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.094212+0000", 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98241396199999997 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.094323+0000", 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110746 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.094682+0000", 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035934299999999998 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.098431+0000", 2024-09-21T19:13:36.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037483 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.100233+0000", 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001801836 2024-09-21T19:13:36.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101458+0000", 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00122518 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101511+0000", 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.376e-05 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101567+0000", 2024-09-21T19:13:36.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5741e-05 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:207355 11.16 11:6d52b466:::benchmark_data_smithi081_181703_object83734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:56.124754+0000", 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.925539645999997, 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98088025000000001, 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.140 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207355 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124754+0000", 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124754+0000", 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124756+0000", 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5740000000000001e-06 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124760+0000", 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6360000000000002e-06 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124770+0000", 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0190000000000001e-05 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124778+0000", 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8029999999999997e-06 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124818+0000", 2024-09-21T19:13:36.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9258000000000003e-05 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.124825+0000", 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3529999999999997e-06 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.100613+0000", 2024-09-21T19:13:36.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97578828500000003 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.100728+0000", 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011435899999999999 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5", 2024-09-21T19:13:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101021+0000", 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029388 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.104574+0000", 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0035524419999999998 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105499+0000", 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092517299999999995 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105543+0000", 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3888999999999999e-05 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105594+0000", 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1527000000000003e-05 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.105634+0000", 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9881e-05 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:207368 11.17 11:eda5abb7:::benchmark_data_smithi081_181703_object83747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:12:56.135634+0000", 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 39.914659725999996, 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96605375599999999, 2024-09-21T19:13:36.150 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.151 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207368 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135634+0000", 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135634+0000", 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135636+0000", 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.945e-06 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135642+0000", 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2920000000000001e-06 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135655+0000", 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2918999999999999e-05 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135662+0000", 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8800000000000002e-06 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135710+0000", 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7902999999999999e-05 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:56.135716+0000", 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5390000000000004e-06 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.096893+0000", 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.96117675000000002 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.096982+0000", 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8318000000000004e-05 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 5,6", 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.097169+0000", 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000187939 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.100184+0000", 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030147479999999998 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.100856+0000", 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00067139000000000001 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101617+0000", 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00076174399999999996 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101647+0000", 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0074999999999999e-05 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:12:57.101688+0000", 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0314000000000001e-05 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:224181 11.5 11:a4959f67:::benchmark_data_smithi081_181703_object100560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-21T19:13:36.157 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:13.187497+0000", 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.862796325000001, 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0721749899999999, 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 224181 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187497+0000", 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187497+0000", 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187498+0000", 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0299999999999999e-07 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187500+0000", 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4090000000000001e-06 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187506+0000", 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6799999999999998e-06 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187511+0000", 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9989999999999999e-06 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187596+0000", 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5462000000000003e-05 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.187707+0000", 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011085499999999999 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,5,6", 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.188081+0000", 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000373278 2024-09-21T19:13:36.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.190362+0000", 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022816960000000002 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.193429+0000", 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030665710000000001 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.199060+0000", 2024-09-21T19:13:36.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056310700000000002 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.199069+0000", 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5489999999999995e-06 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.259552+0000", 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0604823800000001 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.259672+0000", 2024-09-21T19:13:36.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120538 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:224220 11.5 11:a5b52b2c:::benchmark_data_smithi081_181703_object100599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:13.228247+0000", 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.822046759999999, 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0368362289999999, 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.164 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 224220 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228247+0000", 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228247+0000", 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228248+0000", 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.513e-06 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228254+0000", 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2429999999999996e-06 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228268+0000", 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4027e-05 2024-09-21T19:13:36.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228275+0000", 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4080000000000001e-06 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228332+0000", 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6706999999999998e-05 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.228342+0000", 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.73e-06 2024-09-21T19:13:36.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.260021+0000", 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0316790309999999 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.260123+0000", 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102192 2024-09-21T19:13:36.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.260436+0000", 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031327900000000002 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.264924+0000", 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044876400000000002 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.264975+0000", 2024-09-21T19:13:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1075000000000001e-05 2024-09-21T19:13:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.264999+0000", 2024-09-21T19:13:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3938e-05 2024-09-21T19:13:36.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.265042+0000", 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3448000000000001e-05 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.265083+0000", 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0998000000000003e-05 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:224253 11.5 11:a0c9a2da:::benchmark_data_smithi081_181703_object100632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-21T19:13:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:13.259576+0000", 2024-09-21T19:13:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.790717397000002, 2024-09-21T19:13:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.007412843, 2024-09-21T19:13:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.174 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 224253 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.175 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259576+0000", 2024-09-21T19:13:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259576+0000", 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259578+0000", 2024-09-21T19:13:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6640000000000001e-06 2024-09-21T19:13:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259582+0000", 2024-09-21T19:13:36.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4360000000000004e-06 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259592+0000", 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8989999999999996e-06 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259600+0000", 2024-09-21T19:13:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4000000000000003e-06 2024-09-21T19:13:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259642+0000", 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2033999999999998e-05 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.259648+0000", 2024-09-21T19:13:36.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4280000000000001e-06 2024-09-21T19:13:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.261973+0000", 2024-09-21T19:13:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0023251689999999 2024-09-21T19:13:36.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.262040+0000", 2024-09-21T19:13:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6327000000000004e-05 2024-09-21T19:13:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.262208+0000", 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016852699999999999 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.265092+0000", 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028835419999999998 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.265143+0000", 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1650000000000002e-05 2024-09-21T19:13:36.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.266926+0000", 2024-09-21T19:13:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001782641 2024-09-21T19:13:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.266965+0000", 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8693999999999999e-05 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.266989+0000", 2024-09-21T19:13:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4431999999999999e-05 2024-09-21T19:13:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:224286 11.5 11:a0bc0036:::benchmark_data_smithi081_181703_object100665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:13.281326+0000", 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.768967987, 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98575713099999995, 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.189 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 224286 2024-09-21T19:13:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.190 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281326+0000", 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281326+0000", 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281327+0000", 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8700000000000004e-07 2024-09-21T19:13:36.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281331+0000", 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8840000000000004e-06 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281337+0000", 2024-09-21T19:13:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6189999999999996e-06 2024-09-21T19:13:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281343+0000", 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0920000000000003e-06 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281396+0000", 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2734e-05 2024-09-21T19:13:36.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:13.281403+0000", 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8900000000000001e-06 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.263270+0000", 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98186675400000001 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.263331+0000", 2024-09-21T19:13:36.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1427999999999995e-05 2024-09-21T19:13:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 3,6", 2024-09-21T19:13:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.263586+0000", 2024-09-21T19:13:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00025448300000000001 2024-09-21T19:13:36.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.266833+0000", 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032471219999999999 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.266876+0000", 2024-09-21T19:13:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3321000000000001e-05 2024-09-21T19:13:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.267025+0000", 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014943399999999999 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.267059+0000", 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4010000000000001e-05 2024-09-21T19:13:36.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:14.267083+0000", 2024-09-21T19:13:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3373000000000001e-05 2024-09-21T19:13:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:232406 11.19 11:9a24d715:::benchmark_data_smithi081_181703_object108785:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e494)", 2024-09-21T19:13:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:22.898712+0000", 2024-09-21T19:13:36.203 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.151581911999999, 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.054765001, 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232406 2024-09-21T19:13:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898712+0000", 2024-09-21T19:13:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898712+0000", 2024-09-21T19:13:36.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898713+0000", 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7400000000000002e-07 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898716+0000", 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6490000000000001e-06 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898723+0000", 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2779999999999999e-06 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898728+0000", 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9889999999999997e-06 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898790+0000", 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1136999999999994e-05 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.898799+0000", 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7040000000000006e-06 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.953436+0000", 2024-09-21T19:13:36.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.054636616 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.953477+0000", 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0853999999999999e-05 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:232485 11.19 11:9baab22f:::benchmark_data_smithi081_181703_object108864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e494)", 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:22.978541+0000", 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.071753022999999, 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99477338699999995, 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.212 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232485 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978541+0000", 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978541+0000", 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978542+0000", 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.252e-06 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978546+0000", 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2239999999999997e-06 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978558+0000", 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1579e-05 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978575+0000", 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7475000000000001e-05 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978669+0000", 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4210999999999996e-05 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.978679+0000", 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2420000000000006e-06 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.954165+0000", 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97548625099999997 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.954262+0000", 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7214999999999997e-05 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:13:36.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.954629+0000", 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036734799999999998 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.969183+0000", 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014553178999999999 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.970941+0000", 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001758337 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.973209+0000", 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022677040000000002 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.973270+0000", 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1546999999999999e-05 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.973314+0000", 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3822999999999997e-05 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.218 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:232487 11.19 11:9813277a:::benchmark_data_smithi081_181703_object108866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e494)", 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:22.979257+0000", 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.071036883, 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.996351985, 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232487 2024-09-21T19:13:36.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979257+0000", 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979257+0000", 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979258+0000", 2024-09-21T19:13:36.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3629999999999999e-06 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979265+0000", 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.827e-06 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979277+0000", 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2439000000000001e-05 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979285+0000", 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0840000000000003e-06 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979401+0000", 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011689200000000001 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.979410+0000", 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2009999999999999e-06 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.958705+0000", 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97929508300000001 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.958818+0000", 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011351199999999999 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.959108+0000", 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028954499999999999 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.969186+0000", 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010078317999999999 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975313+0000", 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061270939999999996 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975527+0000", 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021396499999999999 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975581+0000", 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3931000000000003e-05 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975609+0000", 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7730999999999999e-05 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:232492 11.19 11:98b31c7f:::benchmark_data_smithi081_181703_object108871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e494)", 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:22.983992+0000", 2024-09-21T19:13:36.225 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.066301944999999, 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.99170212199999996, 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232492 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.983992+0000", 2024-09-21T19:13:36.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.983992+0000", 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.983993+0000", 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1100000000000004e-07 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.983996+0000", 2024-09-21T19:13:36.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7979999999999999e-06 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.984007+0000", 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0672e-05 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.984012+0000", 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9429999999999999e-06 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.984067+0000", 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4676000000000003e-05 2024-09-21T19:13:36.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.984078+0000", 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0954e-05 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.960667+0000", 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97658957800000001 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.960766+0000", 2024-09-21T19:13:36.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8698999999999998e-05 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.961010+0000", 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024446500000000003 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.969189+0000", 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081784409999999998 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975347+0000", 2024-09-21T19:13:36.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061582950000000003 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975640+0000", 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029324700000000001 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975666+0000", 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5459000000000001e-05 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975694+0000", 2024-09-21T19:13:36.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7983999999999998e-05 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.36497.0:232506 11.19 11:9ac76b85:::benchmark_data_smithi081_181703_object108885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e494)", 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2024-09-21T19:13:22.992420+0000", 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.057873324999999, 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.98335840100000005, 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "continuous": false, 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2024-09-21T19:13:36.232 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.36497", 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "v1:172.21.15.81:0/3207462442", 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 232506 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992420+0000", 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992420+0000", 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992422+0000", 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.691e-06 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992427+0000", 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8559999999999998e-06 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992440+0000", 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3072e-05 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992447+0000", 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.041e-06 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992500+0000", 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2886999999999997e-05 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for peered", 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:22.992508+0000", 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.242e-06 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.963061+0000", 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.97055330699999998 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.963186+0000", 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012411 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for subops from 0,2", 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.963455+0000", 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026926000000000001 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op_commit", 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.969193+0000", 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057379689999999999 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975366+0000", 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061729710000000002 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_commit_rec", 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975722+0000", 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035658800000000002 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975757+0000", 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5067999999999997e-05 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2024-09-21T19:13:23.975779+0000", 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1339e-05 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T19:13:36.238 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:36.260 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:35 smithi081 ceph-mon[29384]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:35 smithi081 ceph-mon[29384]: from='client.26605 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:36.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:35 smithi081 ceph-mon[29384]: osdmap e499: 8 total, 8 up, 7 in 2024-09-21T19:13:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:35 smithi081 ceph-mon[39192]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:35 smithi081 ceph-mon[39192]: from='client.26605 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:36.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:35 smithi081 ceph-mon[39192]: osdmap e499: 8 total, 8 up, 7 in 2024-09-21T19:13:36.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.386 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:36.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:36.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:36.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:36.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:36.734 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:36.755 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:36.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:36.878 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:36.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:36.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.001 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:37.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:37.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:37.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T19:13:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.124 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:37.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:37.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:37.235 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:36 smithi081 ceph-mon[39192]: pgmap v1029: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 147 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.40k op/s; 307/7617 objects misplaced (4.030%) 2024-09-21T19:13:37.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:36 smithi081 ceph-mon[29384]: pgmap v1029: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 147 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.40k op/s; 307/7617 objects misplaced (4.030%) 2024-09-21T19:13:37.236 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:13:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:13:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:13:37.247 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:37.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:37.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:36 smithi104 ceph-mon[32438]: pgmap v1029: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 147 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.40k op/s; 307/7617 objects misplaced (4.030%) 2024-09-21T19:13:37.320 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:37.349 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:37.369 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.371 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:37.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:37.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:37.493 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:37.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7 2024-09-21T19:13:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.617 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:37.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.7' 2024-09-21T19:13:37.718 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:37.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:37.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:37.864 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:37.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:37.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:37.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:38.014 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:38.014 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:13:38.014 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2024-09-21T19:13:38.014 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:38.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0 2024-09-21T19:13:38.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.0' 2024-09-21T19:13:38.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:38.234 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:38.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2 2024-09-21T19:13:38.356 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.2' 2024-09-21T19:13:38.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4 2024-09-21T19:13:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.480 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:38.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.4' 2024-09-21T19:13:38.524 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_4' removed 2024-09-21T19:13:38.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:38.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T19:13:38.618 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:38.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3 2024-09-21T19:13:38.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T19:13:38.740 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.3' 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: pgmap v1030: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 91 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 980 op/s; 187/4938 objects misplaced (3.787%) 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T19:13:38.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1051584022' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: pgmap v1030: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 91 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 980 op/s; 187/4938 objects misplaced (3.787%) 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:38 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1051584022' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: pgmap v1030: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 91 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 980 op/s; 187/4938 objects misplaced (3.787%) 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T19:13:38.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1051584022' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T19:13:38.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6 2024-09-21T19:13:38.864 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.6' 2024-09-21T19:13:38.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:38.987 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:38.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:39.023 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 6, 7, 0, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 7, 6, 5] 2024-09-21T19:13:39.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T19:13:39.024 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-21T19:13:39.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd in 5 2024-09-21T19:13:39.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1 2024-09-21T19:13:39.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:13:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:13:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: docker: command not found 2024-09-21T19:13:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:39.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi081 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.1' 2024-09-21T19:13:39.212 DEBUG:teuthology.orchestra.run.smithi104:> sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5 2024-09-21T19:13:39.226 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T19:13:39.241 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T19:13:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:13:39.242 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: docker: command not found 2024-09-21T19:13:39.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi104 with status 1: 'sudo docker kill -s 1 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd.5' 2024-09-21T19:13:39.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T19:13:39.285 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:39.470 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T19:13:39.470 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2024-09-21T19:13:39.470 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2024-09-21T19:13:39.470 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2024-09-21T19:13:39.470 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1051584022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: osdmap e500: 8 total, 8 up, 7 in 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:13:39.707 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1051584022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: osdmap e500: 8 total, 8 up, 7 in 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:13:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:39 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1051584022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: osdmap e500: 8 total, 8 up, 7 in 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:13:39.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:39 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:39.873 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:39.906 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:40.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T19:13:40.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T19:13:40.012+0000 7ff1af1de640 -1 mon.a@0(leader).osd e502 definitely_dead 0 2024-09-21T19:13:40.324 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:40.418 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0: {} 2024-09-21T19:13:40.418 INFO:teuthology.orchestra.run.smithi081.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T19:13:40.434 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1: {} 2024-09-21T19:13:40.435 INFO:teuthology.orchestra.run.smithi081.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T19:13:40.450 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2: {} 2024-09-21T19:13:40.451 INFO:teuthology.orchestra.run.smithi081.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T19:13:40.467 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3: {} 2024-09-21T19:13:40.467 INFO:teuthology.orchestra.run.smithi081.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T19:13:40.485 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4: {} 2024-09-21T19:13:40.485 INFO:teuthology.orchestra.run.smithi081.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T19:13:40.501 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5: {} 2024-09-21T19:13:40.501 INFO:teuthology.orchestra.run.smithi081.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T19:13:40.518 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6: {} 2024-09-21T19:13:40.518 INFO:teuthology.orchestra.run.smithi081.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T19:13:40.535 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7: {} 2024-09-21T19:13:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[29384]: osdmap e501: 8 total, 8 up, 7 in 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[29384]: osdmap e502: 8 total, 8 up, 7 in 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[29384]: pgmap v1034: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 830 B/s rd, 0 op/s 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1028768519' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[29384]: from='client.36782 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3063555175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[39192]: osdmap e501: 8 total, 8 up, 7 in 2024-09-21T19:13:40.595 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[39192]: osdmap e502: 8 total, 8 up, 7 in 2024-09-21T19:13:40.596 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[39192]: pgmap v1034: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 830 B/s rd, 0 op/s 2024-09-21T19:13:40.596 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1028768519' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T19:13:40.596 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[39192]: from='client.36782 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T19:13:40.596 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3063555175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:40.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:40 smithi104 ceph-mon[32438]: osdmap e501: 8 total, 8 up, 7 in 2024-09-21T19:13:40.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:40 smithi104 ceph-mon[32438]: osdmap e502: 8 total, 8 up, 7 in 2024-09-21T19:13:40.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:40 smithi104 ceph-mon[32438]: pgmap v1034: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 830 B/s rd, 0 op/s 2024-09-21T19:13:40.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1028768519' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T19:13:40.748 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:40 smithi104 ceph-mon[32438]: from='client.36782 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T19:13:40.749 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3063555175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T19:13:40.847 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:40.847 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:40.853 INFO:teuthology.orchestra.run.smithi081.stderr:marked in osd.5. 2024-09-21T19:13:41.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T19:13:41.686 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:41.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:13:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:13:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T19:13:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: from='client.36782 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T19:13:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T19:13:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3567636358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:41.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:41 smithi104 ceph-mon[32438]: from='client.36794 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:13:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:13:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T19:13:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: from='client.36782 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T19:13:41.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3567636358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[29384]: from='client.36794 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: from='client.36782 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3567636358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:41.817 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:41 smithi081 ceph-mon[39192]: from='client.36794 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T19:13:41.854 INFO:teuthology.orchestra.run.smithi081.stderr:noscrub is unset 2024-09-21T19:13:42.025 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-21T19:13:42.281 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:42.281 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:42.650 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[39192]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[39192]: from='client.36794 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[39192]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[39192]: pgmap v1037: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[29384]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[29384]: from='client.36794 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[29384]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T19:13:42.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:42 smithi081 ceph-mon[29384]: pgmap v1037: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail 2024-09-21T19:13:42.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T19:13:42.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:42 smithi104 ceph-mon[32438]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T19:13:42.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:42 smithi104 ceph-mon[32438]: from='client.36794 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T19:13:42.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:42 smithi104 ceph-mon[32438]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T19:13:42.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:42 smithi104 ceph-mon[32438]: pgmap v1037: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.4 GiB used, 620 GiB / 626 GiB avail 2024-09-21T19:13:42.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T19:13:43.184 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:43.262 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:43.739 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:43.739 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:44.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 3.1c scrub starts 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 3.1c scrub ok 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 4.3 scrub starts 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 4.3 scrub ok 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 5.e scrub starts 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 5.e scrub ok 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 5.4 scrub starts 2024-09-21T19:13:44.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:43 smithi104 ceph-mon[32438]: 5.4 scrub ok 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 3.1c scrub starts 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 3.1c scrub ok 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 4.3 scrub starts 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 4.3 scrub ok 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 5.e scrub starts 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 5.e scrub ok 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 5.4 scrub starts 2024-09-21T19:13:44.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[29384]: 5.4 scrub ok 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 3.1c scrub starts 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 3.1c scrub ok 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 4.3 scrub starts 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 4.3 scrub ok 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 5.e scrub starts 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 5.e scrub ok 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 5.4 scrub starts 2024-09-21T19:13:44.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:43 smithi081 ceph-mon[39192]: 5.4 scrub ok 2024-09-21T19:13:44.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T19:13:44.880 INFO:teuthology.orchestra.run.smithi081.stderr:nodeep-scrub is unset 2024-09-21T19:13:44.912 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: 6.7 scrub starts 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: 6.7 scrub ok 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: 3.4 scrub starts 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: 3.4 scrub ok 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: pgmap v1038: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 59 KiB/s, 10 objects/s recovering 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3031724886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:45.007 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:44 smithi104 ceph-mon[32438]: from='client.36800 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:45.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: 6.7 scrub starts 2024-09-21T19:13:45.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: 6.7 scrub ok 2024-09-21T19:13:45.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: 3.4 scrub starts 2024-09-21T19:13:45.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: 3.4 scrub ok 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: pgmap v1038: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 59 KiB/s, 10 objects/s recovering 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3031724886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[29384]: from='client.36800 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: 6.7 scrub starts 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: 6.7 scrub ok 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: 3.4 scrub starts 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: 3.4 scrub ok 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: pgmap v1038: 152 pgs: 1 peering, 151 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 59 KiB/s, 10 objects/s recovering 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3031724886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:45.244 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:44 smithi081 ceph-mon[39192]: from='client.36800 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T19:13:45.430 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2024-09-21T19:13:45.430 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[29384]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[29384]: Cluster is now healthy 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[29384]: 4.f scrub starts 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[29384]: 4.f scrub ok 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[29384]: from='client.36800 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[29384]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[39192]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[39192]: Cluster is now healthy 2024-09-21T19:13:46.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[39192]: 4.f scrub starts 2024-09-21T19:13:46.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[39192]: 4.f scrub ok 2024-09-21T19:13:46.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[39192]: from='client.36800 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:46.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:45 smithi081 ceph-mon[39192]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T19:13:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:45 smithi104 ceph-mon[32438]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T19:13:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:45 smithi104 ceph-mon[32438]: Cluster is now healthy 2024-09-21T19:13:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:45 smithi104 ceph-mon[32438]: 4.f scrub starts 2024-09-21T19:13:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:45 smithi104 ceph-mon[32438]: 4.f scrub ok 2024-09-21T19:13:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:45 smithi104 ceph-mon[32438]: from='client.36800 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T19:13:46.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:45 smithi104 ceph-mon[32438]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T19:13:47.026 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T19:13:47.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 21 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: 2.1 scrub starts 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: 2.1 scrub ok 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: pgmap v1040: 152 pgs: 152 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43 KiB/s, 9 objects/s recovering 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: 3.1d scrub starts 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: 3.1d scrub ok 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: 6.1c scrub starts 2024-09-21T19:13:47.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[29384]: 6.1c scrub ok 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: 2.1 scrub starts 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: 2.1 scrub ok 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: pgmap v1040: 152 pgs: 152 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43 KiB/s, 9 objects/s recovering 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: 3.1d scrub starts 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: 3.1d scrub ok 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: 6.1c scrub starts 2024-09-21T19:13:47.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:46 smithi081 ceph-mon[39192]: 6.1c scrub ok 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: 2.1 scrub starts 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: 2.1 scrub ok 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: pgmap v1040: 152 pgs: 152 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43 KiB/s, 9 objects/s recovering 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: 3.1d scrub starts 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: 3.1d scrub ok 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: 6.1c scrub starts 2024-09-21T19:13:47.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:46 smithi104 ceph-mon[32438]: 6.1c scrub ok 2024-09-21T19:13:47.293 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:47.534 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:13:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:13:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:13:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:47 smithi104 ceph-mon[32438]: 4.4 scrub starts 2024-09-21T19:13:48.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:47 smithi104 ceph-mon[32438]: 4.4 scrub ok 2024-09-21T19:13:48.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:47 smithi081 ceph-mon[29384]: 4.4 scrub starts 2024-09-21T19:13:48.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:47 smithi081 ceph-mon[29384]: 4.4 scrub ok 2024-09-21T19:13:48.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:47 smithi081 ceph-mon[39192]: 4.4 scrub starts 2024-09-21T19:13:48.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:47 smithi081 ceph-mon[39192]: 4.4 scrub ok 2024-09-21T19:13:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 1 pgp_num to 21 2024-09-21T19:13:49.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:13:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:13:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:48 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:13:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:48 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:13:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:48 smithi104 ceph-mon[32438]: pgmap v1041: 152 pgs: 152 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 37 KiB/s, 8 objects/s recovering 2024-09-21T19:13:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2908801664' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T19:13:49.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:48 smithi104 ceph-mon[32438]: from='client.36806 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T19:13:49.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:13:49.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:13:49.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[39192]: pgmap v1041: 152 pgs: 152 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 37 KiB/s, 8 objects/s recovering 2024-09-21T19:13:49.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2908801664' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T19:13:49.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[39192]: from='client.36806 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T19:13:49.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:13:49.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:13:49.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[29384]: pgmap v1041: 152 pgs: 152 active+clean; 476 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 37 KiB/s, 8 objects/s recovering 2024-09-21T19:13:49.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2908801664' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T19:13:49.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:48 smithi081 ceph-mon[29384]: from='client.36806 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T19:13:49.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T19:13:50.145 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:49 smithi104 ceph-mon[32438]: 3.5 scrub starts 2024-09-21T19:13:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:49 smithi104 ceph-mon[32438]: 3.5 scrub ok 2024-09-21T19:13:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:49 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:49 smithi104 ceph-mon[32438]: from='client.36806 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T19:13:50.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:49 smithi104 ceph-mon[32438]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T19:13:50.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[29384]: 3.5 scrub starts 2024-09-21T19:13:50.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[29384]: 3.5 scrub ok 2024-09-21T19:13:50.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:50.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[29384]: from='client.36806 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T19:13:50.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[29384]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T19:13:50.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[39192]: 3.5 scrub starts 2024-09-21T19:13:50.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[39192]: 3.5 scrub ok 2024-09-21T19:13:50.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:13:50.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[39192]: from='client.36806 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T19:13:50.267 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:49 smithi081 ceph-mon[39192]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T19:13:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:mon.a: {} 2024-09-21T19:13:50.644 INFO:teuthology.orchestra.run.smithi081.stdout:mon.b: {} 2024-09-21T19:13:50.644 INFO:teuthology.orchestra.run.smithi081.stdout:mon.c: {} 2024-09-21T19:13:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T19:13:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T19:13:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T19:13:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:13:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:13:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T19:13:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: pgmap v1044: 152 pgs: 152 active+clean; 476 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-09-21T19:13:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: 3.16 scrub starts 2024-09-21T19:13:51.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: 3.16 scrub ok 2024-09-21T19:13:51.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:50 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1772563565' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T19:13:51.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: pgmap v1044: 152 pgs: 152 active+clean; 476 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: 3.16 scrub starts 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: 3.16 scrub ok 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1772563565' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: pgmap v1044: 152 pgs: 152 active+clean; 476 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: 3.16 scrub starts 2024-09-21T19:13:51.306 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: 3.16 scrub ok 2024-09-21T19:13:51.307 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:50 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1772563565' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T19:13:51.386 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T19:13:51.636 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[29384]: 1.14 scrub starts 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[29384]: 1.14 scrub ok 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[29384]: 1.7 scrub starts 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[29384]: 1.7 scrub ok 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[39192]: 1.14 scrub starts 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[39192]: 1.14 scrub ok 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[39192]: 1.7 scrub starts 2024-09-21T19:13:52.052 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:51 smithi081 ceph-mon[39192]: 1.7 scrub ok 2024-09-21T19:13:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:51 smithi104 ceph-mon[32438]: 1.14 scrub starts 2024-09-21T19:13:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:51 smithi104 ceph-mon[32438]: 1.14 scrub ok 2024-09-21T19:13:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:51 smithi104 ceph-mon[32438]: 1.7 scrub starts 2024-09-21T19:13:52.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:51 smithi104 ceph-mon[32438]: 1.7 scrub ok 2024-09-21T19:13:52.894 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T19:13:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:52 smithi104 ceph-mon[32438]: 1.c scrub starts 2024-09-21T19:13:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:52 smithi104 ceph-mon[32438]: 1.c scrub ok 2024-09-21T19:13:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:52 smithi104 ceph-mon[32438]: pgmap v1045: 152 pgs: 152 active+clean; 476 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-09-21T19:13:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:52 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2052190883' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:13:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:52 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:53.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:52 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[39192]: 1.c scrub starts 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[39192]: 1.c scrub ok 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[39192]: pgmap v1045: 152 pgs: 152 active+clean; 476 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2052190883' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:53.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[29384]: 1.c scrub starts 2024-09-21T19:13:53.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[29384]: 1.c scrub ok 2024-09-21T19:13:53.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[29384]: pgmap v1045: 152 pgs: 152 active+clean; 476 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-09-21T19:13:53.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2052190883' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:13:53.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:13:53.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:52 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:13:53.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T19:13:53.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: 4.14 scrub starts 2024-09-21T19:13:53.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: 4.14 scrub ok 2024-09-21T19:13:53.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: 1.13 scrub starts 2024-09-21T19:13:53.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: 1.13 scrub ok 2024-09-21T19:13:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2052190883' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:13:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T19:13:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: 3.17 scrub starts 2024-09-21T19:13:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[29384]: 3.17 scrub ok 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: 4.14 scrub starts 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: 4.14 scrub ok 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: 1.13 scrub starts 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: 1.13 scrub ok 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2052190883' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: 3.17 scrub starts 2024-09-21T19:13:54.163 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:53 smithi081 ceph-mon[39192]: 3.17 scrub ok 2024-09-21T19:13:54.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: 4.14 scrub starts 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: 4.14 scrub ok 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: 1.13 scrub starts 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: 1.13 scrub ok 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2052190883' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: 3.17 scrub starts 2024-09-21T19:13:54.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:53 smithi104 ceph-mon[32438]: 3.17 scrub ok 2024-09-21T19:13:54.905 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 1.5 scrub starts 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 1.5 scrub ok 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: pgmap v1047: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 1.11 scrub starts 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 1.11 scrub ok 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 1.2 scrub starts 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 1.2 scrub ok 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 4.1f scrub starts 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: 4.1f scrub ok 2024-09-21T19:13:55.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:54 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3626082347' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 1.5 scrub starts 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 1.5 scrub ok 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: pgmap v1047: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 1.11 scrub starts 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 1.11 scrub ok 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 1.2 scrub starts 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 1.2 scrub ok 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 4.1f scrub starts 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: 4.1f scrub ok 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3626082347' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:13:55.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 1.5 scrub starts 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 1.5 scrub ok 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: pgmap v1047: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 1.11 scrub starts 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 1.11 scrub ok 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 1.2 scrub starts 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 1.2 scrub ok 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 4.1f scrub starts 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: 4.1f scrub ok 2024-09-21T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:54 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3626082347' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T19:13:55.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: 6.14 scrub starts 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: 6.14 scrub ok 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3626082347' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: 5.7 scrub starts 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: 5.7 scrub ok 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: 3.d scrub starts 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[29384]: 3.d scrub ok 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: 6.14 scrub starts 2024-09-21T19:13:56.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: 6.14 scrub ok 2024-09-21T19:13:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3626082347' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:13:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T19:13:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: 5.7 scrub starts 2024-09-21T19:13:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: 5.7 scrub ok 2024-09-21T19:13:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: 3.d scrub starts 2024-09-21T19:13:56.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:55 smithi081 ceph-mon[39192]: 3.d scrub ok 2024-09-21T19:13:56.078 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:56.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: 6.14 scrub starts 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: 6.14 scrub ok 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3626082347' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: 5.7 scrub starts 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: 5.7 scrub ok 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: 3.d scrub starts 2024-09-21T19:13:56.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:55 smithi104 ceph-mon[32438]: 3.d scrub ok 2024-09-21T19:13:56.918 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: 4.2 scrub starts 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: 4.2 scrub ok 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: pgmap v1049: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: 2.0 scrub starts 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: 2.0 scrub ok 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: 6.a scrub starts 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: 6.a scrub ok 2024-09-21T19:13:57.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1526517173' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[39192]: from='client.36836 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: 4.2 scrub starts 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: 4.2 scrub ok 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: pgmap v1049: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: 2.0 scrub starts 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: 2.0 scrub ok 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: 6.a scrub starts 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: 6.a scrub ok 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1526517173' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:57.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:56 smithi081 ceph-mon[29384]: from='client.36836 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:57.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: 4.2 scrub starts 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: 4.2 scrub ok 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: pgmap v1049: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: 2.0 scrub starts 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: 2.0 scrub ok 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: 6.a scrub starts 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: 6.a scrub ok 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1526517173' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:56 smithi104 ceph-mon[32438]: from='client.36836 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:57.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:13:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:13:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:13:57.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T19:13:57.868 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:13:58.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:57 smithi104 ceph-mon[32438]: from='client.36836 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:13:58.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:57 smithi104 ceph-mon[32438]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T19:13:58.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:57 smithi081 ceph-mon[29384]: from='client.36836 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:13:58.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:57 smithi081 ceph-mon[29384]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T19:13:58.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:57 smithi081 ceph-mon[39192]: from='client.36836 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:13:58.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:57 smithi081 ceph-mon[39192]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T19:13:58.928 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T19:13:59.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:13:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-mon[32438]: 5.1e scrub starts 2024-09-21T19:13:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-mon[32438]: 5.1e scrub ok 2024-09-21T19:13:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-mon[32438]: pgmap v1052: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T19:13:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-mon[32438]: 5.10 scrub starts 2024-09-21T19:13:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-mon[32438]: 5.10 scrub ok 2024-09-21T19:13:59.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:58 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2684966930' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:59.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[29384]: 5.1e scrub starts 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[29384]: 5.1e scrub ok 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[29384]: pgmap v1052: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[29384]: 5.10 scrub starts 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[29384]: 5.10 scrub ok 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2684966930' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[39192]: 5.1e scrub starts 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[39192]: 5.1e scrub ok 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[39192]: pgmap v1052: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[39192]: 5.10 scrub starts 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[39192]: 5.10 scrub ok 2024-09-21T19:13:59.290 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:58 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2684966930' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T19:13:59.728 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T19:13:59.993 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:00.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: 4.a scrub starts 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: 4.a scrub ok 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: 4.12 scrub starts 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: 4.12 scrub ok 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2684966930' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[29384]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: 4.a scrub starts 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: 4.a scrub ok 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: 4.12 scrub starts 2024-09-21T19:14:00.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: 4.12 scrub ok 2024-09-21T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2684966930' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:13:59 smithi081 ceph-mon[39192]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: 4.a scrub starts 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: 4.a scrub ok 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: 4.12 scrub starts 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: 4.12 scrub ok 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2684966930' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T19:14:00.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:13:59 smithi104 ceph-mon[32438]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T19:14:00.941 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 6.6 scrub starts 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 6.6 scrub ok 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 5.f scrub starts 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 5.f scrub ok 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 6.1 scrub starts 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 6.1 scrub ok 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 6.2 scrub starts 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: 6.2 scrub ok 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: pgmap v1054: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T19:14:01.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:00 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/745183054' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 6.6 scrub starts 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 6.6 scrub ok 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 5.f scrub starts 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 5.f scrub ok 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 6.1 scrub starts 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 6.1 scrub ok 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 6.2 scrub starts 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: 6.2 scrub ok 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: pgmap v1054: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T19:14:01.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/745183054' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 6.6 scrub starts 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 6.6 scrub ok 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 5.f scrub starts 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 5.f scrub ok 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 6.1 scrub starts 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 6.1 scrub ok 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 6.2 scrub starts 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: 6.2 scrub ok 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: pgmap v1054: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T19:14:01.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:00 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/745183054' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:01.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T19:14:02.036 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/745183054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: 5.1b scrub starts 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: 5.1b scrub ok 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:14:02.066 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/745183054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: 5.1b scrub starts 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: 5.1b scrub ok 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:01 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:14:02.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/745183054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: 5.1b scrub starts 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: 5.1b scrub ok 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:14:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:01 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:14:02.950 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T19:14:03.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:02 smithi104 ceph-mon[32438]: 6.1a scrub starts 2024-09-21T19:14:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:02 smithi104 ceph-mon[32438]: 6.1a scrub ok 2024-09-21T19:14:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:02 smithi104 ceph-mon[32438]: pgmap v1056: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T19:14:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:02 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1045161942' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:03.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:02 smithi104 ceph-mon[32438]: from='client.36854 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[39192]: 6.1a scrub starts 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[39192]: 6.1a scrub ok 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[39192]: pgmap v1056: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1045161942' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[39192]: from='client.36854 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[29384]: 6.1a scrub starts 2024-09-21T19:14:03.291 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[29384]: 6.1a scrub ok 2024-09-21T19:14:03.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[29384]: pgmap v1056: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T19:14:03.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1045161942' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:03.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:02 smithi081 ceph-mon[29384]: from='client.36854 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T19:14:03.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T19:14:03.809 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: 5.d scrub starts 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: 5.d scrub ok 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: 5.1d scrub starts 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: 5.1d scrub ok 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: from='client.36854 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: 3.e scrub starts 2024-09-21T19:14:04.245 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[39192]: 3.e scrub ok 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: 5.d scrub starts 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: 5.d scrub ok 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: 5.1d scrub starts 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: 5.1d scrub ok 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: from='client.36854 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: 3.e scrub starts 2024-09-21T19:14:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:03 smithi081 ceph-mon[29384]: 3.e scrub ok 2024-09-21T19:14:04.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: 5.d scrub starts 2024-09-21T19:14:04.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: 5.d scrub ok 2024-09-21T19:14:04.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: 5.1d scrub starts 2024-09-21T19:14:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: 5.1d scrub ok 2024-09-21T19:14:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: from='client.36854 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T19:14:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T19:14:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: 3.e scrub starts 2024-09-21T19:14:04.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:03 smithi104 ceph-mon[32438]: 3.e scrub ok 2024-09-21T19:14:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: pgmap v1058: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: 6.5 scrub starts 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: 6.5 scrub ok 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: 3.10 scrub starts 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: 3.10 scrub ok 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3472922629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: from='client.36860 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: 4.17 scrub starts 2024-09-21T19:14:05.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:04 smithi104 ceph-mon[32438]: 4.17 scrub ok 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: pgmap v1058: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: 6.5 scrub starts 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: 6.5 scrub ok 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: 3.10 scrub starts 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: 3.10 scrub ok 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3472922629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:05.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: from='client.36860 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: 4.17 scrub starts 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[29384]: 4.17 scrub ok 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: pgmap v1058: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: 6.5 scrub starts 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: 6.5 scrub ok 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: 3.10 scrub starts 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: 3.10 scrub ok 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3472922629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: from='client.36860 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: 4.17 scrub starts 2024-09-21T19:14:05.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:04 smithi081 ceph-mon[39192]: 4.17 scrub ok 2024-09-21T19:14:06.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 3.1a scrub starts 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 3.1a scrub ok 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: from='client.36860 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 3.a scrub starts 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 3.a scrub ok 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 1.9 scrub starts 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 1.9 scrub ok 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 6.1f scrub starts 2024-09-21T19:14:06.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:05 smithi104 ceph-mon[32438]: 6.1f scrub ok 2024-09-21T19:14:06.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 3.1a scrub starts 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 3.1a scrub ok 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: from='client.36860 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 3.a scrub starts 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 3.a scrub ok 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 1.9 scrub starts 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 1.9 scrub ok 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 6.1f scrub starts 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[29384]: 6.1f scrub ok 2024-09-21T19:14:06.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 3.1a scrub starts 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 3.1a scrub ok 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: from='client.36860 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 3.a scrub starts 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 3.a scrub ok 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 1.9 scrub starts 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 1.9 scrub ok 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 6.1f scrub starts 2024-09-21T19:14:06.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:05 smithi081 ceph-mon[39192]: 6.1f scrub ok 2024-09-21T19:14:06.366 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: pgmap v1060: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: 5.c scrub starts 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: 5.c scrub ok 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: 3.13 scrub starts 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: 3.13 scrub ok 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: 6.13 scrub starts 2024-09-21T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[29384]: 6.13 scrub ok 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: pgmap v1060: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: 5.c scrub starts 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: 5.c scrub ok 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: 3.13 scrub starts 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: 3.13 scrub ok 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: 6.13 scrub starts 2024-09-21T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:06 smithi081 ceph-mon[39192]: 6.13 scrub ok 2024-09-21T19:14:07.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: pgmap v1060: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: 5.c scrub starts 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: 5.c scrub ok 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: 3.13 scrub starts 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: 3.13 scrub ok 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: 6.13 scrub starts 2024-09-21T19:14:07.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:06 smithi104 ceph-mon[32438]: 6.13 scrub ok 2024-09-21T19:14:07.565 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:14:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:14:07.974 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T19:14:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:07 smithi104 ceph-mon[32438]: 6.1d scrub starts 2024-09-21T19:14:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:07 smithi104 ceph-mon[32438]: 6.1d scrub ok 2024-09-21T19:14:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:07 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2211299782' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:08.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:07 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:08.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[29384]: 6.1d scrub starts 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[29384]: 6.1d scrub ok 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2211299782' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[39192]: 6.1d scrub starts 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[39192]: 6.1d scrub ok 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2211299782' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T19:14:08.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:07 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:08.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T19:14:09.022 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:09.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 5.19 scrub starts 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 5.19 scrub ok 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 3.6 scrub starts 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 3.6 scrub ok 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: pgmap v1062: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2211299782' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:14:09.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T19:14:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 6.8 scrub starts 2024-09-21T19:14:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 6.8 scrub ok 2024-09-21T19:14:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 5.0 scrub starts 2024-09-21T19:14:09.258 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:08 smithi104 ceph-mon[32438]: 5.0 scrub ok 2024-09-21T19:14:09.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 5.19 scrub starts 2024-09-21T19:14:09.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 5.19 scrub ok 2024-09-21T19:14:09.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 3.6 scrub starts 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 3.6 scrub ok 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: pgmap v1062: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2211299782' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 6.8 scrub starts 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 6.8 scrub ok 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 5.0 scrub starts 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[29384]: 5.0 scrub ok 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 5.19 scrub starts 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 5.19 scrub ok 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 3.6 scrub starts 2024-09-21T19:14:09.283 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 3.6 scrub ok 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: pgmap v1062: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2211299782' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 6.8 scrub starts 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 6.8 scrub ok 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 5.0 scrub starts 2024-09-21T19:14:09.284 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:08 smithi081 ceph-mon[39192]: 5.0 scrub ok 2024-09-21T19:14:09.994 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T19:14:10.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:09 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:09 smithi104 ceph-mon[32438]: 6.12 scrub starts 2024-09-21T19:14:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:09 smithi104 ceph-mon[32438]: 6.12 scrub ok 2024-09-21T19:14:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:09 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/285195593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:10.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:09 smithi104 ceph-mon[32438]: from='client.36872 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[29384]: 6.12 scrub starts 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[29384]: 6.12 scrub ok 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/285195593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[29384]: from='client.36872 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:10.299 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[39192]: 6.12 scrub starts 2024-09-21T19:14:10.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[39192]: 6.12 scrub ok 2024-09-21T19:14:10.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/285195593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:10.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:09 smithi081 ceph-mon[39192]: from='client.36872 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:10.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T19:14:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:11.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: pgmap v1064: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:11.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: from='client.36872 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:14:11.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T19:14:11.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: 5.15 scrub starts 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[29384]: 5.15 scrub ok 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: pgmap v1064: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: from='client.36872 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: 5.15 scrub starts 2024-09-21T19:14:11.168 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:10 smithi081 ceph-mon[39192]: 5.15 scrub ok 2024-09-21T19:14:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: pgmap v1064: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: from='client.36872 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:14:11.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T19:14:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:14:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:14:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: 5.15 scrub starts 2024-09-21T19:14:11.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:10 smithi104 ceph-mon[32438]: 5.15 scrub ok 2024-09-21T19:14:12.013 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T19:14:12.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 5.b scrub starts 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 5.b scrub ok 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 6.18 scrub starts 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 6.18 scrub ok 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 4.13 scrub starts 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 4.13 scrub ok 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3136458769' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: from='client.36878 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 6.1b scrub starts 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[29384]: 6.1b scrub ok 2024-09-21T19:14:12.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 5.b scrub starts 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 5.b scrub ok 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 6.18 scrub starts 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 6.18 scrub ok 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 4.13 scrub starts 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 4.13 scrub ok 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3136458769' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: from='client.36878 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 6.1b scrub starts 2024-09-21T19:14:12.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:12 smithi081 ceph-mon[39192]: 6.1b scrub ok 2024-09-21T19:14:12.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 5.b scrub starts 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 5.b scrub ok 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 6.18 scrub starts 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 6.18 scrub ok 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 4.13 scrub starts 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 4.13 scrub ok 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3136458769' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: from='client.36878 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 6.1b scrub starts 2024-09-21T19:14:12.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:12 smithi104 ceph-mon[32438]: 6.1b scrub ok 2024-09-21T19:14:12.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T19:14:13.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: pgmap v1067: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-09-21T19:14:13.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: from='client.36878 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:14:13.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T19:14:13.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: 3.15 scrub starts 2024-09-21T19:14:13.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: 3.15 scrub ok 2024-09-21T19:14:13.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:14:13.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:14:13.021 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[29384]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T19:14:13.023 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: pgmap v1067: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: from='client.36878 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: 3.15 scrub starts 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: 3.15 scrub ok 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:14:13.277 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:14:13.278 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:13 smithi081 ceph-mon[39192]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T19:14:13.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: pgmap v1067: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-09-21T19:14:13.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: from='client.36878 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T19:14:13.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T19:14:13.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: 3.15 scrub starts 2024-09-21T19:14:13.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: 3.15 scrub ok 2024-09-21T19:14:13.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:14:13.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:14:13.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-mon[32438]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T19:14:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=cleanup t=2024-09-21T19:14:13.41831629Z level=info msg="Completed cleanup jobs" duration=33.07896ms 2024-09-21T19:14:13.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:14:13 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=plugins.update.checker t=2024-09-21T19:14:13.547315525Z level=info msg="Update check succeeded" duration=75.254394ms 2024-09-21T19:14:14.028 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T19:14:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:14 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2931849415' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:14.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:14 smithi081 ceph-mon[29384]: from='client.36884 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:14 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2931849415' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:14.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:14 smithi081 ceph-mon[39192]: from='client.36884 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:14 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2931849415' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:14.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:14 smithi104 ceph-mon[32438]: from='client.36884 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:14.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T19:14:14.993 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:15.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[39192]: 6.0 scrub starts 2024-09-21T19:14:15.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[39192]: 6.0 scrub ok 2024-09-21T19:14:15.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[39192]: pgmap v1070: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:15.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[39192]: from='client.36884 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:14:15.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[39192]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T19:14:15.285 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[39192]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T19:14:15.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[29384]: 6.0 scrub starts 2024-09-21T19:14:15.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[29384]: 6.0 scrub ok 2024-09-21T19:14:15.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[29384]: pgmap v1070: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:15.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[29384]: from='client.36884 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:14:15.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[29384]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T19:14:15.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:15 smithi081 ceph-mon[29384]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T19:14:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:15 smithi104 ceph-mon[32438]: 6.0 scrub starts 2024-09-21T19:14:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:15 smithi104 ceph-mon[32438]: 6.0 scrub ok 2024-09-21T19:14:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:15 smithi104 ceph-mon[32438]: pgmap v1070: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:15 smithi104 ceph-mon[32438]: from='client.36884 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:14:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:15 smithi104 ceph-mon[32438]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T19:14:15.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:15 smithi104 ceph-mon[32438]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T19:14:16.030 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[39192]: 1.8 scrub starts 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[39192]: 1.8 scrub ok 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[39192]: 5.1a scrub starts 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[39192]: 5.1a scrub ok 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/279870591' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[29384]: 1.8 scrub starts 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[29384]: 1.8 scrub ok 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[29384]: 5.1a scrub starts 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[29384]: 5.1a scrub ok 2024-09-21T19:14:16.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:16 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/279870591' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:16.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:16 smithi104 ceph-mon[32438]: 1.8 scrub starts 2024-09-21T19:14:16.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:16 smithi104 ceph-mon[32438]: 1.8 scrub ok 2024-09-21T19:14:16.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:16 smithi104 ceph-mon[32438]: 5.1a scrub starts 2024-09-21T19:14:16.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:16 smithi104 ceph-mon[32438]: 5.1a scrub ok 2024-09-21T19:14:16.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:16 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/279870591' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T19:14:16.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T19:14:17.121 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: 5.5 scrub starts 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: 5.5 scrub ok 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: 1.e scrub starts 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: 1.e scrub ok 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: pgmap v1073: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: 5.1f scrub starts 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: 5.1f scrub ok 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/279870591' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[39192]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T19:14:17.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: 5.5 scrub starts 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: 5.5 scrub ok 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: 1.e scrub starts 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: 1.e scrub ok 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: pgmap v1073: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: 5.1f scrub starts 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: 5.1f scrub ok 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/279870591' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:14:17.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-mon[29384]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T19:14:17.497 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:14:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: 5.5 scrub starts 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: 5.5 scrub ok 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: 1.e scrub starts 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: 1.e scrub ok 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: pgmap v1073: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: 5.1f scrub starts 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: 5.1f scrub ok 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/279870591' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T19:14:17.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:17 smithi104 ceph-mon[32438]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T19:14:18.038 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: 5.6 scrub starts 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: 5.6 scrub ok 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: 4.18 scrub starts 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: 4.18 scrub ok 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: 3.f scrub starts 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: 3.f scrub ok 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2178986973' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: 5.6 scrub starts 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: 5.6 scrub ok 2024-09-21T19:14:18.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: 4.18 scrub starts 2024-09-21T19:14:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: 4.18 scrub ok 2024-09-21T19:14:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: 3.f scrub starts 2024-09-21T19:14:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: 3.f scrub ok 2024-09-21T19:14:18.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:18 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2178986973' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: 5.6 scrub starts 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: 5.6 scrub ok 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: 4.18 scrub starts 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: 4.18 scrub ok 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: 3.f scrub starts 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: 3.f scrub ok 2024-09-21T19:14:18.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2178986973' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:14:18.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T19:14:19.106 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: pgmap v1075: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: 4.e scrub starts 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: 4.e scrub ok 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: 6.11 scrub starts 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: 6.11 scrub ok 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2178986973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[29384]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: pgmap v1075: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: 4.e scrub starts 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: 4.e scrub ok 2024-09-21T19:14:19.220 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: 6.11 scrub starts 2024-09-21T19:14:19.221 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: 6.11 scrub ok 2024-09-21T19:14:19.221 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2178986973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:14:19.221 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:19 smithi081 ceph-mon[39192]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T19:14:19.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:14:18 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: pgmap v1075: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: 4.e scrub starts 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: 4.e scrub ok 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: 6.11 scrub starts 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: 6.11 scrub ok 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2178986973' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:14:19.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:19 smithi104 ceph-mon[32438]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T19:14:20.041 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: 5.1 scrub starts 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: 5.1 scrub ok 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: 1.f scrub starts 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: 1.f scrub ok 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2948222616' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: 3.7 scrub starts 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[29384]: 3.7 scrub ok 2024-09-21T19:14:20.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: 5.1 scrub starts 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: 5.1 scrub ok 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: 1.f scrub starts 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: 1.f scrub ok 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2948222616' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: 3.7 scrub starts 2024-09-21T19:14:20.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:20 smithi081 ceph-mon[39192]: 3.7 scrub ok 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: 5.1 scrub starts 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: 5.1 scrub ok 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: 1.f scrub starts 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: 1.f scrub ok 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2948222616' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: 3.7 scrub starts 2024-09-21T19:14:20.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:20 smithi104 ceph-mon[32438]: 3.7 scrub ok 2024-09-21T19:14:20.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T19:14:21.062 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: 5.3 scrub starts 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: 5.3 scrub ok 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: pgmap v1077: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: 3.0 scrub starts 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: 3.0 scrub ok 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2948222616' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:14:21.308 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: 6.19 scrub starts 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[39192]: 6.19 scrub ok 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: 5.3 scrub starts 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: 5.3 scrub ok 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: pgmap v1077: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: 3.0 scrub starts 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: 3.0 scrub ok 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2948222616' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: 6.19 scrub starts 2024-09-21T19:14:21.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:21 smithi081 ceph-mon[29384]: 6.19 scrub ok 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: 5.3 scrub starts 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: 5.3 scrub ok 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: pgmap v1077: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: 3.0 scrub starts 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: 3.0 scrub ok 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2948222616' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T19:14:21.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: 6.19 scrub starts 2024-09-21T19:14:21.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:21 smithi104 ceph-mon[32438]: 6.19 scrub ok 2024-09-21T19:14:22.063 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T19:14:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:22 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/736230337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:22 smithi104 ceph-mon[32438]: from='client.36908 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:22 smithi104 ceph-mon[32438]: 4.10 scrub starts 2024-09-21T19:14:22.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:22 smithi104 ceph-mon[32438]: 4.10 scrub ok 2024-09-21T19:14:22.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/736230337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[29384]: from='client.36908 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[29384]: 4.10 scrub starts 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[29384]: 4.10 scrub ok 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/736230337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[39192]: from='client.36908 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[39192]: 4.10 scrub starts 2024-09-21T19:14:22.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:22 smithi081 ceph-mon[39192]: 4.10 scrub ok 2024-09-21T19:14:22.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T19:14:23.193 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:23.314 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[39192]: pgmap v1079: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:23.314 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[39192]: from='client.36908 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:14:23.314 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[39192]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T19:14:23.314 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:23.314 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[39192]: 5.17 scrub starts 2024-09-21T19:14:23.314 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[39192]: 5.17 scrub ok 2024-09-21T19:14:23.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[29384]: pgmap v1079: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:23.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[29384]: from='client.36908 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:14:23.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[29384]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T19:14:23.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:23.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[29384]: 5.17 scrub starts 2024-09-21T19:14:23.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:23 smithi081 ceph-mon[29384]: 5.17 scrub ok 2024-09-21T19:14:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:23 smithi104 ceph-mon[32438]: pgmap v1079: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T19:14:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:23 smithi104 ceph-mon[32438]: from='client.36908 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:14:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:23 smithi104 ceph-mon[32438]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T19:14:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:23 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:23 smithi104 ceph-mon[32438]: 5.17 scrub starts 2024-09-21T19:14:23.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:23 smithi104 ceph-mon[32438]: 5.17 scrub ok 2024-09-21T19:14:24.062 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: 4.1b scrub starts 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: 4.1b scrub ok 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: 4.15 scrub starts 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: 4.15 scrub ok 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2728378861' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: from='client.36914 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: 3.12 scrub starts 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[29384]: 3.12 scrub ok 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: 4.1b scrub starts 2024-09-21T19:14:24.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: 4.1b scrub ok 2024-09-21T19:14:24.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: 4.15 scrub starts 2024-09-21T19:14:24.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: 4.15 scrub ok 2024-09-21T19:14:24.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2728378861' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:24.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: from='client.36914 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:24.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: 3.12 scrub starts 2024-09-21T19:14:24.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:24 smithi081 ceph-mon[39192]: 3.12 scrub ok 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: 4.1b scrub starts 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: 4.1b scrub ok 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: 4.15 scrub starts 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: 4.15 scrub ok 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2728378861' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: from='client.36914 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: 3.12 scrub starts 2024-09-21T19:14:24.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:24 smithi104 ceph-mon[32438]: 3.12 scrub ok 2024-09-21T19:14:24.765 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T19:14:24.765 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T19:14:24.765 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T19:14:24.765 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T19:14:24.765 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T19:14:24.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T19:14:25.033 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: pgmap v1081: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: 4.8 scrub starts 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: 4.8 scrub ok 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: from='client.36914 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: 5.9 scrub starts 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[29384]: 5.9 scrub ok 2024-09-21T19:14:25.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: pgmap v1081: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: 4.8 scrub starts 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: 4.8 scrub ok 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: from='client.36914 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: 5.9 scrub starts 2024-09-21T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:25 smithi081 ceph-mon[39192]: 5.9 scrub ok 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: pgmap v1081: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: 4.8 scrub starts 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: 4.8 scrub ok 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: from='client.36914 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: 5.9 scrub starts 2024-09-21T19:14:25.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:25 smithi104 ceph-mon[32438]: 5.9 scrub ok 2024-09-21T19:14:25.722 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:14:25.722 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":529,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","created":"2024-09-21T18:47:59.361278+0000","modified":"2024-09-21T19:14:24.846777+0000","last_up_change":"2024-09-21T19:06:54.357382+0000","last_in_change":"2024-09-21T19:13:40.842345+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:50:55.921148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"506","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"448","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T18:53:04.563630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":60,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T18:53:05.494307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T18:53:07.470408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T18:53:09.418988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T18:53:11.462116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd6626b0-dc43-42d0-a459-9cf3911457ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6801","nonce":390647394}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6802","nonce":390647394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6804","nonce":390647394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6803","nonce":390647394}]},"public_addr":"172.21.15.81:6801/390647394","cluster_addr":"172.21.15.81:6802/390647394","heartbeat_back_addr":"172.21.15.81:6804/390647394","heartbeat_front_addr":"172.21.15.81:6803/390647394","state":["exists","up"]},{"osd":1,"uuid":"2f541064-dc48-4f7f-b29b-065491e148b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6805","nonce":3705537065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6806","nonce":3705537065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6808","nonce":3705537065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6807","nonce":3705537065}]},"public_addr":"172.21.15.81:6805/3705537065","cluster_addr":"172.21.15.81:6806/3705537065","heartbeat_back_addr":"172.21.15.81:6808/3705537065","heartbeat_front_addr":"172.21.15.81:6807/3705537065","state":["exists","up"]},{"osd":2,"uuid":"e7050b14-f463-4de3-8a8d-e05e78507938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6809","nonce":486671153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6810","nonce":486671153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6812","nonce":486671153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6811","nonce":486671153}]},"public_addr":"172.21.15.81:6809/486671153","cluster_addr":"172.21.15.81:6810/486671153","heartbeat_back_addr":"172.21.15.81:6812/486671153","heartbeat_front_addr":"172.21.15.81:6811/486671153","state":["exists","up"]},{"osd":3,"uuid":"75b4cb80-138b-445b-a93a-3c4c598b5e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6813","nonce":1516859335}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6814","nonce":1516859335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6816","nonce":1516859335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6815","nonce":1516859335}]},"public_addr":"172.21.15.81:6813/1516859335","cluster_addr":"172.21.15.81:6814/1516859335","heartbeat_back_addr":"172.21.15.81:6816/1516859335","heartbeat_front_addr":"172.21.15.81:6815/1516859335","state":["exists","up"]},{"osd":4,"uuid":"85220357-38ea-4c90-b422-5d703329f106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":520,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6800","nonce":1904940409}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":1904940409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":1904940409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":1904940409}]},"public_addr":"172.21.15.104:6800/1904940409","cluster_addr":"172.21.15.104:6801/1904940409","heartbeat_back_addr":"172.21.15.104:6803/1904940409","heartbeat_front_addr":"172.21.15.104:6802/1904940409","state":["exists","up"]},{"osd":5,"uuid":"a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":326,"up_from":348,"up_thru":528,"down_at":327,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3649492645}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":3649492645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":3649492645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":3649492645}]},"public_addr":"172.21.15.104:6804/3649492645","cluster_addr":"172.21.15.104:6805/3649492645","heartbeat_back_addr":"172.21.15.104:6807/3649492645","heartbeat_front_addr":"172.21.15.104:6806/3649492645","state":["exists","up"]},{"osd":6,"uuid":"e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":45,"last_clean_end":238,"up_from":261,"up_thru":503,"down_at":239,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":2795996229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":2795996229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":2795996229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":2795996229}]},"public_addr":"172.21.15.104:6808/2795996229","cluster_addr":"172.21.15.104:6809/2795996229","heartbeat_back_addr":"172.21.15.104:6811/2795996229","heartbeat_front_addr":"172.21.15.104:6810/2795996229","state":["exists","up"]},{"osd":7,"uuid":"f15ee432-aa23-4e5a-9df4-478538af2045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":385235741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6813","nonce":385235741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6815","nonce":385235741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6814","nonce":385235741}]},"public_addr":"172.21.15.104:6812/385235741","cluster_addr":"172.21.15.104:6813/385235741","heartbeat_back_addr":"172.21.15.104:6815/385235741","heartbeat_front_addr":"172.21.15.104:6814/385235741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:14.180282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:40.090234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:38.794546+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:00.651883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:10.412487+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:05.771330+0000","dead_epoch":326},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:49.753951+0000","dead_epoch":238},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:12.320097+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":7,"to":6}]},{"pgid":"3.1c","mappings":[{"from":4,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"5.9","mappings":[{"from":7,"to":3}]},{"pgid":"5.c","mappings":[{"from":1,"to":6}]},{"pgid":"5.1f","mappings":[{"from":4,"to":3}]},{"pgid":"6.2","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/188521944":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/116812683":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/628265626":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/1496889754":"2024-09-22T18:53:35.352265+0000","172.21.15.81:6800/2203644131":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3906790599":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3045112983":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/806683817":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/3701153923":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/1611463627":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/1508794297":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/3426409743":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/198739293":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/4266768293":"2024-09-22T18:53:35.352265+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: 5.12 scrub starts 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: 5.12 scrub ok 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: 3.11 scrub starts 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: 3.11 scrub ok 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/526894523' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: 3.1e scrub starts 2024-09-21T19:14:26.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:26 smithi104 ceph-mon[32438]: 3.1e scrub ok 2024-09-21T19:14:26.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: 5.12 scrub starts 2024-09-21T19:14:26.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: 5.12 scrub ok 2024-09-21T19:14:26.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: 3.11 scrub starts 2024-09-21T19:14:26.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: 3.11 scrub ok 2024-09-21T19:14:26.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/526894523' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: 3.1e scrub starts 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[29384]: 3.1e scrub ok 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: 5.12 scrub starts 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: 5.12 scrub ok 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: 3.11 scrub starts 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: 3.11 scrub ok 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/526894523' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: 3.1e scrub starts 2024-09-21T19:14:26.553 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:26 smithi081 ceph-mon[39192]: 3.1e scrub ok 2024-09-21T19:14:26.586 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T19:14:26.586 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T19:14:26.836 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:27.239 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-mon[39192]: pgmap v1084: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-21T19:14:27.239 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-mon[39192]: 6.d scrub starts 2024-09-21T19:14:27.239 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-mon[39192]: 6.d scrub ok 2024-09-21T19:14:27.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-mon[29384]: pgmap v1084: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-21T19:14:27.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-mon[29384]: 6.d scrub starts 2024-09-21T19:14:27.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-mon[29384]: 6.d scrub ok 2024-09-21T19:14:27.240 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:14:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:14:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:27 smithi104 ceph-mon[32438]: pgmap v1084: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-21T19:14:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:27 smithi104 ceph-mon[32438]: 6.d scrub starts 2024-09-21T19:14:27.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:27 smithi104 ceph-mon[32438]: 6.d scrub ok 2024-09-21T19:14:27.625 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:14:27.625 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":529,"fsid":"d839dc86-7849-11ef-baf6-efdc52797490","created":"2024-09-21T18:47:59.361278+0000","modified":"2024-09-21T19:14:24.846777+0000","last_up_change":"2024-09-21T19:06:54.357382+0000","last_in_change":"2024-09-21T19:13:40.842345+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:50:55.921148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"506","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"448","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T18:53:04.563630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":60,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T18:53:05.494307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T18:53:07.470408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T18:53:09.418988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T18:53:11.462116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bd6626b0-dc43-42d0-a459-9cf3911457ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6801","nonce":390647394}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6802","nonce":390647394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6804","nonce":390647394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6803","nonce":390647394}]},"public_addr":"172.21.15.81:6801/390647394","cluster_addr":"172.21.15.81:6802/390647394","heartbeat_back_addr":"172.21.15.81:6804/390647394","heartbeat_front_addr":"172.21.15.81:6803/390647394","state":["exists","up"]},{"osd":1,"uuid":"2f541064-dc48-4f7f-b29b-065491e148b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6805","nonce":3705537065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6806","nonce":3705537065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6808","nonce":3705537065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6807","nonce":3705537065}]},"public_addr":"172.21.15.81:6805/3705537065","cluster_addr":"172.21.15.81:6806/3705537065","heartbeat_back_addr":"172.21.15.81:6808/3705537065","heartbeat_front_addr":"172.21.15.81:6807/3705537065","state":["exists","up"]},{"osd":2,"uuid":"e7050b14-f463-4de3-8a8d-e05e78507938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6809","nonce":486671153}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6810","nonce":486671153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6812","nonce":486671153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6811","nonce":486671153}]},"public_addr":"172.21.15.81:6809/486671153","cluster_addr":"172.21.15.81:6810/486671153","heartbeat_back_addr":"172.21.15.81:6812/486671153","heartbeat_front_addr":"172.21.15.81:6811/486671153","state":["exists","up"]},{"osd":3,"uuid":"75b4cb80-138b-445b-a93a-3c4c598b5e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6813","nonce":1516859335}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6814","nonce":1516859335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6816","nonce":1516859335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.81:6815","nonce":1516859335}]},"public_addr":"172.21.15.81:6813/1516859335","cluster_addr":"172.21.15.81:6814/1516859335","heartbeat_back_addr":"172.21.15.81:6816/1516859335","heartbeat_front_addr":"172.21.15.81:6815/1516859335","state":["exists","up"]},{"osd":4,"uuid":"85220357-38ea-4c90-b422-5d703329f106","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":520,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6800","nonce":1904940409}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":1904940409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":1904940409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":1904940409}]},"public_addr":"172.21.15.104:6800/1904940409","cluster_addr":"172.21.15.104:6801/1904940409","heartbeat_back_addr":"172.21.15.104:6803/1904940409","heartbeat_front_addr":"172.21.15.104:6802/1904940409","state":["exists","up"]},{"osd":5,"uuid":"a9a2ad2d-7a12-4e66-ac74-d9de9b20d35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":326,"up_from":348,"up_thru":528,"down_at":327,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3649492645}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":3649492645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":3649492645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":3649492645}]},"public_addr":"172.21.15.104:6804/3649492645","cluster_addr":"172.21.15.104:6805/3649492645","heartbeat_back_addr":"172.21.15.104:6807/3649492645","heartbeat_front_addr":"172.21.15.104:6806/3649492645","state":["exists","up"]},{"osd":6,"uuid":"e9afe3b9-d6da-4c5f-9b50-6c840b1f33bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":45,"last_clean_end":238,"up_from":261,"up_thru":503,"down_at":239,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":2795996229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":2795996229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":2795996229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":2795996229}]},"public_addr":"172.21.15.104:6808/2795996229","cluster_addr":"172.21.15.104:6809/2795996229","heartbeat_back_addr":"172.21.15.104:6811/2795996229","heartbeat_front_addr":"172.21.15.104:6810/2795996229","state":["exists","up"]},{"osd":7,"uuid":"f15ee432-aa23-4e5a-9df4-478538af2045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":503,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":385235741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6813","nonce":385235741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6815","nonce":385235741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6814","nonce":385235741}]},"public_addr":"172.21.15.104:6812/385235741","cluster_addr":"172.21.15.104:6813/385235741","heartbeat_back_addr":"172.21.15.104:6815/385235741","heartbeat_front_addr":"172.21.15.104:6814/385235741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:14.180282+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:40.090234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:38.794546+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:00.651883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:10.412487+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:05.771330+0000","dead_epoch":326},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:13:49.753951+0000","dead_epoch":238},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T19:14:12.320097+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":7,"to":6}]},{"pgid":"3.1c","mappings":[{"from":4,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"5.9","mappings":[{"from":7,"to":3}]},{"pgid":"5.c","mappings":[{"from":1,"to":6}]},{"pgid":"5.1f","mappings":[{"from":4,"to":3}]},{"pgid":"6.2","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/188521944":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/116812683":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/628265626":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/1496889754":"2024-09-22T18:53:35.352265+0000","172.21.15.81:6800/2203644131":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3906790599":"2024-09-22T18:48:21.208275+0000","172.21.15.81:0/3045112983":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/806683817":"2024-09-22T18:48:45.625700+0000","172.21.15.81:0/3701153923":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/1611463627":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/1508794297":"2024-09-22T18:48:45.625700+0000","172.21.15.81:6800/3426409743":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/198739293":"2024-09-22T18:53:35.352265+0000","172.21.15.81:0/4266768293":"2024-09-22T18:53:35.352265+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T19:14:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:28 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1979242736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T19:14:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:28 smithi081 ceph-mon[39192]: 3.b scrub starts 2024-09-21T19:14:28.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:28 smithi081 ceph-mon[39192]: 3.b scrub ok 2024-09-21T19:14:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:28 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1979242736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T19:14:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:28 smithi081 ceph-mon[29384]: 3.b scrub starts 2024-09-21T19:14:28.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:28 smithi081 ceph-mon[29384]: 3.b scrub ok 2024-09-21T19:14:28.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T19:14:28.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T19:14:28.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T19:14:28.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T19:14:28.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T19:14:28.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T19:14:28.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T19:14:28.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T19:14:28.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:28 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1979242736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T19:14:28.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:28 smithi104 ceph-mon[32438]: 3.b scrub starts 2024-09-21T19:14:28.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:28 smithi104 ceph-mon[32438]: 3.b scrub ok 2024-09-21T19:14:28.899 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:28.900 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:29.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:14:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: pgmap v1085: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: 4.9 scrub starts 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: 4.9 scrub ok 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: 6.4 scrub starts 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: 6.4 scrub ok 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: 4.1d scrub starts 2024-09-21T19:14:29.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:29 smithi104 ceph-mon[32438]: 4.1d scrub ok 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: pgmap v1085: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: 4.9 scrub starts 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: 4.9 scrub ok 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: 6.4 scrub starts 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: 6.4 scrub ok 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: 4.1d scrub starts 2024-09-21T19:14:29.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[39192]: 4.1d scrub ok 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: pgmap v1085: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: 4.9 scrub starts 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: 4.9 scrub ok 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: 6.4 scrub starts 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: 6.4 scrub ok 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: 4.1d scrub starts 2024-09-21T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:29 smithi081 ceph-mon[29384]: 4.1d scrub ok 2024-09-21T19:14:29.700 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:30.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:30 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:30.266 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:30 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:30.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:30 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 5.1c scrub starts 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 5.1c scrub ok 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: pgmap v1086: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 1.3 scrub starts 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 1.3 scrub ok 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 1.d scrub starts 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 1.d scrub ok 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 6.b scrub starts 2024-09-21T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[29384]: 6.b scrub ok 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 5.1c scrub starts 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 5.1c scrub ok 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: pgmap v1086: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 1.3 scrub starts 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 1.3 scrub ok 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 1.d scrub starts 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 1.d scrub ok 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 6.b scrub starts 2024-09-21T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:31 smithi081 ceph-mon[39192]: 6.b scrub ok 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 5.1c scrub starts 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 5.1c scrub ok 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: pgmap v1086: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 1.3 scrub starts 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 1.3 scrub ok 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 1.d scrub starts 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 1.d scrub ok 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 6.b scrub starts 2024-09-21T19:14:31.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:31 smithi104 ceph-mon[32438]: 6.b scrub ok 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[29384]: 5.18 scrub starts 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[29384]: 5.18 scrub ok 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[29384]: 3.18 scrub starts 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[29384]: 3.18 scrub ok 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[39192]: 5.18 scrub starts 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[39192]: 5.18 scrub ok 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[39192]: 3.18 scrub starts 2024-09-21T19:14:32.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:32 smithi081 ceph-mon[39192]: 3.18 scrub ok 2024-09-21T19:14:32.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:32 smithi104 ceph-mon[32438]: 5.18 scrub starts 2024-09-21T19:14:32.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:32 smithi104 ceph-mon[32438]: 5.18 scrub ok 2024-09-21T19:14:32.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:32 smithi104 ceph-mon[32438]: 3.18 scrub starts 2024-09-21T19:14:32.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:32 smithi104 ceph-mon[32438]: 3.18 scrub ok 2024-09-21T19:14:32.513 INFO:teuthology.orchestra.run.smithi081.stdout:227633266949 2024-09-21T19:14:32.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T19:14:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:33 smithi081 ceph-mon[29384]: pgmap v1087: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:14:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:33 smithi081 ceph-mon[29384]: 6.3 scrub starts 2024-09-21T19:14:33.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:33 smithi081 ceph-mon[29384]: 6.3 scrub ok 2024-09-21T19:14:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:33 smithi081 ceph-mon[39192]: pgmap v1087: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:14:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:33 smithi081 ceph-mon[39192]: 6.3 scrub starts 2024-09-21T19:14:33.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:33 smithi081 ceph-mon[39192]: 6.3 scrub ok 2024-09-21T19:14:33.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:33 smithi104 ceph-mon[32438]: pgmap v1087: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T19:14:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:33 smithi104 ceph-mon[32438]: 6.3 scrub starts 2024-09-21T19:14:33.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:33 smithi104 ceph-mon[32438]: 6.3 scrub ok 2024-09-21T19:14:33.879 INFO:teuthology.orchestra.run.smithi081.stdout:1120986464400 2024-09-21T19:14:33.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 4.1c scrub starts 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 4.1c scrub ok 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 2.2 scrub starts 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 2.2 scrub ok 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 1.0 scrub starts 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 1.0 scrub ok 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 6.f scrub starts 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 6.f scrub ok 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 4.19 scrub starts 2024-09-21T19:14:34.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 4.19 scrub ok 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 6.10 scrub starts 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[29384]: 6.10 scrub ok 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 4.1c scrub starts 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 4.1c scrub ok 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 2.2 scrub starts 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 2.2 scrub ok 2024-09-21T19:14:34.105 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 1.0 scrub starts 2024-09-21T19:14:34.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 1.0 scrub ok 2024-09-21T19:14:34.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 6.f scrub starts 2024-09-21T19:14:34.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 6.f scrub ok 2024-09-21T19:14:34.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 4.19 scrub starts 2024-09-21T19:14:34.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 4.19 scrub ok 2024-09-21T19:14:34.106 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 6.10 scrub starts 2024-09-21T19:14:34.107 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:34 smithi081 ceph-mon[39192]: 6.10 scrub ok 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 4.1c scrub starts 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 4.1c scrub ok 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 2.2 scrub starts 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 2.2 scrub ok 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 1.0 scrub starts 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 1.0 scrub ok 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 6.f scrub starts 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 6.f scrub ok 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 4.19 scrub starts 2024-09-21T19:14:34.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 4.19 scrub ok 2024-09-21T19:14:34.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 6.10 scrub starts 2024-09-21T19:14:34.508 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:34 smithi104 ceph-mon[32438]: 6.10 scrub ok 2024-09-21T19:14:35.363 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: pgmap v1088: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: 1.6 scrub starts 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: 1.6 scrub ok 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: 4.1a scrub starts 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: 4.1a scrub ok 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: 4.11 scrub starts 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[39192]: 4.11 scrub ok 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: pgmap v1088: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: 1.6 scrub starts 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: 1.6 scrub ok 2024-09-21T19:14:35.364 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: 4.1a scrub starts 2024-09-21T19:14:35.365 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: 4.1a scrub ok 2024-09-21T19:14:35.365 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: 4.11 scrub starts 2024-09-21T19:14:35.365 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:35 smithi081 ceph-mon[29384]: 4.11 scrub ok 2024-09-21T19:14:35.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: pgmap v1088: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T19:14:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: 1.6 scrub starts 2024-09-21T19:14:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: 1.6 scrub ok 2024-09-21T19:14:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: 4.1a scrub starts 2024-09-21T19:14:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: 4.1a scrub ok 2024-09-21T19:14:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: 4.11 scrub starts 2024-09-21T19:14:35.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:35 smithi104 ceph-mon[32438]: 4.11 scrub ok 2024-09-21T19:14:35.835 INFO:teuthology.orchestra.run.smithi081.stdout:77309411615 2024-09-21T19:14:35.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T19:14:36.328 INFO:teuthology.orchestra.run.smithi081.stdout:55834575139 2024-09-21T19:14:36.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[29384]: 3.14 scrub starts 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[29384]: 3.14 scrub ok 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[29384]: 1.b scrub starts 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[29384]: 1.b scrub ok 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[29384]: 5.8 scrub starts 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[29384]: 5.8 scrub ok 2024-09-21T19:14:36.430 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[39192]: 3.14 scrub starts 2024-09-21T19:14:36.431 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[39192]: 3.14 scrub ok 2024-09-21T19:14:36.431 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[39192]: 1.b scrub starts 2024-09-21T19:14:36.431 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[39192]: 1.b scrub ok 2024-09-21T19:14:36.431 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[39192]: 5.8 scrub starts 2024-09-21T19:14:36.431 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:36 smithi081 ceph-mon[39192]: 5.8 scrub ok 2024-09-21T19:14:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:36 smithi104 ceph-mon[32438]: 3.14 scrub starts 2024-09-21T19:14:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:36 smithi104 ceph-mon[32438]: 3.14 scrub ok 2024-09-21T19:14:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:36 smithi104 ceph-mon[32438]: 1.b scrub starts 2024-09-21T19:14:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:36 smithi104 ceph-mon[32438]: 1.b scrub ok 2024-09-21T19:14:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:36 smithi104 ceph-mon[32438]: 5.8 scrub starts 2024-09-21T19:14:36.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:36 smithi104 ceph-mon[32438]: 5.8 scrub ok 2024-09-21T19:14:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:107374182681 2024-09-21T19:14:36.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T19:14:37.098 INFO:teuthology.orchestra.run.smithi081.stdout:128849019155 2024-09-21T19:14:37.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T19:14:37.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[29384]: pgmap v1089: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[29384]: 4.d scrub starts 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[29384]: 3.1 scrub starts 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[29384]: 3.1 scrub ok 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[29384]: 4.d scrub ok 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:14:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[39192]: pgmap v1089: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[39192]: 4.d scrub starts 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[39192]: 3.1 scrub starts 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[39192]: 3.1 scrub ok 2024-09-21T19:14:37.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:37 smithi081 ceph-mon[39192]: 4.d scrub ok 2024-09-21T19:14:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:37 smithi104 ceph-mon[32438]: pgmap v1089: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-09-21T19:14:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:37 smithi104 ceph-mon[32438]: 4.d scrub starts 2024-09-21T19:14:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:37 smithi104 ceph-mon[32438]: 3.1 scrub starts 2024-09-21T19:14:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:37 smithi104 ceph-mon[32438]: 3.1 scrub ok 2024-09-21T19:14:37.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:37 smithi104 ceph-mon[32438]: 4.d scrub ok 2024-09-21T19:14:37.748 INFO:teuthology.orchestra.run.smithi081.stdout:34359738664 2024-09-21T19:14:37.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T19:14:38.334 INFO:teuthology.orchestra.run.smithi081.stdout:1494648619101 2024-09-21T19:14:38.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T19:14:38.455 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.461 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.482 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.483 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.487 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.488 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.492 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:38 smithi104 ceph-mon[32438]: 6.1e scrub starts 2024-09-21T19:14:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:38 smithi104 ceph-mon[32438]: 6.1e scrub ok 2024-09-21T19:14:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:38 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:38 smithi104 ceph-mon[32438]: 3.2 scrub starts 2024-09-21T19:14:38.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:38 smithi104 ceph-mon[32438]: 3.2 scrub ok 2024-09-21T19:14:38.565 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[39192]: 6.1e scrub starts 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[39192]: 6.1e scrub ok 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[39192]: 3.2 scrub starts 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[39192]: 3.2 scrub ok 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[29384]: 6.1e scrub starts 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[29384]: 6.1e scrub ok 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[29384]: 3.2 scrub starts 2024-09-21T19:14:38.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:38 smithi081 ceph-mon[29384]: 3.2 scrub ok 2024-09-21T19:14:39.256 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:39 smithi104 ceph-mon[32438]: pgmap v1090: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:39.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:14:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:14:39.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:39 smithi081 ceph-mon[29384]: pgmap v1090: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:39.377 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:39 smithi081 ceph-mon[39192]: pgmap v1090: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:39.469 INFO:teuthology.orchestra.run.smithi081.stdout:34359738665 2024-09-21T19:14:40.182 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:40 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:40.182 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:40 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/287397489' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T19:14:40.183 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:40 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:40.183 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:40 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/287397489' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T19:14:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:40 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:40.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:40 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/287397489' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T19:14:40.792 INFO:teuthology.orchestra.run.smithi081.stdout:1120986464401 2024-09-21T19:14:41.093 INFO:teuthology.orchestra.run.smithi081.stdout:128849019157 2024-09-21T19:14:41.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:41 smithi104 ceph-mon[32438]: pgmap v1091: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:41.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1763920825' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T19:14:41.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:41 smithi104 ceph-mon[32438]: 5.11 scrub starts 2024-09-21T19:14:41.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:41 smithi104 ceph-mon[32438]: 5.11 scrub ok 2024-09-21T19:14:41.408 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:41 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3881757311' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T19:14:41.528 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[29384]: pgmap v1091: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:41.611 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1763920825' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T19:14:41.611 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[29384]: 5.11 scrub starts 2024-09-21T19:14:41.611 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[29384]: 5.11 scrub ok 2024-09-21T19:14:41.611 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3881757311' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T19:14:41.611 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[39192]: pgmap v1091: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:41.612 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1763920825' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T19:14:41.612 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[39192]: 5.11 scrub starts 2024-09-21T19:14:41.612 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[39192]: 5.11 scrub ok 2024-09-21T19:14:41.612 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:41 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3881757311' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T19:14:41.809 INFO:teuthology.orchestra.run.smithi081.stdout:227633266951 2024-09-21T19:14:42.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738664 got 34359738665 for osd.0 2024-09-21T19:14:42.151 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:42 smithi104 ceph-mon[32438]: 3.8 scrub starts 2024-09-21T19:14:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:42 smithi104 ceph-mon[32438]: 3.8 scrub ok 2024-09-21T19:14:42.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:42 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/1861689852' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T19:14:42.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:42 smithi081 ceph-mon[39192]: 3.8 scrub starts 2024-09-21T19:14:42.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:42 smithi081 ceph-mon[39192]: 3.8 scrub ok 2024-09-21T19:14:42.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:42 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/1861689852' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T19:14:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:42 smithi081 ceph-mon[29384]: 3.8 scrub starts 2024-09-21T19:14:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:42 smithi081 ceph-mon[29384]: 3.8 scrub ok 2024-09-21T19:14:42.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:42 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/1861689852' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T19:14:42.697 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1120986464400 got 1120986464401 for osd.6 2024-09-21T19:14:42.697 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:43.037 INFO:teuthology.orchestra.run.smithi081.stdout:107374182682 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[29384]: pgmap v1092: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[29384]: 3.c scrub starts 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[29384]: 3.c scrub ok 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[29384]: 5.14 scrub starts 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[29384]: 5.14 scrub ok 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2989302649' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T19:14:43.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[39192]: pgmap v1092: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:43.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[39192]: 3.c scrub starts 2024-09-21T19:14:43.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[39192]: 3.c scrub ok 2024-09-21T19:14:43.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[39192]: 5.14 scrub starts 2024-09-21T19:14:43.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[39192]: 5.14 scrub ok 2024-09-21T19:14:43.317 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:43 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2989302649' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T19:14:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:43 smithi104 ceph-mon[32438]: pgmap v1092: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T19:14:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:43 smithi104 ceph-mon[32438]: 3.c scrub starts 2024-09-21T19:14:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:43 smithi104 ceph-mon[32438]: 3.c scrub ok 2024-09-21T19:14:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:43 smithi104 ceph-mon[32438]: 5.14 scrub starts 2024-09-21T19:14:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:43 smithi104 ceph-mon[32438]: 5.14 scrub ok 2024-09-21T19:14:43.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:43 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2989302649' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T19:14:43.648 INFO:teuthology.orchestra.run.smithi081.stdout:55834575140 2024-09-21T19:14:44.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266949 got 227633266951 for osd.7 2024-09-21T19:14:44.013 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:44.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[29384]: 4.c scrub starts 2024-09-21T19:14:44.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[29384]: 4.c scrub ok 2024-09-21T19:14:44.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[29384]: purged_snaps scrub starts 2024-09-21T19:14:44.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[29384]: purged_snaps scrub ok 2024-09-21T19:14:44.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/2991522104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T19:14:44.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[39192]: 4.c scrub starts 2024-09-21T19:14:44.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[39192]: 4.c scrub ok 2024-09-21T19:14:44.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[39192]: purged_snaps scrub starts 2024-09-21T19:14:44.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[39192]: purged_snaps scrub ok 2024-09-21T19:14:44.300 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:44 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/2991522104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T19:14:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:44 smithi104 ceph-mon[32438]: 4.c scrub starts 2024-09-21T19:14:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:44 smithi104 ceph-mon[32438]: 4.c scrub ok 2024-09-21T19:14:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:44 smithi104 ceph-mon[32438]: purged_snaps scrub starts 2024-09-21T19:14:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:44 smithi104 ceph-mon[32438]: purged_snaps scrub ok 2024-09-21T19:14:44.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:44 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/2991522104' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T19:14:44.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575139 got 55834575140 for osd.1 2024-09-21T19:14:44.828 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:45.303 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182681 got 107374182682 for osd.3 2024-09-21T19:14:45.303 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:45.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: 1.a scrub starts 2024-09-21T19:14:45.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: 1.a scrub ok 2024-09-21T19:14:45.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: pgmap v1093: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: 3.19 scrub starts 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[29384]: 3.19 scrub ok 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: 1.a scrub starts 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: 1.a scrub ok 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: pgmap v1093: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: 3.19 scrub starts 2024-09-21T19:14:45.481 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:45 smithi081 ceph-mon[39192]: 3.19 scrub ok 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: 1.a scrub starts 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: 1.a scrub ok 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: pgmap v1093: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: 3.19 scrub starts 2024-09-21T19:14:45.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:45 smithi104 ceph-mon[32438]: 3.19 scrub ok 2024-09-21T19:14:45.570 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:45.759 INFO:teuthology.orchestra.run.smithi081.stdout:77309411617 2024-09-21T19:14:46.010 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019155 got 128849019157 for osd.4 2024-09-21T19:14:46.011 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: 3.9 scrub starts 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: 3.9 scrub ok 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: 4.6 scrub starts 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: 4.6 scrub ok 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/856321581' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: 4.5 scrub starts 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[39192]: 4.5 scrub ok 2024-09-21T19:14:46.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: 3.9 scrub starts 2024-09-21T19:14:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: 3.9 scrub ok 2024-09-21T19:14:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: 4.6 scrub starts 2024-09-21T19:14:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: 4.6 scrub ok 2024-09-21T19:14:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/856321581' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T19:14:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: 4.5 scrub starts 2024-09-21T19:14:46.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:46 smithi081 ceph-mon[29384]: 4.5 scrub ok 2024-09-21T19:14:46.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: 3.9 scrub starts 2024-09-21T19:14:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: 3.9 scrub ok 2024-09-21T19:14:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: 4.6 scrub starts 2024-09-21T19:14:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: 4.6 scrub ok 2024-09-21T19:14:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/856321581' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T19:14:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: 4.5 scrub starts 2024-09-21T19:14:46.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:46 smithi104 ceph-mon[32438]: 4.5 scrub ok 2024-09-21T19:14:46.694 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411615 got 77309411617 for osd.2 2024-09-21T19:14:46.694 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:47.316 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:14:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:14:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[39192]: pgmap v1096: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T19:14:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[39192]: 5.a scrub starts 2024-09-21T19:14:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[39192]: 5.a scrub ok 2024-09-21T19:14:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[39192]: 1.4 scrub starts 2024-09-21T19:14:47.316 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[39192]: 1.4 scrub ok 2024-09-21T19:14:47.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[29384]: pgmap v1096: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T19:14:47.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[29384]: 5.a scrub starts 2024-09-21T19:14:47.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[29384]: 5.a scrub ok 2024-09-21T19:14:47.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[29384]: 1.4 scrub starts 2024-09-21T19:14:47.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:47 smithi081 ceph-mon[29384]: 1.4 scrub ok 2024-09-21T19:14:47.435 INFO:teuthology.orchestra.run.smithi081.stdout:1494648619104 2024-09-21T19:14:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:47 smithi104 ceph-mon[32438]: pgmap v1096: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T19:14:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:47 smithi104 ceph-mon[32438]: 5.a scrub starts 2024-09-21T19:14:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:47 smithi104 ceph-mon[32438]: 5.a scrub ok 2024-09-21T19:14:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:47 smithi104 ceph-mon[32438]: 1.4 scrub starts 2024-09-21T19:14:47.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:47 smithi104 ceph-mon[32438]: 1.4 scrub ok 2024-09-21T19:14:48.154 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1494648619101 got 1494648619104 for osd.5 2024-09-21T19:14:48.154 DEBUG:teuthology.parallel:result is None 2024-09-21T19:14:48.154 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T19:14:48.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:14:48.406 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:48.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:48 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/3275569230' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T19:14:48.527 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:48 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/3275569230' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T19:14:48.527 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:48 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/3275569230' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T19:14:49.092 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:14:49.094 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:14:49.147 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:14:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug there is no tcmu-runner data available 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: pgmap v1097: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 1.1 scrub starts 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 1.1 scrub ok 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 6.e scrub starts 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 6.e scrub ok 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 6.9 scrub starts 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 6.9 scrub ok 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 5.2 scrub starts 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[29384]: 5.2 scrub ok 2024-09-21T19:14:49.404 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: pgmap v1097: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 1.1 scrub starts 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 1.1 scrub ok 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 6.e scrub starts 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 6.e scrub ok 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 6.9 scrub starts 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 6.9 scrub ok 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 5.2 scrub starts 2024-09-21T19:14:49.405 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:49 smithi081 ceph-mon[39192]: 5.2 scrub ok 2024-09-21T19:14:49.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: pgmap v1097: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 1.1 scrub starts 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 1.1 scrub ok 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 6.e scrub starts 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 6.e scrub ok 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 6.9 scrub starts 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 6.9 scrub ok 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 5.2 scrub starts 2024-09-21T19:14:49.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:49 smithi104 ceph-mon[32438]: 5.2 scrub ok 2024-09-21T19:14:49.815 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":1097,"stamp":"2024-09-21T19:14:47.911764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2838,"num_read_kb":2590,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":537685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2748,"ondisk_log_size":2748,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5932664,"kb_used_data":9836,"kb_used_omap":18,"kb_used_meta":5922669,"kb_avail":744028552,"statfs":{"total":767960285184,"available":761885237248,"internally_reserved":0,"allocated":10072064,"data_stored":6111073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":6064814064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002465"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":900,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851799+0000","last_change":"2024-09-21T19:13:55.923183+0000","last_active":"2024-09-21T19:14:24.851799+0000","last_peered":"2024-09-21T19:14:24.851799+0000","last_clean":"2024-09-21T19:14:24.851799+0000","last_became_active":"2024-09-21T19:13:55.922887+0000","last_became_peered":"2024-09-21T19:13:55.922887+0000","last_unstale":"2024-09-21T19:14:24.851799+0000","last_undegraded":"2024-09-21T19:14:24.851799+0000","last_fullsized":"2024-09-21T19:14:24.851799+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:03.679452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:03.679452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"489'21","reported_seq":604,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857855+0000","last_change":"2024-09-21T19:14:16.919099+0000","last_active":"2024-09-21T19:14:44.857855+0000","last_peered":"2024-09-21T19:14:44.857855+0000","last_clean":"2024-09-21T19:14:44.857855+0000","last_became_active":"2024-09-21T19:14:13.025672+0000","last_became_peered":"2024-09-21T19:14:13.025672+0000","last_unstale":"2024-09-21T19:14:44.857855+0000","last_undegraded":"2024-09-21T19:14:44.857855+0000","last_fullsized":"2024-09-21T19:14:44.857855+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:14:16.919030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:16.919030+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:43.683336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1034,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.855431+0000","last_change":"2024-09-21T19:14:07.832217+0000","last_active":"2024-09-21T19:14:24.855431+0000","last_peered":"2024-09-21T19:14:24.855431+0000","last_clean":"2024-09-21T19:14:24.855431+0000","last_became_active":"2024-09-21T19:13:41.863135+0000","last_became_peered":"2024-09-21T19:13:41.863135+0000","last_unstale":"2024-09-21T19:14:24.855431+0000","last_undegraded":"2024-09-21T19:14:24.855431+0000","last_fullsized":"2024-09-21T19:14:24.855431+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:07.832163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:07.832163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.682688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":917,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858002+0000","last_change":"2024-09-21T19:14:13.020739+0000","last_active":"2024-09-21T19:14:44.858002+0000","last_peered":"2024-09-21T19:14:44.858002+0000","last_clean":"2024-09-21T19:14:44.858002+0000","last_became_active":"2024-09-21T19:14:13.020316+0000","last_became_peered":"2024-09-21T19:14:13.020316+0000","last_unstale":"2024-09-21T19:14:44.858002+0000","last_undegraded":"2024-09-21T19:14:44.858002+0000","last_fullsized":"2024-09-21T19:14:44.858002+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:01.795304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:01.795304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:11.420757+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1009,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.850494+0000","last_change":"2024-09-21T19:14:11.909638+0000","last_active":"2024-09-21T19:14:24.850494+0000","last_peered":"2024-09-21T19:14:24.850494+0000","last_clean":"2024-09-21T19:14:24.850494+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:14:24.850494+0000","last_undegraded":"2024-09-21T19:14:24.850494+0000","last_fullsized":"2024-09-21T19:14:24.850494+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:11.909595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:11.909595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.170553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1037,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:25.844226+0000","last_change":"2024-09-21T19:14:25.844226+0000","last_active":"2024-09-21T19:14:25.844226+0000","last_peered":"2024-09-21T19:14:25.844226+0000","last_clean":"2024-09-21T19:14:25.844226+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:14:25.844226+0000","last_undegraded":"2024-09-21T19:14:25.844226+0000","last_fullsized":"2024-09-21T19:14:25.844226+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:25.844178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:25.844178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:31.060089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"490'39","reported_seq":1143,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:33.797542+0000","last_change":"2024-09-21T19:14:33.797542+0000","last_active":"2024-09-21T19:14:33.797542+0000","last_peered":"2024-09-21T19:14:33.797542+0000","last_clean":"2024-09-21T19:14:33.797542+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:14:33.797542+0000","last_undegraded":"2024-09-21T19:14:33.797542+0000","last_fullsized":"2024-09-21T19:14:33.797542+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:14:33.797488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.797488+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:02.755324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":953,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857852+0000","last_change":"2024-09-21T19:14:30.904760+0000","last_active":"2024-09-21T19:14:44.857852+0000","last_peered":"2024-09-21T19:14:44.857852+0000","last_clean":"2024-09-21T19:14:44.857852+0000","last_became_active":"2024-09-21T19:14:13.025353+0000","last_became_peered":"2024-09-21T19:14:13.025353+0000","last_unstale":"2024-09-21T19:14:44.857852+0000","last_undegraded":"2024-09-21T19:14:44.857852+0000","last_fullsized":"2024-09-21T19:14:44.857852+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:30.904704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.904704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.524084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":906,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.869202+0000","last_change":"2024-09-21T19:14:24.867523+0000","last_active":"2024-09-21T19:14:24.869202+0000","last_peered":"2024-09-21T19:14:24.869202+0000","last_clean":"2024-09-21T19:14:24.869202+0000","last_became_active":"2024-09-21T19:14:24.867187+0000","last_became_peered":"2024-09-21T19:14:24.867187+0000","last_unstale":"2024-09-21T19:14:24.869202+0000","last_undegraded":"2024-09-21T19:14:24.869202+0000","last_fullsized":"2024-09-21T19:14:24.869202+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:45.962479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:45.962479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:48.252339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"489'21","reported_seq":611,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858257+0000","last_change":"2024-09-21T19:14:34.009659+0000","last_active":"2024-09-21T19:14:44.858257+0000","last_peered":"2024-09-21T19:14:44.858257+0000","last_clean":"2024-09-21T19:14:44.858257+0000","last_became_active":"2024-09-21T19:14:13.024633+0000","last_became_peered":"2024-09-21T19:14:13.024633+0000","last_unstale":"2024-09-21T19:14:44.858257+0000","last_undegraded":"2024-09-21T19:14:44.858257+0000","last_fullsized":"2024-09-21T19:14:44.858257+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:14:34.009595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:34.009595+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:41.871788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":70,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.854472+0000","last_change":"2024-09-21T19:14:01.034756+0000","last_active":"2024-09-21T19:14:24.854472+0000","last_peered":"2024-09-21T19:14:24.854472+0000","last_clean":"2024-09-21T19:14:24.854472+0000","last_became_active":"2024-09-21T19:13:41.863805+0000","last_became_peered":"2024-09-21T19:13:41.863805+0000","last_unstale":"2024-09-21T19:14:24.854472+0000","last_undegraded":"2024-09-21T19:14:24.854472+0000","last_fullsized":"2024-09-21T19:14:24.854472+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:01.034700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:01.034700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:08.295623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":857,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.850867+0000","last_change":"2024-09-21T19:14:10.868263+0000","last_active":"2024-09-21T19:14:24.850867+0000","last_peered":"2024-09-21T19:14:24.850867+0000","last_clean":"2024-09-21T19:14:24.850867+0000","last_became_active":"2024-09-21T19:13:55.927611+0000","last_became_peered":"2024-09-21T19:13:55.927611+0000","last_unstale":"2024-09-21T19:14:24.850867+0000","last_undegraded":"2024-09-21T19:14:24.850867+0000","last_fullsized":"2024-09-21T19:14:24.850867+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:10.868212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:10.868212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:26.089747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":27,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.869278+0000","last_change":"2024-09-21T19:14:24.868817+0000","last_active":"2024-09-21T19:14:24.869278+0000","last_peered":"2024-09-21T19:14:24.869278+0000","last_clean":"2024-09-21T19:14:24.869278+0000","last_became_active":"2024-09-21T19:14:24.868485+0000","last_became_peered":"2024-09-21T19:14:24.868485+0000","last_unstale":"2024-09-21T19:14:24.869278+0000","last_undegraded":"2024-09-21T19:14:24.869278+0000","last_fullsized":"2024-09-21T19:14:24.869278+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:42.453534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:42.453534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:46.945085+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"489'9","reported_seq":551,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858303+0000","last_change":"2024-09-21T19:14:22.902175+0000","last_active":"2024-09-21T19:14:44.858303+0000","last_peered":"2024-09-21T19:14:44.858303+0000","last_clean":"2024-09-21T19:14:44.858303+0000","last_became_active":"2024-09-21T19:14:13.021371+0000","last_became_peered":"2024-09-21T19:14:13.021371+0000","last_unstale":"2024-09-21T19:14:44.858303+0000","last_undegraded":"2024-09-21T19:14:44.858303+0000","last_fullsized":"2024-09-21T19:14:44.858303+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'9","last_scrub_stamp":"2024-09-21T19:14:22.902106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:22.902106+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:31.169757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1004,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071874+0000","last_change":"2024-09-21T19:14:15.147298+0000","last_active":"2024-09-21T19:14:45.071874+0000","last_peered":"2024-09-21T19:14:45.071874+0000","last_clean":"2024-09-21T19:14:45.071874+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:14:45.071874+0000","last_undegraded":"2024-09-21T19:14:45.071874+0000","last_fullsized":"2024-09-21T19:14:45.071874+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:15.147233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.147233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.518936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":535,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867070+0000","last_change":"2024-09-21T19:14:24.864710+0000","last_active":"2024-09-21T19:14:24.867070+0000","last_peered":"2024-09-21T19:14:24.867070+0000","last_clean":"2024-09-21T19:14:24.867070+0000","last_became_active":"2024-09-21T19:14:24.864388+0000","last_became_peered":"2024-09-21T19:14:24.864388+0000","last_unstale":"2024-09-21T19:14:24.867070+0000","last_undegraded":"2024-09-21T19:14:24.867070+0000","last_fullsized":"2024-09-21T19:14:24.867070+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:20.967629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:20.967629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.156518+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":954,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859041+0000","last_change":"2024-09-21T19:14:37.053297+0000","last_active":"2024-09-21T19:14:44.859041+0000","last_peered":"2024-09-21T19:14:44.859041+0000","last_clean":"2024-09-21T19:14:44.859041+0000","last_became_active":"2024-09-21T19:14:13.028761+0000","last_became_peered":"2024-09-21T19:14:13.028761+0000","last_unstale":"2024-09-21T19:14:44.859041+0000","last_undegraded":"2024-09-21T19:14:44.859041+0000","last_fullsized":"2024-09-21T19:14:44.859041+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:37.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:37.053229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:51.615622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":865,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.854599+0000","last_change":"2024-09-21T19:13:55.929500+0000","last_active":"2024-09-21T19:14:24.854599+0000","last_peered":"2024-09-21T19:14:24.854599+0000","last_clean":"2024-09-21T19:14:24.854599+0000","last_became_active":"2024-09-21T19:13:55.927362+0000","last_became_peered":"2024-09-21T19:13:55.927362+0000","last_unstale":"2024-09-21T19:14:24.854599+0000","last_undegraded":"2024-09-21T19:14:24.854599+0000","last_fullsized":"2024-09-21T19:14:24.854599+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:42.596058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:42.596058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"490'39","reported_seq":505,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:32.365216+0000","last_change":"2024-09-21T19:14:32.365216+0000","last_active":"2024-09-21T19:14:32.365216+0000","last_peered":"2024-09-21T19:14:32.365216+0000","last_clean":"2024-09-21T19:14:32.365216+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:14:32.365216+0000","last_undegraded":"2024-09-21T19:14:32.365216+0000","last_fullsized":"2024-09-21T19:14:32.365216+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:14:32.365158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:32.365158+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.979432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1031,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856572+0000","last_change":"2024-09-21T19:14:02.786478+0000","last_active":"2024-09-21T19:14:24.856572+0000","last_peered":"2024-09-21T19:14:24.856572+0000","last_clean":"2024-09-21T19:14:24.856572+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:14:24.856572+0000","last_undegraded":"2024-09-21T19:14:24.856572+0000","last_fullsized":"2024-09-21T19:14:24.856572+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:02.786428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:02.786428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:21.973273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1020,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853067+0000","last_change":"2024-09-21T19:14:05.846447+0000","last_active":"2024-09-21T19:14:24.853067+0000","last_peered":"2024-09-21T19:14:24.853067+0000","last_clean":"2024-09-21T19:14:24.853067+0000","last_became_active":"2024-09-21T19:13:41.875083+0000","last_became_peered":"2024-09-21T19:13:41.875083+0000","last_unstale":"2024-09-21T19:14:24.853067+0000","last_undegraded":"2024-09-21T19:14:24.853067+0000","last_fullsized":"2024-09-21T19:14:24.853067+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:05.846390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:05.846390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.180814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":940,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858564+0000","last_change":"2024-09-21T19:14:13.022440+0000","last_active":"2024-09-21T19:14:44.858564+0000","last_peered":"2024-09-21T19:14:44.858564+0000","last_clean":"2024-09-21T19:14:44.858564+0000","last_became_active":"2024-09-21T19:14:13.021848+0000","last_became_peered":"2024-09-21T19:14:13.021848+0000","last_unstale":"2024-09-21T19:14:44.858564+0000","last_undegraded":"2024-09-21T19:14:44.858564+0000","last_fullsized":"2024-09-21T19:14:44.858564+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:04.794479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.794479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:05.068465+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"490'24","reported_seq":1104,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:28.809841+0000","last_change":"2024-09-21T19:14:28.809841+0000","last_active":"2024-09-21T19:14:28.809841+0000","last_peered":"2024-09-21T19:14:28.809841+0000","last_clean":"2024-09-21T19:14:28.809841+0000","last_became_active":"2024-09-21T19:10:52.013122+0000","last_became_peered":"2024-09-21T19:10:52.013122+0000","last_unstale":"2024-09-21T19:14:28.809841+0000","last_undegraded":"2024-09-21T19:14:28.809841+0000","last_fullsized":"2024-09-21T19:14:28.809841+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'24","last_scrub_stamp":"2024-09-21T19:14:28.809749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:28.809749+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.853614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":940,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858184+0000","last_change":"2024-09-21T19:14:29.889914+0000","last_active":"2024-09-21T19:14:44.858184+0000","last_peered":"2024-09-21T19:14:44.858184+0000","last_clean":"2024-09-21T19:14:44.858184+0000","last_became_active":"2024-09-21T19:14:13.022424+0000","last_became_peered":"2024-09-21T19:14:13.022424+0000","last_unstale":"2024-09-21T19:14:44.858184+0000","last_undegraded":"2024-09-21T19:14:44.858184+0000","last_fullsized":"2024-09-21T19:14:44.858184+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:29.889856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:29.889856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:37.493731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":1064,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072251+0000","last_change":"2024-09-21T19:13:46.233497+0000","last_active":"2024-09-21T19:14:45.072251+0000","last_peered":"2024-09-21T19:14:45.072251+0000","last_clean":"2024-09-21T19:14:45.072251+0000","last_became_active":"2024-09-21T19:13:41.876488+0000","last_became_peered":"2024-09-21T19:13:41.876488+0000","last_unstale":"2024-09-21T19:14:45.072251+0000","last_undegraded":"2024-09-21T19:14:45.072251+0000","last_fullsized":"2024-09-21T19:14:45.072251+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:13:46.233444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:46.233444+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:06.991782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":1029,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.855532+0000","last_change":"2024-09-21T19:10:27.115013+0000","last_active":"2024-09-21T19:14:24.855532+0000","last_peered":"2024-09-21T19:14:24.855532+0000","last_clean":"2024-09-21T19:14:24.855532+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:14:24.855532+0000","last_undegraded":"2024-09-21T19:14:24.855532+0000","last_fullsized":"2024-09-21T19:14:24.855532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:10:27.114925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:10:27.114925+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:38.030720+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"490'30","reported_seq":917,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851593+0000","last_change":"2024-09-21T19:13:55.930365+0000","last_active":"2024-09-21T19:14:24.851593+0000","last_peered":"2024-09-21T19:14:24.851593+0000","last_clean":"2024-09-21T19:14:24.851593+0000","last_became_active":"2024-09-21T19:13:55.927195+0000","last_became_peered":"2024-09-21T19:13:55.927195+0000","last_unstale":"2024-09-21T19:14:24.851593+0000","last_undegraded":"2024-09-21T19:14:24.851593+0000","last_fullsized":"2024-09-21T19:14:24.851593+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:13:18.789971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:18.789971+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:38.273112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":1170,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.297184+0000","last_change":"2024-09-21T19:14:15.941975+0000","last_active":"2024-09-21T19:14:44.297184+0000","last_peered":"2024-09-21T19:14:44.297184+0000","last_clean":"2024-09-21T19:14:44.297184+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:14:44.297184+0000","last_undegraded":"2024-09-21T19:14:44.297184+0000","last_fullsized":"2024-09-21T19:14:44.297184+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:14:15.941912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.941912+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.900683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":436,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867114+0000","last_change":"2024-09-21T19:14:24.865707+0000","last_active":"2024-09-21T19:14:24.867114+0000","last_peered":"2024-09-21T19:14:24.867114+0000","last_clean":"2024-09-21T19:14:24.867114+0000","last_became_active":"2024-09-21T19:14:24.864948+0000","last_became_peered":"2024-09-21T19:14:24.864948+0000","last_unstale":"2024-09-21T19:14:24.867114+0000","last_undegraded":"2024-09-21T19:14:24.867114+0000","last_fullsized":"2024-09-21T19:14:24.867114+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:07.511235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:07.511235+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"490'39","reported_seq":1083,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.855384+0000","last_change":"2024-09-21T19:13:44.857267+0000","last_active":"2024-09-21T19:14:24.855384+0000","last_peered":"2024-09-21T19:14:24.855384+0000","last_clean":"2024-09-21T19:14:24.855384+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:14:24.855384+0000","last_undegraded":"2024-09-21T19:14:24.855384+0000","last_fullsized":"2024-09-21T19:14:24.855384+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:13:44.857192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:44.857192+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:08.272995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1003,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:41.711114+0000","last_change":"2024-09-21T19:14:41.711114+0000","last_active":"2024-09-21T19:14:41.711114+0000","last_peered":"2024-09-21T19:14:41.711114+0000","last_clean":"2024-09-21T19:14:41.711114+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:14:41.711114+0000","last_undegraded":"2024-09-21T19:14:41.711114+0000","last_fullsized":"2024-09-21T19:14:41.711114+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:41.711069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:41.711069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:07.293338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"521'16","reported_seq":1215,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859127+0000","last_change":"2024-09-21T19:14:13.024964+0000","last_active":"2024-09-21T19:14:44.859127+0000","last_peered":"2024-09-21T19:14:44.859127+0000","last_clean":"2024-09-21T19:14:44.859127+0000","last_became_active":"2024-09-21T19:14:13.024453+0000","last_became_peered":"2024-09-21T19:14:13.024453+0000","last_unstale":"2024-09-21T19:14:44.859127+0000","last_undegraded":"2024-09-21T19:14:44.859127+0000","last_fullsized":"2024-09-21T19:14:44.859127+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15","last_scrub_stamp":"2024-09-21T19:13:42.896452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:42.896452+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:04.665743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":667,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:25.924672+0000","last_change":"2024-09-21T19:14:25.924613+0000","last_active":"2024-09-21T19:14:25.924672+0000","last_peered":"2024-09-21T19:14:25.924672+0000","last_clean":"2024-09-21T19:14:25.924672+0000","last_became_active":"2024-09-21T19:14:24.861084+0000","last_became_peered":"2024-09-21T19:14:24.861084+0000","last_unstale":"2024-09-21T19:14:25.924672+0000","last_undegraded":"2024-09-21T19:14:25.924672+0000","last_fullsized":"2024-09-21T19:14:25.924672+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:25.924556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:25.924556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:35.123028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":726,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858969+0000","last_change":"2024-09-21T19:14:13.028074+0000","last_active":"2024-09-21T19:14:44.858969+0000","last_peered":"2024-09-21T19:14:44.858969+0000","last_clean":"2024-09-21T19:14:44.858969+0000","last_became_active":"2024-09-21T19:14:13.027734+0000","last_became_peered":"2024-09-21T19:14:13.027734+0000","last_unstale":"2024-09-21T19:14:44.858969+0000","last_undegraded":"2024-09-21T19:14:44.858969+0000","last_fullsized":"2024-09-21T19:14:44.858969+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:53.424066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:53.424066+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:12.087769+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"490'42","reported_seq":1110,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853072+0000","last_change":"2024-09-21T19:11:44.959245+0000","last_active":"2024-09-21T19:14:24.853072+0000","last_peered":"2024-09-21T19:14:24.853072+0000","last_clean":"2024-09-21T19:14:24.853072+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:14:24.853072+0000","last_undegraded":"2024-09-21T19:14:24.853072+0000","last_fullsized":"2024-09-21T19:14:24.853072+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:44.959061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.959061+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1025,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853421+0000","last_change":"2024-09-21T19:14:19.904069+0000","last_active":"2024-09-21T19:14:24.853421+0000","last_peered":"2024-09-21T19:14:24.853421+0000","last_clean":"2024-09-21T19:14:24.853421+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:14:24.853421+0000","last_undegraded":"2024-09-21T19:14:24.853421+0000","last_fullsized":"2024-09-21T19:14:24.853421+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:19.904020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:19.904020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:34.444141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":928,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857925+0000","last_change":"2024-09-21T19:14:18.906020+0000","last_active":"2024-09-21T19:14:44.857925+0000","last_peered":"2024-09-21T19:14:44.857925+0000","last_clean":"2024-09-21T19:14:44.857925+0000","last_became_active":"2024-09-21T19:14:13.023475+0000","last_became_peered":"2024-09-21T19:14:13.023475+0000","last_unstale":"2024-09-21T19:14:44.857925+0000","last_undegraded":"2024-09-21T19:14:44.857925+0000","last_fullsized":"2024-09-21T19:14:44.857925+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:18.905955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:18.905955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:40.196872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":937,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858179+0000","last_change":"2024-09-21T19:14:13.025483+0000","last_active":"2024-09-21T19:14:44.858179+0000","last_peered":"2024-09-21T19:14:44.858179+0000","last_clean":"2024-09-21T19:14:44.858179+0000","last_became_active":"2024-09-21T19:14:13.024395+0000","last_became_peered":"2024-09-21T19:14:13.024395+0000","last_unstale":"2024-09-21T19:14:44.858179+0000","last_undegraded":"2024-09-21T19:14:44.858179+0000","last_fullsized":"2024-09-21T19:14:44.858179+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:59.879323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:59.879323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:01.209491+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":814,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851061+0000","last_change":"2024-09-21T19:13:41.868709+0000","last_active":"2024-09-21T19:14:24.851061+0000","last_peered":"2024-09-21T19:14:24.851061+0000","last_clean":"2024-09-21T19:14:24.851061+0000","last_became_active":"2024-09-21T19:13:41.867782+0000","last_became_peered":"2024-09-21T19:13:41.867782+0000","last_unstale":"2024-09-21T19:14:24.851061+0000","last_undegraded":"2024-09-21T19:14:24.851061+0000","last_fullsized":"2024-09-21T19:14:24.851061+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:11:48.886536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:11:48.886536+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"490'42","reported_seq":615,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858344+0000","last_change":"2024-09-21T19:14:13.025698+0000","last_active":"2024-09-21T19:14:44.858344+0000","last_peered":"2024-09-21T19:14:44.858344+0000","last_clean":"2024-09-21T19:14:44.858344+0000","last_became_active":"2024-09-21T19:14:13.025130+0000","last_became_peered":"2024-09-21T19:14:13.025130+0000","last_unstale":"2024-09-21T19:14:44.858344+0000","last_undegraded":"2024-09-21T19:14:44.858344+0000","last_fullsized":"2024-09-21T19:14:44.858344+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'42","last_scrub_stamp":"2024-09-21T19:13:21.835098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:21.835098+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":870,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.852101+0000","last_change":"2024-09-21T19:14:07.874082+0000","last_active":"2024-09-21T19:14:24.852101+0000","last_peered":"2024-09-21T19:14:24.852101+0000","last_clean":"2024-09-21T19:14:24.852101+0000","last_became_active":"2024-09-21T19:13:55.926089+0000","last_became_peered":"2024-09-21T19:13:55.926089+0000","last_unstale":"2024-09-21T19:14:24.852101+0000","last_undegraded":"2024-09-21T19:14:24.852101+0000","last_fullsized":"2024-09-21T19:14:24.852101+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:07.874024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:07.874024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.625951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":1256,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:37.708140+0000","last_change":"2024-09-21T19:14:08.907447+0000","last_active":"2024-09-21T19:14:37.708140+0000","last_peered":"2024-09-21T19:14:37.708140+0000","last_clean":"2024-09-21T19:14:37.708140+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:14:37.708140+0000","last_undegraded":"2024-09-21T19:14:37.708140+0000","last_fullsized":"2024-09-21T19:14:37.708140+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:14:08.907398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:08.907398+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:19.040641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1027,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071752+0000","last_change":"2024-09-21T19:14:32.072031+0000","last_active":"2024-09-21T19:14:45.071752+0000","last_peered":"2024-09-21T19:14:45.071752+0000","last_clean":"2024-09-21T19:14:45.071752+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:14:45.071752+0000","last_undegraded":"2024-09-21T19:14:45.071752+0000","last_fullsized":"2024-09-21T19:14:45.071752+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:32.071962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:32.071962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.875506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":819,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.297344+0000","last_change":"2024-09-21T19:13:50.963002+0000","last_active":"2024-09-21T19:14:44.297344+0000","last_peered":"2024-09-21T19:14:44.297344+0000","last_clean":"2024-09-21T19:14:44.297344+0000","last_became_active":"2024-09-21T19:11:47.123128+0000","last_became_peered":"2024-09-21T19:11:47.123128+0000","last_unstale":"2024-09-21T19:14:44.297344+0000","last_undegraded":"2024-09-21T19:14:44.297344+0000","last_fullsized":"2024-09-21T19:14:44.297344+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:50.962927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:50.962927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:57.328613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"490'30","reported_seq":1109,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.857339+0000","last_change":"2024-09-21T19:13:55.727501+0000","last_active":"2024-09-21T19:14:24.857339+0000","last_peered":"2024-09-21T19:14:24.857339+0000","last_clean":"2024-09-21T19:14:24.857339+0000","last_became_active":"2024-09-21T19:13:41.872842+0000","last_became_peered":"2024-09-21T19:13:41.872842+0000","last_unstale":"2024-09-21T19:14:24.857339+0000","last_undegraded":"2024-09-21T19:14:24.857339+0000","last_fullsized":"2024-09-21T19:14:24.857339+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:13:55.727428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:55.727428+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:00.346884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":526,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867994+0000","last_change":"2024-09-21T19:14:24.866553+0000","last_active":"2024-09-21T19:14:24.867994+0000","last_peered":"2024-09-21T19:14:24.867994+0000","last_clean":"2024-09-21T19:14:24.867994+0000","last_became_active":"2024-09-21T19:14:24.865145+0000","last_became_peered":"2024-09-21T19:14:24.865145+0000","last_unstale":"2024-09-21T19:14:24.867994+0000","last_undegraded":"2024-09-21T19:14:24.867994+0000","last_fullsized":"2024-09-21T19:14:24.867994+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:48.104135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:48.104135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:58.516201+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"511'18","reported_seq":1122,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:38.040584+0000","last_change":"2024-09-21T19:14:19.838907+0000","last_active":"2024-09-21T19:14:38.040584+0000","last_peered":"2024-09-21T19:14:38.040584+0000","last_clean":"2024-09-21T19:14:38.040584+0000","last_became_active":"2024-09-21T19:13:55.929191+0000","last_became_peered":"2024-09-21T19:13:55.929191+0000","last_unstale":"2024-09-21T19:14:38.040584+0000","last_undegraded":"2024-09-21T19:14:38.040584+0000","last_fullsized":"2024-09-21T19:14:38.040584+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'18","last_scrub_stamp":"2024-09-21T19:14:19.838854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:19.838854+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:48.270840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":881,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851176+0000","last_change":"2024-09-21T19:14:13.860409+0000","last_active":"2024-09-21T19:14:24.851176+0000","last_peered":"2024-09-21T19:14:24.851176+0000","last_clean":"2024-09-21T19:14:24.851176+0000","last_became_active":"2024-09-21T19:13:55.924872+0000","last_became_peered":"2024-09-21T19:13:55.924872+0000","last_unstale":"2024-09-21T19:14:24.851176+0000","last_undegraded":"2024-09-21T19:14:24.851176+0000","last_fullsized":"2024-09-21T19:14:24.851176+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:13.860363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:13.860363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:32.068056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":805,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.297540+0000","last_change":"2024-09-21T19:14:33.996267+0000","last_active":"2024-09-21T19:14:44.297540+0000","last_peered":"2024-09-21T19:14:44.297540+0000","last_clean":"2024-09-21T19:14:44.297540+0000","last_became_active":"2024-09-21T19:11:47.128147+0000","last_became_peered":"2024-09-21T19:11:47.128147+0000","last_unstale":"2024-09-21T19:14:44.297540+0000","last_undegraded":"2024-09-21T19:14:44.297540+0000","last_fullsized":"2024-09-21T19:14:44.297540+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:33.996196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.996196+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:03.082199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"490'39","reported_seq":951,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856747+0000","last_change":"2024-09-21T19:13:55.930226+0000","last_active":"2024-09-21T19:14:24.856747+0000","last_peered":"2024-09-21T19:14:24.856747+0000","last_clean":"2024-09-21T19:14:24.856747+0000","last_became_active":"2024-09-21T19:13:55.928792+0000","last_became_peered":"2024-09-21T19:13:55.928792+0000","last_unstale":"2024-09-21T19:14:24.856747+0000","last_undegraded":"2024-09-21T19:14:24.856747+0000","last_fullsized":"2024-09-21T19:14:24.856747+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:13:42.860161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:42.860161+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:44.680581+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1050,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856938+0000","last_change":"2024-09-21T19:13:43.885912+0000","last_active":"2024-09-21T19:14:24.856938+0000","last_peered":"2024-09-21T19:14:24.856938+0000","last_clean":"2024-09-21T19:14:24.856938+0000","last_became_active":"2024-09-21T19:13:41.872311+0000","last_became_peered":"2024-09-21T19:13:41.872311+0000","last_unstale":"2024-09-21T19:14:24.856938+0000","last_undegraded":"2024-09-21T19:14:24.856938+0000","last_fullsized":"2024-09-21T19:14:24.856938+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:43.885865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:43.885865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:52.951358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":928,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.295886+0000","last_change":"2024-09-21T19:13:41.867779+0000","last_active":"2024-09-21T19:14:44.295886+0000","last_peered":"2024-09-21T19:14:44.295886+0000","last_clean":"2024-09-21T19:14:44.295886+0000","last_became_active":"2024-09-21T19:13:41.867084+0000","last_became_peered":"2024-09-21T19:13:41.867084+0000","last_unstale":"2024-09-21T19:14:44.295886+0000","last_undegraded":"2024-09-21T19:14:44.295886+0000","last_fullsized":"2024-09-21T19:14:44.295886+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:17.771078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:17.771078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1024,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.855336+0000","last_change":"2024-09-21T19:13:59.800260+0000","last_active":"2024-09-21T19:14:24.855336+0000","last_peered":"2024-09-21T19:14:24.855336+0000","last_clean":"2024-09-21T19:14:24.855336+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:14:24.855336+0000","last_undegraded":"2024-09-21T19:14:24.855336+0000","last_fullsized":"2024-09-21T19:14:24.855336+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:59.800207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:59.800207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.108621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":736,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.868054+0000","last_change":"2024-09-21T19:14:24.866764+0000","last_active":"2024-09-21T19:14:24.868054+0000","last_peered":"2024-09-21T19:14:24.868054+0000","last_clean":"2024-09-21T19:14:24.868054+0000","last_became_active":"2024-09-21T19:14:24.866425+0000","last_became_peered":"2024-09-21T19:14:24.866425+0000","last_unstale":"2024-09-21T19:14:24.868054+0000","last_undegraded":"2024-09-21T19:14:24.868054+0000","last_fullsized":"2024-09-21T19:14:24.868054+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:07.756889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:07.756889+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"490'54","reported_seq":1402,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:37.708145+0000","last_change":"2024-09-21T19:13:46.857209+0000","last_active":"2024-09-21T19:14:37.708145+0000","last_peered":"2024-09-21T19:14:37.708145+0000","last_clean":"2024-09-21T19:14:37.708145+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:14:37.708145+0000","last_undegraded":"2024-09-21T19:14:37.708145+0000","last_fullsized":"2024-09-21T19:14:37.708145+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'54","last_scrub_stamp":"2024-09-21T19:13:46.857095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:46.857095+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:00.931094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":953,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858591+0000","last_change":"2024-09-21T19:14:13.027115+0000","last_active":"2024-09-21T19:14:44.858591+0000","last_peered":"2024-09-21T19:14:44.858591+0000","last_clean":"2024-09-21T19:14:44.858591+0000","last_became_active":"2024-09-21T19:14:13.025981+0000","last_became_peered":"2024-09-21T19:14:13.025981+0000","last_unstale":"2024-09-21T19:14:44.858591+0000","last_undegraded":"2024-09-21T19:14:44.858591+0000","last_fullsized":"2024-09-21T19:14:44.858591+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:26.913553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:26.913553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":562,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:32.975360+0000","last_change":"2024-09-21T19:14:32.975360+0000","last_active":"2024-09-21T19:14:32.975360+0000","last_peered":"2024-09-21T19:14:32.975360+0000","last_clean":"2024-09-21T19:14:32.975360+0000","last_became_active":"2024-09-21T19:14:24.863058+0000","last_became_peered":"2024-09-21T19:14:24.863058+0000","last_unstale":"2024-09-21T19:14:32.975360+0000","last_undegraded":"2024-09-21T19:14:32.975360+0000","last_fullsized":"2024-09-21T19:14:32.975360+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:14:32.975298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:14:32.975298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.567425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":867,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851701+0000","last_change":"2024-09-21T19:14:15.775344+0000","last_active":"2024-09-21T19:14:24.851701+0000","last_peered":"2024-09-21T19:14:24.851701+0000","last_clean":"2024-09-21T19:14:24.851701+0000","last_became_active":"2024-09-21T19:13:55.926560+0000","last_became_peered":"2024-09-21T19:13:55.926560+0000","last_unstale":"2024-09-21T19:14:24.851701+0000","last_undegraded":"2024-09-21T19:14:24.851701+0000","last_fullsized":"2024-09-21T19:14:24.851701+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:15.775294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.775294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.789016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":991,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.852956+0000","last_change":"2024-09-21T19:13:58.984558+0000","last_active":"2024-09-21T19:14:24.852956+0000","last_peered":"2024-09-21T19:14:24.852956+0000","last_clean":"2024-09-21T19:14:24.852956+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:14:24.852956+0000","last_undegraded":"2024-09-21T19:14:24.852956+0000","last_fullsized":"2024-09-21T19:14:24.852956+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:58.984508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:58.984508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:06.348475+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":1110,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071634+0000","last_change":"2024-09-21T19:13:54.002452+0000","last_active":"2024-09-21T19:14:45.071634+0000","last_peered":"2024-09-21T19:14:45.071634+0000","last_clean":"2024-09-21T19:14:45.071634+0000","last_became_active":"2024-09-21T19:11:47.117467+0000","last_became_peered":"2024-09-21T19:11:47.117467+0000","last_unstale":"2024-09-21T19:14:45.071634+0000","last_undegraded":"2024-09-21T19:14:45.071634+0000","last_fullsized":"2024-09-21T19:14:45.071634+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:54.002392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.002392+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:16.895095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"489'21","reported_seq":553,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.855281+0000","last_change":"2024-09-21T19:13:47.868425+0000","last_active":"2024-09-21T19:14:24.855281+0000","last_peered":"2024-09-21T19:14:24.855281+0000","last_clean":"2024-09-21T19:14:24.855281+0000","last_became_active":"2024-09-21T19:13:41.874168+0000","last_became_peered":"2024-09-21T19:13:41.874168+0000","last_unstale":"2024-09-21T19:14:24.855281+0000","last_undegraded":"2024-09-21T19:14:24.855281+0000","last_fullsized":"2024-09-21T19:14:24.855281+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:13:47.868368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:47.868368+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:49.773783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":1066,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.855726+0000","last_change":"2024-09-21T19:14:20.003988+0000","last_active":"2024-09-21T19:14:24.855726+0000","last_peered":"2024-09-21T19:14:24.855726+0000","last_clean":"2024-09-21T19:14:24.855726+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:14:24.855726+0000","last_undegraded":"2024-09-21T19:14:24.855726+0000","last_fullsized":"2024-09-21T19:14:24.855726+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:20.003924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:20.003924+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:39.737121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":460,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.859109+0000","last_change":"2024-09-21T19:13:45.514102+0000","last_active":"2024-09-21T19:14:24.859109+0000","last_peered":"2024-09-21T19:14:24.859109+0000","last_clean":"2024-09-21T19:14:24.859109+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:14:24.859109+0000","last_undegraded":"2024-09-21T19:14:24.859109+0000","last_fullsized":"2024-09-21T19:14:24.859109+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:13:45.514051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:13:45.514051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:58.431122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":481,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.859897+0000","last_change":"2024-09-21T19:14:16.442387+0000","last_active":"2024-09-21T19:14:24.859897+0000","last_peered":"2024-09-21T19:14:24.859897+0000","last_clean":"2024-09-21T19:14:24.859897+0000","last_became_active":"2024-09-21T19:13:41.876613+0000","last_became_peered":"2024-09-21T19:13:41.876613+0000","last_unstale":"2024-09-21T19:14:24.859897+0000","last_undegraded":"2024-09-21T19:14:24.859897+0000","last_fullsized":"2024-09-21T19:14:24.859897+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:16.442336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:16.442336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.523208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":1025,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072429+0000","last_change":"2024-09-21T19:14:04.026314+0000","last_active":"2024-09-21T19:14:45.072429+0000","last_peered":"2024-09-21T19:14:45.072429+0000","last_clean":"2024-09-21T19:14:45.072429+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:14:45.072429+0000","last_undegraded":"2024-09-21T19:14:45.072429+0000","last_fullsized":"2024-09-21T19:14:45.072429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:14:04.026250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.026250+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:08.075250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":820,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:30.104353+0000","last_change":"2024-09-21T19:14:30.104353+0000","last_active":"2024-09-21T19:14:30.104353+0000","last_peered":"2024-09-21T19:14:30.104353+0000","last_clean":"2024-09-21T19:14:30.104353+0000","last_became_active":"2024-09-21T19:11:47.124560+0000","last_became_peered":"2024-09-21T19:11:47.124560+0000","last_unstale":"2024-09-21T19:14:30.104353+0000","last_undegraded":"2024-09-21T19:14:30.104353+0000","last_fullsized":"2024-09-21T19:14:30.104353+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:30.104280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.104280+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:58.386649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"490'36","reported_seq":1042,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851902+0000","last_change":"2024-09-21T19:13:07.182968+0000","last_active":"2024-09-21T19:14:24.851902+0000","last_peered":"2024-09-21T19:14:24.851902+0000","last_clean":"2024-09-21T19:14:24.851902+0000","last_became_active":"2024-09-21T19:13:07.170435+0000","last_became_peered":"2024-09-21T19:13:07.170435+0000","last_unstale":"2024-09-21T19:14:24.851902+0000","last_undegraded":"2024-09-21T19:14:24.851902+0000","last_fullsized":"2024-09-21T19:14:24.851902+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T19:11:45.692118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:45.692118+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:38.859825+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":875,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:36.059558+0000","last_change":"2024-09-21T19:14:36.059514+0000","last_active":"2024-09-21T19:14:36.059558+0000","last_peered":"2024-09-21T19:14:36.059558+0000","last_clean":"2024-09-21T19:14:36.059558+0000","last_became_active":"2024-09-21T19:13:55.929379+0000","last_became_peered":"2024-09-21T19:13:55.929379+0000","last_unstale":"2024-09-21T19:14:36.059558+0000","last_undegraded":"2024-09-21T19:14:36.059558+0000","last_fullsized":"2024-09-21T19:14:36.059558+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:36.059468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:36.059468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:49.888615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":2331,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.242602+0000","last_change":"2024-09-21T19:13:56.018952+0000","last_active":"2024-09-21T19:14:45.242602+0000","last_peered":"2024-09-21T19:14:45.242602+0000","last_clean":"2024-09-21T19:14:45.242602+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:14:45.242602+0000","last_undegraded":"2024-09-21T19:14:45.242602+0000","last_fullsized":"2024-09-21T19:14:45.242602+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:13:56.018881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:13:56.018881+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.956576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1287,"num_read_kb":1282,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":70,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.857804+0000","last_change":"2024-09-21T19:13:54.989792+0000","last_active":"2024-09-21T19:14:24.857804+0000","last_peered":"2024-09-21T19:14:24.857804+0000","last_clean":"2024-09-21T19:14:24.857804+0000","last_became_active":"2024-09-21T19:13:41.871779+0000","last_became_peered":"2024-09-21T19:13:41.871779+0000","last_unstale":"2024-09-21T19:14:24.857804+0000","last_undegraded":"2024-09-21T19:14:24.857804+0000","last_fullsized":"2024-09-21T19:14:24.857804+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:54.989741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.989741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:01.716245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1040,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:28.078803+0000","last_change":"2024-09-21T19:14:28.078803+0000","last_active":"2024-09-21T19:14:28.078803+0000","last_peered":"2024-09-21T19:14:28.078803+0000","last_clean":"2024-09-21T19:14:28.078803+0000","last_became_active":"2024-09-21T19:13:41.871757+0000","last_became_peered":"2024-09-21T19:13:41.871757+0000","last_unstale":"2024-09-21T19:14:28.078803+0000","last_undegraded":"2024-09-21T19:14:28.078803+0000","last_fullsized":"2024-09-21T19:14:28.078803+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:28.078742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:28.078742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.539999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"490'42","reported_seq":1263,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.297448+0000","last_change":"2024-09-21T19:11:02.540038+0000","last_active":"2024-09-21T19:14:44.297448+0000","last_peered":"2024-09-21T19:14:44.297448+0000","last_clean":"2024-09-21T19:14:44.297448+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:14:44.297448+0000","last_undegraded":"2024-09-21T19:14:44.297448+0000","last_fullsized":"2024-09-21T19:14:44.297448+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:02.539974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:02.539974+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1040,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:37.790869+0000","last_change":"2024-09-21T19:14:37.790869+0000","last_active":"2024-09-21T19:14:37.790869+0000","last_peered":"2024-09-21T19:14:37.790869+0000","last_clean":"2024-09-21T19:14:37.790869+0000","last_became_active":"2024-09-21T19:13:41.869386+0000","last_became_peered":"2024-09-21T19:13:41.869386+0000","last_unstale":"2024-09-21T19:14:37.790869+0000","last_undegraded":"2024-09-21T19:14:37.790869+0000","last_fullsized":"2024-09-21T19:14:37.790869+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:37.790818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:37.790818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:02.538838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":1132,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072223+0000","last_change":"2024-09-21T19:14:33.060110+0000","last_active":"2024-09-21T19:14:45.072223+0000","last_peered":"2024-09-21T19:14:45.072223+0000","last_clean":"2024-09-21T19:14:45.072223+0000","last_became_active":"2024-09-21T19:11:47.120134+0000","last_became_peered":"2024-09-21T19:11:47.120134+0000","last_unstale":"2024-09-21T19:14:45.072223+0000","last_undegraded":"2024-09-21T19:14:45.072223+0000","last_fullsized":"2024-09-21T19:14:45.072223+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:33.060021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.060021+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:41.088380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1045,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072287+0000","last_change":"2024-09-21T19:13:43.248336+0000","last_active":"2024-09-21T19:14:45.072287+0000","last_peered":"2024-09-21T19:14:45.072287+0000","last_clean":"2024-09-21T19:14:45.072287+0000","last_became_active":"2024-09-21T19:13:41.872230+0000","last_became_peered":"2024-09-21T19:13:41.872230+0000","last_unstale":"2024-09-21T19:14:45.072287+0000","last_undegraded":"2024-09-21T19:14:45.072287+0000","last_fullsized":"2024-09-21T19:14:45.072287+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:43.248293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:43.248293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":511,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.869339+0000","last_change":"2024-09-21T19:14:24.867743+0000","last_active":"2024-09-21T19:14:24.869339+0000","last_peered":"2024-09-21T19:14:24.869339+0000","last_clean":"2024-09-21T19:14:24.869339+0000","last_became_active":"2024-09-21T19:14:24.867241+0000","last_became_peered":"2024-09-21T19:14:24.867241+0000","last_unstale":"2024-09-21T19:14:24.869339+0000","last_undegraded":"2024-09-21T19:14:24.869339+0000","last_fullsized":"2024-09-21T19:14:24.869339+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:43.092935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:43.092935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:50.834654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":840,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:35.117767+0000","last_change":"2024-09-21T19:14:35.117767+0000","last_active":"2024-09-21T19:14:35.117767+0000","last_peered":"2024-09-21T19:14:35.117767+0000","last_clean":"2024-09-21T19:14:35.117767+0000","last_became_active":"2024-09-21T19:13:41.871481+0000","last_became_peered":"2024-09-21T19:13:41.871481+0000","last_unstale":"2024-09-21T19:14:35.117767+0000","last_undegraded":"2024-09-21T19:14:35.117767+0000","last_fullsized":"2024-09-21T19:14:35.117767+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:35.117720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:35.117720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:51.442926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"490'27","reported_seq":607,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859143+0000","last_change":"2024-09-21T19:14:17.918424+0000","last_active":"2024-09-21T19:14:44.859143+0000","last_peered":"2024-09-21T19:14:44.859143+0000","last_clean":"2024-09-21T19:14:44.859143+0000","last_became_active":"2024-09-21T19:14:13.028931+0000","last_became_peered":"2024-09-21T19:14:13.028931+0000","last_unstale":"2024-09-21T19:14:44.859143+0000","last_undegraded":"2024-09-21T19:14:44.859143+0000","last_fullsized":"2024-09-21T19:14:44.859143+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:14:17.918358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:17.918358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:44.809780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":957,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.073356+0000","last_change":"2024-09-21T19:14:45.073356+0000","last_active":"2024-09-21T19:14:45.073356+0000","last_peered":"2024-09-21T19:14:45.073356+0000","last_clean":"2024-09-21T19:14:45.073356+0000","last_became_active":"2024-09-21T19:14:13.024178+0000","last_became_peered":"2024-09-21T19:14:13.024178+0000","last_unstale":"2024-09-21T19:14:45.073356+0000","last_undegraded":"2024-09-21T19:14:45.073356+0000","last_fullsized":"2024-09-21T19:14:45.073356+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:45.073297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:45.073297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.601678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":916,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.869314+0000","last_change":"2024-09-21T19:14:24.867578+0000","last_active":"2024-09-21T19:14:24.869314+0000","last_peered":"2024-09-21T19:14:24.869314+0000","last_clean":"2024-09-21T19:14:24.869314+0000","last_became_active":"2024-09-21T19:14:24.867093+0000","last_became_peered":"2024-09-21T19:14:24.867093+0000","last_unstale":"2024-09-21T19:14:24.869314+0000","last_undegraded":"2024-09-21T19:14:24.869314+0000","last_fullsized":"2024-09-21T19:14:24.869314+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:59.126971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:59.126971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:28.418699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1003,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851148+0000","last_change":"2024-09-21T19:13:41.868100+0000","last_active":"2024-09-21T19:14:24.851148+0000","last_peered":"2024-09-21T19:14:24.851148+0000","last_clean":"2024-09-21T19:14:24.851148+0000","last_became_active":"2024-09-21T19:13:41.866996+0000","last_became_peered":"2024-09-21T19:13:41.866996+0000","last_unstale":"2024-09-21T19:14:24.851148+0000","last_undegraded":"2024-09-21T19:14:24.851148+0000","last_fullsized":"2024-09-21T19:14:24.851148+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:08.220479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:08.220479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":436,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867963+0000","last_change":"2024-09-21T19:14:24.866856+0000","last_active":"2024-09-21T19:14:24.867963+0000","last_peered":"2024-09-21T19:14:24.867963+0000","last_clean":"2024-09-21T19:14:24.867963+0000","last_became_active":"2024-09-21T19:14:24.865037+0000","last_became_peered":"2024-09-21T19:14:24.865037+0000","last_unstale":"2024-09-21T19:14:24.867963+0000","last_undegraded":"2024-09-21T19:14:24.867963+0000","last_fullsized":"2024-09-21T19:14:24.867963+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:14.425583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:14.425583+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:24.104232+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"490'45","reported_seq":746,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858839+0000","last_change":"2024-09-21T19:14:36.060038+0000","last_active":"2024-09-21T19:14:44.858839+0000","last_peered":"2024-09-21T19:14:44.858839+0000","last_clean":"2024-09-21T19:14:44.858839+0000","last_became_active":"2024-09-21T19:14:13.022192+0000","last_became_peered":"2024-09-21T19:14:13.022192+0000","last_unstale":"2024-09-21T19:14:44.858839+0000","last_undegraded":"2024-09-21T19:14:44.858839+0000","last_fullsized":"2024-09-21T19:14:44.858839+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'45","last_scrub_stamp":"2024-09-21T19:14:36.059972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:36.059972+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:03.727967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":941,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.296269+0000","last_change":"2024-09-21T19:14:04.994693+0000","last_active":"2024-09-21T19:14:44.296269+0000","last_peered":"2024-09-21T19:14:44.296269+0000","last_clean":"2024-09-21T19:14:44.296269+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:14:44.296269+0000","last_undegraded":"2024-09-21T19:14:44.296269+0000","last_fullsized":"2024-09-21T19:14:44.296269+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:04.994628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.994628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:11.148862+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":466,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.295691+0000","last_change":"2024-09-21T19:14:05.946266+0000","last_active":"2024-09-21T19:14:44.295691+0000","last_peered":"2024-09-21T19:14:44.295691+0000","last_clean":"2024-09-21T19:14:44.295691+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:14:44.295691+0000","last_undegraded":"2024-09-21T19:14:44.295691+0000","last_fullsized":"2024-09-21T19:14:44.295691+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:05.946211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:05.946211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:29.215685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":472,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:33.413391+0000","last_change":"2024-09-21T19:14:33.413325+0000","last_active":"2024-09-21T19:14:33.413391+0000","last_peered":"2024-09-21T19:14:33.413391+0000","last_clean":"2024-09-21T19:14:33.413391+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:14:33.413391+0000","last_undegraded":"2024-09-21T19:14:33.413391+0000","last_fullsized":"2024-09-21T19:14:33.413391+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:33.413271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.413271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:54.479246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":1126,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071669+0000","last_change":"2024-09-21T19:14:05.000670+0000","last_active":"2024-09-21T19:14:45.071669+0000","last_peered":"2024-09-21T19:14:45.071669+0000","last_clean":"2024-09-21T19:14:45.071669+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:14:45.071669+0000","last_undegraded":"2024-09-21T19:14:45.071669+0000","last_fullsized":"2024-09-21T19:14:45.071669+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:05.000568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:05.000568+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:32.973330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"490'30","reported_seq":593,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858308+0000","last_change":"2024-09-21T19:14:43.056924+0000","last_active":"2024-09-21T19:14:44.858308+0000","last_peered":"2024-09-21T19:14:44.858308+0000","last_clean":"2024-09-21T19:14:44.858308+0000","last_became_active":"2024-09-21T19:14:13.027486+0000","last_became_peered":"2024-09-21T19:14:13.027486+0000","last_unstale":"2024-09-21T19:14:44.858308+0000","last_undegraded":"2024-09-21T19:14:44.858308+0000","last_fullsized":"2024-09-21T19:14:44.858308+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:14:43.056857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:43.056857+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.472766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":1039,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:27.801807+0000","last_change":"2024-09-21T19:14:27.801807+0000","last_active":"2024-09-21T19:14:27.801807+0000","last_peered":"2024-09-21T19:14:27.801807+0000","last_clean":"2024-09-21T19:14:27.801807+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:14:27.801807+0000","last_undegraded":"2024-09-21T19:14:27.801807+0000","last_fullsized":"2024-09-21T19:14:27.801807+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:27.801751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:27.801751+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.975381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"504'14","reported_seq":604,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:42.709114+0000","last_change":"2024-09-21T19:14:02.435541+0000","last_active":"2024-09-21T19:14:42.709114+0000","last_peered":"2024-09-21T19:14:42.709114+0000","last_clean":"2024-09-21T19:14:42.709114+0000","last_became_active":"2024-09-21T19:13:41.877504+0000","last_became_peered":"2024-09-21T19:13:41.877504+0000","last_unstale":"2024-09-21T19:14:42.709114+0000","last_undegraded":"2024-09-21T19:14:42.709114+0000","last_fullsized":"2024-09-21T19:14:42.709114+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'14","last_scrub_stamp":"2024-09-21T19:14:02.435491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:02.435491+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:05.135485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":921,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858923+0000","last_change":"2024-09-21T19:14:13.023798+0000","last_active":"2024-09-21T19:14:44.858923+0000","last_peered":"2024-09-21T19:14:44.858923+0000","last_clean":"2024-09-21T19:14:44.858923+0000","last_became_active":"2024-09-21T19:14:13.022622+0000","last_became_peered":"2024-09-21T19:14:13.022622+0000","last_unstale":"2024-09-21T19:14:44.858923+0000","last_undegraded":"2024-09-21T19:14:44.858923+0000","last_fullsized":"2024-09-21T19:14:44.858923+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:06.812349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:06.812349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":102,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858913+0000","last_change":"2024-09-21T19:14:15.887014+0000","last_active":"2024-09-21T19:14:44.858913+0000","last_peered":"2024-09-21T19:14:44.858913+0000","last_clean":"2024-09-21T19:14:44.858913+0000","last_became_active":"2024-09-21T19:14:13.023805+0000","last_became_peered":"2024-09-21T19:14:13.023805+0000","last_unstale":"2024-09-21T19:14:44.858913+0000","last_undegraded":"2024-09-21T19:14:44.858913+0000","last_fullsized":"2024-09-21T19:14:44.858913+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:15.886949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.886949+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:19.925990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"490'21","reported_seq":878,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856801+0000","last_change":"2024-09-21T19:13:55.929905+0000","last_active":"2024-09-21T19:14:24.856801+0000","last_peered":"2024-09-21T19:14:24.856801+0000","last_clean":"2024-09-21T19:14:24.856801+0000","last_became_active":"2024-09-21T19:13:55.926999+0000","last_became_peered":"2024-09-21T19:13:55.926999+0000","last_unstale":"2024-09-21T19:14:24.856801+0000","last_undegraded":"2024-09-21T19:14:24.856801+0000","last_fullsized":"2024-09-21T19:14:24.856801+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:41.665004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.665004+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":967,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:41.947759+0000","last_change":"2024-09-21T19:14:41.947759+0000","last_active":"2024-09-21T19:14:41.947759+0000","last_peered":"2024-09-21T19:14:41.947759+0000","last_clean":"2024-09-21T19:14:41.947759+0000","last_became_active":"2024-09-21T19:14:24.862420+0000","last_became_peered":"2024-09-21T19:14:24.862420+0000","last_unstale":"2024-09-21T19:14:41.947759+0000","last_undegraded":"2024-09-21T19:14:41.947759+0000","last_fullsized":"2024-09-21T19:14:41.947759+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:41.947704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:41.947704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:11.301009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":456,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.858870+0000","last_change":"2024-09-21T19:13:17.345255+0000","last_active":"2024-09-21T19:14:24.858870+0000","last_peered":"2024-09-21T19:14:24.858870+0000","last_clean":"2024-09-21T19:14:24.858870+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:14:24.858870+0000","last_undegraded":"2024-09-21T19:14:24.858870+0000","last_fullsized":"2024-09-21T19:14:24.858870+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:17.345126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:17.345126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:42.575928+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":872,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851434+0000","last_change":"2024-09-21T19:13:55.930484+0000","last_active":"2024-09-21T19:14:24.851434+0000","last_peered":"2024-09-21T19:14:24.851434+0000","last_clean":"2024-09-21T19:14:24.851434+0000","last_became_active":"2024-09-21T19:13:55.929513+0000","last_became_peered":"2024-09-21T19:13:55.929513+0000","last_unstale":"2024-09-21T19:14:24.851434+0000","last_undegraded":"2024-09-21T19:14:24.851434+0000","last_fullsized":"2024-09-21T19:14:24.851434+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.588820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.588820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":847,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.297248+0000","last_change":"2024-09-21T19:14:18.978118+0000","last_active":"2024-09-21T19:14:44.297248+0000","last_peered":"2024-09-21T19:14:44.297248+0000","last_clean":"2024-09-21T19:14:44.297248+0000","last_became_active":"2024-09-21T19:13:41.873982+0000","last_became_peered":"2024-09-21T19:13:41.873982+0000","last_unstale":"2024-09-21T19:14:44.297248+0000","last_undegraded":"2024-09-21T19:14:44.297248+0000","last_fullsized":"2024-09-21T19:14:44.297248+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:18.978048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:18.978048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:37.065594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"490'51","reported_seq":976,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.296106+0000","last_change":"2024-09-21T19:13:58.129757+0000","last_active":"2024-09-21T19:14:44.296106+0000","last_peered":"2024-09-21T19:14:44.296106+0000","last_clean":"2024-09-21T19:14:44.296106+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:14:44.296106+0000","last_undegraded":"2024-09-21T19:14:44.296106+0000","last_fullsized":"2024-09-21T19:14:44.296106+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'51","last_scrub_stamp":"2024-09-21T19:13:58.129684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:58.129684+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:13.959269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1049,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072773+0000","last_change":"2024-09-21T19:13:55.013063+0000","last_active":"2024-09-21T19:14:45.072773+0000","last_peered":"2024-09-21T19:14:45.072773+0000","last_clean":"2024-09-21T19:14:45.072773+0000","last_became_active":"2024-09-21T19:13:41.868691+0000","last_became_peered":"2024-09-21T19:13:41.868691+0000","last_unstale":"2024-09-21T19:14:45.072773+0000","last_undegraded":"2024-09-21T19:14:45.072773+0000","last_fullsized":"2024-09-21T19:14:45.072773+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:55.013011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:55.013011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:18.337157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":481,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853633+0000","last_change":"2024-09-21T19:14:10.426107+0000","last_active":"2024-09-21T19:14:24.853633+0000","last_peered":"2024-09-21T19:14:24.853633+0000","last_clean":"2024-09-21T19:14:24.853633+0000","last_became_active":"2024-09-21T19:13:41.875147+0000","last_became_peered":"2024-09-21T19:13:41.875147+0000","last_unstale":"2024-09-21T19:14:24.853633+0000","last_undegraded":"2024-09-21T19:14:24.853633+0000","last_fullsized":"2024-09-21T19:14:24.853633+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:10.426061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:10.426061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:13.761821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1015,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072155+0000","last_change":"2024-09-21T19:14:08.030116+0000","last_active":"2024-09-21T19:14:45.072155+0000","last_peered":"2024-09-21T19:14:45.072155+0000","last_clean":"2024-09-21T19:14:45.072155+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:14:45.072155+0000","last_undegraded":"2024-09-21T19:14:45.072155+0000","last_fullsized":"2024-09-21T19:14:45.072155+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:08.030060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:08.030060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:13.046600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":805,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851372+0000","last_change":"2024-09-21T19:13:50.999379+0000","last_active":"2024-09-21T19:14:24.851372+0000","last_peered":"2024-09-21T19:14:24.851372+0000","last_clean":"2024-09-21T19:14:24.851372+0000","last_became_active":"2024-09-21T19:11:49.145230+0000","last_became_peered":"2024-09-21T19:11:49.145230+0000","last_unstale":"2024-09-21T19:14:24.851372+0000","last_undegraded":"2024-09-21T19:14:24.851372+0000","last_fullsized":"2024-09-21T19:14:24.851372+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:50.999329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:50.999329+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:52.341111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"489'24","reported_seq":81,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858650+0000","last_change":"2024-09-21T19:14:27.915104+0000","last_active":"2024-09-21T19:14:44.858650+0000","last_peered":"2024-09-21T19:14:44.858650+0000","last_clean":"2024-09-21T19:14:44.858650+0000","last_became_active":"2024-09-21T19:14:13.022927+0000","last_became_peered":"2024-09-21T19:14:13.022927+0000","last_unstale":"2024-09-21T19:14:44.858650+0000","last_undegraded":"2024-09-21T19:14:44.858650+0000","last_fullsized":"2024-09-21T19:14:44.858650+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'24","last_scrub_stamp":"2024-09-21T19:14:27.915028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:27.915028+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.299437+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1037,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072793+0000","last_change":"2024-09-21T19:14:03.006048+0000","last_active":"2024-09-21T19:14:45.072793+0000","last_peered":"2024-09-21T19:14:45.072793+0000","last_clean":"2024-09-21T19:14:45.072793+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:14:45.072793+0000","last_undegraded":"2024-09-21T19:14:45.072793+0000","last_fullsized":"2024-09-21T19:14:45.072793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:03.005968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:03.005968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:19.650408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":464,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:35.439447+0000","last_change":"2024-09-21T19:14:35.439447+0000","last_active":"2024-09-21T19:14:35.439447+0000","last_peered":"2024-09-21T19:14:35.439447+0000","last_clean":"2024-09-21T19:14:35.439447+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:14:35.439447+0000","last_undegraded":"2024-09-21T19:14:35.439447+0000","last_fullsized":"2024-09-21T19:14:35.439447+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:35.439400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:35.439400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:04.051547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":983,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:30.785324+0000","last_change":"2024-09-21T19:14:30.785324+0000","last_active":"2024-09-21T19:14:30.785324+0000","last_peered":"2024-09-21T19:14:30.785324+0000","last_clean":"2024-09-21T19:14:30.785324+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:14:30.785324+0000","last_undegraded":"2024-09-21T19:14:30.785324+0000","last_fullsized":"2024-09-21T19:14:30.785324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:30.785275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.785275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.319888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":190,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072203+0000","last_change":"2024-09-21T19:14:30.147373+0000","last_active":"2024-09-21T19:14:45.072203+0000","last_peered":"2024-09-21T19:14:45.072203+0000","last_clean":"2024-09-21T19:14:45.072203+0000","last_became_active":"2024-09-21T19:11:49.139511+0000","last_became_peered":"2024-09-21T19:11:49.139511+0000","last_unstale":"2024-09-21T19:14:45.072203+0000","last_undegraded":"2024-09-21T19:14:45.072203+0000","last_fullsized":"2024-09-21T19:14:45.072203+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:30.147305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.147305+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:39.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"490'39","reported_seq":624,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.864423+0000","last_change":"2024-09-21T19:14:24.862367+0000","last_active":"2024-09-21T19:14:24.864423+0000","last_peered":"2024-09-21T19:14:24.864423+0000","last_clean":"2024-09-21T19:14:24.864423+0000","last_became_active":"2024-09-21T19:14:24.861608+0000","last_became_peered":"2024-09-21T19:14:24.861608+0000","last_unstale":"2024-09-21T19:14:24.864423+0000","last_undegraded":"2024-09-21T19:14:24.864423+0000","last_fullsized":"2024-09-21T19:14:24.864423+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:14:24.051467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:24.051467+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.782268+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":1051,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072945+0000","last_change":"2024-09-21T19:14:17.075150+0000","last_active":"2024-09-21T19:14:45.072945+0000","last_peered":"2024-09-21T19:14:45.072945+0000","last_clean":"2024-09-21T19:14:45.072945+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:14:45.072945+0000","last_undegraded":"2024-09-21T19:14:45.072945+0000","last_fullsized":"2024-09-21T19:14:45.072945+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:14:17.075085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:17.075085+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:27.084716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":570,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:37.708169+0000","last_change":"2024-09-21T19:14:24.871790+0000","last_active":"2024-09-21T19:14:37.708169+0000","last_peered":"2024-09-21T19:14:37.708169+0000","last_clean":"2024-09-21T19:14:37.708169+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:14:37.708169+0000","last_undegraded":"2024-09-21T19:14:37.708169+0000","last_fullsized":"2024-09-21T19:14:37.708169+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:14:24.871708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:24.871708+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:44.378176+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":985,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.864285+0000","last_change":"2024-09-21T19:14:24.863784+0000","last_active":"2024-09-21T19:14:24.864285+0000","last_peered":"2024-09-21T19:14:24.864285+0000","last_clean":"2024-09-21T19:14:24.864285+0000","last_became_active":"2024-09-21T19:14:24.863412+0000","last_became_peered":"2024-09-21T19:14:24.863412+0000","last_unstale":"2024-09-21T19:14:24.864285+0000","last_undegraded":"2024-09-21T19:14:24.864285+0000","last_fullsized":"2024-09-21T19:14:24.864285+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:56.076731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:56.076731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.553102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":63,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.858509+0000","last_change":"2024-09-21T19:13:41.869004+0000","last_active":"2024-09-21T19:14:24.858509+0000","last_peered":"2024-09-21T19:14:24.858509+0000","last_clean":"2024-09-21T19:14:24.858509+0000","last_became_active":"2024-09-21T19:13:41.868497+0000","last_became_peered":"2024-09-21T19:13:41.868497+0000","last_unstale":"2024-09-21T19:14:24.858509+0000","last_undegraded":"2024-09-21T19:14:24.858509+0000","last_fullsized":"2024-09-21T19:14:24.858509+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:24.088416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:24.088416+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:44.499795+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":978,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.295184+0000","last_change":"2024-09-21T19:14:04.037324+0000","last_active":"2024-09-21T19:14:44.295184+0000","last_peered":"2024-09-21T19:14:44.295184+0000","last_clean":"2024-09-21T19:14:44.295184+0000","last_became_active":"2024-09-21T19:13:41.868092+0000","last_became_peered":"2024-09-21T19:13:41.868092+0000","last_unstale":"2024-09-21T19:14:44.295184+0000","last_undegraded":"2024-09-21T19:14:44.295184+0000","last_fullsized":"2024-09-21T19:14:44.295184+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:04.037267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.037267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.087896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"490'18","reported_seq":1033,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853287+0000","last_change":"2024-09-21T19:14:04.860511+0000","last_active":"2024-09-21T19:14:24.853287+0000","last_peered":"2024-09-21T19:14:24.853287+0000","last_clean":"2024-09-21T19:14:24.853287+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:14:24.853287+0000","last_undegraded":"2024-09-21T19:14:24.853287+0000","last_fullsized":"2024-09-21T19:14:24.853287+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'18","last_scrub_stamp":"2024-09-21T19:14:04.860444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.860444+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.498619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":519,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867876+0000","last_change":"2024-09-21T19:14:24.865786+0000","last_active":"2024-09-21T19:14:24.867876+0000","last_peered":"2024-09-21T19:14:24.867876+0000","last_clean":"2024-09-21T19:14:24.867876+0000","last_became_active":"2024-09-21T19:14:24.865457+0000","last_became_peered":"2024-09-21T19:14:24.865457+0000","last_unstale":"2024-09-21T19:14:24.867876+0000","last_undegraded":"2024-09-21T19:14:24.867876+0000","last_fullsized":"2024-09-21T19:14:24.867876+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:18.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:18.170847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:29.926764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":995,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072328+0000","last_change":"2024-09-21T19:10:22.627798+0000","last_active":"2024-09-21T19:14:45.072328+0000","last_peered":"2024-09-21T19:14:45.072328+0000","last_clean":"2024-09-21T19:14:45.072328+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:14:45.072328+0000","last_undegraded":"2024-09-21T19:14:45.072328+0000","last_fullsized":"2024-09-21T19:14:45.072328+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:22.627740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:22.627740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":818,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856467+0000","last_change":"2024-09-21T19:13:52.802017+0000","last_active":"2024-09-21T19:14:24.856467+0000","last_peered":"2024-09-21T19:14:24.856467+0000","last_clean":"2024-09-21T19:14:24.856467+0000","last_became_active":"2024-09-21T19:11:49.138877+0000","last_became_peered":"2024-09-21T19:11:49.138877+0000","last_unstale":"2024-09-21T19:14:24.856467+0000","last_undegraded":"2024-09-21T19:14:24.856467+0000","last_fullsized":"2024-09-21T19:14:24.856467+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:52.801960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:52.801960+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:07.421811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"489'21","reported_seq":885,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.852003+0000","last_change":"2024-09-21T19:13:55.931247+0000","last_active":"2024-09-21T19:14:24.852003+0000","last_peered":"2024-09-21T19:14:24.852003+0000","last_clean":"2024-09-21T19:14:24.852003+0000","last_became_active":"2024-09-21T19:13:55.928306+0000","last_became_peered":"2024-09-21T19:13:55.928306+0000","last_unstale":"2024-09-21T19:14:24.852003+0000","last_undegraded":"2024-09-21T19:14:24.852003+0000","last_fullsized":"2024-09-21T19:14:24.852003+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:12:36.624435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:12:36.624435+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1024,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071895+0000","last_change":"2024-09-21T19:14:25.096034+0000","last_active":"2024-09-21T19:14:45.071895+0000","last_peered":"2024-09-21T19:14:45.071895+0000","last_clean":"2024-09-21T19:14:45.071895+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:14:45.071895+0000","last_undegraded":"2024-09-21T19:14:45.071895+0000","last_fullsized":"2024-09-21T19:14:45.071895+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:25.095979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:25.095979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.153733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":994,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853070+0000","last_change":"2024-09-21T19:14:22.870075+0000","last_active":"2024-09-21T19:14:24.853070+0000","last_peered":"2024-09-21T19:14:24.853070+0000","last_clean":"2024-09-21T19:14:24.853070+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:14:24.853070+0000","last_undegraded":"2024-09-21T19:14:24.853070+0000","last_fullsized":"2024-09-21T19:14:24.853070+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:22.870022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:22.870022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:28.288452+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":455,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.860093+0000","last_change":"2024-09-21T19:13:54.395414+0000","last_active":"2024-09-21T19:14:24.860093+0000","last_peered":"2024-09-21T19:14:24.860093+0000","last_clean":"2024-09-21T19:14:24.860093+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:14:24.860093+0000","last_undegraded":"2024-09-21T19:14:24.860093+0000","last_fullsized":"2024-09-21T19:14:24.860093+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:54.395365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.395365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:10.292853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":1091,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867153+0000","last_change":"2024-09-21T19:14:24.864595+0000","last_active":"2024-09-21T19:14:24.867153+0000","last_peered":"2024-09-21T19:14:24.867153+0000","last_clean":"2024-09-21T19:14:24.867153+0000","last_became_active":"2024-09-21T19:14:24.864021+0000","last_became_peered":"2024-09-21T19:14:24.864021+0000","last_unstale":"2024-09-21T19:14:24.867153+0000","last_undegraded":"2024-09-21T19:14:24.867153+0000","last_fullsized":"2024-09-21T19:14:24.867153+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:25.372586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:25.372586+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"490'21","reported_seq":591,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.867053+0000","last_change":"2024-09-21T19:14:24.864960+0000","last_active":"2024-09-21T19:14:24.867053+0000","last_peered":"2024-09-21T19:14:24.867053+0000","last_clean":"2024-09-21T19:14:24.867053+0000","last_became_active":"2024-09-21T19:14:24.864311+0000","last_became_peered":"2024-09-21T19:14:24.864311+0000","last_unstale":"2024-09-21T19:14:24.867053+0000","last_undegraded":"2024-09-21T19:14:24.867053+0000","last_fullsized":"2024-09-21T19:14:24.867053+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'21","last_scrub_stamp":"2024-09-21T19:14:23.037770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:23.037770+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:40.847380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":874,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851391+0000","last_change":"2024-09-21T19:14:23.929705+0000","last_active":"2024-09-21T19:14:24.851391+0000","last_peered":"2024-09-21T19:14:24.851391+0000","last_clean":"2024-09-21T19:14:24.851391+0000","last_became_active":"2024-09-21T19:13:55.927491+0000","last_became_peered":"2024-09-21T19:13:55.927491+0000","last_unstale":"2024-09-21T19:14:24.851391+0000","last_undegraded":"2024-09-21T19:14:24.851391+0000","last_fullsized":"2024-09-21T19:14:24.851391+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:23.929658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:23.929658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:28.867014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1286,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:42.748095+0000","last_change":"2024-09-21T19:14:42.748095+0000","last_active":"2024-09-21T19:14:42.748095+0000","last_peered":"2024-09-21T19:14:42.748095+0000","last_clean":"2024-09-21T19:14:42.748095+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:14:42.748095+0000","last_undegraded":"2024-09-21T19:14:42.748095+0000","last_fullsized":"2024-09-21T19:14:42.748095+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:14:42.748033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:42.748033+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:56.785153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":931,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859153+0000","last_change":"2024-09-21T19:14:13.022645+0000","last_active":"2024-09-21T19:14:44.859153+0000","last_peered":"2024-09-21T19:14:44.859153+0000","last_clean":"2024-09-21T19:14:44.859153+0000","last_became_active":"2024-09-21T19:14:13.022313+0000","last_became_peered":"2024-09-21T19:14:13.022313+0000","last_unstale":"2024-09-21T19:14:44.859153+0000","last_undegraded":"2024-09-21T19:14:44.859153+0000","last_fullsized":"2024-09-21T19:14:44.859153+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:05.516911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:05.516911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":759,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851144+0000","last_change":"2024-09-21T19:13:55.929698+0000","last_active":"2024-09-21T19:14:24.851144+0000","last_peered":"2024-09-21T19:14:24.851144+0000","last_clean":"2024-09-21T19:14:24.851144+0000","last_became_active":"2024-09-21T19:13:55.928943+0000","last_became_peered":"2024-09-21T19:13:55.928943+0000","last_unstale":"2024-09-21T19:14:24.851144+0000","last_undegraded":"2024-09-21T19:14:24.851144+0000","last_fullsized":"2024-09-21T19:14:24.851144+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:53.945511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:53.945511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:09.908937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"490'30","reported_seq":1055,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851582+0000","last_change":"2024-09-21T19:13:52.024298+0000","last_active":"2024-09-21T19:14:24.851582+0000","last_peered":"2024-09-21T19:14:24.851582+0000","last_clean":"2024-09-21T19:14:24.851582+0000","last_became_active":"2024-09-21T19:10:38.923267+0000","last_became_peered":"2024-09-21T19:10:38.923267+0000","last_unstale":"2024-09-21T19:14:24.851582+0000","last_undegraded":"2024-09-21T19:14:24.851582+0000","last_fullsized":"2024-09-21T19:14:24.851582+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:13:52.024239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:52.024239+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:57.960257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1027,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072163+0000","last_change":"2024-09-21T19:14:06.025842+0000","last_active":"2024-09-21T19:14:45.072163+0000","last_peered":"2024-09-21T19:14:45.072163+0000","last_clean":"2024-09-21T19:14:45.072163+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:14:45.072163+0000","last_undegraded":"2024-09-21T19:14:45.072163+0000","last_fullsized":"2024-09-21T19:14:45.072163+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:06.025758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:06.025758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.796919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"529'16","reported_seq":698,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:37.708210+0000","last_change":"2024-09-21T19:14:24.863104+0000","last_active":"2024-09-21T19:14:37.708210+0000","last_peered":"2024-09-21T19:14:37.708210+0000","last_clean":"2024-09-21T19:14:37.708210+0000","last_became_active":"2024-09-21T19:14:24.862762+0000","last_became_peered":"2024-09-21T19:14:24.862762+0000","last_unstale":"2024-09-21T19:14:37.708210+0000","last_undegraded":"2024-09-21T19:14:37.708210+0000","last_fullsized":"2024-09-21T19:14:37.708210+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15","last_scrub_stamp":"2024-09-21T19:14:10.819759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:10.819759+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.526679+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":865,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.852205+0000","last_change":"2024-09-21T19:13:55.931622+0000","last_active":"2024-09-21T19:14:24.852205+0000","last_peered":"2024-09-21T19:14:24.852205+0000","last_clean":"2024-09-21T19:14:24.852205+0000","last_became_active":"2024-09-21T19:13:55.930144+0000","last_became_peered":"2024-09-21T19:13:55.930144+0000","last_unstale":"2024-09-21T19:14:24.852205+0000","last_undegraded":"2024-09-21T19:14:24.852205+0000","last_fullsized":"2024-09-21T19:14:24.852205+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:05.734582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:05.734582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"490'27","reported_seq":610,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858270+0000","last_change":"2024-09-21T19:14:13.027160+0000","last_active":"2024-09-21T19:14:44.858270+0000","last_peered":"2024-09-21T19:14:44.858270+0000","last_clean":"2024-09-21T19:14:44.858270+0000","last_became_active":"2024-09-21T19:14:13.026094+0000","last_became_peered":"2024-09-21T19:14:13.026094+0000","last_unstale":"2024-09-21T19:14:44.858270+0000","last_undegraded":"2024-09-21T19:14:44.858270+0000","last_fullsized":"2024-09-21T19:14:44.858270+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:14:11.010860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:11.010860+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:16.776521+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":938,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858541+0000","last_change":"2024-09-21T19:14:35.025832+0000","last_active":"2024-09-21T19:14:44.858541+0000","last_peered":"2024-09-21T19:14:44.858541+0000","last_clean":"2024-09-21T19:14:44.858541+0000","last_became_active":"2024-09-21T19:14:13.027969+0000","last_became_peered":"2024-09-21T19:14:13.027969+0000","last_unstale":"2024-09-21T19:14:44.858541+0000","last_undegraded":"2024-09-21T19:14:44.858541+0000","last_fullsized":"2024-09-21T19:14:44.858541+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:35.025773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:35.025773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:45.369290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1035,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.980743+0000","last_change":"2024-09-21T19:14:24.980743+0000","last_active":"2024-09-21T19:14:24.980743+0000","last_peered":"2024-09-21T19:14:24.980743+0000","last_clean":"2024-09-21T19:14:24.980743+0000","last_became_active":"2024-09-21T19:13:41.872018+0000","last_became_peered":"2024-09-21T19:13:41.872018+0000","last_unstale":"2024-09-21T19:14:24.980743+0000","last_undegraded":"2024-09-21T19:14:24.980743+0000","last_fullsized":"2024-09-21T19:14:24.980743+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:24.980696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:24.980696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:26.350269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1030,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.853440+0000","last_change":"2024-09-21T19:14:17.987752+0000","last_active":"2024-09-21T19:14:24.853440+0000","last_peered":"2024-09-21T19:14:24.853440+0000","last_clean":"2024-09-21T19:14:24.853440+0000","last_became_active":"2024-09-21T19:13:41.866921+0000","last_became_peered":"2024-09-21T19:13:41.866921+0000","last_unstale":"2024-09-21T19:14:24.853440+0000","last_undegraded":"2024-09-21T19:14:24.853440+0000","last_fullsized":"2024-09-21T19:14:24.853440+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:17.987700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:17.987700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:36.820401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"489'21","reported_seq":1101,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.857498+0000","last_change":"2024-09-21T19:13:58.757296+0000","last_active":"2024-09-21T19:14:24.857498+0000","last_peered":"2024-09-21T19:14:24.857498+0000","last_clean":"2024-09-21T19:14:24.857498+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:14:24.857498+0000","last_undegraded":"2024-09-21T19:14:24.857498+0000","last_fullsized":"2024-09-21T19:14:24.857498+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:13:58.757236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:58.757236+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:14.998365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1020,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072624+0000","last_change":"2024-09-21T19:14:12.080118+0000","last_active":"2024-09-21T19:14:45.072624+0000","last_peered":"2024-09-21T19:14:45.072624+0000","last_clean":"2024-09-21T19:14:45.072624+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:14:45.072624+0000","last_undegraded":"2024-09-21T19:14:45.072624+0000","last_fullsized":"2024-09-21T19:14:45.072624+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:12.080074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:12.080074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:31.509730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1009,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.850676+0000","last_change":"2024-09-21T19:13:25.223371+0000","last_active":"2024-09-21T19:14:24.850676+0000","last_peered":"2024-09-21T19:14:24.850676+0000","last_clean":"2024-09-21T19:14:24.850676+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:14:24.850676+0000","last_undegraded":"2024-09-21T19:14:24.850676+0000","last_fullsized":"2024-09-21T19:14:24.850676+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:25.223303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:25.223303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:35.595719+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":893,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:34.028328+0000","last_change":"2024-09-21T19:14:34.028328+0000","last_active":"2024-09-21T19:14:34.028328+0000","last_peered":"2024-09-21T19:14:34.028328+0000","last_clean":"2024-09-21T19:14:34.028328+0000","last_became_active":"2024-09-21T19:13:55.926284+0000","last_became_peered":"2024-09-21T19:13:55.926284+0000","last_unstale":"2024-09-21T19:14:34.028328+0000","last_undegraded":"2024-09-21T19:14:34.028328+0000","last_fullsized":"2024-09-21T19:14:34.028328+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:34.028260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:34.028260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:48.787994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":178,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.860484+0000","last_change":"2024-09-21T19:13:50.409307+0000","last_active":"2024-09-21T19:14:24.860484+0000","last_peered":"2024-09-21T19:14:24.860484+0000","last_clean":"2024-09-21T19:14:24.860484+0000","last_became_active":"2024-09-21T19:11:49.139887+0000","last_became_peered":"2024-09-21T19:11:49.139887+0000","last_unstale":"2024-09-21T19:14:24.860484+0000","last_undegraded":"2024-09-21T19:14:24.860484+0000","last_fullsized":"2024-09-21T19:14:24.860484+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:50.409255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:50.409255+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:14.987768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"490'27","reported_seq":1053,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:34.805044+0000","last_change":"2024-09-21T19:14:34.805044+0000","last_active":"2024-09-21T19:14:34.805044+0000","last_peered":"2024-09-21T19:14:34.805044+0000","last_clean":"2024-09-21T19:14:34.805044+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:14:34.805044+0000","last_undegraded":"2024-09-21T19:14:34.805044+0000","last_fullsized":"2024-09-21T19:14:34.805044+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:14:34.804991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:34.804991+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:45.990196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":70,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856305+0000","last_change":"2024-09-21T19:13:49.927583+0000","last_active":"2024-09-21T19:14:24.856305+0000","last_peered":"2024-09-21T19:14:24.856305+0000","last_clean":"2024-09-21T19:14:24.856305+0000","last_became_active":"2024-09-21T19:13:41.868259+0000","last_became_peered":"2024-09-21T19:13:41.868259+0000","last_unstale":"2024-09-21T19:14:24.856305+0000","last_undegraded":"2024-09-21T19:14:24.856305+0000","last_fullsized":"2024-09-21T19:14:24.856305+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:49.927510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:49.927510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:57.501058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1040,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071392+0000","last_change":"2024-09-21T19:13:57.970268+0000","last_active":"2024-09-21T19:14:45.071392+0000","last_peered":"2024-09-21T19:14:45.071392+0000","last_clean":"2024-09-21T19:14:45.071392+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:14:45.071392+0000","last_undegraded":"2024-09-21T19:14:45.071392+0000","last_fullsized":"2024-09-21T19:14:45.071392+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:57.970213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:57.970213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:12.056921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1016,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.850865+0000","last_change":"2024-09-21T19:14:06.844136+0000","last_active":"2024-09-21T19:14:24.850865+0000","last_peered":"2024-09-21T19:14:24.850865+0000","last_clean":"2024-09-21T19:14:24.850865+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:14:24.850865+0000","last_undegraded":"2024-09-21T19:14:24.850865+0000","last_fullsized":"2024-09-21T19:14:24.850865+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:06.844093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:06.844093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:14.591231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"489'12","reported_seq":1046,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.852955+0000","last_change":"2024-09-21T19:14:21.909131+0000","last_active":"2024-09-21T19:14:24.852955+0000","last_peered":"2024-09-21T19:14:24.852955+0000","last_clean":"2024-09-21T19:14:24.852955+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:14:24.852955+0000","last_undegraded":"2024-09-21T19:14:24.852955+0000","last_fullsized":"2024-09-21T19:14:24.852955+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'12","last_scrub_stamp":"2024-09-21T19:14:21.909069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:21.909069+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:48.183106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":867,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.851536+0000","last_change":"2024-09-21T19:13:55.925158+0000","last_active":"2024-09-21T19:14:24.851536+0000","last_peered":"2024-09-21T19:14:24.851536+0000","last_clean":"2024-09-21T19:14:24.851536+0000","last_became_active":"2024-09-21T19:13:55.924905+0000","last_became_peered":"2024-09-21T19:13:55.924905+0000","last_unstale":"2024-09-21T19:14:24.851536+0000","last_undegraded":"2024-09-21T19:14:24.851536+0000","last_fullsized":"2024-09-21T19:14:24.851536+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:52.925166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:52.925166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.600131+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":923,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.295365+0000","last_change":"2024-09-21T19:14:41.030909+0000","last_active":"2024-09-21T19:14:44.295365+0000","last_peered":"2024-09-21T19:14:44.295365+0000","last_clean":"2024-09-21T19:14:44.295365+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:14:44.295365+0000","last_undegraded":"2024-09-21T19:14:44.295365+0000","last_fullsized":"2024-09-21T19:14:44.295365+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:41.030849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:41.030849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:51.166190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1015,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071750+0000","last_change":"2024-09-21T19:14:09.047666+0000","last_active":"2024-09-21T19:14:45.071750+0000","last_peered":"2024-09-21T19:14:45.071750+0000","last_clean":"2024-09-21T19:14:45.071750+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:14:45.071750+0000","last_undegraded":"2024-09-21T19:14:45.071750+0000","last_fullsized":"2024-09-21T19:14:45.071750+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:09.047596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:09.047596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:16.779417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1030,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.856381+0000","last_change":"2024-09-21T19:14:06.807070+0000","last_active":"2024-09-21T19:14:24.856381+0000","last_peered":"2024-09-21T19:14:24.856381+0000","last_clean":"2024-09-21T19:14:24.856381+0000","last_became_active":"2024-09-21T19:13:41.871213+0000","last_became_peered":"2024-09-21T19:13:41.871213+0000","last_unstale":"2024-09-21T19:14:24.856381+0000","last_undegraded":"2024-09-21T19:14:24.856381+0000","last_fullsized":"2024-09-21T19:14:24.856381+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:06.807020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:06.807020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.583546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1024,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:31.782570+0000","last_change":"2024-09-21T19:14:31.782570+0000","last_active":"2024-09-21T19:14:31.782570+0000","last_peered":"2024-09-21T19:14:31.782570+0000","last_clean":"2024-09-21T19:14:31.782570+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:14:31.782570+0000","last_undegraded":"2024-09-21T19:14:31.782570+0000","last_fullsized":"2024-09-21T19:14:31.782570+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:31.782517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:31.782517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:36.257793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"490'27","reported_seq":1054,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.297287+0000","last_change":"2024-09-21T19:13:54.058371+0000","last_active":"2024-09-21T19:14:44.297287+0000","last_peered":"2024-09-21T19:14:44.297287+0000","last_clean":"2024-09-21T19:14:44.297287+0000","last_became_active":"2024-09-21T19:13:41.874446+0000","last_became_peered":"2024-09-21T19:13:41.874446+0000","last_unstale":"2024-09-21T19:14:44.297287+0000","last_undegraded":"2024-09-21T19:14:44.297287+0000","last_fullsized":"2024-09-21T19:14:44.297287+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:13:54.058294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.058294+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:58.870576+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":889,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-21T19:14:24.852194+0000","last_change":"2024-09-21T19:13:57.805964+0000","last_active":"2024-09-21T19:14:24.852194+0000","last_peered":"2024-09-21T19:14:24.852194+0000","last_clean":"2024-09-21T19:14:24.852194+0000","last_became_active":"2024-09-21T19:13:55.928191+0000","last_became_peered":"2024-09-21T19:13:55.928191+0000","last_unstale":"2024-09-21T19:14:24.852194+0000","last_undegraded":"2024-09-21T19:14:24.852194+0000","last_fullsized":"2024-09-21T19:14:24.852194+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:57.805917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:57.805917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:05.549998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1527,"num_read_kb":1284,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":6766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1287,"num_read_kb":1282,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266953,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830492,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92914660,"statfs":{"total":95995035648,"available":95144611840,"internally_reserved":0,"allocated":1052672,"data_stored":552380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":849344970},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464403,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515052,"kb_used_data":1308,"kb_used_omap":4,"kb_used_meta":513723,"kb_avail":93230100,"statfs":{"total":95995035648,"available":95467622400,"internally_reserved":0,"allocated":1339392,"data_stored":866031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":526053028},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619104,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383596,"kb_used_data":1244,"kb_used_omap":4,"kb_used_meta":382331,"kb_avail":93361556,"statfs":{"total":95995035648,"available":95602233344,"internally_reserved":0,"allocated":1273856,"data_stored":800949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":391507619},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019159,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686112,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93059040,"statfs":{"total":95995035648,"available":95292456960,"internally_reserved":0,"allocated":1060864,"data_stored":552880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182683,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894932,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92850220,"statfs":{"total":95995035648,"available":95078625280,"internally_reserved":0,"allocated":1642496,"data_stored":1130167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":914749898},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411617,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":754356,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92990796,"statfs":{"total":95995035648,"available":95222575104,"internally_reserved":0,"allocated":1282048,"data_stored":801504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906844,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838308,"statfs":{"total":95995035648,"available":95066427392,"internally_reserved":0,"allocated":1060864,"data_stored":539793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961280,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92783872,"statfs":{"total":95995035648,"available":95010684928,"internally_reserved":0,"allocated":1359872,"data_stored":867369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:14:49.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T19:14:50.068 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[29384]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[29384]: 1.10 scrub starts 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[29384]: 1.10 scrub ok 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[29384]: from='client.37028 v1:172.21.15.81:0/3500625822' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[29384]: 4.0 scrub starts 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[29384]: 4.0 scrub ok 2024-09-21T19:14:50.467 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[39192]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:50.468 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[39192]: 1.10 scrub starts 2024-09-21T19:14:50.468 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[39192]: 1.10 scrub ok 2024-09-21T19:14:50.468 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[39192]: from='client.37028 v1:172.21.15.81:0/3500625822' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:14:50.468 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[39192]: 4.0 scrub starts 2024-09-21T19:14:50.468 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:50 smithi081 ceph-mon[39192]: 4.0 scrub ok 2024-09-21T19:14:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:50 smithi104 ceph-mon[32438]: from='client.24487 v1:172.21.15.104:0/2347605155' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T19:14:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:50 smithi104 ceph-mon[32438]: 1.10 scrub starts 2024-09-21T19:14:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:50 smithi104 ceph-mon[32438]: 1.10 scrub ok 2024-09-21T19:14:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:50 smithi104 ceph-mon[32438]: from='client.37028 v1:172.21.15.81:0/3500625822' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:14:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:50 smithi104 ceph-mon[32438]: 4.0 scrub starts 2024-09-21T19:14:50.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:50 smithi104 ceph-mon[32438]: 4.0 scrub ok 2024-09-21T19:14:50.739 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-21T19:14:50.741 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-21T19:14:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:51 smithi104 ceph-mon[32438]: pgmap v1098: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:51 smithi104 ceph-mon[32438]: 6.15 scrub starts 2024-09-21T19:14:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:51 smithi104 ceph-mon[32438]: 6.15 scrub ok 2024-09-21T19:14:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:51 smithi104 ceph-mon[32438]: 6.c scrub starts 2024-09-21T19:14:51.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:51 smithi104 ceph-mon[32438]: 6.c scrub ok 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[39192]: pgmap v1098: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[39192]: 6.15 scrub starts 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[39192]: 6.15 scrub ok 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[39192]: 6.c scrub starts 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[39192]: 6.c scrub ok 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[29384]: pgmap v1098: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[29384]: 6.15 scrub starts 2024-09-21T19:14:51.531 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[29384]: 6.15 scrub ok 2024-09-21T19:14:51.532 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[29384]: 6.c scrub starts 2024-09-21T19:14:51.532 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:51 smithi081 ceph-mon[29384]: 6.c scrub ok 2024-09-21T19:14:51.559 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":1098,"stamp":"2024-09-21T19:14:49.912223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2838,"num_read_kb":2590,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":537685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2748,"ondisk_log_size":2748,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5932832,"kb_used_data":9980,"kb_used_omap":18,"kb_used_meta":5922669,"kb_avail":744028384,"statfs":{"total":767960285184,"available":761885065216,"internally_reserved":0,"allocated":10219520,"data_stored":6191353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18448,"internal_metadata":6064814064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002112"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":904,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154739+0000","last_change":"2024-09-21T19:13:55.923183+0000","last_active":"2024-09-21T19:14:45.154739+0000","last_peered":"2024-09-21T19:14:45.154739+0000","last_clean":"2024-09-21T19:14:45.154739+0000","last_became_active":"2024-09-21T19:13:55.922887+0000","last_became_peered":"2024-09-21T19:13:55.922887+0000","last_unstale":"2024-09-21T19:14:45.154739+0000","last_undegraded":"2024-09-21T19:14:45.154739+0000","last_fullsized":"2024-09-21T19:14:45.154739+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:03.679452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:03.679452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"489'21","reported_seq":604,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857855+0000","last_change":"2024-09-21T19:14:16.919099+0000","last_active":"2024-09-21T19:14:44.857855+0000","last_peered":"2024-09-21T19:14:44.857855+0000","last_clean":"2024-09-21T19:14:44.857855+0000","last_became_active":"2024-09-21T19:14:13.025672+0000","last_became_peered":"2024-09-21T19:14:13.025672+0000","last_unstale":"2024-09-21T19:14:44.857855+0000","last_undegraded":"2024-09-21T19:14:44.857855+0000","last_fullsized":"2024-09-21T19:14:44.857855+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:14:16.919030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:16.919030+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:43.683336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1038,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975429+0000","last_change":"2024-09-21T19:14:07.832217+0000","last_active":"2024-09-21T19:14:44.975429+0000","last_peered":"2024-09-21T19:14:44.975429+0000","last_clean":"2024-09-21T19:14:44.975429+0000","last_became_active":"2024-09-21T19:13:41.863135+0000","last_became_peered":"2024-09-21T19:13:41.863135+0000","last_unstale":"2024-09-21T19:14:44.975429+0000","last_undegraded":"2024-09-21T19:14:44.975429+0000","last_fullsized":"2024-09-21T19:14:44.975429+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:07.832163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:07.832163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.682688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":917,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858002+0000","last_change":"2024-09-21T19:14:13.020739+0000","last_active":"2024-09-21T19:14:44.858002+0000","last_peered":"2024-09-21T19:14:44.858002+0000","last_clean":"2024-09-21T19:14:44.858002+0000","last_became_active":"2024-09-21T19:14:13.020316+0000","last_became_peered":"2024-09-21T19:14:13.020316+0000","last_unstale":"2024-09-21T19:14:44.858002+0000","last_undegraded":"2024-09-21T19:14:44.858002+0000","last_fullsized":"2024-09-21T19:14:44.858002+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:01.795304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:01.795304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:11.420757+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1013,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856591+0000","last_change":"2024-09-21T19:14:11.909638+0000","last_active":"2024-09-21T19:14:44.856591+0000","last_peered":"2024-09-21T19:14:44.856591+0000","last_clean":"2024-09-21T19:14:44.856591+0000","last_became_active":"2024-09-21T19:02:40.877208+0000","last_became_peered":"2024-09-21T19:02:40.877208+0000","last_unstale":"2024-09-21T19:14:44.856591+0000","last_undegraded":"2024-09-21T19:14:44.856591+0000","last_fullsized":"2024-09-21T19:14:44.856591+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:11.909595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:11.909595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.170553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1041,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857547+0000","last_change":"2024-09-21T19:14:25.844226+0000","last_active":"2024-09-21T19:14:44.857547+0000","last_peered":"2024-09-21T19:14:44.857547+0000","last_clean":"2024-09-21T19:14:44.857547+0000","last_became_active":"2024-09-21T19:05:42.903963+0000","last_became_peered":"2024-09-21T19:05:42.903963+0000","last_unstale":"2024-09-21T19:14:44.857547+0000","last_undegraded":"2024-09-21T19:14:44.857547+0000","last_fullsized":"2024-09-21T19:14:44.857547+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:25.844178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:25.844178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:31.060089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"490'39","reported_seq":1147,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855827+0000","last_change":"2024-09-21T19:14:33.797542+0000","last_active":"2024-09-21T19:14:44.855827+0000","last_peered":"2024-09-21T19:14:44.855827+0000","last_clean":"2024-09-21T19:14:44.855827+0000","last_became_active":"2024-09-21T19:05:42.911501+0000","last_became_peered":"2024-09-21T19:05:42.911501+0000","last_unstale":"2024-09-21T19:14:44.855827+0000","last_undegraded":"2024-09-21T19:14:44.855827+0000","last_fullsized":"2024-09-21T19:14:44.855827+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:14:33.797488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.797488+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:02.755324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":953,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857852+0000","last_change":"2024-09-21T19:14:30.904760+0000","last_active":"2024-09-21T19:14:44.857852+0000","last_peered":"2024-09-21T19:14:44.857852+0000","last_clean":"2024-09-21T19:14:44.857852+0000","last_became_active":"2024-09-21T19:14:13.025353+0000","last_became_peered":"2024-09-21T19:14:13.025353+0000","last_unstale":"2024-09-21T19:14:44.857852+0000","last_undegraded":"2024-09-21T19:14:44.857852+0000","last_fullsized":"2024-09-21T19:14:44.857852+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:30.904704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.904704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.524084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":910,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662020+0000","last_change":"2024-09-21T19:14:24.867523+0000","last_active":"2024-09-21T19:14:45.662020+0000","last_peered":"2024-09-21T19:14:45.662020+0000","last_clean":"2024-09-21T19:14:45.662020+0000","last_became_active":"2024-09-21T19:14:24.867187+0000","last_became_peered":"2024-09-21T19:14:24.867187+0000","last_unstale":"2024-09-21T19:14:45.662020+0000","last_undegraded":"2024-09-21T19:14:45.662020+0000","last_fullsized":"2024-09-21T19:14:45.662020+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:45.962479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:45.962479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:48.252339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"489'21","reported_seq":611,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858257+0000","last_change":"2024-09-21T19:14:34.009659+0000","last_active":"2024-09-21T19:14:44.858257+0000","last_peered":"2024-09-21T19:14:44.858257+0000","last_clean":"2024-09-21T19:14:44.858257+0000","last_became_active":"2024-09-21T19:14:13.024633+0000","last_became_peered":"2024-09-21T19:14:13.024633+0000","last_unstale":"2024-09-21T19:14:44.858257+0000","last_undegraded":"2024-09-21T19:14:44.858257+0000","last_fullsized":"2024-09-21T19:14:44.858257+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:14:34.009595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:34.009595+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:41.871788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":74,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661843+0000","last_change":"2024-09-21T19:14:01.034756+0000","last_active":"2024-09-21T19:14:45.661843+0000","last_peered":"2024-09-21T19:14:45.661843+0000","last_clean":"2024-09-21T19:14:45.661843+0000","last_became_active":"2024-09-21T19:13:41.863805+0000","last_became_peered":"2024-09-21T19:13:41.863805+0000","last_unstale":"2024-09-21T19:14:45.661843+0000","last_undegraded":"2024-09-21T19:14:45.661843+0000","last_fullsized":"2024-09-21T19:14:45.661843+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:01.034700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:01.034700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:08.295623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":861,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.153742+0000","last_change":"2024-09-21T19:14:10.868263+0000","last_active":"2024-09-21T19:14:45.153742+0000","last_peered":"2024-09-21T19:14:45.153742+0000","last_clean":"2024-09-21T19:14:45.153742+0000","last_became_active":"2024-09-21T19:13:55.927611+0000","last_became_peered":"2024-09-21T19:13:55.927611+0000","last_unstale":"2024-09-21T19:14:45.153742+0000","last_undegraded":"2024-09-21T19:14:45.153742+0000","last_fullsized":"2024-09-21T19:14:45.153742+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:10.868212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:10.868212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:26.089747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":31,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662052+0000","last_change":"2024-09-21T19:14:24.868817+0000","last_active":"2024-09-21T19:14:45.662052+0000","last_peered":"2024-09-21T19:14:45.662052+0000","last_clean":"2024-09-21T19:14:45.662052+0000","last_became_active":"2024-09-21T19:14:24.868485+0000","last_became_peered":"2024-09-21T19:14:24.868485+0000","last_unstale":"2024-09-21T19:14:45.662052+0000","last_undegraded":"2024-09-21T19:14:45.662052+0000","last_fullsized":"2024-09-21T19:14:45.662052+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:42.453534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:42.453534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:46.945085+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"489'9","reported_seq":551,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858303+0000","last_change":"2024-09-21T19:14:22.902175+0000","last_active":"2024-09-21T19:14:44.858303+0000","last_peered":"2024-09-21T19:14:44.858303+0000","last_clean":"2024-09-21T19:14:44.858303+0000","last_became_active":"2024-09-21T19:14:13.021371+0000","last_became_peered":"2024-09-21T19:14:13.021371+0000","last_unstale":"2024-09-21T19:14:44.858303+0000","last_undegraded":"2024-09-21T19:14:44.858303+0000","last_fullsized":"2024-09-21T19:14:44.858303+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'9","last_scrub_stamp":"2024-09-21T19:14:22.902106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:22.902106+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:31.169757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1004,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071874+0000","last_change":"2024-09-21T19:14:15.147298+0000","last_active":"2024-09-21T19:14:45.071874+0000","last_peered":"2024-09-21T19:14:45.071874+0000","last_clean":"2024-09-21T19:14:45.071874+0000","last_became_active":"2024-09-21T18:53:11.412693+0000","last_became_peered":"2024-09-21T18:53:11.412693+0000","last_unstale":"2024-09-21T19:14:45.071874+0000","last_undegraded":"2024-09-21T19:14:45.071874+0000","last_fullsized":"2024-09-21T19:14:45.071874+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:15.147233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.147233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.518936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":539,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661440+0000","last_change":"2024-09-21T19:14:24.864710+0000","last_active":"2024-09-21T19:14:45.661440+0000","last_peered":"2024-09-21T19:14:45.661440+0000","last_clean":"2024-09-21T19:14:45.661440+0000","last_became_active":"2024-09-21T19:14:24.864388+0000","last_became_peered":"2024-09-21T19:14:24.864388+0000","last_unstale":"2024-09-21T19:14:45.661440+0000","last_undegraded":"2024-09-21T19:14:45.661440+0000","last_fullsized":"2024-09-21T19:14:45.661440+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:20.967629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:20.967629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.156518+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":954,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859041+0000","last_change":"2024-09-21T19:14:37.053297+0000","last_active":"2024-09-21T19:14:44.859041+0000","last_peered":"2024-09-21T19:14:44.859041+0000","last_clean":"2024-09-21T19:14:44.859041+0000","last_became_active":"2024-09-21T19:14:13.028761+0000","last_became_peered":"2024-09-21T19:14:13.028761+0000","last_unstale":"2024-09-21T19:14:44.859041+0000","last_undegraded":"2024-09-21T19:14:44.859041+0000","last_fullsized":"2024-09-21T19:14:44.859041+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:37.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:37.053229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:51.615622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":869,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154632+0000","last_change":"2024-09-21T19:13:55.929500+0000","last_active":"2024-09-21T19:14:45.154632+0000","last_peered":"2024-09-21T19:14:45.154632+0000","last_clean":"2024-09-21T19:14:45.154632+0000","last_became_active":"2024-09-21T19:13:55.927362+0000","last_became_peered":"2024-09-21T19:13:55.927362+0000","last_unstale":"2024-09-21T19:14:45.154632+0000","last_undegraded":"2024-09-21T19:14:45.154632+0000","last_fullsized":"2024-09-21T19:14:45.154632+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:42.596058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:11:42.596058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"490'39","reported_seq":509,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855751+0000","last_change":"2024-09-21T19:14:32.365216+0000","last_active":"2024-09-21T19:14:44.855751+0000","last_peered":"2024-09-21T19:14:44.855751+0000","last_clean":"2024-09-21T19:14:44.855751+0000","last_became_active":"2024-09-21T19:05:42.909451+0000","last_became_peered":"2024-09-21T19:05:42.909451+0000","last_unstale":"2024-09-21T19:14:44.855751+0000","last_undegraded":"2024-09-21T19:14:44.855751+0000","last_fullsized":"2024-09-21T19:14:44.855751+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:14:32.365158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:32.365158+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.979432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1035,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975844+0000","last_change":"2024-09-21T19:14:02.786478+0000","last_active":"2024-09-21T19:14:44.975844+0000","last_peered":"2024-09-21T19:14:44.975844+0000","last_clean":"2024-09-21T19:14:44.975844+0000","last_became_active":"2024-09-21T18:55:51.535642+0000","last_became_peered":"2024-09-21T18:55:51.535642+0000","last_unstale":"2024-09-21T19:14:44.975844+0000","last_undegraded":"2024-09-21T19:14:44.975844+0000","last_fullsized":"2024-09-21T19:14:44.975844+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:02.786428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:02.786428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:21.973273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1024,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856341+0000","last_change":"2024-09-21T19:14:05.846447+0000","last_active":"2024-09-21T19:14:44.856341+0000","last_peered":"2024-09-21T19:14:44.856341+0000","last_clean":"2024-09-21T19:14:44.856341+0000","last_became_active":"2024-09-21T19:13:41.875083+0000","last_became_peered":"2024-09-21T19:13:41.875083+0000","last_unstale":"2024-09-21T19:14:44.856341+0000","last_undegraded":"2024-09-21T19:14:44.856341+0000","last_fullsized":"2024-09-21T19:14:44.856341+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:05.846390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:05.846390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.180814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":940,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858564+0000","last_change":"2024-09-21T19:14:13.022440+0000","last_active":"2024-09-21T19:14:44.858564+0000","last_peered":"2024-09-21T19:14:44.858564+0000","last_clean":"2024-09-21T19:14:44.858564+0000","last_became_active":"2024-09-21T19:14:13.021848+0000","last_became_peered":"2024-09-21T19:14:13.021848+0000","last_unstale":"2024-09-21T19:14:44.858564+0000","last_undegraded":"2024-09-21T19:14:44.858564+0000","last_fullsized":"2024-09-21T19:14:44.858564+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:04.794479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.794479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:05.068465+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"490'24","reported_seq":1108,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857329+0000","last_change":"2024-09-21T19:14:28.809841+0000","last_active":"2024-09-21T19:14:44.857329+0000","last_peered":"2024-09-21T19:14:44.857329+0000","last_clean":"2024-09-21T19:14:44.857329+0000","last_became_active":"2024-09-21T19:10:52.013122+0000","last_became_peered":"2024-09-21T19:10:52.013122+0000","last_unstale":"2024-09-21T19:14:44.857329+0000","last_undegraded":"2024-09-21T19:14:44.857329+0000","last_fullsized":"2024-09-21T19:14:44.857329+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'24","last_scrub_stamp":"2024-09-21T19:14:28.809749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:28.809749+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.853614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":940,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858184+0000","last_change":"2024-09-21T19:14:29.889914+0000","last_active":"2024-09-21T19:14:44.858184+0000","last_peered":"2024-09-21T19:14:44.858184+0000","last_clean":"2024-09-21T19:14:44.858184+0000","last_became_active":"2024-09-21T19:14:13.022424+0000","last_became_peered":"2024-09-21T19:14:13.022424+0000","last_unstale":"2024-09-21T19:14:44.858184+0000","last_undegraded":"2024-09-21T19:14:44.858184+0000","last_fullsized":"2024-09-21T19:14:44.858184+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:29.889856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:29.889856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:37.493731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":1064,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072251+0000","last_change":"2024-09-21T19:13:46.233497+0000","last_active":"2024-09-21T19:14:45.072251+0000","last_peered":"2024-09-21T19:14:45.072251+0000","last_clean":"2024-09-21T19:14:45.072251+0000","last_became_active":"2024-09-21T19:13:41.876488+0000","last_became_peered":"2024-09-21T19:13:41.876488+0000","last_unstale":"2024-09-21T19:14:45.072251+0000","last_undegraded":"2024-09-21T19:14:45.072251+0000","last_fullsized":"2024-09-21T19:14:45.072251+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:13:46.233444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:46.233444+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:06.991782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"58'1","reported_seq":1041,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.976933+0000","last_change":"2024-09-21T19:14:44.976933+0000","last_active":"2024-09-21T19:14:44.976933+0000","last_peered":"2024-09-21T19:14:44.976933+0000","last_clean":"2024-09-21T19:14:44.976933+0000","last_became_active":"2024-09-21T18:53:07.396697+0000","last_became_peered":"2024-09-21T18:53:07.396697+0000","last_unstale":"2024-09-21T19:14:44.976933+0000","last_undegraded":"2024-09-21T19:14:44.976933+0000","last_fullsized":"2024-09-21T19:14:44.976933+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:44.976883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:44.976883+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:58.712827+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"490'30","reported_seq":921,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154580+0000","last_change":"2024-09-21T19:13:55.930365+0000","last_active":"2024-09-21T19:14:45.154580+0000","last_peered":"2024-09-21T19:14:45.154580+0000","last_clean":"2024-09-21T19:14:45.154580+0000","last_became_active":"2024-09-21T19:13:55.927195+0000","last_became_peered":"2024-09-21T19:13:55.927195+0000","last_unstale":"2024-09-21T19:14:45.154580+0000","last_undegraded":"2024-09-21T19:14:45.154580+0000","last_fullsized":"2024-09-21T19:14:45.154580+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:13:18.789971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:18.789971+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"317'11","reported_seq":1173,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161734+0000","last_change":"2024-09-21T19:14:15.941975+0000","last_active":"2024-09-21T19:14:45.161734+0000","last_peered":"2024-09-21T19:14:45.161734+0000","last_clean":"2024-09-21T19:14:45.161734+0000","last_became_active":"2024-09-21T19:05:37.846030+0000","last_became_peered":"2024-09-21T19:05:37.846030+0000","last_unstale":"2024-09-21T19:14:45.161734+0000","last_undegraded":"2024-09-21T19:14:45.161734+0000","last_fullsized":"2024-09-21T19:14:45.161734+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:14:15.941912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.941912+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.900683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":448,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661633+0000","last_change":"2024-09-21T19:14:43.856899+0000","last_active":"2024-09-21T19:14:45.661633+0000","last_peered":"2024-09-21T19:14:45.661633+0000","last_clean":"2024-09-21T19:14:45.661633+0000","last_became_active":"2024-09-21T19:14:24.864948+0000","last_became_peered":"2024-09-21T19:14:24.864948+0000","last_unstale":"2024-09-21T19:14:45.661633+0000","last_undegraded":"2024-09-21T19:14:45.661633+0000","last_fullsized":"2024-09-21T19:14:45.661633+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:43.856828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:43.856828+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:46.511000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"490'39","reported_seq":1087,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.976014+0000","last_change":"2024-09-21T19:13:44.857267+0000","last_active":"2024-09-21T19:14:44.976014+0000","last_peered":"2024-09-21T19:14:44.976014+0000","last_clean":"2024-09-21T19:14:44.976014+0000","last_became_active":"2024-09-21T18:53:09.410973+0000","last_became_peered":"2024-09-21T18:53:09.410973+0000","last_unstale":"2024-09-21T19:14:44.976014+0000","last_undegraded":"2024-09-21T19:14:44.976014+0000","last_fullsized":"2024-09-21T19:14:44.976014+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:13:44.857192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:44.857192+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:08.272995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1007,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855374+0000","last_change":"2024-09-21T19:14:41.711114+0000","last_active":"2024-09-21T19:14:44.855374+0000","last_peered":"2024-09-21T19:14:44.855374+0000","last_clean":"2024-09-21T19:14:44.855374+0000","last_became_active":"2024-09-21T18:53:07.398450+0000","last_became_peered":"2024-09-21T18:53:07.398450+0000","last_unstale":"2024-09-21T19:14:44.855374+0000","last_undegraded":"2024-09-21T19:14:44.855374+0000","last_fullsized":"2024-09-21T19:14:44.855374+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:41.711069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:41.711069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:07.293338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"521'16","reported_seq":1215,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859127+0000","last_change":"2024-09-21T19:14:13.024964+0000","last_active":"2024-09-21T19:14:44.859127+0000","last_peered":"2024-09-21T19:14:44.859127+0000","last_clean":"2024-09-21T19:14:44.859127+0000","last_became_active":"2024-09-21T19:14:13.024453+0000","last_became_peered":"2024-09-21T19:14:13.024453+0000","last_unstale":"2024-09-21T19:14:44.859127+0000","last_undegraded":"2024-09-21T19:14:44.859127+0000","last_fullsized":"2024-09-21T19:14:44.859127+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15","last_scrub_stamp":"2024-09-21T19:13:42.896452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:42.896452+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:04.665743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":671,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662150+0000","last_change":"2024-09-21T19:14:25.924613+0000","last_active":"2024-09-21T19:14:45.662150+0000","last_peered":"2024-09-21T19:14:45.662150+0000","last_clean":"2024-09-21T19:14:45.662150+0000","last_became_active":"2024-09-21T19:14:24.861084+0000","last_became_peered":"2024-09-21T19:14:24.861084+0000","last_unstale":"2024-09-21T19:14:45.662150+0000","last_undegraded":"2024-09-21T19:14:45.662150+0000","last_fullsized":"2024-09-21T19:14:45.662150+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:25.924556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:25.924556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:35.123028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":726,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858969+0000","last_change":"2024-09-21T19:14:13.028074+0000","last_active":"2024-09-21T19:14:44.858969+0000","last_peered":"2024-09-21T19:14:44.858969+0000","last_clean":"2024-09-21T19:14:44.858969+0000","last_became_active":"2024-09-21T19:14:13.027734+0000","last_became_peered":"2024-09-21T19:14:13.027734+0000","last_unstale":"2024-09-21T19:14:44.858969+0000","last_undegraded":"2024-09-21T19:14:44.858969+0000","last_fullsized":"2024-09-21T19:14:44.858969+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:53.424066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:53.424066+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:12.087769+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"490'42","reported_seq":1114,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855074+0000","last_change":"2024-09-21T19:11:44.959245+0000","last_active":"2024-09-21T19:14:44.855074+0000","last_peered":"2024-09-21T19:14:44.855074+0000","last_clean":"2024-09-21T19:14:44.855074+0000","last_became_active":"2024-09-21T18:55:51.532449+0000","last_became_peered":"2024-09-21T18:55:51.532449+0000","last_unstale":"2024-09-21T19:14:44.855074+0000","last_undegraded":"2024-09-21T19:14:44.855074+0000","last_fullsized":"2024-09-21T19:14:44.855074+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T19:11:44.959061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:44.959061+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1029,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856797+0000","last_change":"2024-09-21T19:14:19.904069+0000","last_active":"2024-09-21T19:14:44.856797+0000","last_peered":"2024-09-21T19:14:44.856797+0000","last_clean":"2024-09-21T19:14:44.856797+0000","last_became_active":"2024-09-21T18:55:51.532014+0000","last_became_peered":"2024-09-21T18:55:51.532014+0000","last_unstale":"2024-09-21T19:14:44.856797+0000","last_undegraded":"2024-09-21T19:14:44.856797+0000","last_fullsized":"2024-09-21T19:14:44.856797+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:19.904020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:19.904020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:34.444141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":928,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857925+0000","last_change":"2024-09-21T19:14:18.906020+0000","last_active":"2024-09-21T19:14:44.857925+0000","last_peered":"2024-09-21T19:14:44.857925+0000","last_clean":"2024-09-21T19:14:44.857925+0000","last_became_active":"2024-09-21T19:14:13.023475+0000","last_became_peered":"2024-09-21T19:14:13.023475+0000","last_unstale":"2024-09-21T19:14:44.857925+0000","last_undegraded":"2024-09-21T19:14:44.857925+0000","last_fullsized":"2024-09-21T19:14:44.857925+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:18.905955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:18.905955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:40.196872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":937,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858179+0000","last_change":"2024-09-21T19:14:13.025483+0000","last_active":"2024-09-21T19:14:44.858179+0000","last_peered":"2024-09-21T19:14:44.858179+0000","last_clean":"2024-09-21T19:14:44.858179+0000","last_became_active":"2024-09-21T19:14:13.024395+0000","last_became_peered":"2024-09-21T19:14:13.024395+0000","last_unstale":"2024-09-21T19:14:44.858179+0000","last_undegraded":"2024-09-21T19:14:44.858179+0000","last_fullsized":"2024-09-21T19:14:44.858179+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:59.879323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:59.879323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:01.209491+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":826,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:46.818620+0000","last_change":"2024-09-21T19:14:46.818620+0000","last_active":"2024-09-21T19:14:46.818620+0000","last_peered":"2024-09-21T19:14:46.818620+0000","last_clean":"2024-09-21T19:14:46.818620+0000","last_became_active":"2024-09-21T19:13:41.867782+0000","last_became_peered":"2024-09-21T19:13:41.867782+0000","last_unstale":"2024-09-21T19:14:46.818620+0000","last_undegraded":"2024-09-21T19:14:46.818620+0000","last_fullsized":"2024-09-21T19:14:46.818620+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:46.818566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:46.818566+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:03.512809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"490'42","reported_seq":615,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858344+0000","last_change":"2024-09-21T19:14:13.025698+0000","last_active":"2024-09-21T19:14:44.858344+0000","last_peered":"2024-09-21T19:14:44.858344+0000","last_clean":"2024-09-21T19:14:44.858344+0000","last_became_active":"2024-09-21T19:14:13.025130+0000","last_became_peered":"2024-09-21T19:14:13.025130+0000","last_unstale":"2024-09-21T19:14:44.858344+0000","last_undegraded":"2024-09-21T19:14:44.858344+0000","last_fullsized":"2024-09-21T19:14:44.858344+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'42","last_scrub_stamp":"2024-09-21T19:13:21.835098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:21.835098+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"58'1","reported_seq":874,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154810+0000","last_change":"2024-09-21T19:14:07.874082+0000","last_active":"2024-09-21T19:14:45.154810+0000","last_peered":"2024-09-21T19:14:45.154810+0000","last_clean":"2024-09-21T19:14:45.154810+0000","last_became_active":"2024-09-21T19:13:55.926089+0000","last_became_peered":"2024-09-21T19:13:55.926089+0000","last_unstale":"2024-09-21T19:14:45.154810+0000","last_undegraded":"2024-09-21T19:14:45.154810+0000","last_fullsized":"2024-09-21T19:14:45.154810+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:07.874024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:07.874024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.625951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":1261,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855583+0000","last_change":"2024-09-21T19:14:08.907447+0000","last_active":"2024-09-21T19:14:44.855583+0000","last_peered":"2024-09-21T19:14:44.855583+0000","last_clean":"2024-09-21T19:14:44.855583+0000","last_became_active":"2024-09-21T18:53:11.411178+0000","last_became_peered":"2024-09-21T18:53:11.411178+0000","last_unstale":"2024-09-21T19:14:44.855583+0000","last_undegraded":"2024-09-21T19:14:44.855583+0000","last_fullsized":"2024-09-21T19:14:44.855583+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-21T19:14:08.907398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:08.907398+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:19.040641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1027,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071752+0000","last_change":"2024-09-21T19:14:32.072031+0000","last_active":"2024-09-21T19:14:45.071752+0000","last_peered":"2024-09-21T19:14:45.071752+0000","last_clean":"2024-09-21T19:14:45.071752+0000","last_became_active":"2024-09-21T19:05:42.904389+0000","last_became_peered":"2024-09-21T19:05:42.904389+0000","last_unstale":"2024-09-21T19:14:45.071752+0000","last_undegraded":"2024-09-21T19:14:45.071752+0000","last_fullsized":"2024-09-21T19:14:45.071752+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:32.071962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:32.071962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.875506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":821,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161841+0000","last_change":"2024-09-21T19:13:50.963002+0000","last_active":"2024-09-21T19:14:45.161841+0000","last_peered":"2024-09-21T19:14:45.161841+0000","last_clean":"2024-09-21T19:14:45.161841+0000","last_became_active":"2024-09-21T19:11:47.123128+0000","last_became_peered":"2024-09-21T19:11:47.123128+0000","last_unstale":"2024-09-21T19:14:45.161841+0000","last_undegraded":"2024-09-21T19:14:45.161841+0000","last_fullsized":"2024-09-21T19:14:45.161841+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:50.962927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:50.962927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:57.328613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"490'30","reported_seq":1113,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975613+0000","last_change":"2024-09-21T19:13:55.727501+0000","last_active":"2024-09-21T19:14:44.975613+0000","last_peered":"2024-09-21T19:14:44.975613+0000","last_clean":"2024-09-21T19:14:44.975613+0000","last_became_active":"2024-09-21T19:13:41.872842+0000","last_became_peered":"2024-09-21T19:13:41.872842+0000","last_unstale":"2024-09-21T19:14:44.975613+0000","last_undegraded":"2024-09-21T19:14:44.975613+0000","last_fullsized":"2024-09-21T19:14:44.975613+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:13:55.727428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:55.727428+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:00.346884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":530,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662143+0000","last_change":"2024-09-21T19:14:24.866553+0000","last_active":"2024-09-21T19:14:45.662143+0000","last_peered":"2024-09-21T19:14:45.662143+0000","last_clean":"2024-09-21T19:14:45.662143+0000","last_became_active":"2024-09-21T19:14:24.865145+0000","last_became_peered":"2024-09-21T19:14:24.865145+0000","last_unstale":"2024-09-21T19:14:45.662143+0000","last_undegraded":"2024-09-21T19:14:45.662143+0000","last_fullsized":"2024-09-21T19:14:45.662143+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:48.104135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:48.104135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:58.516201+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"511'18","reported_seq":1127,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154574+0000","last_change":"2024-09-21T19:14:19.838907+0000","last_active":"2024-09-21T19:14:45.154574+0000","last_peered":"2024-09-21T19:14:45.154574+0000","last_clean":"2024-09-21T19:14:45.154574+0000","last_became_active":"2024-09-21T19:13:55.929191+0000","last_became_peered":"2024-09-21T19:13:55.929191+0000","last_unstale":"2024-09-21T19:14:45.154574+0000","last_undegraded":"2024-09-21T19:14:45.154574+0000","last_fullsized":"2024-09-21T19:14:45.154574+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'18","last_scrub_stamp":"2024-09-21T19:14:19.838854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:19.838854+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:48.270840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":885,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.153974+0000","last_change":"2024-09-21T19:14:13.860409+0000","last_active":"2024-09-21T19:14:45.153974+0000","last_peered":"2024-09-21T19:14:45.153974+0000","last_clean":"2024-09-21T19:14:45.153974+0000","last_became_active":"2024-09-21T19:13:55.924872+0000","last_became_peered":"2024-09-21T19:13:55.924872+0000","last_unstale":"2024-09-21T19:14:45.153974+0000","last_undegraded":"2024-09-21T19:14:45.153974+0000","last_fullsized":"2024-09-21T19:14:45.153974+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:13.860363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:13.860363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:32.068056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":807,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161576+0000","last_change":"2024-09-21T19:14:33.996267+0000","last_active":"2024-09-21T19:14:45.161576+0000","last_peered":"2024-09-21T19:14:45.161576+0000","last_clean":"2024-09-21T19:14:45.161576+0000","last_became_active":"2024-09-21T19:11:47.128147+0000","last_became_peered":"2024-09-21T19:11:47.128147+0000","last_unstale":"2024-09-21T19:14:45.161576+0000","last_undegraded":"2024-09-21T19:14:45.161576+0000","last_fullsized":"2024-09-21T19:14:45.161576+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:33.996196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.996196+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:03.082199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"490'39","reported_seq":955,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154709+0000","last_change":"2024-09-21T19:13:55.930226+0000","last_active":"2024-09-21T19:14:45.154709+0000","last_peered":"2024-09-21T19:14:45.154709+0000","last_clean":"2024-09-21T19:14:45.154709+0000","last_became_active":"2024-09-21T19:13:55.928792+0000","last_became_peered":"2024-09-21T19:13:55.928792+0000","last_unstale":"2024-09-21T19:14:45.154709+0000","last_undegraded":"2024-09-21T19:14:45.154709+0000","last_fullsized":"2024-09-21T19:14:45.154709+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:13:42.860161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:42.860161+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1054,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.976104+0000","last_change":"2024-09-21T19:13:43.885912+0000","last_active":"2024-09-21T19:14:44.976104+0000","last_peered":"2024-09-21T19:14:44.976104+0000","last_clean":"2024-09-21T19:14:44.976104+0000","last_became_active":"2024-09-21T19:13:41.872311+0000","last_became_peered":"2024-09-21T19:13:41.872311+0000","last_unstale":"2024-09-21T19:14:44.976104+0000","last_undegraded":"2024-09-21T19:14:44.976104+0000","last_fullsized":"2024-09-21T19:14:44.976104+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:43.885865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:43.885865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:52.951358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":938,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:48.990980+0000","last_change":"2024-09-21T19:14:48.990980+0000","last_active":"2024-09-21T19:14:48.990980+0000","last_peered":"2024-09-21T19:14:48.990980+0000","last_clean":"2024-09-21T19:14:48.990980+0000","last_became_active":"2024-09-21T19:13:41.867084+0000","last_became_peered":"2024-09-21T19:13:41.867084+0000","last_unstale":"2024-09-21T19:14:48.990980+0000","last_undegraded":"2024-09-21T19:14:48.990980+0000","last_fullsized":"2024-09-21T19:14:48.990980+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:48.990928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:48.990928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:59.167786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1028,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975190+0000","last_change":"2024-09-21T19:13:59.800260+0000","last_active":"2024-09-21T19:14:44.975190+0000","last_peered":"2024-09-21T19:14:44.975190+0000","last_clean":"2024-09-21T19:14:44.975190+0000","last_became_active":"2024-09-21T19:05:42.904981+0000","last_became_peered":"2024-09-21T19:05:42.904981+0000","last_unstale":"2024-09-21T19:14:44.975190+0000","last_undegraded":"2024-09-21T19:14:44.975190+0000","last_fullsized":"2024-09-21T19:14:44.975190+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:59.800207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:59.800207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.108621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":748,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:47.953819+0000","last_change":"2024-09-21T19:14:47.953819+0000","last_active":"2024-09-21T19:14:47.953819+0000","last_peered":"2024-09-21T19:14:47.953819+0000","last_clean":"2024-09-21T19:14:47.953819+0000","last_became_active":"2024-09-21T19:14:24.866425+0000","last_became_peered":"2024-09-21T19:14:24.866425+0000","last_unstale":"2024-09-21T19:14:47.953819+0000","last_undegraded":"2024-09-21T19:14:47.953819+0000","last_fullsized":"2024-09-21T19:14:47.953819+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:47.953756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:47.953756+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:05.714587+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"490'54","reported_seq":1407,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975604+0000","last_change":"2024-09-21T19:13:46.857209+0000","last_active":"2024-09-21T19:14:44.975604+0000","last_peered":"2024-09-21T19:14:44.975604+0000","last_clean":"2024-09-21T19:14:44.975604+0000","last_became_active":"2024-09-21T19:05:42.910573+0000","last_became_peered":"2024-09-21T19:05:42.910573+0000","last_unstale":"2024-09-21T19:14:44.975604+0000","last_undegraded":"2024-09-21T19:14:44.975604+0000","last_fullsized":"2024-09-21T19:14:44.975604+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'54","last_scrub_stamp":"2024-09-21T19:13:46.857095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:46.857095+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:00.931094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":953,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858591+0000","last_change":"2024-09-21T19:14:13.027115+0000","last_active":"2024-09-21T19:14:44.858591+0000","last_peered":"2024-09-21T19:14:44.858591+0000","last_clean":"2024-09-21T19:14:44.858591+0000","last_became_active":"2024-09-21T19:14:13.025981+0000","last_became_peered":"2024-09-21T19:14:13.025981+0000","last_unstale":"2024-09-21T19:14:44.858591+0000","last_undegraded":"2024-09-21T19:14:44.858591+0000","last_fullsized":"2024-09-21T19:14:44.858591+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:26.913553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:26.913553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"60'2","reported_seq":566,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661682+0000","last_change":"2024-09-21T19:14:32.975360+0000","last_active":"2024-09-21T19:14:45.661682+0000","last_peered":"2024-09-21T19:14:45.661682+0000","last_clean":"2024-09-21T19:14:45.661682+0000","last_became_active":"2024-09-21T19:14:24.863058+0000","last_became_peered":"2024-09-21T19:14:24.863058+0000","last_unstale":"2024-09-21T19:14:45.661682+0000","last_undegraded":"2024-09-21T19:14:45.661682+0000","last_fullsized":"2024-09-21T19:14:45.661682+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-21T19:14:32.975298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:14:32.975298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.567425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":871,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154377+0000","last_change":"2024-09-21T19:14:15.775344+0000","last_active":"2024-09-21T19:14:45.154377+0000","last_peered":"2024-09-21T19:14:45.154377+0000","last_clean":"2024-09-21T19:14:45.154377+0000","last_became_active":"2024-09-21T19:13:55.926560+0000","last_became_peered":"2024-09-21T19:13:55.926560+0000","last_unstale":"2024-09-21T19:14:45.154377+0000","last_undegraded":"2024-09-21T19:14:45.154377+0000","last_fullsized":"2024-09-21T19:14:45.154377+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:15.775294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.775294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.789016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":995,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856710+0000","last_change":"2024-09-21T19:13:58.984558+0000","last_active":"2024-09-21T19:14:44.856710+0000","last_peered":"2024-09-21T19:14:44.856710+0000","last_clean":"2024-09-21T19:14:44.856710+0000","last_became_active":"2024-09-21T18:53:13.426840+0000","last_became_peered":"2024-09-21T18:53:13.426840+0000","last_unstale":"2024-09-21T19:14:44.856710+0000","last_undegraded":"2024-09-21T19:14:44.856710+0000","last_fullsized":"2024-09-21T19:14:44.856710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:58.984508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:58.984508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:06.348475+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":1110,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071634+0000","last_change":"2024-09-21T19:13:54.002452+0000","last_active":"2024-09-21T19:14:45.071634+0000","last_peered":"2024-09-21T19:14:45.071634+0000","last_clean":"2024-09-21T19:14:45.071634+0000","last_became_active":"2024-09-21T19:11:47.117467+0000","last_became_peered":"2024-09-21T19:11:47.117467+0000","last_unstale":"2024-09-21T19:14:45.071634+0000","last_undegraded":"2024-09-21T19:14:45.071634+0000","last_fullsized":"2024-09-21T19:14:45.071634+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:54.002392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.002392+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:16.895095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"489'21","reported_seq":557,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975773+0000","last_change":"2024-09-21T19:13:47.868425+0000","last_active":"2024-09-21T19:14:44.975773+0000","last_peered":"2024-09-21T19:14:44.975773+0000","last_clean":"2024-09-21T19:14:44.975773+0000","last_became_active":"2024-09-21T19:13:41.874168+0000","last_became_peered":"2024-09-21T19:13:41.874168+0000","last_unstale":"2024-09-21T19:14:44.975773+0000","last_undegraded":"2024-09-21T19:14:44.975773+0000","last_fullsized":"2024-09-21T19:14:44.975773+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:13:47.868368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:47.868368+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:49.773783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"58'1","reported_seq":1070,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975070+0000","last_change":"2024-09-21T19:14:20.003988+0000","last_active":"2024-09-21T19:14:44.975070+0000","last_peered":"2024-09-21T19:14:44.975070+0000","last_clean":"2024-09-21T19:14:44.975070+0000","last_became_active":"2024-09-21T19:05:42.905274+0000","last_became_peered":"2024-09-21T19:05:42.905274+0000","last_unstale":"2024-09-21T19:14:44.975070+0000","last_undegraded":"2024-09-21T19:14:44.975070+0000","last_fullsized":"2024-09-21T19:14:44.975070+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:20.003924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:20.003924+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:39.737121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"58'1","reported_seq":464,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855387+0000","last_change":"2024-09-21T19:13:45.514102+0000","last_active":"2024-09-21T19:14:44.855387+0000","last_peered":"2024-09-21T19:14:44.855387+0000","last_clean":"2024-09-21T19:14:44.855387+0000","last_became_active":"2024-09-21T19:05:42.902943+0000","last_became_peered":"2024-09-21T19:05:42.902943+0000","last_unstale":"2024-09-21T19:14:44.855387+0000","last_undegraded":"2024-09-21T19:14:44.855387+0000","last_fullsized":"2024-09-21T19:14:44.855387+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:13:45.514051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:13:45.514051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:58.431122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023747260000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":485,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855933+0000","last_change":"2024-09-21T19:14:16.442387+0000","last_active":"2024-09-21T19:14:44.855933+0000","last_peered":"2024-09-21T19:14:44.855933+0000","last_clean":"2024-09-21T19:14:44.855933+0000","last_became_active":"2024-09-21T19:13:41.876613+0000","last_became_peered":"2024-09-21T19:13:41.876613+0000","last_unstale":"2024-09-21T19:14:44.855933+0000","last_undegraded":"2024-09-21T19:14:44.855933+0000","last_fullsized":"2024-09-21T19:14:44.855933+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:16.442336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:16.442336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.523208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"65'1","reported_seq":1025,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072429+0000","last_change":"2024-09-21T19:14:04.026314+0000","last_active":"2024-09-21T19:14:45.072429+0000","last_peered":"2024-09-21T19:14:45.072429+0000","last_clean":"2024-09-21T19:14:45.072429+0000","last_became_active":"2024-09-21T19:02:40.888330+0000","last_became_peered":"2024-09-21T19:02:40.888330+0000","last_unstale":"2024-09-21T19:14:45.072429+0000","last_undegraded":"2024-09-21T19:14:45.072429+0000","last_fullsized":"2024-09-21T19:14:45.072429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-09-21T19:14:04.026250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.026250+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:08.075250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":824,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975505+0000","last_change":"2024-09-21T19:14:30.104353+0000","last_active":"2024-09-21T19:14:44.975505+0000","last_peered":"2024-09-21T19:14:44.975505+0000","last_clean":"2024-09-21T19:14:44.975505+0000","last_became_active":"2024-09-21T19:11:47.124560+0000","last_became_peered":"2024-09-21T19:11:47.124560+0000","last_unstale":"2024-09-21T19:14:44.975505+0000","last_undegraded":"2024-09-21T19:14:44.975505+0000","last_fullsized":"2024-09-21T19:14:44.975505+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:30.104280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.104280+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:58.386649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"490'36","reported_seq":1054,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.155773+0000","last_change":"2024-09-21T19:14:45.155773+0000","last_active":"2024-09-21T19:14:45.155773+0000","last_peered":"2024-09-21T19:14:45.155773+0000","last_clean":"2024-09-21T19:14:45.155773+0000","last_became_active":"2024-09-21T19:13:07.170435+0000","last_became_peered":"2024-09-21T19:13:07.170435+0000","last_unstale":"2024-09-21T19:14:45.155773+0000","last_undegraded":"2024-09-21T19:14:45.155773+0000","last_fullsized":"2024-09-21T19:14:45.155773+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'36","last_scrub_stamp":"2024-09-21T19:14:45.155707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:45.155707+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:49.294849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":879,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.153962+0000","last_change":"2024-09-21T19:14:36.059514+0000","last_active":"2024-09-21T19:14:45.153962+0000","last_peered":"2024-09-21T19:14:45.153962+0000","last_clean":"2024-09-21T19:14:45.153962+0000","last_became_active":"2024-09-21T19:13:55.929379+0000","last_became_peered":"2024-09-21T19:13:55.929379+0000","last_unstale":"2024-09-21T19:14:45.153962+0000","last_undegraded":"2024-09-21T19:14:45.153962+0000","last_fullsized":"2024-09-21T19:14:45.153962+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:36.059468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:36.059468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:49.888615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"65'5","reported_seq":2331,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.242602+0000","last_change":"2024-09-21T19:13:56.018952+0000","last_active":"2024-09-21T19:14:45.242602+0000","last_peered":"2024-09-21T19:14:45.242602+0000","last_clean":"2024-09-21T19:14:45.242602+0000","last_became_active":"2024-09-21T18:55:51.522394+0000","last_became_peered":"2024-09-21T18:55:51.522394+0000","last_unstale":"2024-09-21T19:14:45.242602+0000","last_undegraded":"2024-09-21T19:14:45.242602+0000","last_fullsized":"2024-09-21T19:14:45.242602+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-21T19:13:56.018881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:05.370660+0000","last_clean_scrub_stamp":"2024-09-21T19:13:56.018881+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.956576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051921599999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1287,"num_read_kb":1282,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":74,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662505+0000","last_change":"2024-09-21T19:13:54.989792+0000","last_active":"2024-09-21T19:14:45.662505+0000","last_peered":"2024-09-21T19:14:45.662505+0000","last_clean":"2024-09-21T19:14:45.662505+0000","last_became_active":"2024-09-21T19:13:41.871779+0000","last_became_peered":"2024-09-21T19:13:41.871779+0000","last_unstale":"2024-09-21T19:14:45.662505+0000","last_undegraded":"2024-09-21T19:14:45.662505+0000","last_fullsized":"2024-09-21T19:14:45.662505+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:54.989741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.989741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:01.716245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1044,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975488+0000","last_change":"2024-09-21T19:14:28.078803+0000","last_active":"2024-09-21T19:14:44.975488+0000","last_peered":"2024-09-21T19:14:44.975488+0000","last_clean":"2024-09-21T19:14:44.975488+0000","last_became_active":"2024-09-21T19:13:41.871757+0000","last_became_peered":"2024-09-21T19:13:41.871757+0000","last_unstale":"2024-09-21T19:14:44.975488+0000","last_undegraded":"2024-09-21T19:14:44.975488+0000","last_fullsized":"2024-09-21T19:14:44.975488+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:28.078742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:28.078742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:53.539999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"490'42","reported_seq":1274,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.970136+0000","last_change":"2024-09-21T19:14:45.970080+0000","last_active":"2024-09-21T19:14:45.970136+0000","last_peered":"2024-09-21T19:14:45.970136+0000","last_clean":"2024-09-21T19:14:45.970136+0000","last_became_active":"2024-09-21T19:02:40.879632+0000","last_became_peered":"2024-09-21T19:02:40.879632+0000","last_unstale":"2024-09-21T19:14:45.970136+0000","last_undegraded":"2024-09-21T19:14:45.970136+0000","last_fullsized":"2024-09-21T19:14:45.970136+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'42","last_scrub_stamp":"2024-09-21T19:14:45.969995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:45.969995+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:09.372964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1044,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856283+0000","last_change":"2024-09-21T19:14:37.790869+0000","last_active":"2024-09-21T19:14:44.856283+0000","last_peered":"2024-09-21T19:14:44.856283+0000","last_clean":"2024-09-21T19:14:44.856283+0000","last_became_active":"2024-09-21T19:13:41.869386+0000","last_became_peered":"2024-09-21T19:13:41.869386+0000","last_unstale":"2024-09-21T19:14:44.856283+0000","last_undegraded":"2024-09-21T19:14:44.856283+0000","last_fullsized":"2024-09-21T19:14:44.856283+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:37.790818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:37.790818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:02.538838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":1132,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072223+0000","last_change":"2024-09-21T19:14:33.060110+0000","last_active":"2024-09-21T19:14:45.072223+0000","last_peered":"2024-09-21T19:14:45.072223+0000","last_clean":"2024-09-21T19:14:45.072223+0000","last_became_active":"2024-09-21T19:11:47.120134+0000","last_became_peered":"2024-09-21T19:11:47.120134+0000","last_unstale":"2024-09-21T19:14:45.072223+0000","last_undegraded":"2024-09-21T19:14:45.072223+0000","last_fullsized":"2024-09-21T19:14:45.072223+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:33.060021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.060021+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:41.088380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1045,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072287+0000","last_change":"2024-09-21T19:13:43.248336+0000","last_active":"2024-09-21T19:14:45.072287+0000","last_peered":"2024-09-21T19:14:45.072287+0000","last_clean":"2024-09-21T19:14:45.072287+0000","last_became_active":"2024-09-21T19:13:41.872230+0000","last_became_peered":"2024-09-21T19:13:41.872230+0000","last_unstale":"2024-09-21T19:14:45.072287+0000","last_undegraded":"2024-09-21T19:14:45.072287+0000","last_fullsized":"2024-09-21T19:14:45.072287+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:43.248293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:43.248293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":515,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662428+0000","last_change":"2024-09-21T19:14:24.867743+0000","last_active":"2024-09-21T19:14:45.662428+0000","last_peered":"2024-09-21T19:14:45.662428+0000","last_clean":"2024-09-21T19:14:45.662428+0000","last_became_active":"2024-09-21T19:14:24.867241+0000","last_became_peered":"2024-09-21T19:14:24.867241+0000","last_unstale":"2024-09-21T19:14:45.662428+0000","last_undegraded":"2024-09-21T19:14:45.662428+0000","last_fullsized":"2024-09-21T19:14:45.662428+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:43.092935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:43.092935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:50.834654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":844,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975276+0000","last_change":"2024-09-21T19:14:35.117767+0000","last_active":"2024-09-21T19:14:44.975276+0000","last_peered":"2024-09-21T19:14:44.975276+0000","last_clean":"2024-09-21T19:14:44.975276+0000","last_became_active":"2024-09-21T19:13:41.871481+0000","last_became_peered":"2024-09-21T19:13:41.871481+0000","last_unstale":"2024-09-21T19:14:44.975276+0000","last_undegraded":"2024-09-21T19:14:44.975276+0000","last_fullsized":"2024-09-21T19:14:44.975276+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:35.117720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:35.117720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:51.442926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"490'27","reported_seq":607,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859143+0000","last_change":"2024-09-21T19:14:17.918424+0000","last_active":"2024-09-21T19:14:44.859143+0000","last_peered":"2024-09-21T19:14:44.859143+0000","last_clean":"2024-09-21T19:14:44.859143+0000","last_became_active":"2024-09-21T19:14:13.028931+0000","last_became_peered":"2024-09-21T19:14:13.028931+0000","last_unstale":"2024-09-21T19:14:44.859143+0000","last_undegraded":"2024-09-21T19:14:44.859143+0000","last_fullsized":"2024-09-21T19:14:44.859143+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:14:17.918358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:17.918358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:44.809780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":957,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.073356+0000","last_change":"2024-09-21T19:14:45.073356+0000","last_active":"2024-09-21T19:14:45.073356+0000","last_peered":"2024-09-21T19:14:45.073356+0000","last_clean":"2024-09-21T19:14:45.073356+0000","last_became_active":"2024-09-21T19:14:13.024178+0000","last_became_peered":"2024-09-21T19:14:13.024178+0000","last_unstale":"2024-09-21T19:14:45.073356+0000","last_undegraded":"2024-09-21T19:14:45.073356+0000","last_fullsized":"2024-09-21T19:14:45.073356+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:45.073297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:45.073297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.601678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":920,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662325+0000","last_change":"2024-09-21T19:14:24.867578+0000","last_active":"2024-09-21T19:14:45.662325+0000","last_peered":"2024-09-21T19:14:45.662325+0000","last_clean":"2024-09-21T19:14:45.662325+0000","last_became_active":"2024-09-21T19:14:24.867093+0000","last_became_peered":"2024-09-21T19:14:24.867093+0000","last_unstale":"2024-09-21T19:14:45.662325+0000","last_undegraded":"2024-09-21T19:14:45.662325+0000","last_fullsized":"2024-09-21T19:14:45.662325+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:59.126971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:59.126971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:28.418699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1007,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857240+0000","last_change":"2024-09-21T19:13:41.868100+0000","last_active":"2024-09-21T19:14:44.857240+0000","last_peered":"2024-09-21T19:14:44.857240+0000","last_clean":"2024-09-21T19:14:44.857240+0000","last_became_active":"2024-09-21T19:13:41.866996+0000","last_became_peered":"2024-09-21T19:13:41.866996+0000","last_unstale":"2024-09-21T19:14:44.857240+0000","last_undegraded":"2024-09-21T19:14:44.857240+0000","last_fullsized":"2024-09-21T19:14:44.857240+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:08.220479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:08.220479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":440,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661424+0000","last_change":"2024-09-21T19:14:24.866856+0000","last_active":"2024-09-21T19:14:45.661424+0000","last_peered":"2024-09-21T19:14:45.661424+0000","last_clean":"2024-09-21T19:14:45.661424+0000","last_became_active":"2024-09-21T19:14:24.865037+0000","last_became_peered":"2024-09-21T19:14:24.865037+0000","last_unstale":"2024-09-21T19:14:45.661424+0000","last_undegraded":"2024-09-21T19:14:45.661424+0000","last_fullsized":"2024-09-21T19:14:45.661424+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:14.425583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:14.425583+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:24.104232+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"490'45","reported_seq":746,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858839+0000","last_change":"2024-09-21T19:14:36.060038+0000","last_active":"2024-09-21T19:14:44.858839+0000","last_peered":"2024-09-21T19:14:44.858839+0000","last_clean":"2024-09-21T19:14:44.858839+0000","last_became_active":"2024-09-21T19:14:13.022192+0000","last_became_peered":"2024-09-21T19:14:13.022192+0000","last_unstale":"2024-09-21T19:14:44.858839+0000","last_undegraded":"2024-09-21T19:14:44.858839+0000","last_fullsized":"2024-09-21T19:14:44.858839+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'45","last_scrub_stamp":"2024-09-21T19:14:36.059972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:36.059972+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:03.727967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":943,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161560+0000","last_change":"2024-09-21T19:14:04.994693+0000","last_active":"2024-09-21T19:14:45.161560+0000","last_peered":"2024-09-21T19:14:45.161560+0000","last_clean":"2024-09-21T19:14:45.161560+0000","last_became_active":"2024-09-21T19:02:40.878892+0000","last_became_peered":"2024-09-21T19:02:40.878892+0000","last_unstale":"2024-09-21T19:14:45.161560+0000","last_undegraded":"2024-09-21T19:14:45.161560+0000","last_fullsized":"2024-09-21T19:14:45.161560+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:04.994628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.994628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:11.148862+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":468,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161449+0000","last_change":"2024-09-21T19:14:05.946266+0000","last_active":"2024-09-21T19:14:45.161449+0000","last_peered":"2024-09-21T19:14:45.161449+0000","last_clean":"2024-09-21T19:14:45.161449+0000","last_became_active":"2024-09-21T19:05:37.836982+0000","last_became_peered":"2024-09-21T19:05:37.836982+0000","last_unstale":"2024-09-21T19:14:45.161449+0000","last_undegraded":"2024-09-21T19:14:45.161449+0000","last_fullsized":"2024-09-21T19:14:45.161449+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:05.946211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:05.946211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:29.215685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":476,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855238+0000","last_change":"2024-09-21T19:14:33.413325+0000","last_active":"2024-09-21T19:14:44.855238+0000","last_peered":"2024-09-21T19:14:44.855238+0000","last_clean":"2024-09-21T19:14:44.855238+0000","last_became_active":"2024-09-21T19:05:42.902106+0000","last_became_peered":"2024-09-21T19:05:42.902106+0000","last_unstale":"2024-09-21T19:14:44.855238+0000","last_undegraded":"2024-09-21T19:14:44.855238+0000","last_fullsized":"2024-09-21T19:14:44.855238+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:33.413271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:33.413271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:54.479246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":1126,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071669+0000","last_change":"2024-09-21T19:14:05.000670+0000","last_active":"2024-09-21T19:14:45.071669+0000","last_peered":"2024-09-21T19:14:45.071669+0000","last_clean":"2024-09-21T19:14:45.071669+0000","last_became_active":"2024-09-21T19:05:42.903764+0000","last_became_peered":"2024-09-21T19:05:42.903764+0000","last_unstale":"2024-09-21T19:14:45.071669+0000","last_undegraded":"2024-09-21T19:14:45.071669+0000","last_fullsized":"2024-09-21T19:14:45.071669+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:05.000568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:05.000568+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:32.973330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"490'30","reported_seq":593,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858308+0000","last_change":"2024-09-21T19:14:43.056924+0000","last_active":"2024-09-21T19:14:44.858308+0000","last_peered":"2024-09-21T19:14:44.858308+0000","last_clean":"2024-09-21T19:14:44.858308+0000","last_became_active":"2024-09-21T19:14:13.027486+0000","last_became_peered":"2024-09-21T19:14:13.027486+0000","last_unstale":"2024-09-21T19:14:44.858308+0000","last_undegraded":"2024-09-21T19:14:44.858308+0000","last_fullsized":"2024-09-21T19:14:44.858308+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:14:43.056857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:43.056857+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.472766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"58'1","reported_seq":1043,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857040+0000","last_change":"2024-09-21T19:14:27.801807+0000","last_active":"2024-09-21T19:14:44.857040+0000","last_peered":"2024-09-21T19:14:44.857040+0000","last_clean":"2024-09-21T19:14:44.857040+0000","last_became_active":"2024-09-21T18:55:51.531166+0000","last_became_peered":"2024-09-21T18:55:51.531166+0000","last_unstale":"2024-09-21T19:14:44.857040+0000","last_undegraded":"2024-09-21T19:14:44.857040+0000","last_fullsized":"2024-09-21T19:14:44.857040+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-21T19:14:27.801751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:27.801751+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:33.975381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"504'14","reported_seq":609,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855623+0000","last_change":"2024-09-21T19:14:02.435541+0000","last_active":"2024-09-21T19:14:44.855623+0000","last_peered":"2024-09-21T19:14:44.855623+0000","last_clean":"2024-09-21T19:14:44.855623+0000","last_became_active":"2024-09-21T19:13:41.877504+0000","last_became_peered":"2024-09-21T19:13:41.877504+0000","last_unstale":"2024-09-21T19:14:44.855623+0000","last_undegraded":"2024-09-21T19:14:44.855623+0000","last_fullsized":"2024-09-21T19:14:44.855623+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'14","last_scrub_stamp":"2024-09-21T19:14:02.435491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:02.435491+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:05.135485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":921,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858923+0000","last_change":"2024-09-21T19:14:13.023798+0000","last_active":"2024-09-21T19:14:44.858923+0000","last_peered":"2024-09-21T19:14:44.858923+0000","last_clean":"2024-09-21T19:14:44.858923+0000","last_became_active":"2024-09-21T19:14:13.022622+0000","last_became_peered":"2024-09-21T19:14:13.022622+0000","last_unstale":"2024-09-21T19:14:44.858923+0000","last_undegraded":"2024-09-21T19:14:44.858923+0000","last_fullsized":"2024-09-21T19:14:44.858923+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:06.812349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:06.812349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":102,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858913+0000","last_change":"2024-09-21T19:14:15.887014+0000","last_active":"2024-09-21T19:14:44.858913+0000","last_peered":"2024-09-21T19:14:44.858913+0000","last_clean":"2024-09-21T19:14:44.858913+0000","last_became_active":"2024-09-21T19:14:13.023805+0000","last_became_peered":"2024-09-21T19:14:13.023805+0000","last_unstale":"2024-09-21T19:14:44.858913+0000","last_undegraded":"2024-09-21T19:14:44.858913+0000","last_fullsized":"2024-09-21T19:14:44.858913+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:15.886949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:15.886949+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:19.925990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"490'21","reported_seq":882,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154807+0000","last_change":"2024-09-21T19:13:55.929905+0000","last_active":"2024-09-21T19:14:45.154807+0000","last_peered":"2024-09-21T19:14:45.154807+0000","last_clean":"2024-09-21T19:14:45.154807+0000","last_became_active":"2024-09-21T19:13:55.926999+0000","last_became_peered":"2024-09-21T19:13:55.926999+0000","last_unstale":"2024-09-21T19:14:45.154807+0000","last_undegraded":"2024-09-21T19:14:45.154807+0000","last_fullsized":"2024-09-21T19:14:45.154807+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:11:41.665004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:11:41.665004+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":971,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661994+0000","last_change":"2024-09-21T19:14:41.947759+0000","last_active":"2024-09-21T19:14:45.661994+0000","last_peered":"2024-09-21T19:14:45.661994+0000","last_clean":"2024-09-21T19:14:45.661994+0000","last_became_active":"2024-09-21T19:14:24.862420+0000","last_became_peered":"2024-09-21T19:14:24.862420+0000","last_unstale":"2024-09-21T19:14:45.661994+0000","last_undegraded":"2024-09-21T19:14:45.661994+0000","last_fullsized":"2024-09-21T19:14:45.661994+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:41.947704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:41.947704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:11.301009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":468,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:46.676493+0000","last_change":"2024-09-21T19:14:46.676493+0000","last_active":"2024-09-21T19:14:46.676493+0000","last_peered":"2024-09-21T19:14:46.676493+0000","last_clean":"2024-09-21T19:14:46.676493+0000","last_became_active":"2024-09-21T19:05:42.908662+0000","last_became_peered":"2024-09-21T19:05:42.908662+0000","last_unstale":"2024-09-21T19:14:46.676493+0000","last_undegraded":"2024-09-21T19:14:46.676493+0000","last_fullsized":"2024-09-21T19:14:46.676493+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:46.676416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:46.676416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:08.738702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":876,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.153912+0000","last_change":"2024-09-21T19:13:55.930484+0000","last_active":"2024-09-21T19:14:45.153912+0000","last_peered":"2024-09-21T19:14:45.153912+0000","last_clean":"2024-09-21T19:14:45.153912+0000","last_became_active":"2024-09-21T19:13:55.929513+0000","last_became_peered":"2024-09-21T19:13:55.929513+0000","last_unstale":"2024-09-21T19:14:45.153912+0000","last_undegraded":"2024-09-21T19:14:45.153912+0000","last_fullsized":"2024-09-21T19:14:45.153912+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:11:31.588820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:11:31.588820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":849,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161748+0000","last_change":"2024-09-21T19:14:18.978118+0000","last_active":"2024-09-21T19:14:45.161748+0000","last_peered":"2024-09-21T19:14:45.161748+0000","last_clean":"2024-09-21T19:14:45.161748+0000","last_became_active":"2024-09-21T19:13:41.873982+0000","last_became_peered":"2024-09-21T19:13:41.873982+0000","last_unstale":"2024-09-21T19:14:45.161748+0000","last_undegraded":"2024-09-21T19:14:45.161748+0000","last_fullsized":"2024-09-21T19:14:45.161748+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:18.978048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:18.978048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:37.065594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"490'51","reported_seq":978,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161330+0000","last_change":"2024-09-21T19:13:58.129757+0000","last_active":"2024-09-21T19:14:45.161330+0000","last_peered":"2024-09-21T19:14:45.161330+0000","last_clean":"2024-09-21T19:14:45.161330+0000","last_became_active":"2024-09-21T19:02:40.878545+0000","last_became_peered":"2024-09-21T19:02:40.878545+0000","last_unstale":"2024-09-21T19:14:45.161330+0000","last_undegraded":"2024-09-21T19:14:45.161330+0000","last_fullsized":"2024-09-21T19:14:45.161330+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'51","last_scrub_stamp":"2024-09-21T19:13:58.129684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:58.129684+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:13.959269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1049,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072773+0000","last_change":"2024-09-21T19:13:55.013063+0000","last_active":"2024-09-21T19:14:45.072773+0000","last_peered":"2024-09-21T19:14:45.072773+0000","last_clean":"2024-09-21T19:14:45.072773+0000","last_became_active":"2024-09-21T19:13:41.868691+0000","last_became_peered":"2024-09-21T19:13:41.868691+0000","last_unstale":"2024-09-21T19:14:45.072773+0000","last_undegraded":"2024-09-21T19:14:45.072773+0000","last_fullsized":"2024-09-21T19:14:45.072773+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:55.013011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:55.013011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:18.337157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":485,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855515+0000","last_change":"2024-09-21T19:14:10.426107+0000","last_active":"2024-09-21T19:14:44.855515+0000","last_peered":"2024-09-21T19:14:44.855515+0000","last_clean":"2024-09-21T19:14:44.855515+0000","last_became_active":"2024-09-21T19:13:41.875147+0000","last_became_peered":"2024-09-21T19:13:41.875147+0000","last_unstale":"2024-09-21T19:14:44.855515+0000","last_undegraded":"2024-09-21T19:14:44.855515+0000","last_fullsized":"2024-09-21T19:14:44.855515+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:10.426061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:10.426061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:13.761821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1015,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072155+0000","last_change":"2024-09-21T19:14:08.030116+0000","last_active":"2024-09-21T19:14:45.072155+0000","last_peered":"2024-09-21T19:14:45.072155+0000","last_clean":"2024-09-21T19:14:45.072155+0000","last_became_active":"2024-09-21T19:05:42.903718+0000","last_became_peered":"2024-09-21T19:05:42.903718+0000","last_unstale":"2024-09-21T19:14:45.072155+0000","last_undegraded":"2024-09-21T19:14:45.072155+0000","last_fullsized":"2024-09-21T19:14:45.072155+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:08.030060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:08.030060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:13.046600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":809,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857416+0000","last_change":"2024-09-21T19:13:50.999379+0000","last_active":"2024-09-21T19:14:44.857416+0000","last_peered":"2024-09-21T19:14:44.857416+0000","last_clean":"2024-09-21T19:14:44.857416+0000","last_became_active":"2024-09-21T19:11:49.145230+0000","last_became_peered":"2024-09-21T19:11:49.145230+0000","last_unstale":"2024-09-21T19:14:44.857416+0000","last_undegraded":"2024-09-21T19:14:44.857416+0000","last_fullsized":"2024-09-21T19:14:44.857416+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:50.999329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:50.999329+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:52.341111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"489'24","reported_seq":81,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858650+0000","last_change":"2024-09-21T19:14:27.915104+0000","last_active":"2024-09-21T19:14:44.858650+0000","last_peered":"2024-09-21T19:14:44.858650+0000","last_clean":"2024-09-21T19:14:44.858650+0000","last_became_active":"2024-09-21T19:14:13.022927+0000","last_became_peered":"2024-09-21T19:14:13.022927+0000","last_unstale":"2024-09-21T19:14:44.858650+0000","last_undegraded":"2024-09-21T19:14:44.858650+0000","last_fullsized":"2024-09-21T19:14:44.858650+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'24","last_scrub_stamp":"2024-09-21T19:14:27.915028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:27.915028+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.299437+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1037,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072793+0000","last_change":"2024-09-21T19:14:03.006048+0000","last_active":"2024-09-21T19:14:45.072793+0000","last_peered":"2024-09-21T19:14:45.072793+0000","last_clean":"2024-09-21T19:14:45.072793+0000","last_became_active":"2024-09-21T18:53:07.399903+0000","last_became_peered":"2024-09-21T18:53:07.399903+0000","last_unstale":"2024-09-21T19:14:45.072793+0000","last_undegraded":"2024-09-21T19:14:45.072793+0000","last_fullsized":"2024-09-21T19:14:45.072793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:03.005968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:03.005968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:19.650408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":468,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855193+0000","last_change":"2024-09-21T19:14:35.439447+0000","last_active":"2024-09-21T19:14:44.855193+0000","last_peered":"2024-09-21T19:14:44.855193+0000","last_clean":"2024-09-21T19:14:44.855193+0000","last_became_active":"2024-09-21T19:05:42.902145+0000","last_became_peered":"2024-09-21T19:05:42.902145+0000","last_unstale":"2024-09-21T19:14:44.855193+0000","last_undegraded":"2024-09-21T19:14:44.855193+0000","last_fullsized":"2024-09-21T19:14:44.855193+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:35.439400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:35.439400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:16:04.051547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":987,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856703+0000","last_change":"2024-09-21T19:14:30.785324+0000","last_active":"2024-09-21T19:14:44.856703+0000","last_peered":"2024-09-21T19:14:44.856703+0000","last_clean":"2024-09-21T19:14:44.856703+0000","last_became_active":"2024-09-21T18:53:13.428946+0000","last_became_peered":"2024-09-21T18:53:13.428946+0000","last_unstale":"2024-09-21T19:14:44.856703+0000","last_undegraded":"2024-09-21T19:14:44.856703+0000","last_fullsized":"2024-09-21T19:14:44.856703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:30.785275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.785275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.319888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":190,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072203+0000","last_change":"2024-09-21T19:14:30.147373+0000","last_active":"2024-09-21T19:14:45.072203+0000","last_peered":"2024-09-21T19:14:45.072203+0000","last_clean":"2024-09-21T19:14:45.072203+0000","last_became_active":"2024-09-21T19:11:49.139511+0000","last_became_peered":"2024-09-21T19:11:49.139511+0000","last_unstale":"2024-09-21T19:14:45.072203+0000","last_undegraded":"2024-09-21T19:14:45.072203+0000","last_fullsized":"2024-09-21T19:14:45.072203+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:14:30.147305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:14:30.147305+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:39.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"490'39","reported_seq":628,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661299+0000","last_change":"2024-09-21T19:14:24.862367+0000","last_active":"2024-09-21T19:14:45.661299+0000","last_peered":"2024-09-21T19:14:45.661299+0000","last_clean":"2024-09-21T19:14:45.661299+0000","last_became_active":"2024-09-21T19:14:24.861608+0000","last_became_peered":"2024-09-21T19:14:24.861608+0000","last_unstale":"2024-09-21T19:14:45.661299+0000","last_undegraded":"2024-09-21T19:14:45.661299+0000","last_fullsized":"2024-09-21T19:14:45.661299+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'39","last_scrub_stamp":"2024-09-21T19:14:24.051467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:24.051467+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.782268+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":1051,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072945+0000","last_change":"2024-09-21T19:14:17.075150+0000","last_active":"2024-09-21T19:14:45.072945+0000","last_peered":"2024-09-21T19:14:45.072945+0000","last_clean":"2024-09-21T19:14:45.072945+0000","last_became_active":"2024-09-21T18:55:51.535636+0000","last_became_peered":"2024-09-21T18:55:51.535636+0000","last_unstale":"2024-09-21T19:14:45.072945+0000","last_undegraded":"2024-09-21T19:14:45.072945+0000","last_fullsized":"2024-09-21T19:14:45.072945+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-21T19:14:17.075085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:17.075085+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:27.084716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"317'11","reported_seq":575,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856601+0000","last_change":"2024-09-21T19:14:24.871790+0000","last_active":"2024-09-21T19:14:44.856601+0000","last_peered":"2024-09-21T19:14:44.856601+0000","last_clean":"2024-09-21T19:14:44.856601+0000","last_became_active":"2024-09-21T19:05:37.837585+0000","last_became_peered":"2024-09-21T19:05:37.837585+0000","last_unstale":"2024-09-21T19:14:44.856601+0000","last_undegraded":"2024-09-21T19:14:44.856601+0000","last_fullsized":"2024-09-21T19:14:44.856601+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-21T19:14:24.871708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:24.871708+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:44.378176+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":989,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661763+0000","last_change":"2024-09-21T19:14:24.863784+0000","last_active":"2024-09-21T19:14:45.661763+0000","last_peered":"2024-09-21T19:14:45.661763+0000","last_clean":"2024-09-21T19:14:45.661763+0000","last_became_active":"2024-09-21T19:14:24.863412+0000","last_became_peered":"2024-09-21T19:14:24.863412+0000","last_unstale":"2024-09-21T19:14:45.661763+0000","last_undegraded":"2024-09-21T19:14:45.661763+0000","last_fullsized":"2024-09-21T19:14:45.661763+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:56.076731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:56.076731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.553102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":67,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662016+0000","last_change":"2024-09-21T19:13:41.869004+0000","last_active":"2024-09-21T19:14:45.662016+0000","last_peered":"2024-09-21T19:14:45.662016+0000","last_clean":"2024-09-21T19:14:45.662016+0000","last_became_active":"2024-09-21T19:13:41.868497+0000","last_became_peered":"2024-09-21T19:13:41.868497+0000","last_unstale":"2024-09-21T19:14:45.662016+0000","last_undegraded":"2024-09-21T19:14:45.662016+0000","last_fullsized":"2024-09-21T19:14:45.662016+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:24.088416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:24.088416+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":980,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.160929+0000","last_change":"2024-09-21T19:14:04.037324+0000","last_active":"2024-09-21T19:14:45.160929+0000","last_peered":"2024-09-21T19:14:45.160929+0000","last_clean":"2024-09-21T19:14:45.160929+0000","last_became_active":"2024-09-21T19:13:41.868092+0000","last_became_peered":"2024-09-21T19:13:41.868092+0000","last_unstale":"2024-09-21T19:14:45.160929+0000","last_undegraded":"2024-09-21T19:14:45.160929+0000","last_fullsized":"2024-09-21T19:14:45.160929+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:04.037267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.037267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:30.087896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"490'18","reported_seq":1037,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856663+0000","last_change":"2024-09-21T19:14:04.860511+0000","last_active":"2024-09-21T19:14:44.856663+0000","last_peered":"2024-09-21T19:14:44.856663+0000","last_clean":"2024-09-21T19:14:44.856663+0000","last_became_active":"2024-09-21T18:53:09.717611+0000","last_became_peered":"2024-09-21T18:53:09.717611+0000","last_unstale":"2024-09-21T19:14:44.856663+0000","last_undegraded":"2024-09-21T19:14:44.856663+0000","last_fullsized":"2024-09-21T19:14:44.856663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'18","last_scrub_stamp":"2024-09-21T19:14:04.860444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:04.860444+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:25.498619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":523,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662293+0000","last_change":"2024-09-21T19:14:24.865786+0000","last_active":"2024-09-21T19:14:45.662293+0000","last_peered":"2024-09-21T19:14:45.662293+0000","last_clean":"2024-09-21T19:14:45.662293+0000","last_became_active":"2024-09-21T19:14:24.865457+0000","last_became_peered":"2024-09-21T19:14:24.865457+0000","last_unstale":"2024-09-21T19:14:45.662293+0000","last_undegraded":"2024-09-21T19:14:45.662293+0000","last_fullsized":"2024-09-21T19:14:45.662293+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:18.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:18.170847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":995,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072328+0000","last_change":"2024-09-21T19:10:22.627798+0000","last_active":"2024-09-21T19:14:45.072328+0000","last_peered":"2024-09-21T19:14:45.072328+0000","last_clean":"2024-09-21T19:14:45.072328+0000","last_became_active":"2024-09-21T19:02:40.888821+0000","last_became_peered":"2024-09-21T19:02:40.888821+0000","last_unstale":"2024-09-21T19:14:45.072328+0000","last_undegraded":"2024-09-21T19:14:45.072328+0000","last_fullsized":"2024-09-21T19:14:45.072328+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:10:22.627740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:10:22.627740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":822,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975615+0000","last_change":"2024-09-21T19:13:52.802017+0000","last_active":"2024-09-21T19:14:44.975615+0000","last_peered":"2024-09-21T19:14:44.975615+0000","last_clean":"2024-09-21T19:14:44.975615+0000","last_became_active":"2024-09-21T19:11:49.138877+0000","last_became_peered":"2024-09-21T19:11:49.138877+0000","last_unstale":"2024-09-21T19:14:44.975615+0000","last_undegraded":"2024-09-21T19:14:44.975615+0000","last_fullsized":"2024-09-21T19:14:44.975615+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:52.801960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:52.801960+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:07.421811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"489'21","reported_seq":889,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154796+0000","last_change":"2024-09-21T19:13:55.931247+0000","last_active":"2024-09-21T19:14:45.154796+0000","last_peered":"2024-09-21T19:14:45.154796+0000","last_clean":"2024-09-21T19:14:45.154796+0000","last_became_active":"2024-09-21T19:13:55.928306+0000","last_became_peered":"2024-09-21T19:13:55.928306+0000","last_unstale":"2024-09-21T19:14:45.154796+0000","last_undegraded":"2024-09-21T19:14:45.154796+0000","last_fullsized":"2024-09-21T19:14:45.154796+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T19:12:36.624435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:12:36.624435+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1024,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071895+0000","last_change":"2024-09-21T19:14:25.096034+0000","last_active":"2024-09-21T19:14:45.071895+0000","last_peered":"2024-09-21T19:14:45.071895+0000","last_clean":"2024-09-21T19:14:45.071895+0000","last_became_active":"2024-09-21T19:02:40.885678+0000","last_became_peered":"2024-09-21T19:02:40.885678+0000","last_unstale":"2024-09-21T19:14:45.071895+0000","last_undegraded":"2024-09-21T19:14:45.071895+0000","last_fullsized":"2024-09-21T19:14:45.071895+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:25.095979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:25.095979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:47.153733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":998,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856450+0000","last_change":"2024-09-21T19:14:22.870075+0000","last_active":"2024-09-21T19:14:44.856450+0000","last_peered":"2024-09-21T19:14:44.856450+0000","last_clean":"2024-09-21T19:14:44.856450+0000","last_became_active":"2024-09-21T18:53:11.410154+0000","last_became_peered":"2024-09-21T18:53:11.410154+0000","last_unstale":"2024-09-21T19:14:44.856450+0000","last_undegraded":"2024-09-21T19:14:44.856450+0000","last_fullsized":"2024-09-21T19:14:44.856450+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:22.870022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:22.870022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:28.288452+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":459,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855596+0000","last_change":"2024-09-21T19:13:54.395414+0000","last_active":"2024-09-21T19:14:44.855596+0000","last_peered":"2024-09-21T19:14:44.855596+0000","last_clean":"2024-09-21T19:14:44.855596+0000","last_became_active":"2024-09-21T19:05:42.903823+0000","last_became_peered":"2024-09-21T19:05:42.903823+0000","last_unstale":"2024-09-21T19:14:44.855596+0000","last_undegraded":"2024-09-21T19:14:44.855596+0000","last_fullsized":"2024-09-21T19:14:44.855596+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:54.395365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.395365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:10.292853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":1095,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661268+0000","last_change":"2024-09-21T19:14:24.864595+0000","last_active":"2024-09-21T19:14:45.661268+0000","last_peered":"2024-09-21T19:14:45.661268+0000","last_clean":"2024-09-21T19:14:45.661268+0000","last_became_active":"2024-09-21T19:14:24.864021+0000","last_became_peered":"2024-09-21T19:14:24.864021+0000","last_unstale":"2024-09-21T19:14:45.661268+0000","last_undegraded":"2024-09-21T19:14:45.661268+0000","last_fullsized":"2024-09-21T19:14:45.661268+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:25.372586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:25.372586+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"490'21","reported_seq":595,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.661988+0000","last_change":"2024-09-21T19:14:24.864960+0000","last_active":"2024-09-21T19:14:45.661988+0000","last_peered":"2024-09-21T19:14:45.661988+0000","last_clean":"2024-09-21T19:14:45.661988+0000","last_became_active":"2024-09-21T19:14:24.864311+0000","last_became_peered":"2024-09-21T19:14:24.864311+0000","last_unstale":"2024-09-21T19:14:45.661988+0000","last_undegraded":"2024-09-21T19:14:45.661988+0000","last_fullsized":"2024-09-21T19:14:45.661988+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'21","last_scrub_stamp":"2024-09-21T19:14:23.037770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:23.037770+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:40.847380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":878,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154078+0000","last_change":"2024-09-21T19:14:23.929705+0000","last_active":"2024-09-21T19:14:45.154078+0000","last_peered":"2024-09-21T19:14:45.154078+0000","last_clean":"2024-09-21T19:14:45.154078+0000","last_became_active":"2024-09-21T19:13:55.927491+0000","last_became_peered":"2024-09-21T19:13:55.927491+0000","last_unstale":"2024-09-21T19:14:45.154078+0000","last_undegraded":"2024-09-21T19:14:45.154078+0000","last_fullsized":"2024-09-21T19:14:45.154078+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:23.929658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:23.929658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:28.867014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"321'10","reported_seq":1291,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857329+0000","last_change":"2024-09-21T19:14:42.748095+0000","last_active":"2024-09-21T19:14:44.857329+0000","last_peered":"2024-09-21T19:14:44.857329+0000","last_clean":"2024-09-21T19:14:44.857329+0000","last_became_active":"2024-09-21T19:05:42.911591+0000","last_became_peered":"2024-09-21T19:05:42.911591+0000","last_unstale":"2024-09-21T19:14:44.857329+0000","last_undegraded":"2024-09-21T19:14:44.857329+0000","last_fullsized":"2024-09-21T19:14:44.857329+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'10","last_scrub_stamp":"2024-09-21T19:14:42.748033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:42.748033+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:56.785153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":931,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.859153+0000","last_change":"2024-09-21T19:14:13.022645+0000","last_active":"2024-09-21T19:14:44.859153+0000","last_peered":"2024-09-21T19:14:44.859153+0000","last_clean":"2024-09-21T19:14:44.859153+0000","last_became_active":"2024-09-21T19:14:13.022313+0000","last_became_peered":"2024-09-21T19:14:13.022313+0000","last_unstale":"2024-09-21T19:14:44.859153+0000","last_undegraded":"2024-09-21T19:14:44.859153+0000","last_fullsized":"2024-09-21T19:14:44.859153+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:05.516911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:05.516911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":763,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.153716+0000","last_change":"2024-09-21T19:13:55.929698+0000","last_active":"2024-09-21T19:14:45.153716+0000","last_peered":"2024-09-21T19:14:45.153716+0000","last_clean":"2024-09-21T19:14:45.153716+0000","last_became_active":"2024-09-21T19:13:55.928943+0000","last_became_peered":"2024-09-21T19:13:55.928943+0000","last_unstale":"2024-09-21T19:14:45.153716+0000","last_undegraded":"2024-09-21T19:14:45.153716+0000","last_fullsized":"2024-09-21T19:14:45.153716+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:53.945511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:53.945511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:09.908937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"490'30","reported_seq":1059,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857627+0000","last_change":"2024-09-21T19:13:52.024298+0000","last_active":"2024-09-21T19:14:44.857627+0000","last_peered":"2024-09-21T19:14:44.857627+0000","last_clean":"2024-09-21T19:14:44.857627+0000","last_became_active":"2024-09-21T19:10:38.923267+0000","last_became_peered":"2024-09-21T19:10:38.923267+0000","last_unstale":"2024-09-21T19:14:44.857627+0000","last_undegraded":"2024-09-21T19:14:44.857627+0000","last_fullsized":"2024-09-21T19:14:44.857627+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'30","last_scrub_stamp":"2024-09-21T19:13:52.024239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:52.024239+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:57.960257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1027,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072163+0000","last_change":"2024-09-21T19:14:06.025842+0000","last_active":"2024-09-21T19:14:45.072163+0000","last_peered":"2024-09-21T19:14:45.072163+0000","last_clean":"2024-09-21T19:14:45.072163+0000","last_became_active":"2024-09-21T19:05:42.903577+0000","last_became_peered":"2024-09-21T19:05:42.903577+0000","last_unstale":"2024-09-21T19:14:45.072163+0000","last_undegraded":"2024-09-21T19:14:45.072163+0000","last_fullsized":"2024-09-21T19:14:45.072163+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:06.025758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:06.025758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:23.796919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"529'16","reported_seq":703,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662304+0000","last_change":"2024-09-21T19:14:24.863104+0000","last_active":"2024-09-21T19:14:45.662304+0000","last_peered":"2024-09-21T19:14:45.662304+0000","last_clean":"2024-09-21T19:14:45.662304+0000","last_became_active":"2024-09-21T19:14:24.862762+0000","last_became_peered":"2024-09-21T19:14:24.862762+0000","last_unstale":"2024-09-21T19:14:45.662304+0000","last_undegraded":"2024-09-21T19:14:45.662304+0000","last_fullsized":"2024-09-21T19:14:45.662304+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15","last_scrub_stamp":"2024-09-21T19:14:10.819759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:10.819759+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.526679+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":869,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154924+0000","last_change":"2024-09-21T19:13:55.931622+0000","last_active":"2024-09-21T19:14:45.154924+0000","last_peered":"2024-09-21T19:14:45.154924+0000","last_clean":"2024-09-21T19:14:45.154924+0000","last_became_active":"2024-09-21T19:13:55.930144+0000","last_became_peered":"2024-09-21T19:13:55.930144+0000","last_unstale":"2024-09-21T19:14:45.154924+0000","last_undegraded":"2024-09-21T19:14:45.154924+0000","last_fullsized":"2024-09-21T19:14:45.154924+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:05.734582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:13:05.734582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"490'27","reported_seq":610,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858270+0000","last_change":"2024-09-21T19:14:13.027160+0000","last_active":"2024-09-21T19:14:44.858270+0000","last_peered":"2024-09-21T19:14:44.858270+0000","last_clean":"2024-09-21T19:14:44.858270+0000","last_became_active":"2024-09-21T19:14:13.026094+0000","last_became_peered":"2024-09-21T19:14:13.026094+0000","last_unstale":"2024-09-21T19:14:44.858270+0000","last_undegraded":"2024-09-21T19:14:44.858270+0000","last_fullsized":"2024-09-21T19:14:44.858270+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:14:11.010860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:11.010860+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:16.776521+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":938,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.858541+0000","last_change":"2024-09-21T19:14:35.025832+0000","last_active":"2024-09-21T19:14:44.858541+0000","last_peered":"2024-09-21T19:14:44.858541+0000","last_clean":"2024-09-21T19:14:44.858541+0000","last_became_active":"2024-09-21T19:14:13.027969+0000","last_became_peered":"2024-09-21T19:14:13.027969+0000","last_unstale":"2024-09-21T19:14:44.858541+0000","last_undegraded":"2024-09-21T19:14:44.858541+0000","last_fullsized":"2024-09-21T19:14:44.858541+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:35.025773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:35.025773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:45.369290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1039,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975344+0000","last_change":"2024-09-21T19:14:24.980743+0000","last_active":"2024-09-21T19:14:44.975344+0000","last_peered":"2024-09-21T19:14:44.975344+0000","last_clean":"2024-09-21T19:14:44.975344+0000","last_became_active":"2024-09-21T19:13:41.872018+0000","last_became_peered":"2024-09-21T19:13:41.872018+0000","last_unstale":"2024-09-21T19:14:44.975344+0000","last_undegraded":"2024-09-21T19:14:44.975344+0000","last_fullsized":"2024-09-21T19:14:44.975344+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:24.980696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:24.980696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:26.350269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1034,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856343+0000","last_change":"2024-09-21T19:14:17.987752+0000","last_active":"2024-09-21T19:14:44.856343+0000","last_peered":"2024-09-21T19:14:44.856343+0000","last_clean":"2024-09-21T19:14:44.856343+0000","last_became_active":"2024-09-21T19:13:41.866921+0000","last_became_peered":"2024-09-21T19:13:41.866921+0000","last_unstale":"2024-09-21T19:14:44.856343+0000","last_undegraded":"2024-09-21T19:14:44.856343+0000","last_fullsized":"2024-09-21T19:14:44.856343+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:17.987700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:17.987700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:36.820401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"489'21","reported_seq":1105,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975723+0000","last_change":"2024-09-21T19:13:58.757296+0000","last_active":"2024-09-21T19:14:44.975723+0000","last_peered":"2024-09-21T19:14:44.975723+0000","last_clean":"2024-09-21T19:14:44.975723+0000","last_became_active":"2024-09-21T19:05:42.910632+0000","last_became_peered":"2024-09-21T19:05:42.910632+0000","last_unstale":"2024-09-21T19:14:44.975723+0000","last_undegraded":"2024-09-21T19:14:44.975723+0000","last_fullsized":"2024-09-21T19:14:44.975723+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'21","last_scrub_stamp":"2024-09-21T19:13:58.757236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:58.757236+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:14.998365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1020,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.072624+0000","last_change":"2024-09-21T19:14:12.080118+0000","last_active":"2024-09-21T19:14:45.072624+0000","last_peered":"2024-09-21T19:14:45.072624+0000","last_clean":"2024-09-21T19:14:45.072624+0000","last_became_active":"2024-09-21T18:53:07.405909+0000","last_became_peered":"2024-09-21T18:53:07.405909+0000","last_unstale":"2024-09-21T19:14:45.072624+0000","last_undegraded":"2024-09-21T19:14:45.072624+0000","last_fullsized":"2024-09-21T19:14:45.072624+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:12.080074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:12.080074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:31.509730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1013,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856807+0000","last_change":"2024-09-21T19:13:25.223371+0000","last_active":"2024-09-21T19:14:44.856807+0000","last_peered":"2024-09-21T19:14:44.856807+0000","last_clean":"2024-09-21T19:14:44.856807+0000","last_became_active":"2024-09-21T18:55:51.515654+0000","last_became_peered":"2024-09-21T18:55:51.515654+0000","last_unstale":"2024-09-21T19:14:44.856807+0000","last_undegraded":"2024-09-21T19:14:44.856807+0000","last_fullsized":"2024-09-21T19:14:44.856807+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:25.223303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:25.223303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":897,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.153927+0000","last_change":"2024-09-21T19:14:34.028328+0000","last_active":"2024-09-21T19:14:45.153927+0000","last_peered":"2024-09-21T19:14:45.153927+0000","last_clean":"2024-09-21T19:14:45.153927+0000","last_became_active":"2024-09-21T19:13:55.926284+0000","last_became_peered":"2024-09-21T19:13:55.926284+0000","last_unstale":"2024-09-21T19:14:45.153927+0000","last_undegraded":"2024-09-21T19:14:45.153927+0000","last_fullsized":"2024-09-21T19:14:45.153927+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:34.028260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:34.028260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:48.787994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":182,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856068+0000","last_change":"2024-09-21T19:13:50.409307+0000","last_active":"2024-09-21T19:14:44.856068+0000","last_peered":"2024-09-21T19:14:44.856068+0000","last_clean":"2024-09-21T19:14:44.856068+0000","last_became_active":"2024-09-21T19:11:49.139887+0000","last_became_peered":"2024-09-21T19:11:49.139887+0000","last_unstale":"2024-09-21T19:14:44.856068+0000","last_undegraded":"2024-09-21T19:14:44.856068+0000","last_fullsized":"2024-09-21T19:14:44.856068+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-21T19:13:50.409255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:50:56.519979+0000","last_clean_scrub_stamp":"2024-09-21T19:13:50.409255+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:14.987768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"490'27","reported_seq":1057,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.856068+0000","last_change":"2024-09-21T19:14:34.805044+0000","last_active":"2024-09-21T19:14:44.856068+0000","last_peered":"2024-09-21T19:14:44.856068+0000","last_clean":"2024-09-21T19:14:44.856068+0000","last_became_active":"2024-09-21T19:02:40.881882+0000","last_became_peered":"2024-09-21T19:02:40.881882+0000","last_unstale":"2024-09-21T19:14:44.856068+0000","last_undegraded":"2024-09-21T19:14:44.856068+0000","last_fullsized":"2024-09-21T19:14:44.856068+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:14:34.804991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:34.804991+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:45.990196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":74,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.662309+0000","last_change":"2024-09-21T19:13:49.927583+0000","last_active":"2024-09-21T19:14:45.662309+0000","last_peered":"2024-09-21T19:14:45.662309+0000","last_clean":"2024-09-21T19:14:45.662309+0000","last_became_active":"2024-09-21T19:13:41.868259+0000","last_became_peered":"2024-09-21T19:13:41.868259+0000","last_unstale":"2024-09-21T19:14:45.662309+0000","last_undegraded":"2024-09-21T19:14:45.662309+0000","last_fullsized":"2024-09-21T19:14:45.662309+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:49.927510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:49.927510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:57.501058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1040,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071392+0000","last_change":"2024-09-21T19:13:57.970268+0000","last_active":"2024-09-21T19:14:45.071392+0000","last_peered":"2024-09-21T19:14:45.071392+0000","last_clean":"2024-09-21T19:14:45.071392+0000","last_became_active":"2024-09-21T19:02:40.887575+0000","last_became_peered":"2024-09-21T19:02:40.887575+0000","last_unstale":"2024-09-21T19:14:45.071392+0000","last_undegraded":"2024-09-21T19:14:45.071392+0000","last_fullsized":"2024-09-21T19:14:45.071392+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:57.970213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:57.970213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:12.056921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1020,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.857036+0000","last_change":"2024-09-21T19:14:06.844136+0000","last_active":"2024-09-21T19:14:44.857036+0000","last_peered":"2024-09-21T19:14:44.857036+0000","last_clean":"2024-09-21T19:14:44.857036+0000","last_became_active":"2024-09-21T19:02:40.877262+0000","last_became_peered":"2024-09-21T19:02:40.877262+0000","last_unstale":"2024-09-21T19:14:44.857036+0000","last_undegraded":"2024-09-21T19:14:44.857036+0000","last_fullsized":"2024-09-21T19:14:44.857036+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:06.844093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:06.844093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:14.591231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"489'12","reported_seq":1050,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.854966+0000","last_change":"2024-09-21T19:14:21.909131+0000","last_active":"2024-09-21T19:14:44.854966+0000","last_peered":"2024-09-21T19:14:44.854966+0000","last_clean":"2024-09-21T19:14:44.854966+0000","last_became_active":"2024-09-21T19:02:59.639831+0000","last_became_peered":"2024-09-21T19:02:59.639831+0000","last_unstale":"2024-09-21T19:14:44.854966+0000","last_undegraded":"2024-09-21T19:14:44.854966+0000","last_fullsized":"2024-09-21T19:14:44.854966+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'12","last_scrub_stamp":"2024-09-21T19:14:21.909069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:14:21.909069+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:48.183106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":871,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154673+0000","last_change":"2024-09-21T19:13:55.925158+0000","last_active":"2024-09-21T19:14:45.154673+0000","last_peered":"2024-09-21T19:14:45.154673+0000","last_clean":"2024-09-21T19:14:45.154673+0000","last_became_active":"2024-09-21T19:13:55.924905+0000","last_became_peered":"2024-09-21T19:13:55.924905+0000","last_unstale":"2024-09-21T19:14:45.154673+0000","last_undegraded":"2024-09-21T19:14:45.154673+0000","last_fullsized":"2024-09-21T19:14:45.154673+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:52.925166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:13:52.925166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.600131+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":925,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.160873+0000","last_change":"2024-09-21T19:14:41.030909+0000","last_active":"2024-09-21T19:14:45.160873+0000","last_peered":"2024-09-21T19:14:45.160873+0000","last_clean":"2024-09-21T19:14:45.160873+0000","last_became_active":"2024-09-21T19:02:40.881131+0000","last_became_peered":"2024-09-21T19:02:40.881131+0000","last_unstale":"2024-09-21T19:14:45.160873+0000","last_undegraded":"2024-09-21T19:14:45.160873+0000","last_fullsized":"2024-09-21T19:14:45.160873+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:41.030849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:14:41.030849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:51.166190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1015,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.071750+0000","last_change":"2024-09-21T19:14:09.047666+0000","last_active":"2024-09-21T19:14:45.071750+0000","last_peered":"2024-09-21T19:14:45.071750+0000","last_clean":"2024-09-21T19:14:45.071750+0000","last_became_active":"2024-09-21T19:05:42.903997+0000","last_became_peered":"2024-09-21T19:05:42.903997+0000","last_unstale":"2024-09-21T19:14:45.071750+0000","last_undegraded":"2024-09-21T19:14:45.071750+0000","last_fullsized":"2024-09-21T19:14:45.071750+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:09.047596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:09.047596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:16.779417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1034,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.975999+0000","last_change":"2024-09-21T19:14:06.807070+0000","last_active":"2024-09-21T19:14:44.975999+0000","last_peered":"2024-09-21T19:14:44.975999+0000","last_clean":"2024-09-21T19:14:44.975999+0000","last_became_active":"2024-09-21T19:13:41.871213+0000","last_became_peered":"2024-09-21T19:13:41.871213+0000","last_unstale":"2024-09-21T19:14:44.975999+0000","last_undegraded":"2024-09-21T19:14:44.975999+0000","last_fullsized":"2024-09-21T19:14:44.975999+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:06.807020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:12.406226+0000","last_clean_scrub_stamp":"2024-09-21T19:14:06.807020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:15.583546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1028,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:44.855295+0000","last_change":"2024-09-21T19:14:31.782570+0000","last_active":"2024-09-21T19:14:44.855295+0000","last_peered":"2024-09-21T19:14:44.855295+0000","last_clean":"2024-09-21T19:14:44.855295+0000","last_became_active":"2024-09-21T18:55:51.515796+0000","last_became_peered":"2024-09-21T18:55:51.515796+0000","last_unstale":"2024-09-21T19:14:44.855295+0000","last_undegraded":"2024-09-21T19:14:44.855295+0000","last_fullsized":"2024-09-21T19:14:44.855295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:14:31.782517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:06.375817+0000","last_clean_scrub_stamp":"2024-09-21T19:14:31.782517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:36.257793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"490'27","reported_seq":1056,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.161785+0000","last_change":"2024-09-21T19:13:54.058371+0000","last_active":"2024-09-21T19:14:45.161785+0000","last_peered":"2024-09-21T19:14:45.161785+0000","last_clean":"2024-09-21T19:14:45.161785+0000","last_became_active":"2024-09-21T19:13:41.874446+0000","last_became_peered":"2024-09-21T19:13:41.874446+0000","last_unstale":"2024-09-21T19:14:45.161785+0000","last_undegraded":"2024-09-21T19:14:45.161785+0000","last_fullsized":"2024-09-21T19:14:45.161785+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'27","last_scrub_stamp":"2024-09-21T19:13:54.058294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:08.386251+0000","last_clean_scrub_stamp":"2024-09-21T19:13:54.058294+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:14:58.870576+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":893,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-09-21T19:14:45.154917+0000","last_change":"2024-09-21T19:13:57.805964+0000","last_active":"2024-09-21T19:14:45.154917+0000","last_peered":"2024-09-21T19:14:45.154917+0000","last_clean":"2024-09-21T19:14:45.154917+0000","last_became_active":"2024-09-21T19:13:55.928191+0000","last_became_peered":"2024-09-21T19:13:55.928191+0000","last_unstale":"2024-09-21T19:14:45.154917+0000","last_undegraded":"2024-09-21T19:14:45.154917+0000","last_fullsized":"2024-09-21T19:14:45.154917+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T19:13:57.805917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:53:10.392064+0000","last_clean_scrub_stamp":"2024-09-21T19:13:57.805917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T19:15:05.549998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1527,"num_read_kb":1284,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":6766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1287,"num_read_kb":1282,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266953,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830492,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92914660,"statfs":{"total":95995035648,"available":95144611840,"internally_reserved":0,"allocated":1052672,"data_stored":552380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":849344970},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":261,"seq":1120986464404,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515076,"kb_used_data":1332,"kb_used_omap":4,"kb_used_meta":513723,"kb_avail":93230076,"statfs":{"total":95995035648,"available":95467597824,"internally_reserved":0,"allocated":1363968,"data_stored":879411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":526053028},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":348,"seq":1494648619105,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383624,"kb_used_data":1268,"kb_used_omap":4,"kb_used_meta":382331,"kb_avail":93361528,"statfs":{"total":95995035648,"available":95602204672,"internally_reserved":0,"allocated":1298432,"data_stored":814329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":391507619},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019159,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686112,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93059040,"statfs":{"total":95995035648,"available":95292456960,"internally_reserved":0,"allocated":1060864,"data_stored":552880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182684,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894956,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92850196,"statfs":{"total":95995035648,"available":95078600704,"internally_reserved":0,"allocated":1667072,"data_stored":1143547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":914749898},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411618,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":754384,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92990768,"statfs":{"total":95995035648,"available":95222546432,"internally_reserved":0,"allocated":1306624,"data_stored":814884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906876,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838276,"statfs":{"total":95995035648,"available":95066394624,"internally_reserved":0,"allocated":1085440,"data_stored":553173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961312,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92783840,"statfs":{"total":95995035648,"available":95010652160,"internally_reserved":0,"allocated":1384448,"data_stored":880749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T19:14:51.565 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T19:14:51.565 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe shell --fsid d839dc86-7849-11ef-baf6-efdc52797490 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T19:14:51.811 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[39192]: from='client.17172 v1:172.21.15.81:0/3414467017' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[39192]: 3.1b scrub starts 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[39192]: 3.1b scrub ok 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[39192]: 4.7 scrub starts 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[39192]: 4.7 scrub ok 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[29384]: from='client.17172 v1:172.21.15.81:0/3414467017' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:14:52.194 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[29384]: 3.1b scrub starts 2024-09-21T19:14:52.195 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[29384]: 3.1b scrub ok 2024-09-21T19:14:52.195 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[29384]: 4.7 scrub starts 2024-09-21T19:14:52.195 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-mon[29384]: 4.7 scrub ok 2024-09-21T19:14:52.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:52 smithi104 ceph-mon[32438]: from='client.17172 v1:172.21.15.81:0/3414467017' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T19:14:52.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:52 smithi104 ceph-mon[32438]: 3.1b scrub starts 2024-09-21T19:14:52.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:52 smithi104 ceph-mon[32438]: 3.1b scrub ok 2024-09-21T19:14:52.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:52 smithi104 ceph-mon[32438]: 4.7 scrub starts 2024-09-21T19:14:52.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:52 smithi104 ceph-mon[32438]: 4.7 scrub ok 2024-09-21T19:14:52.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:19:14:52] ENGINE Bus STOPPING 2024-09-21T19:14:52.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:19:14:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T19:14:52.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:19:14:52] ENGINE Bus STOPPED 2024-09-21T19:14:52.816 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:19:14:52] ENGINE Bus STARTING 2024-09-21T19:14:53.132 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:19:14:52] ENGINE Serving on http://:::9283 2024-09-21T19:14:53.132 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:52 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: [21/Sep/2024:19:14:52] ENGINE Bus STARTED 2024-09-21T19:14:53.160 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T19:14:53.169 INFO:tasks.cephadm:Teardown begin 2024-09-21T19:14:53.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T19:14:53.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T19:14:53.281 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T19:14:53.281 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T19:14:53.321 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T19:14:53.340 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T19:14:53.340 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T19:14:53.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a 2024-09-21T19:14:53.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: pgmap v1099: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T19:14:53.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: 5.16 scrub starts 2024-09-21T19:14:53.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: 5.16 scrub ok 2024-09-21T19:14:53.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: 4.b scrub starts 2024-09-21T19:14:53.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: 4.b scrub ok 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/609163053' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='client.? v1:172.21.15.81:0/609163053' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[29384]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: pgmap v1099: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: 5.16 scrub starts 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: 5.16 scrub ok 2024-09-21T19:14:53.384 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: 4.b scrub starts 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: 4.b scrub ok 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/609163053' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='client.? v1:172.21.15.81:0/609163053' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:53.385 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-mon[39192]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: pgmap v1099: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: 5.16 scrub starts 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: 5.16 scrub ok 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: 4.b scrub starts 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: 4.b scrub ok 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/609163053' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='client.? v1:172.21.15.81:0/609163053' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='mgr.14655 ' entity='mgr.y' 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T19:14:53.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:53 smithi104 ceph-mon[32438]: from='mgr.14655 v1:172.21.15.81:0/4163664256' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T19:14:53.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 systemd[1]: Stopping Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:14:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T19:14:53.658+0000 7ff1b49e9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:14:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a[29380]: 2024-09-21T19:14:53.658+0000 7ff1b49e9640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T19:14:54.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:53 smithi081 podman[208652]: 2024-09-21 19:14:53.760676008 +0000 UTC m=+0.206304597 container died 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:14:54.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:54 smithi081 podman[208652]: 2024-09-21 19:14:54.292480074 +0000 UTC m=+0.738108688 container cleanup 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T19:14:54.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:54 smithi081 bash[208652]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a 2024-09-21T19:14:54.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T19:14:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:54 smithi081 podman[208664]: 2024-09-21 19:14:54.535873612 +0000 UTC m=+0.773075304 container remove 49485a6a8237f911c2d4662cb22460b8d5c7fd0f826162abfc135544908bae86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True) 2024-09-21T19:14:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:54 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T19:14:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:54 smithi081 systemd[1]: Stopped Ceph mon.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:14:54.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 21 19:14:54 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.a.service: Consumed 30.490s CPU time. 2024-09-21T19:14:54.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:14:54.868 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T19:14:54.868 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T19:14:54.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.c 2024-09-21T19:14:55.191 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:54 smithi081 systemd[1]: Stopping Ceph mon.c for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:14:55.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:55 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c[39188]: 2024-09-21T19:14:55.189+0000 7f14fcba1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:14:55.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:55 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c[39188]: 2024-09-21T19:14:55.189+0000 7f14fcba1640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T19:14:55.566 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:55 smithi081 podman[208773]: 2024-09-21 19:14:55.35262114 +0000 UTC m=+0.299236957 container died 003e5b6bfcba94bdb53bde6a24381279822a609ecd7fea3a0a3b9aa4e7688747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T19:14:56.009 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:55 smithi081 podman[208773]: 2024-09-21 19:14:55.723074817 +0000 UTC m=+0.669690638 container cleanup 003e5b6bfcba94bdb53bde6a24381279822a609ecd7fea3a0a3b9aa4e7688747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T19:14:56.009 INFO:journalctl@ceph.mon.c.smithi081.stdout:Sep 21 19:14:55 smithi081 bash[208773]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-c 2024-09-21T19:14:56.146 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T19:14:56.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:14:56.182 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T19:14:56.183 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T19:14:56.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.b 2024-09-21T19:14:56.506 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:56 smithi104 systemd[1]: Stopping Ceph mon.b for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:14:56.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-b[32433]: 2024-09-21T19:14:56.502+0000 7ff61e7ce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:14:56.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-b[32433]: 2024-09-21T19:14:56.502+0000 7ff61e7ce640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T19:14:56.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:56 smithi104 podman[154133]: 2024-09-21 19:14:56.708021047 +0000 UTC m=+0.312840614 container died 70529628d81389e8318cab5aae4faf354bd82d1117839e67ed1e53df6d16a5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:14:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:57 smithi104 podman[154133]: 2024-09-21 19:14:57.178498934 +0000 UTC m=+0.783318490 container cleanup 70529628d81389e8318cab5aae4faf354bd82d1117839e67ed1e53df6d16a5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-b, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T19:14:57.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 19:14:57 smithi104 bash[154133]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-mon-b 2024-09-21T19:14:57.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: ::ffff:172.21.15.104 - - [21/Sep/2024:19:14:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T19:14:57.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T19:14:57.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:14:57.652 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T19:14:57.652 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T19:14:57.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y 2024-09-21T19:14:57.980 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:57 smithi081 systemd[1]: Stopping Ceph mgr.y for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:14:57.980 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T19:14:57.977+0000 7f9c877db640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:14:58.297 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:57 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y[29612]: 2024-09-21T19:14:57.977+0000 7f9c877db640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T19:14:58.297 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:58 smithi081 podman[208893]: 2024-09-21 19:14:58.009559105 +0000 UTC m=+0.171291941 container died 03c9b587d393ad02898829aee9d2be18bc7212a435a7bd59c3d71d37b66d1105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T19:14:58.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:58 smithi081 podman[208893]: 2024-09-21 19:14:58.495281373 +0000 UTC m=+0.657014250 container cleanup 03c9b587d393ad02898829aee9d2be18bc7212a435a7bd59c3d71d37b66d1105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0ecbb435437226bc091a0c2d4549ba25bf724efe, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True) 2024-09-21T19:14:58.566 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Sep 21 19:14:58 smithi081 bash[208893]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-y 2024-09-21T19:14:58.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T19:14:59.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:14:59.007 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T19:14:59.007 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T19:14:59.007 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.x 2024-09-21T19:14:59.333 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 19:14:59 smithi104 systemd[1]: Stopping Ceph mgr.x for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:14:59.644 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 19:14:59 smithi104 podman[154255]: 2024-09-21 19:14:59.331389096 +0000 UTC m=+0.129318904 container died 571615ecb52535a5d695d8641f69f75e8f1929243e255e2b0e879cd5bc787f14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-09-21T19:15:00.006 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 19:14:59 smithi104 podman[154255]: 2024-09-21 19:14:59.867290994 +0000 UTC m=+0.665220794 container cleanup 571615ecb52535a5d695d8641f69f75e8f1929243e255e2b0e879cd5bc787f14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T19:15:00.007 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 19:14:59 smithi104 bash[154255]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-mgr-x 2024-09-21T19:15:00.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T19:15:00.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:00.309 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T19:15:00.309 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T19:15:00.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.0 2024-09-21T19:15:00.610 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:00 smithi081 systemd[1]: Stopping Ceph osd.0 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:01.065 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:00 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0[42449]: 2024-09-21T19:15:00.608+0000 7fdffaef4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:01.066 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:00 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0[42449]: 2024-09-21T19:15:00.608+0000 7fdffaef4640 -1 osd.0 531 *** Got signal Terminated *** 2024-09-21T19:15:01.066 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:00 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0[42449]: 2024-09-21T19:15:00.608+0000 7fdffaef4640 -1 osd.0 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:06.565 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:06 smithi081 podman[209014]: 2024-09-21 19:15:06.268160416 +0000 UTC m=+5.766520818 container died e2e97cff78601cf65c8e6bac22f371f725a42e10eb7ef55cc31f42fddcd1362c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T19:15:07.004 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:06 smithi081 podman[209014]: 2024-09-21 19:15:06.759300689 +0000 UTC m=+6.257661126 container cleanup e2e97cff78601cf65c8e6bac22f371f725a42e10eb7ef55cc31f42fddcd1362c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T19:15:07.004 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:06 smithi081 bash[209014]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0 2024-09-21T19:15:07.316 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209030]: 2024-09-21 19:15:07.002573176 +0000 UTC m=+0.728499239 container remove e2e97cff78601cf65c8e6bac22f371f725a42e10eb7ef55cc31f42fddcd1362c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T19:15:07.578 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209118]: 2024-09-21 19:15:07.227113066 +0000 UTC m=+0.016176276 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:07.579 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209118]: 2024-09-21 19:15:07.328576579 +0000 UTC m=+0.117639770 container create 51b083c412c5e7b10cb9182ee8482dfe8db76b09f2ecb252bf854379651f4eb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T19:15:07.579 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209118]: 2024-09-21 19:15:07.572782818 +0000 UTC m=+0.361846022 container init 51b083c412c5e7b10cb9182ee8482dfe8db76b09f2ecb252bf854379651f4eb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T19:15:07.579 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209118]: 2024-09-21 19:15:07.576934306 +0000 UTC m=+0.365997504 container start 51b083c412c5e7b10cb9182ee8482dfe8db76b09f2ecb252bf854379651f4eb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T19:15:07.866 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209118]: 2024-09-21 19:15:07.706321726 +0000 UTC m=+0.495384930 container attach 51b083c412c5e7b10cb9182ee8482dfe8db76b09f2ecb252bf854379651f4eb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-21T19:15:08.202 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 21 19:15:07 smithi081 podman[209118]: 2024-09-21 19:15:07.865333936 +0000 UTC m=+0.654397143 container died 51b083c412c5e7b10cb9182ee8482dfe8db76b09f2ecb252bf854379651f4eb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T19:15:08.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T19:15:08.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:08.618 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T19:15:08.618 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T19:15:08.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.1 2024-09-21T19:15:08.816 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:08 smithi081 systemd[1]: Stopping Ceph osd.1 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:09.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:08 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1[48235]: 2024-09-21T19:15:08.963+0000 7fdd885a8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:09.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:08 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1[48235]: 2024-09-21T19:15:08.963+0000 7fdd885a8640 -1 osd.1 531 *** Got signal Terminated *** 2024-09-21T19:15:09.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:08 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1[48235]: 2024-09-21T19:15:08.963+0000 7fdd885a8640 -1 osd.1 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:14.816 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:14 smithi081 podman[209231]: 2024-09-21 19:15:14.55307518 +0000 UTC m=+5.697716260 container died 0b45315986b745a13899c3a9dcff1a47717f9a5b77aa80cfb747cf87f56d1e88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T19:15:15.315 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 podman[209231]: 2024-09-21 19:15:15.022663771 +0000 UTC m=+6.167304847 container cleanup 0b45315986b745a13899c3a9dcff1a47717f9a5b77aa80cfb747cf87f56d1e88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T19:15:15.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 bash[209231]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1 2024-09-21T19:15:15.683 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 podman[209248]: 2024-09-21 19:15:15.324454687 +0000 UTC m=+0.765265159 container remove 0b45315986b745a13899c3a9dcff1a47717f9a5b77aa80cfb747cf87f56d1e88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T19:15:16.013 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 podman[209334]: 2024-09-21 19:15:15.584152976 +0000 UTC m=+0.015779557 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:16.013 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 podman[209334]: 2024-09-21 19:15:15.691928104 +0000 UTC m=+0.123554670 container create 61c97d9e48e29d2be9c3a289156069414ccce5beb7086bf6949834313a51033e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T19:15:16.013 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 podman[209334]: 2024-09-21 19:15:15.911027242 +0000 UTC m=+0.342653815 container init 61c97d9e48e29d2be9c3a289156069414ccce5beb7086bf6949834313a51033e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T19:15:16.013 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:15 smithi081 podman[209334]: 2024-09-21 19:15:15.914672676 +0000 UTC m=+0.346299242 container start 61c97d9e48e29d2be9c3a289156069414ccce5beb7086bf6949834313a51033e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T19:15:16.013 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:16 smithi081 podman[209334]: 2024-09-21 19:15:16.011070528 +0000 UTC m=+0.442697112 container attach 61c97d9e48e29d2be9c3a289156069414ccce5beb7086bf6949834313a51033e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T19:15:16.316 INFO:journalctl@ceph.osd.1.smithi081.stdout:Sep 21 19:15:16 smithi081 podman[209334]: 2024-09-21 19:15:16.101642991 +0000 UTC m=+0.533269560 container died 61c97d9e48e29d2be9c3a289156069414ccce5beb7086bf6949834313a51033e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T19:15:16.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T19:15:16.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:16.862 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T19:15:16.862 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T19:15:16.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.2 2024-09-21T19:15:17.065 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:16 smithi081 systemd[1]: Stopping Ceph osd.2 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:17.565 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2[54024]: 2024-09-21T19:15:17.192+0000 7ffafad2a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:17.566 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2[54024]: 2024-09-21T19:15:17.192+0000 7ffafad2a640 -1 osd.2 531 *** Got signal Terminated *** 2024-09-21T19:15:17.566 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:17 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2[54024]: 2024-09-21T19:15:17.192+0000 7ffafad2a640 -1 osd.2 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:19.757 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.387Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:19.757 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.389Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:19.757 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.397Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:19.757 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.397Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:19.757 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.397Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:19.757 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.397Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:19.758 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:15:19 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:15:19.397Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.81:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.81:8765: connect: connection refused" 2024-09-21T19:15:23.066 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:22 smithi081 podman[209434]: 2024-09-21 19:15:22.805227659 +0000 UTC m=+5.721483578 container died 2e84d77441c55a95365eb8e4c89a3cc504d09b8eb5437603f434876952a6011e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T19:15:23.522 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:23 smithi081 podman[209434]: 2024-09-21 19:15:23.159873504 +0000 UTC m=+6.076129406 container cleanup 2e84d77441c55a95365eb8e4c89a3cc504d09b8eb5437603f434876952a6011e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T19:15:23.522 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:23 smithi081 bash[209434]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2 2024-09-21T19:15:23.815 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:23 smithi081 podman[209451]: 2024-09-21 19:15:23.519949251 +0000 UTC m=+0.710033064 container remove 2e84d77441c55a95365eb8e4c89a3cc504d09b8eb5437603f434876952a6011e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T19:15:24.141 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:23 smithi081 podman[209540]: 2024-09-21 19:15:23.729876284 +0000 UTC m=+0.015560433 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:24.141 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:23 smithi081 podman[209540]: 2024-09-21 19:15:23.845839848 +0000 UTC m=+0.131523982 container create e58ffb9db81b9e0e0736e723633ee9e46a476dd991c10aef3f9f528a0f5f4ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T19:15:24.141 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:24 smithi081 podman[209540]: 2024-09-21 19:15:24.056386189 +0000 UTC m=+0.342070322 container init e58ffb9db81b9e0e0736e723633ee9e46a476dd991c10aef3f9f528a0f5f4ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T19:15:24.142 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:24 smithi081 podman[209540]: 2024-09-21 19:15:24.059839512 +0000 UTC m=+0.345523654 container start e58ffb9db81b9e0e0736e723633ee9e46a476dd991c10aef3f9f528a0f5f4ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43) 2024-09-21T19:15:24.142 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:24 smithi081 podman[209540]: 2024-09-21 19:15:24.139988962 +0000 UTC m=+0.425673120 container attach e58ffb9db81b9e0e0736e723633ee9e46a476dd991c10aef3f9f528a0f5f4ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2) 2024-09-21T19:15:24.536 INFO:journalctl@ceph.osd.2.smithi081.stdout:Sep 21 19:15:24 smithi081 podman[209540]: 2024-09-21 19:15:24.246945497 +0000 UTC m=+0.532629647 container died e58ffb9db81b9e0e0736e723633ee9e46a476dd991c10aef3f9f528a0f5f4ec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:15:24.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T19:15:24.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:24.874 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T19:15:24.874 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T19:15:24.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.3 2024-09-21T19:15:25.223 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:24 smithi081 systemd[1]: Stopping Ceph osd.3 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:25.566 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:25 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:25.220+0000 7fc72e610640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:25.566 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:25 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:25.220+0000 7fc72e610640 -1 osd.3 531 *** Got signal Terminated *** 2024-09-21T19:15:25.566 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:25 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:25.220+0000 7fc72e610640 -1 osd.3 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:27.315 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:26 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:26.844+0000 7fc726c1d640 -1 osd.3 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:02.295732+0000 front 2024-09-21T19:15:02.295754+0000 (oldest deadline 2024-09-21T19:15:26.395884+0000) 2024-09-21T19:15:27.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:27 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:27.061+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:28.316 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:27 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:27.860+0000 7fc726c1d640 -1 osd.3 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:02.295732+0000 front 2024-09-21T19:15:02.295754+0000 (oldest deadline 2024-09-21T19:15:26.395884+0000) 2024-09-21T19:15:28.506 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:28.034+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:28.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:28 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:28.020+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:29.315 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:28 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:28.850+0000 7fc726c1d640 -1 osd.3 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:02.295732+0000 front 2024-09-21T19:15:02.295754+0000 (oldest deadline 2024-09-21T19:15:26.395884+0000) 2024-09-21T19:15:29.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:29 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:29.036+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:29.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:29 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:29.042+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:29.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:29 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:29.007+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:30.061 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:30 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:30.015+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:30.061 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:30 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:30.001+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:30.061 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:29 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:29.795+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:30.315 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:29 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3[59406]: 2024-09-21T19:15:29.882+0000 7fc726c1d640 -1 osd.3 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:02.295732+0000 front 2024-09-21T19:15:02.295754+0000 (oldest deadline 2024-09-21T19:15:26.395884+0000) 2024-09-21T19:15:30.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:30 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:30.058+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:31.032 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:30 smithi081 podman[209641]: 2024-09-21 19:15:30.760129834 +0000 UTC m=+5.665849118 container died 6adb212949f2008841011e0522f2531aecc99e3f872fc6ed2726e04941e4920b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T19:15:31.073 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:31.028+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:31.073 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:30 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:30.801+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:31.073 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:31.038+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:31.315 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:31 smithi081 podman[209641]: 2024-09-21 19:15:31.229498036 +0000 UTC m=+6.135217306 container cleanup 6adb212949f2008841011e0522f2531aecc99e3f872fc6ed2726e04941e4920b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T19:15:31.316 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:31 smithi081 bash[209641]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3 2024-09-21T19:15:31.506 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:31.071+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:31.815 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:31 smithi081 podman[209657]: 2024-09-21 19:15:31.523095454 +0000 UTC m=+0.766690495 container remove 6adb212949f2008841011e0522f2531aecc99e3f872fc6ed2726e04941e4920b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T19:15:32.031 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:31 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:31.771+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:32.031 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:32 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:32.000+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:32.161 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:31 smithi081 podman[209744]: 2024-09-21 19:15:31.737428513 +0000 UTC m=+0.016162413 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:32.161 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:31 smithi081 podman[209744]: 2024-09-21 19:15:31.84050791 +0000 UTC m=+0.119241849 container create d4708ebb7d5dd642cbc7fd12d207e8c3d7e90371ee256ccaa77bc1f18e500667 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:15:32.161 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:32 smithi081 podman[209744]: 2024-09-21 19:15:32.075875776 +0000 UTC m=+0.354609672 container init d4708ebb7d5dd642cbc7fd12d207e8c3d7e90371ee256ccaa77bc1f18e500667 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T19:15:32.161 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:32 smithi081 podman[209744]: 2024-09-21 19:15:32.079474951 +0000 UTC m=+0.358208842 container start d4708ebb7d5dd642cbc7fd12d207e8c3d7e90371ee256ccaa77bc1f18e500667 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T19:15:32.161 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:32 smithi081 podman[209744]: 2024-09-21 19:15:32.159395347 +0000 UTC m=+0.438129265 container attach d4708ebb7d5dd642cbc7fd12d207e8c3d7e90371ee256ccaa77bc1f18e500667 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T19:15:32.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:32 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:32.029+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:32.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:32 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:32.092+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:32.566 INFO:journalctl@ceph.osd.3.smithi081.stdout:Sep 21 19:15:32 smithi081 podman[209744]: 2024-09-21 19:15:32.27429031 +0000 UTC m=+0.553024201 container died d4708ebb7d5dd642cbc7fd12d207e8c3d7e90371ee256ccaa77bc1f18e500667 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T19:15:32.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T19:15:33.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:33.020 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T19:15:33.020 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T19:15:33.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.4 2024-09-21T19:15:33.063 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:32 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:32.802+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:33.063 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:33.037+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:33.336 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:33.073+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:33.337 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:33 smithi104 systemd[1]: Stopping Ceph osd.4 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:33.337 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:33.061+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:33.756 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:33.334+0000 7fd1b099d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:33.756 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:33.334+0000 7fd1b099d640 -1 osd.4 531 *** Got signal Terminated *** 2024-09-21T19:15:33.756 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:33.334+0000 7fd1b099d640 -1 osd.4 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:34.040 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:33 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:33.763+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:34.506 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:34.121+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:34.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:34.062+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:34.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:34.038+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:35.006 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:34 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:34.747+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:35.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:35 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:35.090+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:35.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:35 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:35.075+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:35.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:35 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:35.056+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:36.058 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:35 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:35.770+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:36.058 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:35 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:35.770+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:36.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:36.115+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:36.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:36.056+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:36.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:36.096+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:37.038 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:36.773+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:37.038 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:36.773+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:37.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:37.101+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:37.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:37.049+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:37.507 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:37.036+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:38.030 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:38.028+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:38.030 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:37.763+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:38.030 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:37 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:37.763+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:38.511 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:38.051+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:38.514 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:38.081+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.202776+0000 front 2024-09-21T19:15:05.202958+0000 (oldest deadline 2024-09-21T19:15:28.702854+0000) 2024-09-21T19:15:38.514 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4[36529]: 2024-09-21T19:15:38.081+0000 7fd1a8faa640 -1 osd.4 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.804870+0000 front 2024-09-21T19:15:13.804841+0000 (oldest deadline 2024-09-21T19:15:37.304932+0000) 2024-09-21T19:15:39.067 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:38 smithi104 podman[154391]: 2024-09-21 19:15:38.987756371 +0000 UTC m=+5.766500621 container died d964a58c4df2113f51414962994cf971e211dad7a599b462f3e3836da4cbf72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T19:15:39.067 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:38.760+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:39.067 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:38 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:38.760+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:39.067 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:39.004+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:39.067 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:39.004+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:39.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:39.064+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:39.758 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:39 smithi104 podman[154391]: 2024-09-21 19:15:39.425321454 +0000 UTC m=+6.204065700 container cleanup d964a58c4df2113f51414962994cf971e211dad7a599b462f3e3836da4cbf72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T19:15:39.758 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:39 smithi104 bash[154391]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4 2024-09-21T19:15:39.758 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:39 smithi104 podman[154407]: 2024-09-21 19:15:39.626936427 +0000 UTC m=+0.638010617 container remove d964a58c4df2113f51414962994cf971e211dad7a599b462f3e3836da4cbf72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T19:15:40.140 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:40.026+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:40.140 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:40.026+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:40.141 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:39.805+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:40.141 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:39.805+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:40.141 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:40 smithi104 podman[154494]: 2024-09-21 19:15:40.027752774 +0000 UTC m=+0.107387274 container create 2a0dc348cbf66cf29f9e6ed2f4416bc6513f7f31ae5aa60af4115fec841f289f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:15:40.141 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:40 smithi104 podman[154494]: 2024-09-21 19:15:39.936599209 +0000 UTC m=+0.016233710 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:40.141 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:39.997+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:40.141 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:39 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:39.997+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:40.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:40 smithi104 podman[154494]: 2024-09-21 19:15:40.272582217 +0000 UTC m=+0.352216718 container init 2a0dc348cbf66cf29f9e6ed2f4416bc6513f7f31ae5aa60af4115fec841f289f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T19:15:40.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:40 smithi104 podman[154494]: 2024-09-21 19:15:40.276116503 +0000 UTC m=+0.355751005 container start 2a0dc348cbf66cf29f9e6ed2f4416bc6513f7f31ae5aa60af4115fec841f289f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T19:15:40.507 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:40 smithi104 podman[154494]: 2024-09-21 19:15:40.346771963 +0000 UTC m=+0.426406482 container attach 2a0dc348cbf66cf29f9e6ed2f4416bc6513f7f31ae5aa60af4115fec841f289f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T19:15:40.763 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 19:15:40 smithi104 podman[154494]: 2024-09-21 19:15:40.51846695 +0000 UTC m=+0.598101452 container died 2a0dc348cbf66cf29f9e6ed2f4416bc6513f7f31ae5aa60af4115fec841f289f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T19:15:41.016 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:40.761+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:41.016 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:40.761+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:41.016 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:40.971+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:41.017 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:40 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:40.971+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:41.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T19:15:41.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:41.202 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T19:15:41.202 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T19:15:41.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5 2024-09-21T19:15:41.282 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:41.014+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:41.282 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:41.014+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:41.724 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:41 smithi104 systemd[1]: Stopping Ceph osd.5 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:41.724 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:41.528+0000 7f179a60f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:41.724 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:41.528+0000 7f179a60f640 -1 osd.5 531 *** Got signal Terminated *** 2024-09-21T19:15:41.724 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:41.528+0000 7f179a60f640 -1 osd.5 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:41.724 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:41.721+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:41.724 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:41.722+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:42.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:41.990+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:42.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:41 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:41.990+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:42.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:42 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:42.028+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:42.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:42 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:42.028+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:43.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:42 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:42.960+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:43.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:42 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:42.960+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:43.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:42 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:42.719+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:43.007 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:42 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:42.719+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:43.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:43.049+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:43.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:43.049+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:44.003 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:43.707+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:44.003 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:43.707+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:44.003 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:43.956+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:44.003 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:43 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:43.956+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:44.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:44.001+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:44.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:44.001+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:44.962 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:44.659+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:44.963 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:44.659+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:44.963 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:44.659+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.569478+0000 front 2024-09-21T19:15:19.569425+0000 (oldest deadline 2024-09-21T19:15:44.269256+0000) 2024-09-21T19:15:45.256 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:45.006+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:45.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:45.006+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:45.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:44.960+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:45.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:44.960+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:45.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:44 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:44.960+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:45.971 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:45.645+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:03.767598+0000 front 2024-09-21T19:15:03.767666+0000 (oldest deadline 2024-09-21T19:15:29.667508+0000) 2024-09-21T19:15:45.971 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:45.645+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:09.667940+0000 front 2024-09-21T19:15:09.668036+0000 (oldest deadline 2024-09-21T19:15:35.568046+0000) 2024-09-21T19:15:45.971 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5[122879]: 2024-09-21T19:15:45.645+0000 7f1792c1c640 -1 osd.5 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.569478+0000 front 2024-09-21T19:15:19.569425+0000 (oldest deadline 2024-09-21T19:15:44.269256+0000) 2024-09-21T19:15:46.256 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:45.974+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:46.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:45.974+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:46.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:45.974+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:46.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:45.969+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:46.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:45.969+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:46.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:45 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:45.969+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:47 smithi104 podman[154597]: 2024-09-21 19:15:47.135409192 +0000 UTC m=+5.713572114 container died 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:47.021+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:47.021+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:47.021+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:46 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:46.932+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:46 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:46.932+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:47.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:46 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:46.932+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:47.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:47 smithi104 podman[154597]: 2024-09-21 19:15:47.519513421 +0000 UTC m=+6.097676385 container cleanup 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T19:15:47.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:47 smithi104 bash[154597]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5 2024-09-21T19:15:48.128 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:47 smithi104 podman[154615]: 2024-09-21 19:15:47.812667352 +0000 UTC m=+0.665141189 container remove 988043902dba2242923418b3998617ecbb3d3a3636b05b0446af5f20b7cef918 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T19:15:48.129 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:48.037+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:48.129 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:48.037+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:48.129 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:48.037+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:48.129 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:47.951+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:48.129 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:47.951+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:48.129 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:47 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:47.951+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:48.400 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:48 smithi104 podman[154703]: 2024-09-21 19:15:48.029728765 +0000 UTC m=+0.015943982 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:48.401 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:48 smithi104 podman[154703]: 2024-09-21 19:15:48.158501812 +0000 UTC m=+0.144717050 container create 9c13e53c55ed53695c4523979dfe12eecf5c0cd26cf95abef4ad783408f942da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T19:15:48.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:48 smithi104 podman[154703]: 2024-09-21 19:15:48.399174834 +0000 UTC m=+0.385390030 container init 9c13e53c55ed53695c4523979dfe12eecf5c0cd26cf95abef4ad783408f942da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T19:15:48.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:48 smithi104 podman[154703]: 2024-09-21 19:15:48.40252115 +0000 UTC m=+0.388736350 container start 9c13e53c55ed53695c4523979dfe12eecf5c0cd26cf95abef4ad783408f942da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T19:15:48.757 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:48 smithi104 podman[154703]: 2024-09-21 19:15:48.474137886 +0000 UTC m=+0.460353103 container attach 9c13e53c55ed53695c4523979dfe12eecf5c0cd26cf95abef4ad783408f942da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T19:15:48.758 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 19:15:48 smithi104 podman[154703]: 2024-09-21 19:15:48.589593343 +0000 UTC m=+0.575808555 container died 9c13e53c55ed53695c4523979dfe12eecf5c0cd26cf95abef4ad783408f942da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-5-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T19:15:49.229 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:48.908+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:49.229 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:48.908+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:49.229 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:48 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:48.908+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:49.229 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:49.050+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:49.229 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:49.050+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:49.230 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:49.050+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:49.253 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T19:15:49.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:49.333 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T19:15:49.333 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T19:15:49.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6 2024-09-21T19:15:49.506 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 systemd[1]: Stopping Ceph osd.6 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:49.941 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:49.646+0000 7f4428d34640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:49.941 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:49.646+0000 7f4428d34640 -1 osd.6 531 *** Got signal Terminated *** 2024-09-21T19:15:49.942 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:49.646+0000 7f4428d34640 -1 osd.6 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:50.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:49.938+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:50.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:49.938+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:50.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:49 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:49.938+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:50.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:50 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:50.033+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:50.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:50 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:50.033+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:50.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:50 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:50.033+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:51.256 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:51.055+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:51.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:51.055+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:51.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:51.055+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:51.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:50 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:50.926+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:51.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:50 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:50.926+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:51.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:50 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:50.926+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:52.256 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:52.014+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:52.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:52.014+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:52.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:52.014+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:52.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:51.900+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:52.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:51.900+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:52.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:51.900+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:52.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:51 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:51.900+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:53.048+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:53.048+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:53.048+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:52.880+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:52.880+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:52.880+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:53.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:52 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:52.880+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:54.094+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.462487+0000 front 2024-09-21T19:15:05.462590+0000 (oldest deadline 2024-09-21T19:15:27.762446+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:54.094+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.063444+0000 front 2024-09-21T19:15:13.063420+0000 (oldest deadline 2024-09-21T19:15:38.363370+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6[101376]: 2024-09-21T19:15:54.094+0000 7f4421341640 -1 osd.6 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:18.363954+0000 front 2024-09-21T19:15:18.364022+0000 (oldest deadline 2024-09-21T19:15:44.263924+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:53.918+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:53.918+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:53.918+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:54.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:53 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:53.918+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:55.256 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:55 smithi104 podman[154805]: 2024-09-21 19:15:55.079192538 +0000 UTC m=+5.523408060 container died 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731) 2024-09-21T19:15:55.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:54.878+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:55.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:54.878+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:55.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:54.878+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:55.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:54 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:54.878+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:55.716 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:55 smithi104 podman[154805]: 2024-09-21 19:15:55.479731044 +0000 UTC m=+5.923946558 container cleanup 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43) 2024-09-21T19:15:55.716 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:55 smithi104 bash[154805]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6 2024-09-21T19:15:56.007 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:55 smithi104 podman[154833]: 2024-09-21 19:15:55.714319137 +0000 UTC m=+0.627504268 container remove 93e516c2aa807f5abda0d99239bbd034f965426b5bde40a729602cd259541ccd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T19:15:56.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:55 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:55.843+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:56.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:55 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:55.843+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:56.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:55 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:55.843+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:56.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:55 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:55.843+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:56.327 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:56 smithi104 podman[154923]: 2024-09-21 19:15:55.926739754 +0000 UTC m=+0.016220525 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:15:56.327 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:56 smithi104 podman[154923]: 2024-09-21 19:15:56.048636559 +0000 UTC m=+0.138117318 container create 77f5e6d572843ce2939e47530051b8b36a90185186dac8e21ce1963708d774ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T19:15:56.327 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:56 smithi104 podman[154923]: 2024-09-21 19:15:56.325885224 +0000 UTC m=+0.415365994 container init 77f5e6d572843ce2939e47530051b8b36a90185186dac8e21ce1963708d774ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T19:15:56.757 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:56 smithi104 podman[154923]: 2024-09-21 19:15:56.32935525 +0000 UTC m=+0.418836010 container start 77f5e6d572843ce2939e47530051b8b36a90185186dac8e21ce1963708d774ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:15:56.757 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:56 smithi104 podman[154923]: 2024-09-21 19:15:56.400876733 +0000 UTC m=+0.490357499 container attach 77f5e6d572843ce2939e47530051b8b36a90185186dac8e21ce1963708d774ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T19:15:56.757 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 19:15:56 smithi104 podman[154923]: 2024-09-21 19:15:56.520479978 +0000 UTC m=+0.609960739 container died 77f5e6d572843ce2939e47530051b8b36a90185186dac8e21ce1963708d774ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.build-date=20240731) 2024-09-21T19:15:57.123 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:56.870+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:57.123 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:56.870+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:57.123 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:56.870+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:57.123 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:56 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:56.870+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:57.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T19:15:57.187 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:15:57.187 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T19:15:57.188 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T19:15:57.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.7 2024-09-21T19:15:57.501 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 systemd[1]: Stopping Ceph osd.7 for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:15:57.501 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.498+0000 7f418bb52640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:15:57.756 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.498+0000 7f418bb52640 -1 osd.7 531 *** Got signal Terminated *** 2024-09-21T19:15:57.756 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.498+0000 7f418bb52640 -1 osd.7 531 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T19:15:58.256 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.863+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:58.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.863+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:58.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.863+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:58.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:57 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:57.863+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:15:59.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:58.891+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:15:59.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:58.891+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:15:59.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:58.891+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:15:59.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:58 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:58.891+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:16:00.256 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:59 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:59.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:16:00.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:59 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:59.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:16:00.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:59 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:59.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:16:00.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:15:59 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:15:59.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:16:01.145 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:00 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:00.817+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:16:01.145 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:00 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:00.817+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:16:01.145 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:00 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:00.817+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:16:01.145 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:00 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:00.817+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:16:02.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:01.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6803 osd.0 since back 2024-09-21T19:15:05.310278+0000 front 2024-09-21T19:15:05.310078+0000 (oldest deadline 2024-09-21T19:15:27.010194+0000) 2024-09-21T19:16:02.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:01.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6807 osd.1 since back 2024-09-21T19:15:13.411683+0000 front 2024-09-21T19:15:13.411784+0000 (oldest deadline 2024-09-21T19:15:39.311625+0000) 2024-09-21T19:16:02.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:01.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6811 osd.2 since back 2024-09-21T19:15:19.312106+0000 front 2024-09-21T19:15:19.312117+0000 (oldest deadline 2024-09-21T19:15:45.712573+0000) 2024-09-21T19:16:02.007 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:01 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7[52410]: 2024-09-21T19:16:01.848+0000 7f4188167640 -1 osd.7 531 heartbeat_check: no reply from 172.21.15.81:6815 osd.3 since back 2024-09-21T19:15:25.713096+0000 front 2024-09-21T19:15:25.713112+0000 (oldest deadline 2024-09-21T19:15:51.012978+0000) 2024-09-21T19:16:03.257 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:02 smithi104 podman[155027]: 2024-09-21 19:16:02.994963448 +0000 UTC m=+5.589086698 container died fe5d47f3a923768893cec65455058f3aa0fdb49addad66056458fa34701561d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T19:16:03.609 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:03 smithi104 podman[155027]: 2024-09-21 19:16:03.406334727 +0000 UTC m=+6.000457969 container cleanup fe5d47f3a923768893cec65455058f3aa0fdb49addad66056458fa34701561d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T19:16:03.609 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:03 smithi104 bash[155027]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7 2024-09-21T19:16:03.918 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:03 smithi104 podman[155043]: 2024-09-21 19:16:03.607481632 +0000 UTC m=+0.607329349 container remove fe5d47f3a923768893cec65455058f3aa0fdb49addad66056458fa34701561d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:16:04.246 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:03 smithi104 podman[155132]: 2024-09-21 19:16:03.916680448 +0000 UTC m=+0.111063364 container create 06ee9277958dd973613418ccf109ed76d7d34072423f312427a4031de38dbccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T19:16:04.247 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:03 smithi104 podman[155132]: 2024-09-21 19:16:03.821399849 +0000 UTC m=+0.015782776 image pull 3fa9090ef91d4e87f14d21d1361976a766f2908d39879e744bc3fd80d5762a25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5 2024-09-21T19:16:04.247 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:04 smithi104 podman[155132]: 2024-09-21 19:16:04.160716032 +0000 UTC m=+0.355098958 container init 06ee9277958dd973613418ccf109ed76d7d34072423f312427a4031de38dbccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T19:16:04.247 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:04 smithi104 podman[155132]: 2024-09-21 19:16:04.164115259 +0000 UTC m=+0.358498180 container start 06ee9277958dd973613418ccf109ed76d7d34072423f312427a4031de38dbccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:16:04.248 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:04 smithi104 podman[155132]: 2024-09-21 19:16:04.24406846 +0000 UTC m=+0.438451384 container attach 06ee9277958dd973613418ccf109ed76d7d34072423f312427a4031de38dbccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T19:16:04.507 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 19:16:04 smithi104 podman[155132]: 2024-09-21 19:16:04.351791106 +0000 UTC m=+0.546174041 container died 06ee9277958dd973613418ccf109ed76d7d34072423f312427a4031de38dbccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T19:16:04.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T19:16:04.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:16:04.890 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T19:16:04.890 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-21T19:16:04.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@rgw.foo.a 2024-09-21T19:16:05.190 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:04 smithi081 systemd[1]: Stopping Ceph rgw.foo.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:05.454 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:05 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a[63853]: 2024-09-21T19:16:05.188+0000 7f2245960640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T19:16:05.454 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:05 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a[63853]: 2024-09-21T19:16:05.188+0000 7f22491cc880 -1 shutting down 2024-09-21T19:16:15.498 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:15 smithi081 bash[209847]: time="2024-09-21T19:16:15Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-21T19:16:15.499 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:15 smithi081 podman[209847]: 2024-09-21 19:16:15.223735586 +0000 UTC m=+10.142900863 container died 76aac42e501cc7fbd4ae64d5babc3c77881bb1211771d76496aa646772ce4048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T19:16:15.815 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:15 smithi081 podman[209847]: 2024-09-21 19:16:15.687903947 +0000 UTC m=+10.607069247 container cleanup 76aac42e501cc7fbd4ae64d5babc3c77881bb1211771d76496aa646772ce4048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T19:16:15.816 INFO:journalctl@ceph.rgw.foo.a.smithi081.stdout:Sep 21 19:16:15 smithi081 bash[209847]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-rgw-foo-a 2024-09-21T19:16:16.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@rgw.foo.a.service' 2024-09-21T19:16:16.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:16:16.172 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-21T19:16:16.172 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T19:16:16.172 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d839dc86-7849-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T19:16:16.467 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 systemd[1]: Stopping Ceph prometheus.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.464Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T19:16:16.734 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.465Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T19:16:16.735 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.467Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T19:16:16.735 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.467Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T19:16:16.735 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a[65147]: ts=2024-09-21T19:16:16.467Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T19:16:16.735 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 podman[155237]: 2024-09-21 19:16:16.475137164 +0000 UTC m=+0.104581658 container died 7747803a501c1cd4b2e20b982793f87eb5dee4ba75f5a20a9b2872e4977ba154 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:17.006 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 podman[155237]: 2024-09-21 19:16:16.831136943 +0000 UTC m=+0.460581436 container cleanup 7747803a501c1cd4b2e20b982793f87eb5dee4ba75f5a20a9b2872e4977ba154 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:17.007 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 19:16:16 smithi104 bash[155237]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T19:16:17.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d839dc86-7849-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T19:16:17.283 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T19:16:17.283 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T19:16:17.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d839dc86-7849-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T19:16:17.485 INFO:teuthology.orchestra.run.smithi081.stdout:Deleting cluster with fsid: d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T19:16:19.346 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:19 smithi081 systemd[1]: Stopping Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:19.347 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:19 smithi081 ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a[78938]: ts=2024-09-21T19:16:19.293Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T19:16:19.347 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:19 smithi081 podman[210352]: 2024-09-21 19:16:19.344873011 +0000 UTC m=+0.132131620 container died 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:20.065 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:19 smithi081 podman[210352]: 2024-09-21 19:16:19.836161796 +0000 UTC m=+0.623420444 container cleanup 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:20.065 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:19 smithi081 bash[210352]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T19:16:20.515 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:20 smithi081 podman[210364]: 2024-09-21 19:16:20.089783799 +0000 UTC m=+0.741305632 container remove 272c2e22bd2a4f93ff823fd6296c29eb1005da74cbfbe4a92d1995e70820ce18 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:20.515 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:20 smithi081 podman[210364]: 2024-09-21 19:16:20.171379017 +0000 UTC m=+0.822900853 volume remove 2005a17249674d689c7c7cc70f7f5490fcc2afee88e7625862dd1d440fc6505b 2024-09-21T19:16:20.515 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:20 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T19:16:20.515 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:20 smithi081 systemd[1]: Stopped Ceph alertmanager.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:16:20.515 INFO:journalctl@ceph.alertmanager.a.smithi081.stdout:Sep 21 19:16:20 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.609s CPU time. 2024-09-21T19:16:20.811 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:20 smithi081 systemd[1]: Stopping Ceph node-exporter.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:21.065 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:20 smithi081 podman[210481]: 2024-09-21 19:16:20.810078003 +0000 UTC m=+0.156501305 container died dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:21.444 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 podman[210481]: 2024-09-21 19:16:21.183051177 +0000 UTC m=+0.529474469 container cleanup dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:21.445 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 bash[210481]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T19:16:21.696 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 podman[210493]: 2024-09-21 19:16:21.442872995 +0000 UTC m=+0.632816395 container remove dbcefa709bdd1a0b2a0e7c806bf30b8bd04fbfd93e3bfb448999ec1905d6f14e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T19:16:21.696 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T19:16:21.696 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T19:16:21.696 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 systemd[1]: Stopped Ceph node-exporter.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:16:21.696 INFO:journalctl@ceph.node-exporter.a.smithi081.stdout:Sep 21 19:16:21 smithi081 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 4.668s CPU time. 2024-09-21T19:16:22.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d839dc86-7849-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T19:16:22.434 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T19:16:24.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:23 smithi104 systemd[1]: Stopping Ceph iscsi.iscsi.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:24.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:24 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a[57274]: debug Shutdown received 2024-09-21T19:16:34.370 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 bash[155658]: time="2024-09-21T19:16:34Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-21T19:16:34.370 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 podman[155658]: 2024-09-21 19:16:34.045274849 +0000 UTC m=+10.121724805 container died f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T19:16:34.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 podman[155658]: 2024-09-21 19:16:34.517193451 +0000 UTC m=+10.593643411 container cleanup f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T19:16:34.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 bash[155658]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a 2024-09-21T19:16:35.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 podman[155672]: 2024-09-21 19:16:34.751833225 +0000 UTC m=+0.704509051 container remove f6261b5e27bc28407862912c0b89e523b75d63ea2c59c0b8ad3b707677ddaa06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b7e696d4014dc129bd30a3126ccae3c43a4a3060f7a811b44f457f7ec9f7f5d5, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-21-1014-0ecbb43, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T19:16:35.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-21T19:16:35.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-21T19:16:35.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 systemd[1]: Stopped Ceph iscsi.iscsi.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:16:35.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi104.stdout:Sep 21 19:16:34 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Consumed 2.659s CPU time. 2024-09-21T19:16:36.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:36 smithi104 systemd[1]: Stopping Ceph grafana.a for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:36.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=server t=2024-09-21T19:16:36.659522688Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-21T19:16:36.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=tracing t=2024-09-21T19:16:36.659925089Z level=info msg="Closing tracing" 2024-09-21T19:16:36.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=ticker t=2024-09-21T19:16:36.659939868Z level=info msg=stopped last_tick=2024-09-21T19:16:30Z 2024-09-21T19:16:36.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:36 smithi104 ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a[59748]: logger=grafana-apiserver t=2024-09-21T19:16:36.659996004Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-21T19:16:36.757 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:36 smithi104 podman[155937]: 2024-09-21 19:16:36.6697964 +0000 UTC m=+0.106978399 container died 7718a9b7807c22ac5f41d1e850fb63bd7622b5d37bbdfb5de2d7ff960c7d7b4c (image=quay.io/ceph/grafana:10.4.8, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T19:16:37.256 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:37 smithi104 podman[155937]: 2024-09-21 19:16:37.042476209 +0000 UTC m=+0.479658211 container cleanup 7718a9b7807c22ac5f41d1e850fb63bd7622b5d37bbdfb5de2d7ff960c7d7b4c (image=quay.io/ceph/grafana:10.4.8, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T19:16:37.257 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:37 smithi104 bash[155937]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a 2024-09-21T19:16:37.640 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:37 smithi104 podman[155948]: 2024-09-21 19:16:37.318689603 +0000 UTC m=+0.645956677 container remove 7718a9b7807c22ac5f41d1e850fb63bd7622b5d37bbdfb5de2d7ff960c7d7b4c (image=quay.io/ceph/grafana:10.4.8, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T19:16:37.640 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:37 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-21T19:16:37.640 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:37 smithi104 systemd[1]: Stopped Ceph grafana.a for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:16:37.640 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 19:16:37 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@grafana.a.service: Consumed 15.770s CPU time. 2024-09-21T19:16:37.892 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:37 smithi104 systemd[1]: Stopping Ceph node-exporter.b for d839dc86-7849-11ef-baf6-efdc52797490... 2024-09-21T19:16:38.256 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:37 smithi104 podman[156065]: 2024-09-21 19:16:37.890679587 +0000 UTC m=+0.116242820 container died a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T19:16:38.713 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 podman[156065]: 2024-09-21 19:16:38.30518679 +0000 UTC m=+0.530750009 container cleanup a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T19:16:38.713 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 bash[156065]: ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T19:16:38.714 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 podman[156078]: 2024-09-21 19:16:38.506303864 +0000 UTC m=+0.619338731 container remove a6261b9e95117c0aecd3bf44b5e0c5ee15bb7664bab652ac5c584cd4be783212 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-d839dc86-7849-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T19:16:38.714 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T19:16:39.006 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T19:16:39.006 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 systemd[1]: Stopped Ceph node-exporter.b for d839dc86-7849-11ef-baf6-efdc52797490. 2024-09-21T19:16:39.007 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 19:16:38 smithi104 systemd[1]: ceph-d839dc86-7849-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 4.591s CPU time. 2024-09-21T19:16:39.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T19:16:39.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T19:16:39.554 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T19:16:39.555 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119/remote/smithi081/crash 2024-09-21T19:16:39.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/crash -- . 2024-09-21T19:16:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:tar: /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T19:16:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:tar: Error is not recoverable: exiting now 2024-09-21T19:16:39.619 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119/remote/smithi104/crash 2024-09-21T19:16:39.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/crash -- . 2024-09-21T19:16:39.665 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/d839dc86-7849-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T19:16:39.666 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-09-21T19:16:39.667 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T19:16:39.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T19:16:39.738 INFO:tasks.cephadm:Compressing logs... 2024-09-21T19:16:39.738 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T19:16:39.742 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/tcmu-runner.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/objectstore_tool.2.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T19:16:39.780 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.audit.log: /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T19:16:39.781 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log 2024-09-21T19:16:39.782 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.6.log: /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T19:16:39.782 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T19:16:39.782 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T19:16:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T19:16:39.783 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T19:16:39.783 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T19:16:39.783 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T19:16:39.784 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T19:16:39.784 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.1.log: /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T19:16:39.786 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T19:16:39.786 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T19:16:39.786 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.c.log: /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T19:16:39.786 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T19:16:39.786 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log 2024-09-21T19:16:39.786 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log 2024-09-21T19:16:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T19:16:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T19:16:39.787 INFO:teuthology.orchestra.run.smithi081.stderr: 59.0% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log.gz 2024-09-21T19:16:39.787 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 62.9% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/tcmu-runner.log.gz 2024-09-21T19:16:39.796 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T19:16:39.888 INFO:teuthology.orchestra.run.smithi104.stderr: 79.8% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T19:16:39.896 INFO:teuthology.orchestra.run.smithi104.stderr: 89.8% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T19:16:39.981 INFO:teuthology.orchestra.run.smithi104.stderr: 91.3% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T19:16:40.039 INFO:teuthology.orchestra.run.smithi081.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T19:16:40.065 INFO:teuthology.orchestra.run.smithi081.stderr: 89.3% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T19:16:40.069 INFO:teuthology.orchestra.run.smithi081.stderr: 91.3% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T19:16:40.128 INFO:teuthology.orchestra.run.smithi081.stderr: 93.9% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T19:16:40.157 INFO:teuthology.orchestra.run.smithi104.stderr: 85.4% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T19:16:40.164 INFO:teuthology.orchestra.run.smithi104.stderr: 91.5% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T19:16:40.280 INFO:teuthology.orchestra.run.smithi081.stderr: 89.7% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T19:16:40.381 INFO:teuthology.orchestra.run.smithi104.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T19:16:40.944 INFO:teuthology.orchestra.run.smithi104.stderr: 94.3% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/objectstore_tool.2.log.gz 2024-09-21T19:16:41.917 INFO:teuthology.orchestra.run.smithi081.stderr: 90.6% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T19:16:46.194 INFO:teuthology.orchestra.run.smithi104.stderr: 90.5% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T19:16:57.136 INFO:teuthology.orchestra.run.smithi081.stderr: 90.0% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T19:20:11.131 INFO:teuthology.orchestra.run.smithi104.stderr: 93.7% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T19:21:22.951 INFO:teuthology.orchestra.run.smithi081.stderr: 92.6% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T19:22:00.005 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T19:24:22.663 INFO:teuthology.orchestra.run.smithi104.stderr: 93.8% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T19:25:19.685 INFO:teuthology.orchestra.run.smithi104.stderr: 92.7% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T19:25:19.688 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-21T19:25:19.688 INFO:teuthology.orchestra.run.smithi104.stderr:real 8m39.931s 2024-09-21T19:25:19.688 INFO:teuthology.orchestra.run.smithi104.stderr:user 5m13.155s 2024-09-21T19:25:19.688 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m15.220s 2024-09-21T19:34:12.195 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T19:34:22.490 INFO:teuthology.orchestra.run.smithi081.stderr: 93.1% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T19:34:38.292 INFO:teuthology.orchestra.run.smithi081.stderr: 93.1% -- replaced with /var/log/ceph/d839dc86-7849-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T19:34:38.294 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-21T19:34:38.294 INFO:teuthology.orchestra.run.smithi081.stderr:real 17m58.540s 2024-09-21T19:34:38.294 INFO:teuthology.orchestra.run.smithi081.stderr:user 7m2.867s 2024-09-21T19:34:38.294 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m20.186s 2024-09-21T19:34:38.295 INFO:tasks.cephadm:Archiving logs... 2024-09-21T19:34:38.295 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119/remote/smithi081/log 2024-09-21T19:34:38.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T19:35:40.963 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119/remote/smithi104/log 2024-09-21T19:35:40.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T19:36:27.613 INFO:tasks.cephadm:Removing cluster... 2024-09-21T19:36:27.613 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d839dc86-7849-11ef-baf6-efdc52797490 --force 2024-09-21T19:36:27.861 INFO:teuthology.orchestra.run.smithi081.stdout:Deleting cluster with fsid: d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T19:36:28.529 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d839dc86-7849-11ef-baf6-efdc52797490 --force 2024-09-21T19:36:28.769 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: d839dc86-7849-11ef-baf6-efdc52797490 2024-09-21T19:36:29.408 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T19:36:29.408 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T19:36:29.424 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T19:36:29.442 INFO:tasks.cephadm:Teardown complete 2024-09-21T19:36:29.442 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T19:36:29.464 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T19:36:29.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T19:36:29.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T19:36:29.560 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T19:36:29.570 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T19:36:29.570 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T19:36:29.572 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T19:36:29.584 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-09-21T19:36:29.586 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-09-21T19:36:29.597 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T19:36:29.597 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-09-21T19:36:29.597 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 +131us[ +134us] +/- 45ms 2024-09-21T19:36:29.597 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 +465us[ +465us] +/- 55ms 2024-09-21T19:36:29.597 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -1053us[-1053us] +/- 91ms 2024-09-21T19:36:29.597 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T19:36:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T19:36:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-09-21T19:36:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 +121us[ +135us] +/- 45ms 2024-09-21T19:36:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 +477us[ +477us] +/- 55ms 2024-09-21T19:36:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -1057us[-1057us] +/- 91ms 2024-09-21T19:36:29.634 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T19:36:29.635 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T19:36:29.646 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T19:36:29.646 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T19:36:29.660 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T19:36:29.764 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T19:36:29.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T19:36:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:36:29.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T19:36:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:36:29.881 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T19:36:29.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T19:36:29.957 INFO:teuthology.task.internal:Duration was 3353.973907 seconds 2024-09-21T19:36:29.957 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T19:36:29.971 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T19:36:29.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T19:36:29.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T19:36:30.023 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T19:36:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T19:36:30.376 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T19:36:30.376 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-09-21T19:36:30.376 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T19:36:30.406 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T19:36:30.406 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T19:36:30.435 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T19:36:30.435 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T19:36:30.448 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T19:36:30.502 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T19:36:30.503 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T19:36:30.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T19:36:32.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T19:36:32.586 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T19:36:32.586 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T19:36:32.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T19:36:32.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T19:36:32.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T19:36:32.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T19:36:32.680 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-09-21T19:36:32.728 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-09-21T19:36:32.753 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T19:36:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:36:32.768 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T19:36:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T19:36:32.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T19:36:32.796 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T19:36:32.796 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119/remote/smithi081 2024-09-21T19:36:32.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T19:36:32.882 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-21_16:57:11-orch:cephadm-wip-adk-testing-2024-09-21-1014-distro-default-smithi/7914119/remote/smithi104 2024-09-21T19:36:32.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T19:36:32.922 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T19:36:32.923 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T19:36:32.925 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T19:36:32.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T19:36:32.984 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T19:36:32.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T19:36:32.996 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T19:36:32.996 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T19:36:32.999 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T19:36:33.011 INFO:teuthology.orchestra.run.smithi081.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 19:36 /home/ubuntu/cephtest 2024-09-21T19:36:33.029 INFO:teuthology.orchestra.run.smithi104.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 19:36 /home/ubuntu/cephtest 2024-09-21T19:36:33.031 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T19:36:33.044 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T19:36:33.103 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3353.973907470703 flavor: default owner: scheduled_adking@teuthology success: true 2024-09-21T19:36:33.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T19:36:33.190 INFO:teuthology.run:pass